thefoundation

Sites for Mozilla Input

2010-08-10T18:46:36Z

As a side project during my internship at Mozilla, I worked with Aakash from Mozilla QA to bring a new feature to the Mozilla Input website.

Oftentimes when users have trouble with a Firefox beta, there is not actually a bug in the beta, but a problem with a specific website (such as broken user agent detection). Even when a problem is related to Firefox, it can be very helpful for QA to see what sites trouble our users the most, and what issues the users face there.

Enter clustering…

To group sentiment by topic, my fellow metrics intern Andres and I made use of Dave Dash’s clustering algorithm, which uses techniques from the search engine world to group related input. That helps to get a quick impression on what’s going on when a site is causing trouble for many users. We also get a lot of positive feedback on sites where the user experience has improved for beta users compared to the release version.

…and Django of course!

It was very cool to do something with Django again. The webdev team is very knowledgeable in this area so I learned a lot working with Fred and Dave. There are some limitations (you still cannot prefetch related objects along the inverse edge of a one-to-many relationship, like with any sensible ORM), but other than that Django has become a pretty solid toolkit. Also I finally got started with Git, which is as of now my version control system of choice.

Hopefully my main project will allow me time to improve Input and the dashboard in the future, there’s a lot of cool stuff planned with it.

And Apos Semicolon: A Cathapostrophe

2010-03-25T16:32:32Z

This morning on Facebook syndication, I reviewed the article on android that I wrote yesterday. And one of the few HTML-incompatible XHTML-properties assaulted my eyes, impersonated by a bunch of entity references.

Specifically, I had escaped the typewriter apostrophe (') using named entity reference syntax ('). Unfortunately, I had forgotten that — while this entity is defined by XHTML 1.0 — it is actually illegal in plain ol’ HTML. This should not have been a problem, as these pages are served using the XHTML 1.0 doctype where ' points to the Unicode code point 0x27, so that you can use single quotes to delimit attributes.

The Django RSS framework however would put a plain "html" content-type into the Atom-Feed, so the references to the apostrophe remained unresolved when the Feed readers converted my contents for display. Instead, they correctly escaped the ampersand, which led to a lot of ugly entity references on my facebook feed.

So for now I am going to reference the apostrophe using the Unicode code point reference ’ (punctuation apostrophe: ’) which is actually recommended over the ASCII-compatible ' (typewriter apostrophe: '). Strictly speaking, I would not even need to use any entityref here, as 0x2019 is not XML syntax. Next I need to figure out if there is a way to configure the Django feeds framework to use XHTML as a content type for Atom feeds and to check if the results are real-world-compatible.

But really, this just shows once more that it is absolutely inhumane to edit XHTML by hand. So I’ll be looking for a suitable WYSIWYM editor to maybe handle this stuff.

Nobody Expects The Production Problems

2008-09-21T00:08:05Z

Sometimes you need to track down problems in a production setup. How the combination of Django and Flup make this difficult, and why I think that both should provide more than log by mail.

Our Setup

Our chief weapon is development! — Cardinal Ximinez of Spain

To the developer, the Django web development Framework offers a set of debugging helpers such as the verbose error page with interactive traceback and the management commands. The automatic reloading of python modules allows you to instantaneously see changes to your application, without even losing running sessions. If that is not enough for you, you might be interested in the powerful Django Command Extensions. The Django Debug Toolbar also looks promising yet rather unfinished. I might come back to that another time.

The Other Setup

... and production! Our two weapons are development and production!

Web applications written in Django are also highly portable: Development can take place on a different OS than production will, using a different version of Python and different database and HTTP systems. With respect to debugging, these alternatives are equalized by the abstraction layers of the Python and Django stack — and fairly well. Once you have sorted out the do's and dont's of your RDBMS, there is the decision on how to handle HTTP requests. Django recommends using the Apache HTTP Server and mod_python. Another performant setup that seems to be quite common is using a modern FastCGI-capable Web Server in combination with the flup FCGI/WSGI bridge. This is what I did.

Production: Turn Off the Debug Switch!

When preparing your production setup, you will want to look into your settings.py. Actually, you will probably have multiple settings files of different names in revision control. Then, you will create a symlink to the settings matching the current setup. In your production settings, you will turn off debugging. This replaces the verbose traceback pages with neat-looking 500 views which will — hopefully — never see the light of day! And otherwise, you will receive a nice error report somehow, right?

Of course you will probably encounter a production error in your site, be it a bunch of downloaded third party applications or your own magnificent creation. In my case, I soon had problems with an index on a text column.

The Third Setup

... and ruthless stage testing! Our three weapons are development, production, and ruthless stage testing.

To catch such problems before they occur with your production setup, it is highly desirable to use a stage setup. This setup has its own database, usually a copy of your production database, updated on demand. The other settings should mirror your production setup. Run your staging application on the production machine or — if you can afford it — on a separate identical system and make it accessible to selected testers only. For example, you might make it listen to the local IP only and ssh-tunnel from your development machine there. On this machine, you can turn on debugging whenever needed.

Problem One: Logging by E-mail

Of course, there might still be problems that are discovered in your production setup. Unfortunately, out of the box Django reports errors only via e-mail. To me, this has some critical drawbacks:

You do not know if the error reporting works unless a problem occurs. In this case, you will only know that error reporting works if it works. Avoid this problem by enabling 404 reporting and then testing an invalid URL!
It is not secure! To my knowledge there is no PGP-support for the error mails sent by Django. The messages might contain sensitive data that should not be sent unencrypted.
If your production setup does not work right away, you will need debugging information in fast iterations. Obtaining such information by mail is inconvenient as it is tiresome to link a browser-click to an e-mail that you might receive minutes later.
Your machine might not be allowed to send mails! Many sysops put firewall rules in place to prevent machines in their domain from becoming spam robots. This is what I did on my production machine, so fortunately I knew in advance that debugging by mail would not work. I am not sure if every django admin is aware of this potential problem.
If you host a high traffic site and an error occurs (for example if you database does not respond due to slashdotting effects or denial of service attacks), the volumes of e-mail it generates might be so high that the error reporting competes with your application on IP-connections, worsening your problems.

Solution: Exception-Logging Middleware

For my basic needs, I have written a simple middleware that should handle any exceptions raised by views in your production setup. Please note that this middleware relies on the Exception Helpers module written by Dmitry Dvoinikov to format tracebacks. The traceback information is written into the logfile specified by the (custom) LOG_FILE setting. If the LOG_LEVEL is at least as verbose as DEBUG, the request object is dumped as well.

Problem Two: Exceptions Outside of Views

We are now prepared for errors that might occur when running your views production. But there is a different class of exceptions that cannot be handled by our middleware because of their nature:

On your Site, you probably want caching, ETags and transfer compression. So you enable the corresponding middleware, hopefully in the right order, as explained by Fredrik Lundh. And you might just add some middleware from djangosnippets or some of your own. Perhaps you want to prettyprint or to simplify your output, or to yield custom error pages for Ajax-Requests.

Solution: Enable Flup Traceback Pages

Any exceptions raised during middleware processing are not caught and handled by Django, but instead propagated up to the flup FastCGI handler. By default, Flup would present an error page like Django does in debugging mode, which is quite helpful (if not as beautiful). But Django disables this error page explicitly during startup. I filed a patch that allows to specify a debug flag to your FastCGI process to remedy this issue.

Summary

Amongst our weaponry are such diverse elements as development, production, ruthless stage testing, almost fanatical verboseness to the log and nice red flup tracebacks, my oh my oh my.

To anticipate and tackle production bugs, the solution consists of these parts:

separate staging and production setups
use exception logging middleware
enable flup error pages in the stage setup

This is still not perfect. Ideally, flup would also log errors in production mode. Weirdly, flup also knows log by e-mail, and in this case it is not even configurable from your Django app! I will probably complement this article when I find an elegant way to fix this.

Still Cracks in Our Foundation

2008-09-09T14:57:48Z

Welcome to thefoundation.de, our aggregate blog on media and internet technology. There are still many cracks to fill...

...but in accordance with the recent release of the Django web development framework, we decided to push our site out of the door as well. This blog is much of a spare time effort and all of us have enough other things on our hands, so that even Web 2.0 baseline commodities such as comments, trackback and pingback are still pending.

And although I generally agree with the notion that one should not always reinvent the wheel, we chose to develop this blogging application by ourselves. The main reason is that it is an easy way to get started with Django and thus with one of the more recent MVC frameworks. Another is that we wanted to be able to define relations and interactions among our four journals without limitations. And of course, there are some really nice Django applications that are more easily integrated with your own glue. There is also a more conceptual bonus which might be regarded as a disadvantage by some: Every design decision that has to be done, we have to do. There are lots of defaults in modern publishing applications, leading to a confluence in style and functionality where there could be diversity.

But, for that to happen here we still has some catching up to do. See you then!