Nobody Expects The Production Problems

written on Sunday, 21st September, 2008

Sometimes you need to track down problems in a production setup. How the combination of Django and Flup make this difficult, and why I think that both should provide more than log by mail.

Our Setup

Our chief weapon is development! — Cardinal Ximinez of Spain

To the developer, the Django web development Framework offers a set of debugging helpers such as the verbose error page with interactive traceback and the management commands. The automatic reloading of python modules allows you to instantaneously see changes to your application, without even losing running sessions. If that is not enough for you, you might be interested in the powerful Django Command Extensions. The Django Debug Toolbar also looks promising yet rather unfinished. I might come back to that another time.

The Other Setup

... and production! Our two weapons are development and production!

Web applications written in Django are also highly portable: Development can take place on a different OS than production will, using a different version of Python and different database and HTTP systems. With respect to debugging, these alternatives are equalized by the abstraction layers of the Python and Django stack — and fairly well. Once you have sorted out the do's and dont's of your RDBMS, there is the decision on how to handle HTTP requests. Django recommends using the Apache HTTP Server and mod_python. Another performant setup that seems to be quite common is using a modern FastCGI-capable Web Server in combination with the flup FCGI/WSGI bridge. This is what I did.

Production: Turn Off the Debug Switch!

When preparing your production setup, you will want to look into your settings.py. Actually, you will probably have multiple settings files of different names in revision control. Then, you will create a symlink to the settings matching the current setup. In your production settings, you will turn off debugging. This replaces the verbose traceback pages with neat-looking 500 views which will — hopefully — never see the light of day! And otherwise, you will receive a nice error report somehow, right?

Of course you will probably encounter a production error in your site, be it a bunch of downloaded third party applications or your own magnificent creation. In my case, I soon had problems with an index on a text column.

The Third Setup

... and ruthless stage testing! Our three weapons are development, production, and ruthless stage testing.

To catch such problems before they occur with your production setup, it is highly desirable to use a stage setup. This setup has its own database, usually a copy of your production database, updated on demand. The other settings should mirror your production setup. Run your staging application on the production machine or — if you can afford it — on a separate identical system and make it accessible to selected testers only. For example, you might make it listen to the local IP only and ssh-tunnel from your development machine there. On this machine, you can turn on debugging whenever needed.

Problem One: Logging by E-mail

Of course, there might still be problems that are discovered in your production setup. Unfortunately, out of the box Django reports errors only via e-mail. To me, this has some critical drawbacks:

• You do not know if the error reporting works unless a problem occurs. In this case, you will only know that error reporting works if it works. Avoid this problem by enabling 404 reporting and then testing an invalid URL!
• It is not secure! To my knowledge there is no PGP-support for the error mails sent by Django. The messages might contain sensitive data that should not be sent unencrypted.
• If your production setup does not work right away, you will need debugging information in fast iterations. Obtaining such information by mail is inconvenient as it is tiresome to link a browser-click to an e-mail that you might receive minutes later.
• Your machine might not be allowed to send mails! Many sysops put firewall rules in place to prevent machines in their domain from becoming spam robots. This is what I did on my production machine, so fortunately I knew in advance that debugging by mail would not work. I am not sure if every django admin is aware of this potential problem.
• If you host a high traffic site and an error occurs (for example if you database does not respond due to slashdotting effects or denial of service attacks), the volumes of e-mail it generates might be so high that the error reporting competes with your application on IP-connections, worsening your problems.

Solution: Exception-Logging Middleware

For my basic needs, I have written a simple middleware that should handle any exceptions raised by views in your production setup. Please note that this middleware relies on the Exception Helpers module written by Dmitry Dvoinikov to format tracebacks. The traceback information is written into the logfile specified by the (custom) LOG_FILE setting. If the LOG_LEVEL is at least as verbose as DEBUG, the request object is dumped as well.

Problem Two: Exceptions Outside of Views

We are now prepared for errors that might occur when running your views production. But there is a different class of exceptions that cannot be handled by our middleware because of their nature:

On your Site, you probably want caching, ETags and transfer compression. So you enable the corresponding middleware, hopefully in the right order, as explained by Fredrik Lundh. And you might just add some middleware from djangosnippets or some of your own. Perhaps you want to prettyprint or to simplify your output, or to yield custom error pages for Ajax-Requests.

Solution: Enable Flup Traceback Pages

Any exceptions raised during middleware processing are not caught and handled by Django, but instead propagated up to the flup FastCGI handler. By default, Flup would present an error page like Django does in debugging mode, which is quite helpful (if not as beautiful). But Django disables this error page explicitly during startup. I filed a patch that allows to specify a debug flag to your FastCGI process to remedy this issue.

Summary

Amongst our weaponry are such diverse elements as development, production, ruthless stage testing, almost fanatical verboseness to the log and nice red flup tracebacks, my oh my oh my.

To anticipate and tackle production bugs, the solution consists of these parts:

• separate staging and production setups
• use exception logging middleware
• enable flup error pages in the stage setup

This is still not perfect. Ideally, flup would also log errors in production mode. Weirdly, flup also knows log by e-mail, and in this case it is not even configurable from your Django app! I will probably complement this article when I find an elegant way to fix this.