thefoundation

Scalable Text Clustering

2011-03-01T17:25:05Z

The updated version of the Grouperfish clustering plans, now published on the Mozilla Blog of Data.

The Background

During my work as metrics liaison with the Firefox Input team, an exciting requirement has come up: scalable online clustering of the millions of feedback items that the users of Firefox share with us.

When designing a service at the metrics team, besides functional requirements (accept text messages, produce clusters) we consider scalability and durability. In fact, scalability concerns play a major role in wanting to replace the current solution (which has done a fine job so far) and not picking another powerful existing tool: We expect the influx of messages (already heading towards 2 million) to increase up to 50x once Firefox 4 is released.

On to Architecture

There is a slide outlining what the system (called Grouperfish) is planned to look like. As this service is to be developed quickly and in iterations, even major parts of the system might be replaced in the future though. This is the rationale for our first version, to be released sometime around the Firefox 4 release:

Concurrency

We want to be able to handle tens of thousands of GET’s and thousands of POST’s per second, provided we have enough commodity hardware at our disposal.

To accept incoming documents and queue them for clustering, Node.JS fits the bill. Its event-based concurrency model dominates thread- and process-based designs in IO-bound tasks such as this. Also, depending on the storage you pick, requests might pause to wait on garbage collection or to rewrite store files. Node can handle a lot of waiting requests because it does not use system level threads (or even processes) for concurrency.

Storage

Grouperfish must store millions of documents in hundreds of thousands of collections. The generated clusters may reference thousands of documents each, each ranging from a few bytes to about a megabyte. Also, we want to store processing data for clustering.

When planning for more data than fits into your collective RAM, you usually have two options (SQL not being one of them since RAM has become pretty big):

Dynamo-style key/value stores like Riak and Cassandra allow to store replicated values with high write rates, and also to quickly retrieve individual items from disk. You do not need to worry about one machine getting too much attention (e.g. when one of your services gets slashdotted), thanks to consistent hashing. Riak even has a notion of buckets, keys and values: We would intuitively use buckets for collections of documents (and of clusters), and values for individual documents (and clusters). No wonder we looked at this more closely. Unfortunately though, Riak’s buckets are more of a namespacing device than anything else. It is expensive to get all elements of a bucket, since they are neither indexed by a common key nor stored together on disk. The Riak design can be a bit misleading in this regard, as buckets are in fact spread throughout the key space. To retrieve all keys in a bucket, Riak will check every single key — possibly scanning gigabytes of main memory (for the very recent Riak search to help, you’d need to blow up your values quite a bit). And you still only have the keys. To get possibly millions of associated values, you need to move your little disk heads a lot. This is not always as bad as it sounds because Riak gives you streaming access to the data as it comes in. But in general, the smaller your buckets in relation to the entire key space, the higher the cost of retrieving many of them.

The other major contender are column-oriented data stores of the BigTable family, the most prominent of which is Apache HBase (the aforementioned Cassandra is actually somewhat in-between, having properties from both worlds). The two main differences for users of HBase vs. Dynamo style stores as far as we are concerned: 1. Data is stored per column family: to retrieve the vector representations of a million documents, we do not have to scan through a million document texts. 2. Records are sorted by key, much like in a traditional database (but optimized for fast inserts, using LSM trees). This is a blessing and a curse. A blessing, because we can scan over contiguous collections of documents. A curse, because we are vulnerable to hotspotting on popular collections. To counter this, we need to make sure that there are random parts in our row keys, e.g. using UUID’s. Because HBase divides tables into regions as they grow and hands them off to other nodes, this method avoids hotspots. And we do not lose the streaming advantage as long as we use common prefixes per collection.

Given our access patterns (insert documents, update clusters, re-process entire collections, fetch lists of clusters), efficient sequential access to selected parts of the data is very important. Sorted, column oriented storage seems to be the way to go. There are other pros and cons (single point of failure, write throughput, hardware requirements), but if we don’t cater to our use case, those won’t ever matter.

Clustering

Grouperfish must be able to handle small numbers of large corpora (millions of documents), as well as large numbers of small corpora (millions of collections). The generated clusters may contain thousands of messages each.

This is practically a no-brainer: Apache Mahout supports in-memory operation (for smaller clusters) as well as distributed clustering (using Apache hadoop, for larger clusters). Mahout can update existing clusters with new documents and generate labels for our clusters. Of course, Mahout is a java-library, so we need to run it within a JVM. To simplify management and introspection, we will run our clustering workers in jetty web containers.

Scheduling

We need to be able to add workers to increase clustering frequency. When there are more new messages than can be clustered right away, we want them to be queued. Also, we have Node.JS and we have Java/Mahout. We want our queue to bridge the gap.

Messaging has become a big topic as systems have become larger and more distributed. We want to use messages to decouple write requests from processing them. There is a very elegant solution to maintain queues, offered by the in-memory data store Redis. Redis is somewhat like a developers dream of shared memory. No encoding and decoding of lists, maps and values as they enter and leave the stores — just operate on your data structures within shared memory. Unfortunately, Redis queues are really just a linked list with a blocking POP operation. While that is very nice, we want to track and resubmit failed tasks when a worker node falls victim to rampaging rodents.

The considerations of choosing RabbitMQ to realize a task queue are worth an article of their own. Suffice to say, it has Node- and Java-bindings, and it supports message Acknowledgement from workers. We still want to use Redis to keep track of collection size, to cache the actual incoming data (no need to ask hbase if we use it right away), and for locking of colections, so that every collection is only modified by one worker at a time. We also might use it to cache frequently requested clusters.

More Thoughts

Selecting these components, I learned that it is important to choose technologies in an unbiased fashion, and to reconsider decisions when a technology has no answer for your requirement. For example, I originally wanted to use just Riak for storage — I like its simplicity and style, and the bucket metaphor — but the enumeration of large buckets would be too slow for an online system. It might be fine for a batch-only system, or a system that just does not operate on collections of varying size as much.

For a Message queue, ØMQ sounded awesome, offering low latency and powerful constructs, but I quickly realized that it is not really what I understand a message queue to be, but rather a very smart abstraction over traditional sockets. Probably someone will eventually build a distributed task queue on top of it though.

Diaspora — Can the Social Graph be Our Web of Trust?

2010-08-23T01:30:22Z

On Friday we had Max, Ilya and Raphael from Diaspora over at Mozilla. They talked about their effort in creating a distributed social network. Where I think they are on the right track, and where they should think even bigger.

Why we need Diaspora

Personally, I see three major challenges that everyone passionate about the open internet needs to make up their mind about:

The erosion of Net Neutrality
Participants switching to closed environments of apps and appliances, becoming mere consumers (*)
People entrusting their personal data and social activity to Facebook, forced to choose between control and connectedness

In the context of the Diaspora talk, I’ll focus on the third issue.

We need Diaspora because people need to be in control over with whom they share personal information. Every time Facebook sneaks in a new default that breaks privacy, we grudgingly change the settings again — and stay, not wanting to lose our friends. Or we just don’t know about it and leave it as it is. Combined with the social monopoly that Facebook has established, this makes privacy and security optional features, subject to change like any other.

How Diaspora can help already

The main distinguishing factor of Diaspora compared to Facebook et al. is in that it decouples your social graph from the network provider, bringing back real competition to the social space. Like with E-Mail, there can be lots of network providers, loosely connected over push-interfaces. Whenever a pod (the equivalent to an e-mail-provider in Diaspora) should violate your trust, you can just switch to another one, or set up your own pod.

What could be done better

On the downside, this means that you have to trust your pod as well as all your friend’s pods. No big deal? Well, where the same server software is used on a distributed network, it is very prone to exploit of vulnerabilities due to patch delay and misconfiguration (correctly setting up TLS is still a big challenge, not only for regular people).

Secure HTTP is great when a large, anonymous group of people needs to trust a central service. It allows us to do online banking and purchases, free from eavesdropping and man-in-the-middle attacks. However, it is not peer-to-peer: When you fetch your mail over a secure IMAP connection, you might be sure that your password is protected, but you do not know who actually sent you that e-mail (think about it: that is the reason why phishing works). When you get it from Google Mail, you might be using TLS, but Google is still able to read your every conversation.

How PGP can solve this

I propose that Diaspora pods should be dumb post boxes that are not able to actually look into status updates, private messages, friend lists and so on. How? The technology for that has been available for quite some time and is called PGP.

Basically, PGP allows you to send and receive messages that cannot be decrypted by the servers that route them. So, if you were to encrypt your message inside your browser, you would establish secure end-to-end communication. No need to trust the shady pods that some of your friends decided to use, not knowing any better. But encryption in a web client? That sounds awfully slow! Well, Firefox Sync does it already with your entire browsing history (the pass phrase to your key is never sent to the server), and I would imagine that JavaScript interpreters have become fast enough to emulate the cryptographic capabilities of a PC from 1991.

I do have ideas on how to approach search and incremental profile updates with this, and on the new security considerations that apply here (Can you always trust your browser? Could a pod not make you use an insecure web client that transmits your passphrase?). However, that is rather technical, possibly material for a follow up post.

The social network is a key signing party

The problem with PGP has always been that people have been unable to exchange public keys in a manner that is both trustworthy and extensive. Because a web of trust can often not be established, people refrain from using encrypted e-mail. Turns out that social networks come with a mechanism that is just made for this: Friending. In the secure social network, accepting a friend request would be equivalent to exchanging keys. Usually you are referred to friends from people you already know, so there already is a basic level of trust.

This means that online social networks can be transformed from a jeopardy to our security to a vehicle of the same. This idea is of course also not entirely new. What might be new is the idea of building the social web entirely on top of PGP rather than just integrating that as an optional feature.

Any Comments?

I have not gotten around to add Commenting or Pingback to this blog, but I would love to incorporate any (links to) comments in a follow up post, please write to michael at this domain.

Update:

If I understand correctly, the diaspora guys are already planning to use GPG for cryptography somewhere. This is a pretty good start. If they really already plan on generating keys for everyone, then they would only need to pull the actual encryption into the web client.

(*) Like any intern at Mozilla I had the opportunity to to talk to John Lilly, and I got the impression that Mozilla takes this development very seriously.

Sites for Mozilla Input

2010-08-10T18:46:36Z

As a side project during my internship at Mozilla, I worked with Aakash from Mozilla QA to bring a new feature to the Mozilla Input website.

Oftentimes when users have trouble with a Firefox beta, there is not actually a bug in the beta, but a problem with a specific website (such as broken user agent detection). Even when a problem is related to Firefox, it can be very helpful for QA to see what sites trouble our users the most, and what issues the users face there.

Enter clustering…

To group sentiment by topic, my fellow metrics intern Andres and I made use of Dave Dash’s clustering algorithm, which uses techniques from the search engine world to group related input. That helps to get a quick impression on what’s going on when a site is causing trouble for many users. We also get a lot of positive feedback on sites where the user experience has improved for beta users compared to the release version.

…and Django of course!

It was very cool to do something with Django again. The webdev team is very knowledgeable in this area so I learned a lot working with Fred and Dave. There are some limitations (you still cannot prefetch related objects along the inverse edge of a one-to-many relationship, like with any sensible ORM), but other than that Django has become a pretty solid toolkit. Also I finally got started with Git, which is as of now my version control system of choice.

Hopefully my main project will allow me time to improve Input and the dashboard in the future, there’s a lot of cool stuff planned with it.

Going to Mozilla

2010-06-18T17:07:33Z

Starting on Monday, June 21 I am going to intern at the Mozilla Corporation (MoCo) in Mountain View, California. Yay!

For quite some time I have been following the mozine and later PMO. So I am absolutely thrilled to have this opportunity, and this also means that this blog will get a new topic added. Not only will I get to know many more interns with whom I am going to live in Mountain View, and not only will I participate in the Mozilla project together with all the great people at the MoCo HQ. But also I will be attending the Mozilla Summit, the biennial meeting of people from all over the world that made great projects such as the Firefox web browser and AMO possible.

My internship position will be at the metrics department led by Ken Kovash and quite probably I will be allowed to go into the details of my project there, either at this blog or at a Mozilla blog.

Leaving for CA

If you plan to go abroad to the U.S. for an internship, I suggest you apply for the internship position(s) of your choice at least two months before the actual start of the internship. I was a bit late to the party and that led to a rather tight schedule: As a Germany based student at RWTH Aachen University, I had to invest some time in getting the visa. But fortunately there is a very helpful visa sponsoring partner, so everything went smoothly after all.

I do not know about other areas, but as a student in computer science you can expect compensation for an internship in the U.S. which is not necessarily the case in Germany. I applied at two organizations, and in both cases their offers covered living expenses and the flight to California. So I really do recommend that next spring you visit the web site of any company or organization you always wanted to get to know, and apply for an internship there. Make sure that the professional and academic experience on your resume matches the position you apply for, and prepare for two to three phone interviews.

OneSocialWeb: more than Jabber for Apps

2010-04-30T14:00:21Z

Almost a month ago, the presentation of OneSocialWeb at the android developers conference droidcon.be was one of the most interesting talks there. Recently the XMPP-centric framework has gone open source.

Last year a group of fellow students and myself were tasked with creating an android applications to organize meetings spontaneously (think something like doodle, only mobile and more short term). At that time we were thinking about using XMPP for real time communication, but were hesitant because of the time this would cost us to implement. In the end we used a traditional REST-based web service rather than a peer-to-peer system. Luckily there already is an effort underway which is called OneSocialWeb, funded by the telco provider Vodafone. It allows people to work together on Java-objects using XMPP.

This means that all the work that has to do with XMPP protocol handling and conflict management will be handled by this abstraction layer, while we developers can focus on delivering useful application. You could use this for simple things like associating chat conversations with arbitrary objects in your application. You could also try to model your entire application domain around this collaboration: In his presentation at droidcon Laurent Eschenauer demonstrated this using a collaborative shopping list where each participant could check off items, notifying the others immediately.

The Android platform with its services-model might really help in getting this concept to work, as XMPP protocol handling could be handled by one central service, dispatching updates to any interested Activities. This could well become the bidirectional, decentralized alternative to Apple’s Mobile Push service.

And Apos Semicolon: A Cathapostrophe

2010-03-25T16:32:32Z

This morning on Facebook syndication, I reviewed the article on android that I wrote yesterday. And one of the few HTML-incompatible XHTML-properties assaulted my eyes, impersonated by a bunch of entity references.

Specifically, I had escaped the typewriter apostrophe (') using named entity reference syntax ('). Unfortunately, I had forgotten that — while this entity is defined by XHTML 1.0 — it is actually illegal in plain ol’ HTML. This should not have been a problem, as these pages are served using the XHTML 1.0 doctype where ' points to the Unicode code point 0x27, so that you can use single quotes to delimit attributes.

The Django RSS framework however would put a plain "html" content-type into the Atom-Feed, so the references to the apostrophe remained unresolved when the Feed readers converted my contents for display. Instead, they correctly escaped the ampersand, which led to a lot of ugly entity references on my facebook feed.

So for now I am going to reference the apostrophe using the Unicode code point reference ’ (punctuation apostrophe: ’) which is actually recommended over the ASCII-compatible ' (typewriter apostrophe: '). Strictly speaking, I would not even need to use any entityref here, as 0x2019 is not XML syntax. Next I need to figure out if there is a way to configure the Django feeds framework to use XHTML as a content type for Atom feeds and to check if the results are real-world-compatible.

But really, this just shows once more that it is absolutely inhumane to edit XHTML by hand. So I’ll be looking for a suitable WYSIWYM editor to maybe handle this stuff.

Die Klimaschützer

2009-07-06T17:28:16Z

Start using green energy – install the Carbon Clock on your Facebook-Profile or homepage!

Finally the Carbon Clock is online! The Layout was created by widjet, my task was the Flash and programming stuff. It is my first Facebook-Application! After some struggle with the API the application works fine – needless to say, I am a bit proud!
Depending on the installs of this application, the Carbon Clock shows how many carbon dioxide is prevented from emitting by users of green energy.

Mach mit! Werde Klimaschützer!

Click the Wer macht mit?-Button to install the widget on your homepage or go to the Facebook-Application and install Die Klimaschützer to your profile!

Since I felt lost sometimes during the development, I soon will write down my experiences and techniques in a short article.

It appears I was wrong

2009-04-08T23:18:11Z

Yesterday, Google announced the availability of Java as the new programming language for the App Engine, refuting my guess from last year that it might be JavaScript — though of course, not entirely.

If you take the demand of the users into account, Java is of course the right choice as the next language. It might just alienate lots of Java-Developers if a niche language of the server zoo such as JavaScript was to emerge first. Additionally, Java is one Step short of full (albeit sandboxed) JVM support.

To allow for sandboxing, Google wraps some of its own API's into Java SE or JSR–standardized services such as javax.mail and java.net.URL. Also, there is a white-list currently containing 1323 of the 3700+ Java SE 6 classes. Most of the classes that are not available are from the Swing and AWT suites which a web developer will not need anyway. Instead, Google provides the homebrewn GWT.

Thanks to the free and open source Rhino JavaScript Interpreter written in Java, server side JavaScript on the App Engine is rather easy to achieve now. I guess I might have to check it out and report back about it later, so I just signed up for the Java technology preview on my App Engine account.

There are some videos from the Google Campfire event over at Youtube. Most of the time they are rather interesting, plus Kevin Gibbs does a pretty decent imitation of Steve Jobs during the presentation (voluntary or not).

How to bring AS3 Tweens to an end

2009-03-24T19:28:52Z

Have you ever assigned a tween to a local variable? Have you ever detected non-determining tweens, tweens which do not start at all? Blame ActionScript 3.0's ruthless garbage collector!

Back then in the old days of ActionScript 2.0 scripts like the following were just as easy as useful to program sequences of animations (e. g. assembling animations for site sections):


function transitionToolTip():void {
	var toolTipAlpha:Tween = new Tween( toolTip, "alpha", Regular.easeOut, 0, 1, .75, true );
	toolTipAlpha.addEventListener( TweenEvent.MOTION_FINISHED, handleToolTipTransition );
}
function handleToolTipTransition( e:TweenEvent ):void { … }

Just to make things clear: The ActionScript 3.0 garbage collector does a good job in deleting these local variables. Obviously the ActionScript 2.0 garbage collector should have done this way, too.

In most cases one does not want to stop or modify tweens after firing them, thus one won't need a reference to the exact tween ever again. The next function would be triggered after the tween has finished. Great!
But, nowadays things are different. ActionScript 3.0's garbage collector seriously and unrelentingly works on what it is supposed to work: deleting everything with a lack of relation. In the sample case, shown above, this clearly means that the variable toolTipAlpha, declared with local scope of function transitionToolTip will be deleted – taking along our tween if the garbage collection cycle is awkward. In consequence our tween will not play to its end.

One solution would be the declaration of a class level or global variable to store the reference to the tween. This would look something like this:


var toolTipAlpha:Tween;

function transitionToolTip():void {
	toolTipAlpha = new Tween( toolTip, "alpha", Regular.easeOut, 0, 1, .75, true );
	toolTipAlpha.addEventListener( TweenEvent.MOTION_FINISHED, handleToolTipTransition );
}
function handleToolTipTransition( e:TweenEvent ):void { … }

All the samples are in short form and not taken from real-life ActionScript 3.0 projects thus may not work properly. They are inserted for illustration purposes to show up possible problematics.

A far more elaborated description of the coexistence of tweens, variables and the garbage collector can be found on Scott Morgan's blog in the article AS3 Garbage Collection, the reason your tweens are ending early.

via Scott Morgan's blog – http://www.scottgmorgan.com/

Sorry! Our Ping- and Trackback is not yet working.

Stuck for gift ideas?

2008-12-17T00:35:30Z

Santa will help you out! Get some great last minute tipps from Santa himself directly on your own homepage!

Santa's Last Minute Tipps

Get Santa on your homepage!

This widget was programmed using the Clearspring In-Widget-API. The layout of Santa's MySpace-Site and the widget was created by widjet – a cologne based agency, focused on widget-development.
My part was the implementation of the widget in Adobe Flash. After being doomed with the Clearspring-API I had fun animating Santa and his Tipps!

You can really look forward to see Santa's next Tipps – and be sure not to miss a single one!

Google Street View car spotted in Cologne

2008-10-18T22:50:00Z

I recently saw a strange car standing outside my house. I'm pretty sure it's the Google Street View car taking photos of Cologne! I managed to take some pictures...

I'm sorry for the bad image quality but I only had my mobile on me and the camera is pretty bad...
There is this tripod construction on top of the car roof where presumably the camera rig is mounted on. I wonder how fast the car can go without creating too much motion blur...well, I certainly don't want to be the guy who has to deal with all the photos they take day by day ;-)

Google Street View Car in Cologne

Is JavaScript The Next App Engine Language?

2008-09-21T04:00:28Z

As the language of the browser, JavaScript has become a kind of common denominator among web programmers. On the server it is rare compared to Java, PHP, Ruby or Python. Could this change, since Google has built an implementation of JS and an affordable yet scalable web application infrastructure?

Scripting Language of The Open Web

JavaScript was created as a scripting language for the Netscape 2.0 browser. Other than its namesake Java, it has remained in the browser since then. Web Developers know this language because they have to use it, no matter if they like it. Throughout the late 90s and well into this decade, JavaScript was mostly regarded as a toy language for silly effects and simple form checks. Due to the more recent experience with Ajax applications and the resulting thorough coverage in technical literature, the users of JavaScript have become more professional.

The Server Domain

The development on the server side has — with some notable exceptions — ignored the language JavaScript for many years. Perl was a dominant language in the early days. Then, PHP gained momentum in the fast growing developer community during the internet bubble due to its gentle learning curve, the simple integration with MySQL and the overwhelming availability of inexpensive shared hosting. You could say that in its ubiquity, PHP has mimicked JavaScript, only on the server.
Other languages and tool chains have gained momentum since then, most notably Ruby, Java and Python. They are not as affordable as PHP (except Python on the App Engine), and seem to be used mostly because of their modern MVC frameworks.

Why JavaScript Is Not Popular on The Server

JavaScript has some fundamental weaknesses that have inhibited its inception as a server side programming language:

JavaScript, — or rather ECMAScript — has no real standard library, it is just a language. There are no tools for file or process manipulation. There are no database bindings and there is no shared memory. There is no way to open a socket.
In the past, JavaScript Interpreters have not been known for their performance. Anyone who has seen the message A script on this page may cause Firefox to run slow. would think twice before writing an application in JavaScript.

The Helma framework tries to address the first issue by using the Rhino Interpreter, making the Java SDK available to the JavaScript programmer. The second issue is at least partially resolved as some performance intensive tasks are moved to Java. In Java 6, Rhino is even bundled with Java. Nevertheless Helma and similar integration efforts are not very popular, compared to Rails or Struts. A reason for this might be that Java for the web (the enterprise edition) is a very serious business, while the perception of JavaScript has long been ...well, the opposite of serious. The two cultures do not seem to mix well yet, as good as the technologies might blend.

Why The App Engine Loves JavaScript

When Guido van Rossum adopted Python for the Google App Engine, he had to remove lots of features because they would not work with the sandboxed, replicating environment which is the App Engine. It turns out that the limitations of JavaScript really make it an ideal candidate for the App Engine.

JavaScript is just a language: It cannot create sockets, files or processes. App Engine does not allow for that anyway, so this is a plus!
The language is easily sandboxed. Web Browsers have to do this all the time. This is ideal for a shared hosting environment.
It has a no database bindings included, but a natural serialization format, JSON. This is great for the Google Data Store, because it stores objects, not table rows.
Recently, JavaScript Interpreters (or, lets say JavaScript Machines) have become fast. There is V8 and it is fast.

It is known (at least since Guido van Rossums talk at DjangoCon 2008) that Google plans to adopt more languages for the app engine. Also, there are four primary languages that Google uses for development: Python, Java, C++ and JavaScript. Python is already available. Java is a good candidate for the App Engine, but it is not exactly lightweight certainly not shared nothing. C++ is probably the opposite of any sandboxable language.

In combination with a template engine, a free offer on the App Engine might give the already ubiquitous JavaScript language a final boost. The prospect of one language from model definition to client side computation is certainly compelling!

To encourage further speculations: With Google's V8 machine comes an embedding, process.cc, that contains code necessary to extend a hypothetical HTTP request processing application - which could be part of a web server, for example (...) (V8 Documentation).

Nobody Expects The Production Problems

2008-09-21T00:08:05Z

Sometimes you need to track down problems in a production setup. How the combination of Django and Flup make this difficult, and why I think that both should provide more than log by mail.

Our Setup

Our chief weapon is development! — Cardinal Ximinez of Spain

To the developer, the Django web development Framework offers a set of debugging helpers such as the verbose error page with interactive traceback and the management commands. The automatic reloading of python modules allows you to instantaneously see changes to your application, without even losing running sessions. If that is not enough for you, you might be interested in the powerful Django Command Extensions. The Django Debug Toolbar also looks promising yet rather unfinished. I might come back to that another time.

The Other Setup

... and production! Our two weapons are development and production!

Web applications written in Django are also highly portable: Development can take place on a different OS than production will, using a different version of Python and different database and HTTP systems. With respect to debugging, these alternatives are equalized by the abstraction layers of the Python and Django stack — and fairly well. Once you have sorted out the do's and dont's of your RDBMS, there is the decision on how to handle HTTP requests. Django recommends using the Apache HTTP Server and mod_python. Another performant setup that seems to be quite common is using a modern FastCGI-capable Web Server in combination with the flup FCGI/WSGI bridge. This is what I did.

Production: Turn Off the Debug Switch!

When preparing your production setup, you will want to look into your settings.py. Actually, you will probably have multiple settings files of different names in revision control. Then, you will create a symlink to the settings matching the current setup. In your production settings, you will turn off debugging. This replaces the verbose traceback pages with neat-looking 500 views which will — hopefully — never see the light of day! And otherwise, you will receive a nice error report somehow, right?

Of course you will probably encounter a production error in your site, be it a bunch of downloaded third party applications or your own magnificent creation. In my case, I soon had problems with an index on a text column.

The Third Setup

... and ruthless stage testing! Our three weapons are development, production, and ruthless stage testing.

To catch such problems before they occur with your production setup, it is highly desirable to use a stage setup. This setup has its own database, usually a copy of your production database, updated on demand. The other settings should mirror your production setup. Run your staging application on the production machine or — if you can afford it — on a separate identical system and make it accessible to selected testers only. For example, you might make it listen to the local IP only and ssh-tunnel from your development machine there. On this machine, you can turn on debugging whenever needed.

Problem One: Logging by E-mail

Of course, there might still be problems that are discovered in your production setup. Unfortunately, out of the box Django reports errors only via e-mail. To me, this has some critical drawbacks:

You do not know if the error reporting works unless a problem occurs. In this case, you will only know that error reporting works if it works. Avoid this problem by enabling 404 reporting and then testing an invalid URL!
It is not secure! To my knowledge there is no PGP-support for the error mails sent by Django. The messages might contain sensitive data that should not be sent unencrypted.
If your production setup does not work right away, you will need debugging information in fast iterations. Obtaining such information by mail is inconvenient as it is tiresome to link a browser-click to an e-mail that you might receive minutes later.
Your machine might not be allowed to send mails! Many sysops put firewall rules in place to prevent machines in their domain from becoming spam robots. This is what I did on my production machine, so fortunately I knew in advance that debugging by mail would not work. I am not sure if every django admin is aware of this potential problem.
If you host a high traffic site and an error occurs (for example if you database does not respond due to slashdotting effects or denial of service attacks), the volumes of e-mail it generates might be so high that the error reporting competes with your application on IP-connections, worsening your problems.

Solution: Exception-Logging Middleware

For my basic needs, I have written a simple middleware that should handle any exceptions raised by views in your production setup. Please note that this middleware relies on the Exception Helpers module written by Dmitry Dvoinikov to format tracebacks. The traceback information is written into the logfile specified by the (custom) LOG_FILE setting. If the LOG_LEVEL is at least as verbose as DEBUG, the request object is dumped as well.

Problem Two: Exceptions Outside of Views

We are now prepared for errors that might occur when running your views production. But there is a different class of exceptions that cannot be handled by our middleware because of their nature:

On your Site, you probably want caching, ETags and transfer compression. So you enable the corresponding middleware, hopefully in the right order, as explained by Fredrik Lundh. And you might just add some middleware from djangosnippets or some of your own. Perhaps you want to prettyprint or to simplify your output, or to yield custom error pages for Ajax-Requests.

Solution: Enable Flup Traceback Pages

Any exceptions raised during middleware processing are not caught and handled by Django, but instead propagated up to the flup FastCGI handler. By default, Flup would present an error page like Django does in debugging mode, which is quite helpful (if not as beautiful). But Django disables this error page explicitly during startup. I filed a patch that allows to specify a debug flag to your FastCGI process to remedy this issue.

Summary

Amongst our weaponry are such diverse elements as development, production, ruthless stage testing, almost fanatical verboseness to the log and nice red flup tracebacks, my oh my oh my.

To anticipate and tackle production bugs, the solution consists of these parts:

separate staging and production setups
use exception logging middleware
enable flup error pages in the stage setup

This is still not perfect. Ideally, flup would also log errors in production mode. Weirdly, flup also knows log by e-mail, and in this case it is not even configurable from your Django app! I will probably complement this article when I find an elegant way to fix this.

Integrating a plug-in into JW-Player

2008-09-20T02:06:41Z

If you spent time watching web videos in the popular FLV format you might have stumbled upon the Jeroen Wijering player. Version 4.1 features a plug-in interface – which I found pretty much undocumented.

In the following article I will try to give you a short insight on how to integrate plug-ins into the free and open-source Flash video player by Jeroen Wijering. All techniques described in this article worked with version 4.1.

JW-Player plug-ins

There are some restrictions for commercial use of the player. You have to apply for a commercial license at http://www.jeroenwijering.com/?page=order

What is JW-Player?

In my eyes this player is a masterpiece in structured programming regarding the MVC architectural pattern. The source is as modular as possible so it can easily modified to fit new requirements. As an example it can be adapted to display videos from other sources (e. g. YouTube) by adding a new class as a submodel. The player allows skinning and provides a wide range of settings as well as an useful JavaScript interface. While the majority of possibilities is documented in the Wiki I could not find any helpful information about the plug-ins. Maybe due a lack of time, perhaps because this possibility could be used to emulate the functionality of longtail video – a side project of Jeroen Wijering.
But this is just speculation so let us start over! This article will not give you turnkey sources nor will it be a full tutorial on how this or that could be realized with the JW-Player. I presume that you have already been dealing with Adobe Flash in a basic matter, so you can use my depictions as foundation for your own projects.

How to integrate plug-ins

You will need to have the kroeger 05/63 pixel font installed / activated to not break the default player skin layout!

First you should get the source files and create an HTML file which loads the JW-Player in order to set up a test-bench. In the next step we will create a Adobe Flash file to house our plug-in functionality. First and foremost the size and frame rate does not matter, however I suggest you to choose a size around 300x200 pixels – which should be ~~both, somewhat around the default as well as~~ slightly smaller than the default size and the smallest display, the video player should be used with. This way you can layout your plug-in and perhaps add a script for scalable display later. Wether you will use scripts or not, you need to define a function called initialize. Otherwise the player will throw an error!
The plug-in swf should be published somewhere relative to your test HTML file at best similar to later production environments. In the following code snippets I will assume, that the plug-in is placed in a subdirectory media/flash/plugins/ and is called plug-in.swf.
We will now pass some values to the JW-Player swf making use of the so-called flashvars. (I am sorry, I really tried to find some information about flashvars on the official sites of Adobe. I could not find anything helpful even in the livedocs. Except some outdated samples for Flash 6 and the use with ActionScript 2.0 which in this case are best of a basic use to understand what we are talking about. Perhaps someone can help out here? As there are several common ways to implement Adobe Flash in todays webpages I will attach a short paragraph containing samples of flashvar use at the end of this article.) Let us give the JW-Player a hint where to look for the plug-in. Add the flashvar plugin with the relative path to our plug-in as value, in our case media/flash/plugins/plug-in.swf. Your HTML code should look somewhat like this:

<object type="application/x-shockwave-flash"
	  data="player.swf"
	  width="300px"
	  height="200px" >
	<param name="allowScriptAccess" value="sameDomain" />
	<param name="allowFullScreen" value="true" />
	<param name="flashvars" value="plugins=media/flash/plugins/plug-in.swf" />
	<param name="quality" value="high" />

	<p>Please get the Adobe Flash Player!</p>
</object>

Be sure to use the relative path originated at the HTML document – not the swf!

Yes, it is as easy as that! Now your plug-in should be loaded automatically. But wait! Offline everything works fine, but the Flash player is not able to find the plug-in on the server. After some investigation, we know why there are troubles: We have to navigate through the ActionScript package of the JW-Player to the main view class (com → jeroenwijering → player → View.as). Just at the end of the class header there is a private string directory with an URL assigned as value: http://plugins.longtailvideo.com/. Perhaps this is a hint at how the plug-in interface usually gains utilization. If you set the directory value to be an empty string (like this private var directory:String = "";) the whole thing should work online. (Better way to solve this problem would be, take the deep dive into this piece of code and understand what really is happening here before removing some hardcoded value.)
From now on the rest is just some ActionScript wherewith we can do some magic! Since the code of the video player is well-structured and nearly every event you could imagine is dispatched all you have to do is: listen to the certain part (Model, View, Controller) and invoke some action!

The fun part – ActionScript!

Here we go: Include the View-class and declare a variable to hold the reference to the players View-instance:

	// includes
import com.jeroenwijering.player.View;

	// variables
	// Reference to View-instance of video player
var v_player:View;

At this point, lazy guys, there is no short version, no init, no programmers stenography!

And again, Jeroen Wijering made life with plug-ins easy. If you set up a function called initialize it will be called as soon as your plug-in is properly loaded by the player:

…
	// Called after loading of plugin is completed
function initialize( par_playerView:View ) {
	v_player = View( par_playerView );

	trace( "do something useful" );
}

The simple call to do something does not help much. I will give you an example of what could come next. For example it would be handy if the plug-in content would be centered if the player is resized! Let us assume we have a movie clip containing a graphic (mc_content, movie clip registration point should be on the top left) which should be horizontally centered and positioned somewhere above the vertical center (since there is this nice button to start the playback). But this graphic should disappear on playback start. We have to start with adding some lines to our plug-ins code header, furthermore we will instantiate a transition manager to handle a nice fade out of our plug-in content:

	// imports
	// video player stuff






import com.jeroenwijering.player.View;
import com.jeroenwijering.events.ModelEvent;
import com.jeroenwijering.events.ModelStates;
import com.jeroenwijering.events.ControllerEvent;

	// eye candy
import fl.transitions.Fade;
import fl.transitions.Tween;
import fl.transitions.easing.*;
import fl.transitions.Transition;
import fl.transitions.TransitionManager;

	// variables
	// Reference to View-instance of video player
var v_player:View;

	// Transition manager
var trnsmngr_content:TransitionManager = new TransitionManager( mc_content );

	// functions
	// Configure standard transitions used to fade plugin content in and out
var obj_contentFadeOut:Object = new Object();
obj_contentFadeOut.type = Fade;
obj_contentFadeOut.direction = Transition.OUT;
obj_contentFadeOut.duration = .7;
obj_contentFadeOut.easing = Regular.easeOut;

var obj_contentFadeIn:Object = new Object();
obj_contentFadeIn.type = Fade;
obj_contentFadeIn.direction = Transition.IN;
obj_contentFadeIn.duration = .7;
obj_contentFadeIn.easing = Regular.easeOut;
…

Now we have to set up some functions to handle the positioning of the content elements and kick-off the vanishing of this content element on playback.

…
	// Invoked on resize
function onPlayerResize( par_arg ) {
	alignContent( Number( par_arg.data.width ), Number( par_arg.data.height ) );
}
	// Align content movie clip
function alignContent( par_width:Number, par_height:Number ) {
	mc_content.x = Number( par_width / 2 - mc_content.width / 2 );
	mc_content.y = Number( par_height / 2 - ( mc_content.height + 40 ) );
}

	// Invoked on status change of video player
	//	- if playback has finished: fade in plugin content
	//	- if playback is (re-)started: hide plugin content
function onPlayerStateChange( par_arg ) {
	if( par_arg.data.newstate == "COMPLETED" ) {
		trnsmngr_content.startTransition( obj_contentFadeIn );
	}else if( par_arg.data.newstate == "PLAYING" && par_arg.data.oldstate == "COMPLETED" ) {
		trnsmngr_content.startTransition( obj_contentFadeOut );
	}
}
…

Obviously, the next thing to care about: event listeners! The listeners are registered in our initialize function:

…
	// Called after loading of plugin is completed
function initialize( par_playerView:View ) {
	v_player = View( par_playerView );

	v_player.addControllerListener( ControllerEvent.RESIZE, onPlayerResize );
	v_player.addModelListener( ModelEvent.STATE, onPlayerStateChange );

	alignContent( v_player.config.width, v_player.config.height );
}

Now everything should work fine. The content will be realigned if the player should ever be resized on runtime (not very likely) and faded out on playback start or in respectively on playback end. To initially align the content, there is one line added, which calls alignContent. For convenience, the whole clump of code:

	// imports
	// video player stuff
import com.jeroenwijering.player.View;
import com.jeroenwijering.events.ModelEvent;
import com.jeroenwijering.events.ModelStates;
import com.jeroenwijering.events.ControllerEvent;

	// eye candy
import fl.transitions.Fade;
import fl.transitions.Tween;
import fl.transitions.easing.*;
import fl.transitions.Transition;
import fl.transitions.TransitionManager;

	// variables
	// Reference to View-instance of video player
var v_player:View;

	// Transition manager
var trnsmngr_content:TransitionManager = new TransitionManager( mc_content );

	// functions
	// Configure standard transitions used to fade plugin content in and out
var obj_contentFadeOut:Object = new Object();
obj_contentFadeOut.type = Fade;
obj_contentFadeOut.direction = Transition.OUT;
obj_contentFadeOut.duration = .7;
obj_contentFadeOut.easing = Regular.easeOut;

var obj_contentFadeIn:Object = new Object();
obj_contentFadeIn.type = Fade;
obj_contentFadeIn.direction = Transition.IN;
obj_contentFadeIn.duration = .7;
obj_contentFadeIn.easing = Regular.easeOut;

	// Invoked on resize
function onPlayerResize( par_arg ) {
	alignContent( Number( par_arg.data.width ), Number( par_arg.data.height ) );
}
	// Align content movie clip
function alignContent( par_width:Number, par_height:Number ) {
	mc_content.x = Number( par_width / 2 - mc_content.width / 2 );
	mc_content.y = Number( par_height / 2 - ( mc_content.height + 40 ) );
}

	// Invoked on status change of video player
	//	- if playback has finished: fade in plugin content
	//	- if playback is (re-)started: hide plugin content
function onPlayerStateChange( par_arg ) {
	if( par_arg.data.newstate == "COMPLETED" ) {
		trnsmngr_content.startTransition( obj_contentFadeIn );
	}else if( par_arg.data.newstate == "PLAYING" && par_arg.data.oldstate == "COMPLETED" ) {
		trnsmngr_content.startTransition( obj_contentFadeOut );
	}
}

	// Called after loading of plugin is completed
function initialize( par_playerView:View ) {
	v_player = View( par_playerView );

	v_player.addControllerListener( ControllerEvent.RESIZE, onPlayerResize );
	v_player.addModelListener( ModelEvent.STATE, onPlayerStateChange );

	alignContent( v_player.config.width, v_player.config.height );
}

There are some additional things you should know while working with this nice player and its plug-in possibilities. Let me give you some arbitrary dodges!

v_player.config.[any name]: If you pass your own flashvars to the JW-Player, you can get the value of certain variables by this path plus your variable name at the end. It would be easy to display the videos title if you pass the flashvar title. You just need a textfield and assign the value v_player.config.title.
v_player.config.volume, v_player.config.width, v_player.config.height, v_player.config.fullscreen: There is a bunch of information in the config object of the View-instance. Here you can get the current value, width and height of the player, as well the configuration wether fullscreen is allowed and implemented or not (Nevertheless you cannot alter the state of fullscreen here! See below). To get information about the current display state watch v_player.skin.stage[ "displayState" ]
If you want to know more, have a look in the Player class' (com → jeroenwijering → player → Player.as) defaults object!
v_player.sendEvent( [event], [value] ): Using the function sendEvent you can take control of various parts of the player. Set volume to 50 % by using following code: v_player.sendEvent( "VOLUME", 50 ), set player into fullscreen mode: v_player.sendEvent( "FULLSCREEN", true ) or seek to closest keyframe to second two: v_player.sendEvent( "SEEK", 2 ).
v_player.addViewListener, v_player.addModelListener, v_player.addControllerListener: Like shown in the sample, you have the possibility to add listeners to every part of the player (Model, View, Controller). For further information about the various events, have a look at the events part of the player class package (com → jeroenwijering → events → *).

In the above samples, I assume you work in the plug-in of this tutorial. Otherwise the v_player has to be replaced by your reference to the View-instance of the JW-Player!

This article is not meant as a full featured tutorial to a full working this-and-that-performing plug-in, rather an insight on the different parts of plug-in programming for JW-Player with some short examples.

I am not quite sure about the legal matters on the use of JW-Player with altered code. So please check the information on Jeroen Wijerings homepage or contact him to assure you are not cracking anything besides the legal regulations here!

Flashvars in todays webpages

As mentioned before, there are several ways of integrating Flash files into webpages. Today a lot of people use JavaScript to embed Flash movies, there is the famous swfobject as well as the Standard Adobe Flash way to use AC_FL_RunContent. One main reason to use JavaScript is the outline rectangle the InternetExplorer shows up and which forces you to click on the Flash movie once, before you can interact with the movie. Moreover there is a check if the correct version of Adobe Flash Player is installed on the client machine.
Next we have the former standard way using markup with object- and embed-tags to integrate flash. Finally there is one possibility, first shown in an A List Apart article (for more information please visit: http://latrine.dgx.cz/how-to-correctly-insert-a-flash-into-xhtml) to integrate Flash XHTML-Strict valid with markup and without the embed-tag which is in fact a relic from times with Netscape.
For each of these methods I will give you a short code snippet on how to pass Flashvars. This is known and far from being new, but get this as a compact reference for the most popular ways to integrate Flash.

Flashvars and markup methods for Flash integration

First the markup methods, in the following example the former standard way:

<object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" 
		codebase="http://download.macromedia.com
/pub/shockwave/cabs/flash/swflash.cab#version=6,0,0,0"
		width="300px"
		height="200px" >
	<param name="allowScriptAccess" value="sameDomain" />
	<param name="allowFullScreen" value="true" />
	<param name="movie" value="player.swf" />
	<param name="flashvars" value="plugins=media/flash/plugins/plug-in.swf&title=A%20shortfilm" />

	<embed src="player.swf"
		flashvars="plugins=media/flash/plugins/plug-in.swf&title=A%20shortfilm"
		quality="high"
		width="300px"
		height="200px"
		allowScriptAccess="sameDomain"
		allowFullScreen="true" 
		type="application/x-shockwave-flash"
		pluginspage="http://www.macromedia.com/go/getflashplayer" />
	<param name="quality" value="high" />

	<p>Please get the Adobe Flash Player!</p>
</object>

In this example we pass the variables plugin and title to the swf. You can append as much variables as you needed, as long as you keep the pattern: seperate the variables using the ampersand &, assign value by using the equal sign =: &[variable name]=[value].
Now for the XHTML-Strict compatible method. Analogue to the sample at the beginning of this article, passing Flashvars would look like this:

<object type="application/x-shockwave-flash"
	  data="player.swf"
	  width="300px"
	  height="200px">
	<param name="allowScriptAccess" value="sameDomain" />
	<param name="allowFullScreen" value="true" />
	<param name="flashvars" value="plugins=media/flash/plugins/plug-in.swf&title=A%20shortfilm" />
	<param name="quality" value="high" />

	<p>Please get the Adobe Flash Player!</p>
</object>

Variable names must not contain any special characters (as usual in ActionScript), the values should be URL-encoded.

In the first example you have to declare the variables and values twice to assure, that every browser (using either the object- or the embed-tag) gets the correct values!

The JavaScript way

Since the JavaScript methods are kinda proprietary, you have to find the correct solution for every specific JavaScript Flash integration. So, the following examples feature specific code for the AS_FL_RunContent (first) and the swfobject (second) method:

<script language="javascript">
	if (AC_FL_RunContent == 0) {
		alert("This page requires \"AC_RunActiveContent.js\".");
	} else {
		AC_FL_RunContent(
			'codebase', 'http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0',
			'width', '300',
			'height', '200',
			'src', 'player',
			'quality', 'high',
			'pluginspage', 'http://www.macromedia.com/go/getflashplayer',
			'id', 'player',
			'name', 'player',
			'allowFullScreen', 'true',
			'allowScriptAccess','sameDomain',
			'movie', 'player',
			'FlashVars', 'plugin=media/flash/plugins/plug-in.swf&title=A%20shortfilm'
			); //end AC code
	}
</script>
<noscript>
	<object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
			codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0"
			width="300"
			height="200"
			id="player"
			align="middle" >
		<param name="allowScriptAccess" value="sameDomain" />
		<param name="allowFullScreen" value="true" />
		<param name="movie" value="player.swf" />
		<param name="flashvars" value="plugins=media/flash/plugins/plug-in.swf&title=A%20shortfilm" />

		<embed src="player.swf"
			flashvars="plugins=media/flash/plugins/plug-in.swf&title=A%20shortfilm"
			quality="high"
			width="300"
			height="200"
			name="player"
			allowScriptAccess="sameDomain"
			allowFullScreen="true"
			type="application/x-shockwave-flash"
			pluginspage="http://www.macromedia.com/go/getflashplayer" />
		<param name="quality" value="high" />

		<p>Please get the Adobe Flash Player!</p>
	</object>

Be sure to pass the Flashvars in the JavaScript code, as well as in the object-tag as param-tag and in the embed-tag as attribute flashvars.
Finally, the swfobject method:

<script type="text/javascript" src="swfobject.js"></script>

<div id="playerContainer">
  This is a flash video player.
</div>

<script type="text/javascript">
   var so = new SWFObject("player.swf", "player", "300", "200", "9");
   so.addVariable("plugin", "media/flash/plugins/plug-in.swf");
   so.addVariable("title", "A shortfilm");

   so.write("playerContainer");
</script>

Sample sources

The Flash source codes are uploaded here:
ZIP file containing plugin.fla
Set up an HTML document with flashvars plugins and title:

plugins=[path to plug-in.swf without extension]&title=[video title]

Outcome should be a headline showing the video title on the JW-Player splash screen.

Still Cracks in Our Foundation

2008-09-09T14:57:48Z

Welcome to thefoundation.de, our aggregate blog on media and internet technology. There are still many cracks to fill...

...but in accordance with the recent release of the Django web development framework, we decided to push our site out of the door as well. This blog is much of a spare time effort and all of us have enough other things on our hands, so that even Web 2.0 baseline commodities such as comments, trackback and pingback are still pending.

And although I generally agree with the notion that one should not always reinvent the wheel, we chose to develop this blogging application by ourselves. The main reason is that it is an easy way to get started with Django and thus with one of the more recent MVC frameworks. Another is that we wanted to be able to define relations and interactions among our four journals without limitations. And of course, there are some really nice Django applications that are more easily integrated with your own glue. There is also a more conceptual bonus which might be regarded as a disadvantage by some: Every design decision that has to be done, we have to do. There are lots of defaults in modern publishing applications, leading to a confluence in style and functionality where there could be diversity.

But, for that to happen here we still has some catching up to do. See you then!