Jesper Noehr

Since Python 2.5, Python has had the ‘functools’ module for doing various higher order functions.

For example:

from functools import partial

def adder(first, second):
	return first + second

adder10 = partial(adder, 10)

print adder10(32) # -> 42

Partial evaluation, eh? That’s kinda cool.

On to ‘wraps’ which is the one I’ve found most practical use for. I like decorators, and I use them where applicable. What I don’t like about decorators is that when you get a backtrace, it’ll actually show up as *that* function, and not the function you decorated.

‘wraps’ to the rescue:

from functools import wraps

def some_decorator(f):
	def wrap(*args, **kwargs):
		return f(*args, **kwargs)
	return wraps(f)(wrap)

@some_decorator
def some_function():
	...

Now the function name, docstring, signature, etc. will be that of ‘f’, no longer ‘wrap’! Immensely useful.

I just wanted to do a quick write-up on a couple of things, because:

1. I wanted to announce two upcoming projects of mine, and
2. Getting a new post out there

Piston

Piston’s a django-app I’m writing for Bitbucket. It serves as sort of a “mini-framework” on top of Django for creating RESTful APIs. Well, actually it doesn’t tie you to be RESTful at all, as its url mapping facility hooks directly into Django.

A while back, jacobian wrote an article, “REST worst practices”, outlining some of the things a good implementation would need. I’m happy to say that Piston’s elegantly waltzing its way through the list, checking off his points one by one.

We don’t tie a resource to a model (although you easily can), we have plug-able authentication (with new handlers being a breeze to add), configurable output formats (in form of “emitters”, a simple dict-to-x facility, comes with emitters for JSON, YAML and XML), proper use of HTTP (status codes, headers) and CRUD semantics, and best of all, it ties right in to your Django application.

Anyway, I wrote it for Bitbucket, but it definitely merits an open source release and its own project. It’s behind closed doors right now, but nearing completion. Once we feel it’s good to release, we’ll do a release together with David Larlet, the author of Semantic Django (who else?)

Oberon

Oberon’s also something we use on Bitbucket. It’s a queue-based “application platform” based on Twisted. Vague, huh?

No, we use it for the service integration facility of Bitbucket. Oberon itself is just a daemon, serving as a message-passing facility between the client and what I call “brokers”. A broker is a piece of Python code that must satisfy two things:

1. It must contain a class that subclasses “BaseBroker”, and
2. That class must have a “handle” method receiving a single argument, “payload”

What this allows you to do is pretty nifty. You can load up a few of these brokers, and then using the client API, you can send messages to Oberon, and it’ll take it from there.

For example, we have a couple of brokers, like Twitter, which extracts the information it wants from the payload and uses a Twitter client library to post messages. There’s a Basecamp broker, and the most popular one thus far is the “Issue” broker, which parses commit messages and acts on them. Stuff like “great, all done, fixes #42″ will close up issue 42, and “hm, needs more work, references #37″ will add a comment to issue 37.

Best of all, and my favorite feature is ‘oberonc’, the command line client. It’s pretty basic but it has useful commands like ’stats’, ‘brokers’ and best of all: ‘reload’ — yep, that’s right, you can reload brokers on the fly without disrupting service. It works really well too, due to the way we’ve designed the application. It also means you can load up new brokers that have never been loaded before, so it makes it really interesting to upgrade running systems.

None is this stuff is tied into Bitbucket, so it has a vast variety of uses. It runs on top of ‘twistd’ as well so it should be pretty stable and scalable (it uses stuff like epoll.)

Anyway, Oberon’s also getting its own open source release, together with all the brokers we’ve written for the service integration we’re using on the live system. Those should serve as good examples.

I’ll post about both here, when they’re out.

On BitBucket, we need to handle streaming data through Django. This lowers the memory footprint of the application and makes execution faster.

The problem with this is that several stock middleware in Django “look” at the content before sending it. This is a problem for streaming content, since you’d generally use a generator, and you can’t consume it until the very last minute.

The middleware in Django that does this is ConditionalGetMiddleware which attempts to create an ‘ETag’ header, and CommonMiddleware, which attemps to create a ‘Content-Length’ header.

Here’s an easy way of not executing certain middleware in such cases:

def wsgi_compat_middleware_factory(klass):
    class compatwrapper(klass):
        def process_response(self, req, resp):
            if not whatever_condition:
                return klass.process_response(self, req, resp)
            return resp
    return compatwrapper

This is a “factory”, returning a class that can you use instead of the normal middleware. On BitBucket, the condition is ‘if not req.is_mercurial():’. Replace with whatever makes sense for you.

You use it by doing something like this:

from django.middleware.http import ConditionalGetMiddleware
from django.middleware.common import CommonMiddleware

StreamingConditionalGetMiddleware = wsgi_compat_middleware_factory(ConditionalGetMiddleware)
StreamingCommonMiddleware = wsgi_compat_middleware_factory(CommonMiddleware)

Now you have two new classes – Just install those in place of the stock middleware, and viola.