Python WSGI Middleware for automatic Gzipping

Evan — Thu, 25 Dec 2008 13:11:52 +0000

I've just started learning Python WSGI (PEP-333) and thought the best way to learn would be to write some WSGI tools myself. Most recently, I chose to write a middleware application that converts all output into valid gzipped data. In this article, I will be demonstrating how my middleware gzipper works and how to implement it.

Gzipping arbitrary data

The first obstacle was that Python has no simple way of converting a normal string into a gzipped string. To do this, let's build off of the gzip.GzipFile object, which isn't really designed for this sort of task.

#!/usr/bin/python2.5
from gzip import GzipFile
import StringIO
 
def gzip_string(string, compression_level):
    """ The `gzip` module didn't provide a way to gzip just a string.
        Had to hack together this. I know, it isn't pretty.
    """
    fake_file = StringIO.StringIO()
    gz_file = GzipFile(None, 'wb', compression_level, fileobj=fake_file)
    gz_file.write(string)
    gz_file.close()
    return fake_file.getvalue()

The above function accomplishes this nicely. By using a `StringIO` instance for the `fileobj` in the `GzipFile`, it allows us to make the function think that it is writing to a normal file. Instead, we're grabbing the content through our fake file.

Does the client want gzipped output?

This is a pretty important task. What if the client didn't even ask for gzipped output? Or worse, what if they can't even support it? Cases like these must be taken into consideration. We'll use the following code to check.

def parse_encoding_header(header):
    """ Break up the `HTTP_ACCEPT_ENCODING` header into a dict of
        the form, {'encoding-name':qvalue}.
    """
    encodings = {'identity':1.0}
 
    for encoding in header.split(","):
        if(encoding.find(";") &gt; -1):
            encoding, qvalue = encoding.split(";")
            encoding = encoding.strip()
            qvalue = qvalue.split('=', 1)[1]
            if(qvalue != ""):
                encodings[encoding] = float(qvalue)
            else:
                encodings[encoding] = 1
        else:
            encodings[encoding] = 1
    return encodings
 
def client_wants_gzip(accept_encoding_header):
    """ Check to see if the client can accept gzipped output, and whether
        or not it is even the preferred method. If `identity` is higher, then
        no gzipping should occur.
    """
    encodings = parse_encoding_header(accept_encoding_header)
 
    # Do the actual comparisons
    if('gzip' in encodings):
        return encodings['gzip'] &gt;= encodings['identity']
 
    elif('*' in encodings):
        return encodings['*'] &gt;= encodings['identity']
 
    else:
        return False

What this does is parse the `HTTP_ACCEPT_ENCODING` response header to make sure that it says that gzipping is alright and preferred over no compression. It does this by looking at the qvalue weights for each type of encoding. For more information on this, I suggest you read RFC 2616, section 14 which covers the "Accept-Encoding" header.

Creating the WSGI middleware

Alright, now it's time to use that function an create a middleware application.

class Gzipper(object):
    """ WSGI middleware to wrap around and gzip all output.
        This automatically adds the content-encoding header.
    """
    def __init__(self, app, compresslevel=6):
        self.app = app
        self.compresslevel = compresslevel
 
    def __call__(self, environ, start_response):
        """ Do the actual work. If the host doesn't support gzip as a proper encoding,
            then simply pass over to the next app on the wsgi stack.
        """
        accept_encoding_header = environ.get("HTTP_ACCEPT_ENCODING", "")
        if(not client_wants_gzip(accept_encoding_header)):
                return self.app(environ, start_response)
 
        def _start_response(status, headers, *args, **kwargs):
            """ Wrapper around the original `start_response` function.
                The sole purpose being to add the proper headers automatically.
            """
            headers.append(("Content-Encoding", "gzip"))
            headers.append(("Vary", "Accept-Encoding"))
            return start_response(status, headers, *args, **kwargs)
 
        # Unfortunately, we can't gzip data chunk-by-chunk, so we need to join up whatever
        # is being sent out first. (Remember, WSGI apps return items which can be iterated.)
        data = "".join(self.app(environ, _start_response))
        return [gzip_string(data, self.compresslevel)]

Before doing any encoding, this checks to make sure the client can even support gzipped output by looking at the "HTTP_ACCEPT_ENCODING" header value. What's really cool about how this works is that it adds a content-encoding header for you. This, however, shows a weakness in WSGI as the middleware is required to write a new start_response function that wraps around the old one. WSGI 2.0 clears this up, but this version isn't in use yet.

Using the middleware

Now, let's see how we would implement this with a simple "hello world" WSGI application.

#!/usr/bin/python2.5
from gzipper import Gzipper
from wsgiref.simple_server import WSGIServer, WSGIRequestHandler
 
#
# Let's create a simple WSGI application to test out the Gzipper.
#
def test_app(environ, start_response):
    status = "200 OK"
    headers = [("content-type", "text/html")]
    start_response(status, headers)
 
    return ["Hello world!"]
 
# Set up the WSGI server and add the middleware that wraps around
# the actual application.
httpd = WSGIServer(('', 8080), WSGIRequestHandler)
httpd.set_app(Gzipper(test_app, compresslevel=8))
httpd.serve_forever()

Final thoughts

Middleware can be incredibly useful. I, however, have been seeing numerous people approaching it incorrectly by modifying the environ and having the application relying on it. I believe that DirtSimple.org said it best: "If your application requires that API to be present, then it's not middleware any more!" It would be better to just add that code to a library utilized by the application.

To view all of the code in one block, go to the next page.

Lighttpd Directory Generator using mod_dirlisting With Sorting

Evan — Wed, 03 Sep 2008 21:45:36 +0000

I have put up a new script for those of you who are running Lighttpd. You can find it here. It is a dir-generator that is nearly identical to the default, but with the added bonus that you can sort by name, size, modification time, and file type. One downside is that it doesn't display mime type like the original does since to do that with PHP, it is required that an extension is insalled.

I Before E, Except After C - Not a rule, but a joke

Evan — Fri, 04 Jul 2008 04:59:47 +0000

When I was in high school, I always had fun finding exceptions the "I before e, except after c" rule. I found over a hundred on my own back then just by searching around the Internet. It took me hours. Today I decided to take a better approach and wrote up a small script in to do the work for me. All in all, it found 533 exceptions to this rule.

Click here to view the list of exceptions.

Yeah, that's quite a few. If you're interested in how I did it, continue reading. Don't worry, I won't be sad if you came here just for the list.

The Rule, Broken Down

If the word contains "ei", it must be in the form of "cei" or make a long-A sound.
If the word contains "ie", it cannot be in the form of a "cie".

The Tools Used

The Python Programming Language
Regular Expressions
A file containing all English words [ get here ]

The Code

Writing up the code is pretty simple. Basically all it does is open the dictionary file and check each word against the rule. If it doesn't follow the rule, it logs it in the output file.

import re
 
# Open the input & output file streams
input_handle = file("dictionary.txt", "r")
output_handle = file("i_before_e.txt", "w")
 
# Regexes for removal of valid
bad_ei_regexes = [
 
      # Valid case by default
      re.compile('cei'), 
 
      # Valid case with a long-A sound
      re.compile('^(z|p)ei'),
      re.compile('(?

Evan Fosmark