that’s whacked

Reading gzip compressed data via Javascript

Monday, May 5. 2008  –  Category: Code, Songbird

This weekend while I was working on my Magnatune extension for Songbird, I found I needed to fetch, expand, and parse a remote gzip’d XML document. The fetch was easy (XMLHttpRequest), as was the parse (DOMParser), but I had no idea how to do the expand.

Fortunately, Mossop over on extdev pointed me at Mozilla’s streamConverter services.

Unfortunately there wasn’t much sample code for me to blatantly rip-off^W^W^Wlearn from, so after much bumbling around like the JS amateur that I am, I finally got something working. I’m documenting it here so that hopefully others might find it useful. Or at the very least, I can look it up again when I will inevitably need to do this again

Mossop first pointed out that I wouldn’t be able to use XMLHttpRequest and that I would need to open a channel:

// Get the IO service
var ioService = Cc["@mozilla.org/network/io-service;1"]
        .getService(Ci.nsIIOService);
// Create an nsIURI
var mtUri = ioService.newURI(magnatuneURL, null, null);
// Create a channel from that URI
var chan = ioService.newChannelFromURI(mtUri);

Awesome. The tricky part now is the docs lie. They say there is a gzip to uncompressed stream converter that implements asyncConvertData() and a synchronous convert(). I opt’d for synchronous since it seemed easier to get working off the bat, but kept getting error messages saying it wasn’t implemented. Turns out that’s true. The gzip->uncompressed method only implements asyncConvertData. So now I’d need to define a stream listener (implementing the nsiStreamListener) interface. This is the listener that is invoked for each uncompressed chunk. It needs to implement onStartRequest, onStopRequest, & onDataAvailable where onDataAvailable is passed the uncompressed data:

function StreamListener() {
    this._data = null;
    this._first = true;
}   

StreamListener.prototype = {
    onStartRequest: function(aReq, aContext) {},
    onStopRequest: function(aReq, aContext, aStatusCode) {
        // this._data is my full uncompressed file now, for Magnatune this is my
        // XML file, so now I can go do whatever I want with it.
        Magnatune.Controller.completeSyncWithStore(this._data);
    },  
    onDataAvailable: function(aReq, aContext, aInputStream, aOffset, aCount) {
        var binInputStream = Cc["@mozilla.org/binaryinputstream;1"]
                    .createInstance(Ci.nsIBinaryInputStream);
        binInputStream.setInputStream(aInputStream);
        if (this._first) {
            this._data = binInputStream.readBytes(binInputStream.available());
            this._first = false;
        } else
            this._data += binInputStream.readBytes(binInputStream.available());
        binInputStream.close();
    }
};

So now that I have my channel open, and my stream listener defined - I need to create my nsIStreamConverter service to take the gzip’d data from the channel, and pass it to the stream listener so it can do its thing with the uncompressed data.

// Get the converter service
var converterService = Cc["@mozilla.org/streamConverters;1"]
            .getService(Ci.nsIStreamConverterService);
        
// Instantiate our gzip decompresser converter
var converter = converterService.asyncConvertData("gzip",
            "uncompressed", myListener, null);

So now that we have all our pieces defined, all that’s left to do is pass the converter to the channel and start the pipeline:

// Initiate the asynchronous open.  This will initiate the connection
// to Magnatune, grab the gzip'd data and pass it to our gzip converter
// which will then call the StreamListener, so our completion hook is
// fired in the StreamListener's onStopRequest()
chan.asyncOpen(converter, null);

Awesome. So now every Songbird/Magnatune user will be downloading a 300kb gzip’d file instead of a massive 6MB file each time they sync with the Magnatune DB.