Posts filed under 'Code'
Reading gzip compressed data via Javascript
This weekend while I was working on my Magnatune extension for Songbird, I found I needed to fetch, expand, and parse a remote gzip’d XML document. The fetch was easy (XMLHttpRequest), as was the parse (DOMParser), but I had no idea how to do the expand.
Fortunately, Mossop over on extdev pointed me at Mozilla’s streamConverter services.
Unfortunately there wasn’t much sample code for me to blatantly rip-off^W^W^Wlearn from, so after much bumbling around like the JS amateur that I am, I finally got something working. I’m documenting it here so that hopefully others might find it useful. Or at the very least, I can look it up again when I will inevitably need to do this again 
Mossop first pointed out that I wouldn’t be able to use XMLHttpRequest and that I would need to open a channel:
// Get the IO service
var ioService = Cc["@mozilla.org/network/io-service;1"]
.getService(Ci.nsIIOService);
// Create an nsIURI
var mtUri = ioService.newURI(magnatuneURL, null, null);
// Create a channel from that URI
var chan = ioService.newChannelFromURI(mtUri);Awesome. The tricky part now is the docs lie. They say there is a gzip to uncompressed stream converter that implements asyncConvertData() and a synchronous convert(). I opt’d for synchronous since it seemed easier to get working off the bat, but kept getting error messages saying it wasn’t implemented. Turns out that’s true. The gzip->uncompressed method only implements asyncConvertData. So now I’d need to define a stream listener (implementing the nsiStreamListener) interface. This is the listener that is invoked for each uncompressed chunk. It needs to implement onStartRequest, onStopRequest, & onDataAvailable where onDataAvailable is passed the uncompressed data:
function StreamListener() {
this._data = null;
this._first = true;
}
StreamListener.prototype = {
onStartRequest: function(aReq, aContext) {},
onStopRequest: function(aReq, aContext, aStatusCode) {
// this._data is my full uncompressed file now, for Magnatune this is my
// XML file, so now I can go do whatever I want with it.
Magnatune.Controller.completeSyncWithStore(this._data);
},
onDataAvailable: function(aReq, aContext, aInputStream, aOffset, aCount) {
var binInputStream = Cc["@mozilla.org/binaryinputstream;1"]
.createInstance(Ci.nsIBinaryInputStream);
binInputStream.setInputStream(aInputStream);
if (this._first) {
this._data = binInputStream.readBytes(binInputStream.available());
this._first = false;
} else
this._data += binInputStream.readBytes(binInputStream.available());
binInputStream.close();
}
};So now that I have my channel open, and my stream listener defined - I need to create my nsIStreamConverter service to take the gzip’d data from the channel, and pass it to the stream listener so it can do its thing with the uncompressed data.
// Get the converter service
var converterService = Cc["@mozilla.org/streamConverters;1"]
.getService(Ci.nsIStreamConverterService);
// Instantiate our gzip decompresser converter
var converter = converterService.asyncConvertData("gzip",
"uncompressed", myListener, null);So now that we have all our pieces defined, all that’s left to do is pass the converter to the channel and start the pipeline:
// Initiate the asynchronous open. This will initiate the connection
// to Magnatune, grab the gzip'd data and pass it to our gzip converter
// which will then call the StreamListener, so our completion hook is
// fired in the StreamListener's onStopRequest()
chan.asyncOpen(converter, null);Awesome. So now every Songbird/Magnatune user will be downloading a 300kb gzip’d file instead of a massive 6MB file each time they sync with the Magnatune DB.
10 comments May 5th, 2008
a cross-disciplinary productive day
i’ve had a fun day catching up on all sorts of little odds and ends i’d had on my to do list.
- finished the final step in my migration of all my users’ home directories and web-space to ZFS.
- setup drupal and started building the portal for my neighbourhood. our neighbourhood improvement association was blowing away $45/month on crappy webhosting for a phpBB installation that kept getting hacked, and a mailman list. it’d frustrated me long enough that i finally volunteered to be the webmaster. i’m in the process of building a new website from scratch that will be a full portal to our neighbourhood and provide blogs, message boards, mailing lists, and web space to any resident, organisation/club, or store in our neighbourhood
- did lots of research on various brake pads for my A4. i’m planning on replacing my front brake pads tomorrow. it’s been tough finding a good recommended brake pad that has a firm bite, not-as-much brake dust, and a proper connection for a wear sensor. i’m finding that it’s hard to find a good firm sport brake that will endure my, shall we say, vigorous, driving that also has a wear sensor.
- installed bugzilla on grommit. i finally got tired of hearing about our bug database sucking, and the points about project bug-tracking for externally driven projects (like ksh93) are perfectly valid. yes i realise there is much to be done on b.o.o., but due to the process, policy, and implementation of Sun’s bug tracking system, there are lot more constraints of b.o.o. i do intend to help internally anyway i can to make it better; but in the meantime, hopefully the bugzilla install will let external projects start tracking bugs and just make progress
my random thoughts for the day:
- the new vienna teng cd is incredible. her voice has definitely matured since her first two CDs. the music is less raw now, a little more polished/produced. this is neither good, nor bad… just different. the CD is brilliant. highly recommended.
- there’s nothing quite like a freshly bathed beagle as your soft/plush pillow when you’re taking a nap. lovely.
- it’s getting colder and gloomier out now. i took the dogs to Pt. Isabel, and actually had to wear a beanie. this was also my first official soft-shell-jacket wear of the season. it’s definitely fall.
- i’d always known Pt. Isabel was big… i didn’t realise it was the largest off-leash dog park in the nation. wow. go east bay parks
- the chili/sourdough baguette at the cafe at Pt. Isabel rocks. highly recommended.
Add comment September 9th, 2006
SF in jello
my manager sent this to our group…. entitled “San Francisco in Jell-O”. why were we talking about this? it related to me writing a tool in threaded-Perl, which our resident Perl expert had the following endearing remarks on:
“Plus threaded perl is about as unstable as a Jello Empire State building… “
Add comment October 31st, 2005
build 22 done! (opensol-20050909)
After much crypto review, and a lot of SPARC building… rebuilding… and more rebuilding, I’ve finished the build 22 delivery. New in this release is the following:
- crypto drivers: aes, arcfour, blowfish, des, rsa
- GSS, Kerberos, OpenSSH, OpenSSL, some PAM modules, PKCS11, SASL, crypt, wanboot, & the dprov driver.
You should be able to find it on the OpenSolaris downloads page tomorrow if it gets pushed to SDLC properly. The source browser probably won’t be updated until Monday.
Enjoy!
Add comment September 9th, 2005
more work on grommit and an SMF manifest for mailman
I’ve spent much of the day working on the new grommit box. Ran the big ’smpatch update’, got my BIND fix, and I seem to have all my non-web services/applications up and running. Getting mailman up and running was somewhat of a pain. I used it as a good exercise in writing an SMF manifest though. I ended up deriving one from the network/smtp:sendmail one… undoubtedly, it could be better, but here you go:
< ?xml version='1.0'?>
< !DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
<service name='network/mailman' type='service' version='0'>
<single_instance />
<dependency name='fs-local' grouping='require_all' restart_on='none' type='s
ervice'>
<service_fmri value='svc:/system/filesystem/local'/>
</dependency>
<dependency name='network-service' grouping='require_all' restart_on='none'
type='service'>
<service_fmri value='svc:/network/service'/>
</dependency>
<dependency name='name-services' grouping='require_all' restart_on='refresh'
type='service'>
<service_fmri value='svc:/milestone/name-services'/>
</dependency>
<dependency name='identity' grouping='optional_all' restart_on='refresh' typ
e='service'>
<service_fmri value='svc:/system/identity:domain'/>
</dependency>
<dependency name='system-log' grouping='optional_all' restart_on='none' type
='service'>
<service_fmri value='svc:/system/system-log'/>
</dependency>
<dependency name='sendmail' grouping='require_all' restart_on='none' type='s
ervice'>
<service_fmri value='svc:/network/smtp:sendmail'/>
</dependency>
<instance name='mailman' enabled='true'>
<dependent name='multi-user' restart_on='none' grouping='optional_all'>
<service_fmri value='svc:/milestone/multi-user'/>
</dependent>
<exec_method name='start' type='method' exec='/lib/svc/method/mailman star
t' timeout_seconds='120'>
<method_context />
</exec_method>
<exec_method name='stop' type='method' exec='/lib/svc/method/mailman stop
%{restarter/contract}' timeout_seconds='60'>
<method_context />
</exec_method>
<exec_method name='refresh' type='method' exec='/lib/svc/method/mailman re
start' timeout_seconds='60'>
<method_context />
</exec_method>
<property_group name='startd' type='framework'>
<propval name='ignore_error' type='astring' value='core,signal'/>
</property_group>
<property_group name='general' type='framework'>
<propval name='action_authorization' type='astring' value='solaris.smf.m
anage.mailman'/>
</property_group>
<template>
<common_name>
<loctext xml:lang='C'>GNU mailman mailing list manager</loctext>
</common_name>
</template>
</instance>
<stability value='Unstable'/>
</service>
</service_bundle>
the main things I have left to get running are all the web apps, specifically: the photo gallery, webmail, blogs, and mysql.
4 comments August 14th, 2005
pkgxtra
a while ago i wrote a stupid little shell script called pkgxtra that lets me easily list files owned by a package (either by .pkg filename, or package name (like SUNWcsl)). it also provides a shortcut to see which package owns a given pathname/filename.
here it is, in all its glory:
#!/bin/bash
show_help() {
echo "usage: $0 [-lfp] “;
echo ” -l lists files belonging to package”;
echo ” -f lists files contained in .pkg file”;
echo ” -p show package owner of “;
}
while getopts “lpfh” flag
do
if [ “$flag” == “l” ]; then
MODE=0;
fi
if [ “$flag” == “p” ]; then
MODE=1;
fi
if [ “$flag” == “f” ]; then
MODE=2;
fi
if [ “$flag” == “h” ]; then
show_help
exit 1;
fi
done
shift $(($OPTIND - 1))
if [ “$#” -eq “0″ ]; then
show_help
exit 1;
fi
case “$MODE” in
0) pkgchk -l $1|awk ‘/^Pathname/ {print $2}’;;
1) pkgchk -l -p $1;;
2) pkgchk -l -d $1|awk ‘/^Pathname/ {print $2}’;;
esac
Add comment August 2nd, 2005
NUMA observability tools
I’m happy to see Sasha put up code and binaries for the NUMA observability tools I helped write tests for.
These are really cool utilities to help observe memory placement and optimise “advise” for programs to tweak where they place their memory on NUMA systems. Really cool stuff…
Add comment August 2nd, 2005
skippy
so i’ve been using skippy for a while now on my Linux laptop. i read on [moinak’s]( http://blogs.sun.com/roller/page/moinakg?entry=whenthetaskbarisnot) blog that he had to make a slight modification for Solaris… turns out this isn’t necessary under Xfce since Xfce supports the XGrabKey call.
only problem is that skippy doesn’t support Xsun’s xinerama extension. doh. i make a quick hack to wrap the Xorg Xinerama extensions, and now it works great!
you can grab my skippy-0.5.0.Xsun.tar.gz tarball and try it out. i’ve tested it on Solaris 10/SPARC under Xfce.
Add comment July 27th, 2005
libgtop on Solaris 10
If you are trying to compile libgtop on Solaris 10, you will notice that a bunch of the sysdeps/solaris files won’t compile, specifically sem_limits.c, shm_limits.c, and msg_limits.c. You’ll probably see errours about shminfo, etc. etc.
This is because many of these interfaces were unstable and obsoleted in Solaris 10 in favour of the (much better) rctladm interfaces. In short, you can get use getrctl to get those interfaces now.
You can read some forum msgs on dbForums regarding it.
or look at the Solaris 10 Tunable Parameters Reference Manual for a complete list of what was obsoleted.
I’ve made a patch to libgtop-2.8.3 which should fix this (it does for me, anyway). As usual - this is not an official Sun-endorsed patch in any way, form, shape, blah blah blah. Your mileage may vary.
Add comment July 25th, 2005
converting m4a to mp3
ripped a bunch of CDs on Wendy’s laptop with iTunes, which saved them in m4a format. i see gStreamer supports the m4a format, but i’d rather have it in mp3 just for uniformity.
i know i know, “you should just switch to ogg”
anyway:
perl -e ‘foreach $file (ls *.m4a) { chomp $file; $newfile = $file; $newfile =~ s/.m4a/.mp3/; faad -w "$file" | lame -b 192 - "$newfile"; }’
i know this could be written as a much shorter shell script, but i know perl better than shell scripting, and this works under my linux system - so i’ll stick with it.
6 comments November 29th, 2004
