that’s whacked

an embarrassment of bounces

Tuesday, Sep 12. 2006  –  Category: OpenSolaris

As many of you know, I’ve been working on the ON mercurial repository. For the past couple of weeks, I’ve been working on the external (i.e.: outside Sun) mirrour of the ON Mercurial repository.

This week I’d been ironing out in the kinks in our Mercurial project hosting on opensolaris.org, doing test ‘pushes’ of source code from my internal ON mirrour to the external mirrour.

Today was both a joyous, and ignoble day.

Joyous because I ironed out a bug that had been blocking me for the past couple of days.

Ignoble, because in doing so, I managed to trigger putback notifications for every changeset I pushed. All 2,750 of them. 2,750 emails all destined for me (I had put myself as the notification email address).

Doh.

I sat there and watched 20 emails come in.

Then another 40.

Then another 60.

I noticed my home directory access started lagging. Logging into jurassic (the home directory/web/mail server for all of my building and most of the Solaris Kernel development group) showed a load average spiking up near 100, and almost 5000 processes running under stevel.

Damn. I needed to stop the email, so I put in a procmail recipe to direct them to /dev/null.

5000 sh/procmail processes… all forking unhappily trying to process all my incoming mail… they weren’t going into my mailbox, but they were still wreaking havoc bogging jurassic down.

I thought I would be the good citizen and kill them, so I sent them all a SIGKILL.

Bad idea

Apparently, when you send procmail sigkill, it terminates and sends a bounce-back email message to the originator of the email it was processing. My whizzy cool Mercurial notification hooks sets the originator of the email to be the author of the changeset.

….. and wham, in an instant, I just sent 4958 ‘spam’ email bouncebacks to the 571 people in ON that had done putbacks to the ONNV gate.

One by one I got emails, and phone calls from irate, amused, and annoyed colleagues asking why I was spamming them with bounces. Many thanks to the denizens of #onnv for taking it in stride and laughing it off, especially Danek who received more than his fair share as ON gatekeeper. Many many apologies to cth and eschrock who were the two who had completed the most putbacks, and thus were rewarded with the most bouncebacks. Many thanks to esaxe, akolb, and jjc who were in the conference room with me (we were having a remote PG work day) for laughing and trying to make me feel better about my retardness.

danek and esaxe were the only two to notice my highly-technical 3733t repository name. kudos to you guys.

sigh. … and that has pretty much been my most mortifying moment at Sun so far.