sometimes i really truly hate ITOps

Monday, Apr 10. 2006  –  Category: Sun

so on friday i was running a script out of my homedir. when i came back a few hours later i noticed a bunch of files missing… upon further investigation, i noticed a ton of files missing…. including my directory with all my opensolaris delivery scripts/stuff i’d written in the past 9 months.

i got freaked out and killed my script and sat there in stunned silence for about 15 minutes (well, not complete silence since i was bitching about it on #opensolaris). i then filed a ticket with ITOps to try and recover from a nightly backup.

i got a call about 10 minutes later from someone with ITOps in India who kept insisting on emailing me ‘ls’ output of my directory. i don’t need her to do that. i’m perfectly capable of doing an ls myself. she also kept trying to find out how i was connected to the SWAN (Sun’s network). she didn’t know what punchin was (our next generation VPN) and asked me a few times if i was VPN’d in (to which i replied yes, via punchin). she then asked me a what version of Windows i was running and what program was i using to connect. sigh.

anyway…nothing major was lost… i lost a few changes to some small scripts i use to deliver the nightly and build-synchronised deliveries, but nothing i can’t recreate in an hour or two. but this whole weekend i kept berating myself for writing and then running a script which had managed to trash my home directory. i felt pretty damn stupid.

until i check my email this morning and see:

Dear Jurassic users,

There was a rogue script run by ITOPS which deleted some of the files and directories in user home directories. We have disabled the script and are trying to figure out how to recover from this.

)(#!@&&#%&@#(*%&#!!!!

what the f.

this whole time i thought my script had gone insane and psycho and started trashing my home directory. turns out i was wrong. Sun ITOps had gone insane and psycho and started trashing my home directory.

i know people joke from time to time about their IT departments being unhelpful… we certainly have our fair share of disagreements with ITOps. but this is the first time i’ve genuinely felt like they were out to get me. :-P

correction: i’ve been told it wasn’t ITOps, but somebody’s rogue cron job. i suppose that’s mildly better.

No Trackbacks to “sometimes i really truly hate ITOps”

6 Comments to “sometimes i really truly hate ITOps”

  1. Geoff Arnold Says:

    “somebody’s rogue cron job”. WHOSE rogue cron job, and with what RBAC profile?

    People are always claiming that RBAC is better than sudo because you get fine-grained control. Well, yes – but you lose fine-grained logging. At least with sudo you have to explicitly code the fact that you’re doing something dangerous; with RBAC there’s always the temptation to run with pfksh as your default shell….

  2. Stephen Lau Says:

    i doubt the jurassic-slaves will tell us exactly WHOSE cron job it was. i don’t think it executed with any RBAC/sudo privs. it looks like it was taking out files/directories which were g+w, which is unfortunately, a lot of files/directories.

    once i get my home directory restored, i intend on removing all group writable privs.

  3. Jason Says:

    This is why I only use my work computer for games.

  4. Juno888 Says:

    My parents are separate computer with me coz of my games…

  5. Paul Says:

    Surely there’s some code repository you could check your scripts into so it doesn’t matter if they disappear from your homedir?

  6. Jacob Secret Says:

    Yea, i strongly reccomend you check out Subversion (or svn for short) its a simple unix command that stores copies of your scripts.

    It also allows you to retrieve older versions of the scripts if you accidentally screw it up (:

Leave a Reply


Recent posts