sometimes i really truly hate ITOps
April 10th, 2006 Stephen Lau
so on friday i was running a script out of my homedir. when i came back a few hours later i noticed a bunch of files missing… upon further investigation, i noticed a ton of files missing…. including my directory with all my opensolaris delivery scripts/stuff i’d written in the past 9 months.
i got freaked out and killed my script and sat there in stunned silence for about 15 minutes (well, not complete silence since i was bitching about it on #opensolaris). i then filed a ticket with ITOps to try and recover from a nightly backup.
i got a call about 10 minutes later from someone with ITOps in India who kept insisting on emailing me ‘ls’ output of my directory. i don’t need her to do that. i’m perfectly capable of doing an ls myself. she also kept trying to find out how i was connected to the SWAN (Sun’s network). she didn’t know what punchin was (our next generation VPN) and asked me a few times if i was VPN’d in (to which i replied yes, via punchin). she then asked me a what version of Windows i was running and what program was i using to connect. sigh.
anyway…nothing major was lost… i lost a few changes to some small scripts i use to deliver the nightly and build-synchronised deliveries, but nothing i can’t recreate in an hour or two. but this whole weekend i kept berating myself for writing and then running a script which had managed to trash my home directory. i felt pretty damn stupid.
until i check my email this morning and see:
Dear Jurassic users,
There was a rogue script run by ITOPS which deleted some of the files and directories in user home directories. We have disabled the script and are trying to figure out how to recover from this.
)(#!@&&#%&@#(*%&#!!!!
what the f.
this whole time i thought my script had gone insane and psycho and started trashing my home directory. turns out i was wrong. Sun ITOps had gone insane and psycho and started trashing my home directory.
i know people joke from time to time about their IT departments being unhelpful… we certainly have our fair share of disagreements with ITOps. but this is the first time i’ve genuinely felt like they were out to get me. 
correction: i’ve been told it wasn’t ITOps, but somebody’s rogue cron job. i suppose that’s mildly better.
6 Comments Add your own
-
1.
Geoff Arnold | April 10th, 2006 at 11:55
“somebody’s rogue cron job”. WHOSE rogue cron job, and with what RBAC profile?
People are always claiming that RBAC is better than sudo because you get fine-grained control. Well, yes - but you lose fine-grained logging. At least with sudo you have to explicitly code the fact that you’re doing something dangerous; with RBAC there’s always the temptation to run with
pfkshas your default shell…. -
2.
Stephen Lau | April 10th, 2006 at 12:02
i doubt the jurassic-slaves will tell us exactly WHOSE cron job it was. i don’t think it executed with any RBAC/sudo privs. it looks like it was taking out files/directories which were g+w, which is unfortunately, a lot of files/directories.
once i get my home directory restored, i intend on removing all group writable privs.
-
3.
Jason | April 11th, 2006 at 19:39
This is why I only use my work computer for games.
-
4.
Juno888 | June 17th, 2007 at 20:34
My parents are separate computer with me coz of my games…
-
5.
Paul | August 31st, 2007 at 11:13
Surely there’s some code repository you could check your scripts into so it doesn’t matter if they disappear from your homedir?
-
6.
Jacob Secret | June 23rd, 2008 at 21:19
Yea, i strongly reccomend you check out Subversion (or svn for short) its a simple unix command that stores copies of your scripts.
It also allows you to retrieve older versions of the scripts if you accidentally screw it up (:
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
