that’s whacked

guilty

Wednesday, Jun 29. 2005  –  Category: Sun

Normally I have pretty high standards for the work I produce. I’m feeling guilty today because … well, my work sucks.

I’m supposed to finish up my tasks for my old group by Friday, before we go on our company shutdown for the July 4th week. I’ll then officially be starting in my new OpenSolaris role when we return from the break.

This means that I have to hand off my work to someone else in my group… work that I had originally planned on cleaning up, and automating myself during July/August. So now I’m caught in the middle.. I’ve been writing up a doc over the past couple of days preparing to transition my stuff to another guy in our group, Rob…. and looking at what I have so far… well… it’s ugly. It’s a hack. I’m ashamed.

Rob: if you’re reading this… sorry to have to give you my half-assed code. All I can really say is…. I really did mean it to be better.

it’s all about context

Thursday, Jun 23. 2005  –  Category: Quotage

today’s “totally taken out of context partial quote” is brought to you by poorna:

“all we need is some long poking device….. pop the plum”

okay, so i glossed over some of the words surrounding the quote.

i’m ADD for ADD puffs!

Thursday, Jun 23. 2005  –  Category: Musings

okay, so i tried to take the ADD test erik mentioned, and go to question 32 before i got bored and went back to work (and by work, i mean IM’ing erik about the ADD test :-P)

is that an automatic diagnosis of ADD?

hatred for bonobo

Wednesday, Jun 22. 2005  –  Category: Musings

not the gorilla. the object layer in GNOME.

i either hate it, or i have ADD. i’m not sure. maybe i should take the test erik mentioned.

seriously, all i want to do is query bonobo and see what song Rhythmbox (my mp3 player) is playing. this should be a relatively simple piece of C code. i know it’s simple in Python-Gnome. but for the life of me, i can’t figure out how to do it in C. and no, doing:

system("/usr/bin/rhythmbox --print-playing");

doesn’t count.

all the sample bonobo code I can find on the ‘net is circa 2000. sigh. annoying.

a yellow bowyer

Tuesday, Jun 21. 2005  –  Category: Football

brief context: Lee Bowyer, a oft-prone-to-violent-tendencies football player for Newcastle United a few years ago was acquitted of attacking an Asian student. he was talking to Birmingham City (another team), but in the end decided not to join Birmingham City because he was afraid of Asians ganging up on him and beating him up in Birmingham.

I’m serious…. go read for yourself.

I don’t even know what to say… I mean… Lee… come on. If we wanted to beat you up, we would totally send a giant robot (that’s what Asians do, after all…don’t you read manga?) to kick your ass. More to the point, don’t you think the Asians in Newcastle would have beaten you up by now if we cared?

‘But when he spoke to Steve [the Birmingham manager] right at the start, his first question was `what about the big Asian population in Birmingham?”

confused? so was i

Tuesday, Jun 21. 2005  –  Category: OpenSolaris

Some friends who have been playing around with OpenSolaris have asked me to clarify what everything means. So here’s a (very very very) brief summary of what means what, and some analogies to Linux (specifically Fedora, since I’m a Fedora-fan) and other OS’s:

Solaris 10 - This is our officially supported Sun distribution. This is a free download, and we will release binary patches, updates, etc. etc. The current release is Solaris 10 01/06 which we refer to (internally at least) as “Solaris 10 Update 1″, or s10u1 for short. The initial release was Solaris 10 03/05 which was the FCS release of Solaris 10. Update releases are “service pack” releases that add new features and fix bugs for the main Solaris 10 distribution. Solaris 10 is probably best analogous to Red Hat’s Enterprise Linux type of distribution. Except we provide free downloads in binary form (oh yeah, and support is cheaper)

Solaris Express Community Release - This is the base for OpenSolaris (currently). These are our super bleeding-edge bits… our latest internal build… what will be Solaris 11, currently codenamed Nevada. Currently, as of Feb 22nd 2006, this is Nevada, build 33 (also referred to in shorthand with snv_33, or nv_33). These are released typically every 2 weeks in sync with the ONNV build schedule.

Solaris Express - this is the official Sun ‘bleeding-edge’ bits (as opposed to the OpenSolaris ‘bleeding-edge’ bits. these are provided primarily for customers interested in testing out new features of the new Solaris release without the total cutting-edge-ness of the Solaris Express: Community Release. Currently, the newest release is 02/06 which is Nevada, build 31. These are typically updated every month or two… at a frequency less than Community Release. Solaris Express is a minimally-qualified SXCR, meaning, it boots, and has passed some minimal testing.

OpenSolaris - This is probably best analogous to kernel.org, except with more. It’s our ON (OS/Net) consolidation, and contains the kernel (like kernel.org) as well as other user-land utilities/tools (unlike kernel.org), but is NOT enough to get a whole system bootstrapped currently (like kernel.org). So you’d have to download the Solaris Express Community Edition (Nevada, build 34), and pop the OpenSolaris bits on top. Not unlike downloading Fedora Core, and then downloading the latest Linux kernel and installing it on top. Alternatively, you can download any of the other OpenSolaris-based distributions, such as Schilix, Belenix, or Nexenta GNU/Solaris. We have the OpenSolaris roadmap up for viewing which should also help give an idea for how OpenSolaris will be unfolding over the next year.

I hope that helps explain the relationships somewhat… so what should you use if you want to go Solaris? Depends, what do you need?

Are you running a server, and worried about uptime and stability? use Solaris 10. It’s our official ‘enterprise’ distribution, and is rock solid because it’s not always in flux. Plus it’s supported…. and hey, it’s still free…

Are you an end user, but want to play with the latest features that aren’t in Solaris 10 - but not necessarily interested in jumping into the code? Then you should use Solaris Express Community Edition. Also, a free download… but no guarantees you won’t find a bug, or a panic. Solaris Express is also an option, but may lag in features/bugfixes by a build or two.

And lastly, if you want to jump in and start poking at the code, then go grab Solaris Express Community Edition + OpenSolaris, or one of the previously mentioned OpenSolaris-based distros. They’re all open source, and free.

This blog entry, brought to you by the word

FREE

Update: [0222] Updated to reflect new builds, release timings, and new OpenSolaris distros besides Schilix.

Update: [1206] Updated to reflect accurate release timings

Update: [0621] Updated to reflect the difference between ‘Solaris Express’ and ‘Solaris Express Community Release’.

who? when? christmas?!?!?

Tuesday, Jun 21. 2005  –  Category: Movies&TV

for the past few months, my weekend ritual has always been to uh…get… Doctor Who on Sunday afternoon so I can watch it on my Monday or Tuesday morning BART ride down. this morning I watched the season finale, and the last appearance of Christopher Eccleston as the Doctor.

i admit, it was pretty moving and sad. the geek in me really liked Eccleston. He was gawky, nerdy, and always cobbling together some hack to get them out of their situations. i dunno about this new Doctor… i mean…he’s actually somewhat good looking. anyway…. i actually hear David Tennant is quite a good actor, so i’m crossing my fingers.

the thing that kills me is having to wait until Christmas for the next season to start. sigh. SIX months. now i’ll have to get back to doing real work on my laptop while i ride down. darnit.

more than meets the eye

Monday, Jun 20. 2005  –  Category: Linkage

you have GOT to go watch the new Citroen C4 commercial

it’s awesome. it doesn’t get any better than dancing robots smacking their asses.

note to self: don’t become a spy

Friday, Jun 17. 2005  –  Category: Quotage

Brits abroad warned: Don’t become enemy spy

Funny story from The Register. MI5 is warning Brits abroad to avoid being inadvertendly recruited for foreign intelligence agencies (including presumably, the US).

I feel left out. Nobody has ever recruited me to be a spy.

a tester’s perspective

Monday, Jun 13. 2005  –  Category: OpenSolaris

I figure there will be a barrage of OpenSolaris/Solaris related technical blogs today from all the kernel hackers here at Sun, so this is probably the best time for me to toss one in. This way, if it sucks - it will at least be amortised out by all the highly knowledgable and useful content my peers will produce.

I’ve been in the Solaris Kernel test development/QE group for a couple of years now, and given that we’re about to give our baby out to the world, I thought I should post some info on how to best look after our baby…

My group focuses on developing new tests and test suites to ensure Solaris retains its claim to fame: reliability. Solaris has built its reputation on rock solidness, and our job is to make sure all the cool whiz-bang new features we put into Solaris don’t break anything. I suppose some of the foremost assertions we make in developing tests for new features are:

1) Does the feature do what it claims? (i.e.: test its functionality) 2) Does the feature break anything related? 3) Does the feature break anything UNrelated? 4) Does the feature break anything backwards-compatibility-wise? 5) Is the feature robust?

We have a bunch of test suites which we called regression test suites that are huge extensive test collections that try to assert that the system is doing the right thing. These test APIs, library interfaces, system calls, etc. for the entire system as a whole. These are the fun ones which we run overnight and analyse the results the next morning to see what failed, why it failed, and who can we point the big finger at and yell “YOU BROKE IT!”. These are interesting, and good to run and all - but not terribly interesting for the test developer.

The fun ones are when we get to develop functional and stress tests. Functional tests are the tests where we assert basic functionality, as in, does foobar() do what it claims to do? This goes from the basic API level (as in, the man page for foobar() says it takes an integer, and returns a char* - is this true?) to test cases where we try to validate functional behaviour (as in: in a NUMA capable system, if the thread allocates all the memory out of its home locality group, then the system should allocate memory from the next closest node).

I’ll detail this second case now, as this was an actual case I wrote an assertion for recently, and was one of the more interesting test cases I’ve written. There are some test cases that are cut-and-dry (like the first API level one foobar() I mentioned above), and then there are others that can be more statistical in nature, this is one of those statistical ones. The esteemed kernel hackers I was working with (sorry Krister, I would have linked you, but can’t find your blog…) implemented a new feature called Hierarchical Lgroups (locality groups). Basically this is a new hierarchical abstraction of the lgroup NUMA/MPO (Non-Uniform Memory Access/Memory Placement Optimisation) support that Solaris already had. Previously we had support for two-level locality: local and remote. Hierarchical Lgroups (herein referred to as HLS) now defines an arbitrary-level topology of lgroups.

(if you already know how HLS works, and are just interested in the test case, then skip to the begin actual testcase stuff here text) Basically: you can local, remote, more remote, off-in-Antarctica-remote, and so on so forth. Cool feature. But do we need it?

Yes, because in some machines (basically, ANY >2 processor Opteron system, and who knows what else down the line), we see these hierarchical ‘remote-ness’. Take the 4-CPU Opteron for example, it’s configured in a ring like so:

0----1
|    |
|    |
2----3

So you can see that from CPU0, we have our closest neighbours: CPU 1 & CPU 2. we then have our furthest neighbour, CPU3. I’m sure Jonathan, Sasha, or Eric will be able to give more technical details on the implementation, but I wanted to give some background before I dove into the testcase.

In Solaris, each thread is assigned a home lgroup to run in. This lgroup obviously has at least one CPU (otherwise what would it be running on?) and likely some memory as well (but it doesn’t have to, in the case of an Opteron CPU which has no memory on that node, or a Starcat CPU board with no memory, etc.).

The most basic tenet of NUMA-ness is that you try and use memory closest to the thread. So this means we want to grab memory from our local node. Once we run out of memory, then we hit up the remote node. i.e.: aw man, we ran out of beer in the fridge. time to go hit up the closest bar for happy hour.

But whereas the previous two-level NUMA-ness would have forced us to turn away, go home, and crawl into bed if that closest bar was out of drinks - HLS allows us to keep going further and further away in search of cool frosty liquidy goodness.

So our basic tenet now becomes, get local memory, then go one level away, and then keep on going until we get some memory. In the above 4-CPU case, we see that if we’re homed to CPU0, we will try to get memory from CPU0. Once CPU0 is out of memory, we’ll hit up CPUs 1 & 2. If those are BOTH out of memory, then we go one further and hit up CPU3.

begin actual testcase stuff here So our testcase is interesting. Basically we pick a CPU on the system at random, obviously it should be one that is online (ponline() is your friend). We then verify that our home lgroup (lgrphome()) contains that CPU (lgrpcpus()). Just for assurances, I set a strong affinity for my home lgroup (lgrpaffinityset). We then collect our initial statistics on each lgroup… mainly we’re interested in how much memory each lgroup can contain (lgrpmem_size()). I call this my “memory snapshot”.

Once we’ve completed this, we’ve got our view of the memory universe according to our system. Now we start allocating away. We have our thread start allocating large chunks of memory. The easiest way I found to do this was mmap’ing in huge buggers of private anonymous memory (mmap(MAPPRIVATE|MAPANON)). After each mmap() you check to see if mmap() failed (if it did, then we’ve reached our termination point). If it didn’t, then bzero() that baby so we fault in the memory, and then take another “memory snapshot” to see how much memory is available in each lgroup again. If it succeeded, then we check if we reached the point where we see our home lgroup has little or no memory remaining (by examining the memory snapshot we just took). If it did, then save these statistics - for this is the point at which our home (local) lgroup ran out of memory. Then continue on back up to the top of the loop and allocate more memory.

We sit in this loop until mmap() fails; at which point we know we exhausted all of our memory (or at least, the memory we’re allowed to allocate (32-bit process=2GB, or whatever our resource limits are (setrlimit()). Take another “final” memory snapshot here. Every time we see our current level of memory (i.e.: home lgroup at the first level, then next closest lgroups at the next level, and so on so forth), we take another snapshot.

How much memory did we allocate with mmap()? (totalmemallocated = numberofiterations * allocationsizeperiteration). So now we look at our initial memory snapshot, and compare it to our memory snapshot we took when our home lgroup ran out of memory. 100% of the memory allocated should have been from our home lgroup… so the other lgroups should be relatively untouched (assuming an idle dedicated-test system) whilst the home lgroup dropped to 0. Let’s call this (initialsnapshot[homelgroup] - homelgroupoutofmemsnapshot[homelgroup]) value a. (notice that the homelgroupoutofmemsnapshot[homelgroup] number is 0, so really, a == initial_snapshot[homelgroup]).

Now look at the homelgroupoutofmemsnapshot and compare it to our nextlevelsnapshot. For our 4 CPU system, this would be the snapshot we took when CPUs 1 & 2 ran out of memory. Here we want to verify that of the total memory allocated in this phase (i.e.: the differences between the two snapshots), that 50% came from CPU1 & 50% came from CPU2.

We then do this so on and so forth until we hit all the different levels and have verified that all the memory came from the required lgroups for that level. We also use meminfo(MEMINFO_VLGRP)) to verify that each page of memory we allocated came from the required lgroup, if we want page-by-page accounting. This can be time-consuming though, which is why we assume an idle system and do the statistical analysis of free memory remaining in each lgroup after each phase.

So there, that’s one testcase I wrote that was fun, and a good way to learn all the relevant NUMA APIs.

That may go down in history as one of my longest blog entries ever… damn…