A few non-secrets

A month and a half without a post, yow.  Ok, so here's the thing.  I've had stuff going on in my life, but a lot of it involves friends, and their privacy.  I can't talk about the friend getting surgery, because that's medical.  I can't talk about the acquaintance who I suspect may be trying to manipulate me, because it's just a hunch right now.  There's a lot of stuff I can't talk about going on.

Here's some things I can talk about.

Two of my best friends are getting married in two weeks.  Congratulations, Scott and Mili!  Scott and I have been close friends since fourth grade, and he and Mili have been crazy about each other for years.

My work has been progressing well.  I've still been working on the project I talked about in my last post, but the pilot has been running more slowly than we had expected.  There's been some minor technical changes from the description I made (most notably changing from using a magic uid/gid as a marker to using UFS2 extended attributes), but all the details will be out when we open-source it.

As you know, I've occasionally been writing some music.  Mostly funny stuff, not good stuff, although I do kinda like "Baby Wombats (House Remix)".  I've also been working on a metal song about puppies and rainbows and stuff.  I've got it written, but need to re-record it since what I've recorded so far is just draft quality.

I hope I'll have something more interesting to post soon!

What I'm working on at Juniper

I haven't had much time to update my blog for a while.  It's mostly been work, so I think I'll tell you all about it.

I'm loving my job at Juniper, although I must say it's keeping me very busy lately.  The project I'm on is pretty close to the wire.  In the weeks approaching the Christmas holidays, I was spending every waking minute working on my code to get it ready.  That left me able to go to Christmas and enjoy it without worrying about work (more about my vacation in another post, but the basic summary is, it was terrific!).  Even though my part is working great, there's some other parts to the project that are a bit tense, so I'm helping where I can.  Since I got back from vacation, though, things have been a bit more relaxed for me.

Anyway, I can't go into all the background on the project, but I can tell you about my part.  (This gets a bit technical from here on in, so you can skip it if you're not a computer geek.)  The goal is to make it possible to compile with source code on NFS servers and object code on local disk.  The source then can be edited remotely on workstations, and easily backed up; the objects don't take up expensive NetApp space, and the compile proceeds at local speeds.  This mechanism speeds up compiles by a factor of 2-3.

The core of this is a new filesystem layer that I wrote, lcachefs.  This is a type of loopback filesystem, similar to nullfs or unionfs.  lcachefs does some magic mirroring involving three directories.  As with any filesystem, one of these is the mountpoint: where lcachefs shows its results.  Unlike most filesystems, though, it involves two sources: one is known as the source directory (which holds the NFS-mounted sources), and the other is a storage directory (local disk).

Ok, let's pretend that you first copy your sources from a source directory to the storage directory.  Then you loopback-mount the storage directory over your original source directory.  Then, you compile there.  (You mount it over your original source directory so that the filenames in the debugging information point to the original sources, instead of to a temporary storage directory.)  This looks, to the compiler, just like a regular compile.  However, instead of doing the compile on NFS, you're doing it on local disk.

This has two advantages.  The one that's easy to explain is that you're storing your objects and products on local disk (which is cheap), instead of on NetApps (which, despite all their greatness, are expensive).  Objects can easily be recreated, and take up most of the space in a build tree.

The other part is that you're working entirely on local disk.  This is much, MUCH faster than using NFS.  It's not because of the bulk data transfer (i.e., reads and writes), so higher bandwidth doesn't help.  The problem is with the per-request overhead.  There are a LOT of requests going across the wire in a compile.

Let's consider what happens when you build a file.  Specifically, we'll talk about if you're just building hello.o.  Make will look for Makefile, makefile, BSDmakefile, and .depend.  Then it will look at hello.c and hello.o.  (If it needs to, it may look for hello.s, hello.S, hello.f, and whatever other potential sources you may have.)  It'll look up hello.c.gch (if you have precompiled header support), hello.gcda (if you have coverage support).  Then it has to look for stdio.h.  Well, let's suppose that you have two directories in your -I path; let's call them foo and bar.  Now it needs to check for foo/stdio.h.gch, foo/stdio.h, bar/stdio.h.gch, and bar/stdio.h, all of which are in your source repository (hence usually on NFS) before it can go on to look at /usr/include/stdio.h.  Well, the first thing that stdio.h does (on FreeBSD; your OS may vary) is to include sys/cdefs.h.  You got it... another four lookups in the source repository to find sys/cdefs.h!  Repeat for sys/_null.h, sys/_types.h, and machine/_types.h.  In the end, just to compile hello.o, you've done 39 file lookups into your source directory.  Of those lookups, 21 are for .h files... and hello.c only includes one file, stdio.h!  The average .c file (using the FreeBSD source base as a sample) includes 9 files directly.

That's a lot of work.  These lookups are all blocking I/O, so it's one round trip each; from watching network traces, I'm actually very impressed with how fast NetApps can respond, but it's still a non-zero time.  Moreover, though, you've got to look at the CPU usage.  NFS uses RPC, and the overhead isn't cheap.  In my testing, using NFS to compile something requires four times as much kernel processing-- which works out to twice as much CPU overall-- as using local disk!  (That's all in the kernel, too, and that means that preemption isn't as easy, so scheduling takes a little hit... and don't forget the new network interrupts!)

Finally, NFS is designed around concurrent use.  You can't assume that the contents of an NFS-mounted directory will be the same now as they were 90 seconds ago.  While local disk can use metadata caches very, very effectively (and FreeBSD's filesystems do), when you're using NFS, you have to expire caches a lot.  (There are also some things that FreeBSD could do better when it comes to its NFS client, but I fixed those and only got a 5% boost.)

Ok, now you should be convinced that the speed hit for NFS is bad, and that building on local disk is much, much, faster.  Let's go back to talking about lcachefs.

Earlier, I asked you to think about copying your sources to local disk, and doing the compile there.  That's exactly what lcachefs does, but with a twist: it does this lazily, by which I mean it won't copy files until they're needed.

When you first mount an lcachefs directory, it scans just the top-level of your source directory.  It then creates stub files for each entry in the local storage.  This is just an empty file with a special uid/gid pair that marks it as a stub.  If you ls -l that file, then the filesystem will report the correct uid/gid, size, link count, etc, but in reality the locally-stored file is empty.

When you actually try to read (or otherwise use) a file or directory that's a stub, then it will copy it over to local storage before the system call returns to userland.  In the case of a file, this means it copies the contents and sets the uid/gid to its real, non-stub values.  In the case of a directory, it scans the directory and populates the stubs just like I described in the previous paragraph.

That's the basic overview.  The reality is quite complicated because of hard links, the need to watch out for vnode lock order reversal, and other icky stuff like that.  You can see all the gory details, though, once we open-source lcachefs.  I'm currently awaiting clearance from legal, but I hope to either contribute it to FreeBSD, or release it as an independent open-source project.

Not what I'm doing at Juniper

I don't often make technical blog posts (ok, lately I don't often make posts), because two of the three people who read my blog don't really care.  But sometimes it's nice to be able to brag about what I'm doing at work.  Besides, last post I promised to talk about what I'm doing.

Since then, I haven't made any posts, because I didn't have the time to say what I was doing at work.  Which is stupid, really.  You three people don't read my blog because it's got juicy technical details; that's for the rest of the masses, and they don't care what I might have said I was going to post about until I post it.  I've had other things to write about, but since I said I'd write about work, I didn't want to write about anything else.  I several times thought, "Ooh, I should blog about this!" and then thought, "No, I said my next post would be about my work at Juniper."

Well, I am hereby renouncing that commitment.  What's more, I am advising you three (tonight's loyal fans) to take anything I might tease about as merely speculation on future topics, and not be sure I won't write about something completely different first.

So there.  I've just freed myself to write about other stuff.  I don't really know why I felt it necessary to maintain an obligation that nobody cares about; why would anybody care if I put some posts in between the previous post (when I said I'd talk about my Juniper work) and the future post when I actually do, is totally beyond me.  I just did feel that it's necessary, so now I can go on with more regular posts.

Except it's late and I'm going to bed instead of subjecting you to more of my incredibly convoluted late-night grammar.

Gasp! It's a post!

So here's the thing.

I was miserable at Google.  It's a great company, and I think it's a wonderful place to work.  But the position I was in just wasn't right for me.  I didn't want to blog at the time, because I didn't want people to get the wrong impression about Google.  Seriously, it's a great company; I just wasn't where I should have been.

So at the end of July, I left Google and went back to Juniper.  I was happy at Juniper, and wanted to be happy again.  Besides that, Juniper offered me a promotion, a raise, my choice of projects, and the opportunity to work under a manager I know I work well with.  I'll talk about the work I'm doing for Juniper in my next blog post.

After I went back to Juniper, I've been concentrating on getting into the swing of things, so I still haven't been blogging.  Now I'm doing well, and can start posting again.

Now, here's one last note to the blogosphere: you may note that I'm still using a Google's blogger.com, storing the photos using Google's Picasa Web, and displaying Google AdSense ads.  If I were upset with the company, I wouldn't be doing these things.  I'm strenuously emphasizing that I like the company, because things get taken the wrong way in the blogosphere, and I don't want anybody to get the wrong idea.  I'm also free to badmouth the company if I wanted to, since they're no longer my employer and have no control over my actions.  But I'm not; I'm talking about how great they are.

Anyway, my point is, hello world, it's nice to see you again!

Why This Post Isn't About The Javascript DOM

My original title for this post was "Why The DOM Is The Best Lousy Thing We've Got". I guess I'm not feeling terribly creative with my titles tonight. To be honest, I'm not feeling terribly creative at all tonight. It was a terrific week, but a pretty lousy weekend. I had some unexpected work take up all my time, and it put me in a sour mood for most of the weekend.

It's strange how creativity is so dependent on mood. I'm not just talking about blogging, here; I'm talking about coding, too. When I'm doing well, I can code anything, start to finish, piece of cake. When I'm doing poorly, then even writing hello.c is a huge effort. The work I talked about earlier involved some coding at the end of it— nothing big, just a few dozen lines to shove some data around— but if it weren't for that, I probably would have had a really lousy weekend, instead of the merely lousy one I had.

Ok, now I'm just sounding needlessly whiny. If you start seeing emo bands show up on my weekly top ten chart on the sidebar, send help. Like a box of kittens or something.

Anyway, where was I. Oh, that's right, having a lousy weekend because of work. I really started thinking that maybe I made a mistake taking this job; maybe I was more cut out for the Juniper job than the Google one. Now, here's the irony. When I left Friday, I was writing a document that includes advice to people who feel that way... pretty much saying, "No, it's normal, everybody feels that way, you'll do fine". (This is advice I had previously given Cheryl, and also talked about more in my blog.) And then this weekend I get hit with this wave. Wasn't expecting it.

I did get some useful stuff done for myself this weekend, though. The office is in much better shape, with the bookshelf moved, lighting hung, and desk accessories connected. Even the living room looks cleaner for some reason. At the end of it, I wanted to relax by writing some code. I was going to learn some SVG. Still, as I sat down to work on it, nothing came to mind. I hate hacker's block.

If you haven't read Jhonen Vasquez's "I Feel Sick", it's about an artist who is suffering from artistic withdrawl once she takes a job at a big company. I hadn't read it, and didn't know what it was about, but picked it up tonight to read and forget about my bad mood. (Hint: the "Alice" soundtrack works well with Vasquez.) Y'know, that just didn't help me forget as well as I had expected.

Finally, I decided to write about a blog post I've been wanting to write for a while... I even alluded to it in a post from two weeks ago (when I refer to the "later, more technically-oriented post"). And that's what I've sat down to do here. Hmmm... I seem to have done a great job, haven't I?