Friday, 16 October 2009, 13:55

One of the topics I asked most of my Coders at Work interviewees about was C++. I am not an expert, or even a competent C++ programmer and recognize that my own opinions about C++ are not well-informed enough to be worth much.1 But C++ fascinates me—it’s obviously a hugely successful language: most “serious” desktop apps are still written in C++ despite the recent inroads made by Objective C on OS X and perhaps some C# on Windows; the core of Google’s search engine is written in C++; and C++ dominates the games industry. Yet C++ is also frequently reviled both by those who never use and by those who use it all the time.

That was certainly reflected in the responses I got from my Coders interviewees when I asked them about it. Jamie Zawinski, as I’ve discussed recently, fought tooth and nail to keep C++ out of the Netscape code base (and eventually lost). Some of that was due to the immaturity of C++ compilers and libraries at the time, circa 1994, but it seems also to have to do with his estimation of the language as a language:

C++ is just an abomination. Everything is wrong with it in every way. So I really tried to avoid using that as much as I could and do everything in C at Netscape.

Part of Zawinski’s issue with C++ is that it is simply too complex:

When you’re programming C++ no one can ever agree on which ten percent of the language is safe to use. There’s going to be one guy who decides, “I have to use templates.” And then you discover that there are no two compilers that implement templates the same way.

Note that Zawinski had started his career as a Lisp programmer but also used C for many years while working on Netscape. And he later enjoyed working in Java. So it’s not that C++ was either too high-level or too low-level for him or that he couldn’t wrap his head around object orientation.

Joshua Bloch, who also hacked low level C code for many years before becoming a big-time Java head, told me that he didn’t get into object-oriented programming until quite late: “Java was the first object-oriented language I used with any seriousness, in part because I couldn’t exactly bring myself to use C++.” He echoed Zawinski’s point about how C++ forces programmers to subset the language:

I think C++ was pushed well beyond its complexity threshold and yet there are a lot of people programming it. But what you do is you force people to subset it. So almost every shop that I know of that uses C++ says, “Yes, we’re using C++ but we’re not doing multiple-implementation inheritance and we’re not using operator overloading.” There are just a bunch of features that you’re not going to use because the complexity of the resulting code is too high. And I don’t think it’s good when you have to start doing that. You lose this programmer portability where everyone can read everyone else’s code, which I think is such a good thing.

Ken Thompson, who still mostly uses C despite working at Google which is largely a C++ shop, has had as long an exposure to C++ as just about anyone, having worked with with Bjarne Stroustrup, C++’s inventor, at Bell Labs:

I would try out the language as it was being developed and make comments on it. It was part of the work atmosphere there. And you’d write something and then the next day it wouldn’t work because the language changed. It was very unstable for a very long period of time. At some point I said, no, no more.

In an interview I said exactly that, that I didn’t use it just because it wouldn’t stay still for two days in a row. When Stroustrup read the interview he came screaming into my room about how I was undermining him and what I said mattered and I said it was a bad language. I never said it was a bad language. On and on and on. Since then I kind of avoid that kind of stuff.

At that point in the interview I almost changed the topic. Luckily I took one more try at asking for his actual opinion of C++. His reply:

It certainly has its good points. But by and large I think it’s a bad language. It does a lot of things half well and it’s just a garbage heap of ideas that are mutually exclusive. Everybody I know, whether it’s personal or corporate, selects a subset and these subsets are different. So it’s not a good language to transport an algorithm—to say, “I wrote it; here, take it.” It’s way too big, way too complex. And it’s obviously built by a committee.

Stroustrup campaigned for years and years and years, way beyond any sort of technical contributions he made to the language, to get it adopted and used. And he sort of ran all the standards committees with a whip and a chair. And he said “no” to no one. He put every feature in that language that ever existed. It wasn’t cleanly designed—it was just the union of everything that came along. And I think it suffered drastically from that.

Brendan Eich, the CTO of the Mozilla Corporation, whose Mozilla browser is written almost entirely in C++, talks about “toe loss due to C and C++’s foot guns” and when I asked him if there are any parts of programming that he doesn’t enjoy as much as he used to, he replied:

I don’t know. C++. We’re able to use most of its features—there are too many of them. It’s probably got a better type system than Java. But we’re still screwing around with ’70s debuggers and linkers, and it’s stupid. I don’t know why we put up with it.

At least among my interviewees, even the most positive comments about C++ tended to fall in the category of “damning with faint praise”. I asked Brad Fitzpatrick, who used C++ in college and again now that he’s at Google, whether he likes it:

I don’t mind it. The syntax is terrible and totally inconsistent and the error messages, at least from GCC, are ridiculous. You can get 40 pages of error spew because you forgot some semicolon. But—like anything else—you quickly memorize all the patterns. You don’t even read the words; you just see the structure and think, “Oh, yeah, I probably forgot to close the namespace in a header file.” I think the new C++ spec, even though it adds so much complexity, has a lot of stuff that’ll make it less painful to type—as far as number of keystrokes. The auto variables and the for loops. It’s more like Python style. And the lambdas. It’s enough that I could delude myself into thinking I’m writing in Python, even though it’s C++.

Dan Ingalls, who helped invent modern object oriented programming as part of Alan Kay’s team that developed Smalltalk, never found C++ compelling enough to use but isn’t totally adverse to using it:

I didn’t get that much into it. It seemed like a step forward in various ways from C, but it seemed to be not yet what the promise was, which we were already experiencing. If I had been forced to do another bottom-up implementation, instead of using machine code I would’ve maybe started with C++. And I know a couple of people who are masters of C++ and I love to see how they do things because I think they don’t rely on it for the stuff that it’s not really that good at but totally use it as almost a metaprogramming language.

Joe Armstrong, similarly, has never felt the need to learn C++:

No, C++, I can hardly read or write it. I don’t like C++; it doesn’t feel right. It’s just complicated. I like small simple languages. It didn’t feel small and simple.

And finally Guy Steele, who probably knows more about more languages than anyone I interviewed (or possibly anyone, period), has also not been drawn to C++. But he did go out of his way to try to say something nice about Stroustrup’s effort:

I have not been attracted to C++. I have written some C++ code. Anything I think I might want to write in C++ now could be done about as well and more easily in Java. Unless efficiency were the primary concern.

But I don’t want to be seen as a detractor of Bjarne Stroustrup’s effort. He set himself up a particular goal, which was to make an object-oriented language that would be fully backwards-compatible with C. That was a difficult task to set himself. And given that constraint, I think he came up with an admirable design and it has held up well. But given the kinds of goals that I have in programming, I think the decision to be backwards-compatible with C is a fatal flaw. It’s just a set of difficulties that can’t be overcome.

Obviously with only fifteen interviewees in my book I have only a sampling of possible opinions. There are great programmers who have done great work with C++ and presumably at least some of them would have had more enthusiastic things to say about it if I had spoken with them. But this is what I heard from the people I spoke with.

Last updated 2009-10-16T13:55:11Z.

Thursday, 8 October 2009, 18:09

After my recent posting about what some of the folks I interviewed for Coders at Work had to say about unit testing and TDD, I’ve seen comments various places from people who seem to think that I must never have really tried TDD or that I think unit testing is not useful. Neither of which is really true.

In the last two jobs I had before I quit full-time work to write books and do a bit of consulting I was, among other things, the guy who wrote the test framework used for testing all our Java code. My first framework, which I wrote in the early days of Weblogic, predated JUnit but was similar in intent—my goal was to make it as easy as possible for developers to write fine grained tests and to pinpoint as precisely as possible the cause of test failures so that we could run tests in a continuous build-and-test system that tested every check-in and reported the results. (”Build and test monkeys” we called them at Weblogic and a stuffed monkey made the rounds of the developers’ desks, sitting on the desk of whoever had last broken something.)

In addition to working on the framework itself, I spent a lot of my time at Weblogic trying to figure out how we could have finer-grained tests that would run more quickly and provide easier to track down failures than the rather coarse-grained system tests we had a lot of. (At one point, when I was working on our EJB implementation, I came up with a trick that I’m still a bit proud of: to test our implementation of the state machine implementing the EJB life-cycle, we wrote an EJB that collected a list of tokens for each of the methods called on it by the EJB container and the test, after putting the EJB through its paces, fed that sequence of tokens to a parser generated from the grammar of legal paths through the state machine.)

At my next job, at another start-up founded by one of the Weblogic founders, I was hired early on not only to work on the server part of our product but to be in charge of our software development process. We set up another battery of build-and-test monkeys, using a test framework I had written after leaving Weblogic. And having experienced the benefits of pair programming and lots of testing at Weblogic, I tried to push our process toward something like XP though I don’t think we ever did enough of the practices to really count as an XP shop. Now, with six years of hindsight, I’m still not sure whether I’m glad or sad about that.

In my role as a developer at Kenamea, one of the biggest projects I worked on was implementing a transactional object store. Other than some proof of concept code I wrote by myself, that part of the system was almost entirely pair programmed and was possibly one of the most extensively unit tested parts of our product. I don’t actually recall whether we ever did test-first programming on it but my pair and I tried to be quite strict about writing unit tests for all the code we wrote. And the tests were definitely valuable. They gave us—as unit testing advocates always promise—confidence to dramatically refactor things when necessary and they told us when we had slipped up and broken something. On the other hand, the system was multithreaded which, as Joshua Bloch points out in Coders, always makes things much harder to test. Often the tests were sufficient to show the presence of a bug without pinpointing its location; hours or days of hard thinking were often required to track down those concurrency bugs and when we found them it was usually not at all clear that there were any unit tests we could have written that would have uncovered them.

When I quit Kenamea in order to spend some time hacking Lisp, I was still interested in unit testing and TDD. (There’s a reason that one of the first “practical” chapters in Practical Common Lisp is a simple test framework.) Shortly before I started work on PCL I read Kent Beck’s book Test Driven Development and did a few experiments with “pure” TDD which I recorded in a coding diary. The first was an implementation of an algorithm for generating starting positions for Fischer Random Chess, a version of chess where the pieces in the back row are positioned randomly, subject to a few constraints. Here’s the contemporaneous record of my attempt, complete with false starts and other brainos. (The code is in Common Lisp. Lispers may be shocked to see how little Lisp I knew then, a few months before I started work on PCL. And no guarantees that this code exemplifies good Lisp style or good anything else.)

The next problem I tackled with TDD was The “Impossible Book” problem, posed on the Test-driven Development Yahoo! group around that time. Again, I recorded my attempt as I went. I’m not sure these are great examples of TDD in action; I mention them merely as evidence that I have actually spent some time trying out TDD.

My Impossible Book attempt is also, perhaps, relevant to the current discussion since I was up against the same kind of difficultly as Ron Jeffries was while trying to write his Sudoku solver. Namely, I had no real idea how to solve the problem.

However—perhaps because there was no other infrastructure code to distract myself with—I was lucky enough to realize that no amount of testing was going to help me solve a problem I didn’t know how to solve. So instead I decided, as I say in my coding diary, that “TDD is about driving the implementation”. So I went off and read a bit about how to solve the problem and then used TDD to come up with an implementation. In this case I was lucky that there was already a readily accessible write up of a way to solve the problem. If there hadn’t been I would have had to do more research and probably would have had to learn some math in order to figure out how to apply it to the problem at hand. In other words, the thing that was going to speed me up was not trying harder with TDD but recognizing that I needed to step back and try something else.

Since those experiments with TDD, I’ve mostly been writing books so I haven’t been doing much work on production code. Working on my own coding projects, which tend to be exploratory, not very large, and special purpose (i.e. they only have to work for me) I have not found myself inclined to use TDD or even a lot of unit testing. I’m not claiming that that’s due to a principled appraisal of the benefits and drawbacks of TDD or unit tests; it just hasn’t felt like the problems I’ve had writing the software I’ve wanted to write would be fixed by having more unit tests nor that writing test-first would speed up my exploration.

And that, I guess, brings me back to how I was drawn into this conversation in the first place: I think testing, of all kinds, is an important part of serious software development. I think TDD is, at the very least, an interesting way to write software and I suspect that it might be a very good way to write some kinds of software. But I think the claim that TDD always speeds you up is just bunk. It may be that for the kinds of software Uncle Bob and Tim Bray write, in the kinds of organizations where they work, over the kinds of time scales they care about, it really does always speed things up. I’m happy to believe Uncle Bob when he says that he’s seen the benefits of TDD in his own and in others’ work.

But I also think that when Jamie Zawinski says that writing unit tests would have slowed down the first release of Netscape or when Donald Knuth says that he thinks he saved time by writing out TeX in a notebook without any testing at all until he had figured out how the whole program was going to work, those are data points that need to be accounted for, not dismissed with insults about “living in the stone age” and “being willfully ignorant”. Maybe Zawinski and Knuth are wrong about their own experience. Or maybe they were making different trade offs than Uncle Bob and Bray would chose to make. At any rate, I agree with Tim Bray when he says

If you read the comments around this debate, it’s increasingly obvious that we as a profession don’t have consensus around the value of TDD. Many of our loudmouths (like me for example) have become evangelists probably to the point of being obnoxious. But there remains a strong developer faction out there, mostly just muttering in their beards, who think TDD is another flavor of architecture astronautics that’s gonna slow them down and get in their way.

Maybe TDD’s detractors are, as Uncle Bob claims, analogous to the 19th century surgeons poo-pooing the benefits of washing their hands before surgery but I find Uncle Bob’s rhetorical stance of absolute certainty disconcerting and, ironically, anti-persuasive. That is, I thought better of TDD before I read his recent postings about it. But that’s a silly reason to accept or reject a practice that might do me some good. If I go back to writing production code, I’ll certainly resume my own contemplation on the best way to mix testing with development and wouldn’t be surprised if TDD found a place in my own practice.

Last updated 2009-10-08T18:09:14Z.

Monday, 5 October 2009, 23:56

In his now infamous blog post “The Duct Tape Programmer”, Joel Spolsky quoted Jamie Zawinski from my interview with him in Coders at Work talking about how they didn’t use many unit tests when developing Netscape. “Uncle Bob” Martin, chiming in to say that Spolsky posting was right in general but wrong in almost all his specific claims and criticisms, was particularly riled by Spolsky’s implication that maybe unit tests aren’t 100% necessary at all times:

As for Joel’s consistent dismissal of unit testing, he’s just wrong about that. Unit testing (done TDD style) does not slow you down, it speeds you up. One day I hope Joel eventually realizes this. Programmers who say they don’t have time to write tests are living in the stone age. They might as well be saying that man wasn’t meant to fly.

Tim Bray also jumped in to strongly agree with Uncle Bob on the importance of unit tests, though he couldn’t bring himself to actually agree with much else Uncle Bob said.

Joel is wrong to piss on unit testing, and buys into the common fantasy that it slows you down. It doesn’t slow you down, it speeds you up. It’s been a while since I’ve run a dev team, but it could happen again. If it does, the developers will use TDD or they’ll be looking for another job.

Since this all started from the Zawinski interview in Coders at Work and since there were other people interviewed for the book, I figured it might be interesting to see what some of the other folks I talked to had to say about unit testing and things like TDD (“test driven development” or sometimes “test driven design”, for those of you behind on your acronyms.)

To start with, here’s a bit more context from the Zawinski interview:

Seibel: What about developer-level tests like unit tests?

Zawinski: Nah. We never did any of that. I did occasionally for some things. The date parser for mail headers had a gigantic set of test cases. Back then, at least, no one really paid a whole lot of attention to the standards. So you got all kinds of crap in the headers. And whatever you’re throwing at us, people are going to be annoyed if their mail sorts wrong. So I collected a whole bunch of examples online and just made stuff up and had this giant list of crappily formatted dates and the number I thought that should turn into. And every time I’d change the code I’d run through the tests and some of them would flip. Well, do I agree with that or not?

Seibel: Did that kind of thing get folded into any kind of automated testing?

Zawinski: No, when I was writing unit tests like that for my code they would basically only run when I ran them. We did a little bit of that later with Grendel, the Java rewrite, because it was just so much easier to write a unit test when you write a new class.

Seibel: In retrospect, do you think you suffered at all because of that? Would development have been easier or faster if you guys had been more disciplined about testing?

Zawinski: I don’t think so. I think it would have just slowed us down. There’s a lot to be said for just getting it right the first time. In the early days we were so focused on speed. We had to ship the thing even if it wasn’t perfect. We can ship it later and it would be higher quality but someone else might have eaten our lunch by then.

There’s bound to be stuff where this would have gone faster if we’d had unit tests or smaller modules or whatever. That all sounds great in principle. Given a leisurely development pace, that’s certainly the way to go. But when you’re looking at, “We’ve got to go from zero to done in six weeks,” well, I can’t do that unless I cut something out. And what I’m going to cut out is the stuff that’s not absolutely critical. And unit tests are not critical. If there’s no unit test the customer isn’t going to complain about that. That’s an upstream issue.

I hope I don’t sound like I’m saying, “Testing is for chumps.” It’s not. It’s a matter of priorities. Are you trying to write good software or are you trying to be done by next week? You can’t do both. One of the jokes we made at Netscape a lot was, “We’re absolutely 100 percent committed to quality. We’re going to ship the highest-quality product we can on March 31st.”

So Zawinski says unit testing would have slowed them down. Uncle Bob and Tim Bray both say that unit testing, “doesn’t slow you down, it speeds you up.” Did Zawinski and the rest of the Netscape gang just blow it? They were going all out to develop their software as fast as they could; could they have sped things up with more unit testing? Maybe they were just living in the stone age.

Now, if Uncle Bob and Bray wanted to make a less radical claim than that unit testing always speeds you up, they could point out that unit tests can help you go faster over the longer term, and it’s not clear even Zawinski would disagree. And they’d definitely get some strong support for that claim from the subject of chapter two of Coders, Brad Fitzpatrick. I asked him about any big differences between his early and later programming style:

Fitzpatrick: I’ve also done a lot of testing since LiveJournal. Once I started working with other people especially. And once I realized that code I write never fucking goes away and I’m going to be a maintainer for life. I get comments about blog posts that are almost 10 years old. “Hey, I found this code. I found a bug,” and I’m suddenly maintaining code.

I now maintain so much code, and there’s other people working with it, if there’s anything halfway clever at all, I just assume that somebody else is going to not understand some invariants I have. So basically anytime I do something clever, I make sure I have a test in there to break really loudly and to tell them that they messed up. I had to force a lot of people to write tests, mostly people who were working for me. I would write tests to guard against my own code breaking, and then once they wrote code, I was like, “Are you even sure that works? Write a test. Prove it to me.” At a certain point, people realize, “Holy crap, it does pay off,” especially maintenance costs later.

Another interviewee, Joshua Bloch described how he designs code by starting with the APIs. He claimed this is a sort of “test-first programming and refactoring applied to APIs” since the first thing he does with a newly designed is test whether it would support the use cases that had lead to creating the API in the first place. But since he does all that writing any runnable code, that could also be called old-fashioned, “thinking about what you’re going to do before you do it” programming. Bloch did dispute the claim of those TDD advocates who say the tests produced by TDD can function as a spec for the code under test:

I don’t think tests are even remotely an acceptable substitute for documentation. Once you’re trying to write something that other people can code to, you need precise specs, and the tests should test that the code conforms to those specs.

Elsewhere Bloch described how he used both system and unit testing when he was working on an implementation of transactional shared-memory:

To test the code, I wrote a monstrous “basher.” It ran lots of transactions, each of which contained nested transactions, recursively up to some maximum nesting depth. Each of the nested transactions would lock and read several elements of a shared array in ascending order and add something to each element, preserving the invariant that the sum of all the elements in the array was zero. Each subtransaction was either committed or aborted—90 percent commits, 10 percent aborts, or whatever. Multiple threads ran these transactions concurrently and beat on the array for a prolonged period. Since it was a shared-memory facility that I was testing, I ran multiple multithreaded bashers concurrently, each in its own process.

At reasonable concurrency levels, the basher passed with flying colors. But when I really cranked up the concurrency, I found that occasionally, just occasionally, the basher would fail its consistency check. I had no idea what was going on. Of course I assumed it was my fault because I had written all of this new code.

After the system test demonstrated the presence of a bug he turned to unit tests to find it:

I spent a week or so writing painfully thorough unit tests of each component, and all the tests passed. Then I wrote detailed consistency checks for each internal data structure, so I could call the consistency checks after every mutation until a test failed. Finally I caught a low-level consistency check failing—not repeatably, but in a way that allowed me to analyze what was going on. And I came to the inescapable conclusion that my locks weren’t working. I had concurrent read-modify-write sequences taking place in which two transactions locked, read, and wrote the same value and the last write was clobbering the first.

I had written my own lock manager, so of course I suspected it. But the lock manager was passing its unit tests with flying colors. In the end, I determined that what was broken wasn’t the lock manager, but the underlying mutex implementation! This was before the days when operating systems supported threads, so we had to write our own threading package. It turned out that the engineer responsible for the mutex code had accidentally exchanged the labels on the lock and try-lock routines in the assembly code for our Solaris threading implementation. So every time you thought you were calling lock, you were actually calling try-lock, and vice versa. Which means that when there was actual contention—rare in those days—the second thread just sailed into the critical section as if the first thread didn’t have the lock. The funny thing was that that this meant the whole company had been running without mutexes for a couple weeks, and nobody noticed.

I asked him if he though the author of the mutex code that had been the cause of his problems could or even should have caught the bug with his own unit tests:

I think a good automated unit test of the mutex facility could have saved me from this particular agony, but keep in mind that this was in the early ’90s. It never even occurred to me to blame the engineer involved for not writing good enough unit tests. Even today, writing unit tests for concurrency utilities is an art form.

Donald Knuth, who is also a fan of after-the-fact torture tests, described an approach to coding about as far away from TDD as you can imagine, which he used when originally developing his typesetting system, TeX:

Knuth: When I wrote TeX originally in 1977 and ’78, of course I didn’t have literate programming but I did have structured programming. I wrote it in a big notebook in longhand, in pencil.

Six months later, after I had gone through the whole project, I started typing into the computer. And did the debugging in March of ’78 while I had started writing the program in October of ’77. The code for that is in the Stanford archives—it’s all in pencil—and of course I would come back and change a subroutine as I learned what it should be.

This was a first-generation system, so lots of different architectures were possible and had to be discarded until I’d lived with it for a while and knew what was there. And it was a chicken-and-egg problem—you couldn’t typeset until you had fonts but then you couldn’t have fonts until you could typeset.

But structured programming gave me the idea of invariants and knowing how to make black boxes that I could understand. So I had the confidence that the code would work when I finally would debug it. I felt that I would be saving a lot of time if I waited six months before testing anything. I had enough confidence that the code was approximately right.

Seibel: And the time savings would be because you wouldn’t spend time building scaffolding and stubs to test incomplete code?

Knuth: Right.

So Knuth too disagrees with the notion that unit testing always makes you go faster. Maybe he too is living in the stone age.

Joe Armstrong, on the other hand, says he has moved toward a test-first development style recently:

Seibel: At the point that you start typing code, do you code top-down or bottom-up or middle-out?

Armstrong: Bottom up. I write a little bit and test it, write a little bit and test it. I’ve gone over to this writing test cases first, now. Unit testing. Just write the test cases and then write the code. I feel fairly confident that it works.

The only interviewee who touched directly on TDD versus other approaches was Peter Norvig. He said he does more unit testing than he used to and even said some nice things about TDD but pointed out:

It’s also important to know what you’re doing. When I wrote my Sudoku solver, some bloggers commented on that. They said, “Look at the contrast—here’s Norvig’s Sudoku thing and then there’s this other guy,” whose name I’ve forgotten, one of these test-driven design gurus. He starts off and he says, “Well, I’m going to do Sudoku and I’m going to have this class and first thing I’m going to do is write a bunch of tests.” But then he never got anywhere. He had five different blog posts and in each one he wrote a little bit more and wrote lots of tests but he never got anything working because he didn’t know how to solve the problem.

A bit later Norvig said:

Then bloggers were arguing back and forth about what this means. I don’t think it means much of anything—I think test-driven design is great. I do that a lot more than I used to do. But you can test all you want and if you don’t know how to approach the problem, you’re not going to get a solution.

Ignoring Norvig’s suggestion that the difference between the two attempts doesn’t mean much of anything, it is instructive (or at least interesting, in a rubber-necking kind of way) to look at the two writeups. The “other guy” turns out to be Ron Jeffries, one of the inventors of Extreme Programming, the author of two books on XP, and according to his website an “experienced XP author, trainer, coach, and practitioner”.

Norvig’s writeup is a short essay explaining about 100 lines of Python that can solve any Sudoku. Jeffries writeup, by contrast, is spread over five lengthy blog postings here, here, here, here, and here and ends without coming anywhere close to actually producing a program that can solve any but a tiny subset of all Sudoku problems.

At some level the difference between the two simply boils down—as Norvig suggests—to knowledge: Norvig knew how to solve the problem because it’s a specific instance of a kind of problem he already knew how to solve. Jeffries, obviously, did not. But he did choose to tackle this particular problem using TDD, a technique in which he is supposed to be the expert? Why did he have so little success?

One thing I noticed, reading through Jeffries’s blog posts, was that he got fixated on the problem of how to represent a Sudoku board. He immediately started writing tests of the low-level details of a few functions for manipulating a data structure representing the 9×9 Sudoku board and a few functions for getting at the rows, columns, and boxes of the board. (“Boxes” are what Sudoku players call the 3×3 squares subsquares of the 9×9 board.)

Then he basically wandered around for the rest of his five blog postings fiddling with the representation, making it more “object oriented” and then fixing up the tests to work with the new representation and so on until eventually, it seems, he just got bored and gave up, having made only one minor stab at the problem of actually solving puzzles.

I suspect, having done a small amount of TDD myself, that this is actually a pattern that arises when a programmer tries to apply TDD to a problem they just don’t know how to solve. If I was a high-priced consultant/trainer like Jeffries, I’d probably give this pattern a pithy name like “Going in Circles Means You Don’t Know What You’re Doing”. Because he had no idea how to tackle the real problem, the only kinds of tests he could think of were either the very high-level “the program works” kind which were obviously too much of a leap or low-level tests of nitty-gritty code that is necessary but not at all sufficient for a working solver.

However, since most of what Jeffries spent his time on was the code for representing a Sudoku board and determining which row, column, and box a given square on the board is in, let’s look at that part of Norvig’s code.

Norvig’s basic strategy is to represent a board using a hash table with the keys being row-column pairs like A1, A2, and so on up to I9. After seeing the mess Jeffries makes of trying represent a board this is a refreshingly simple choice. It also seems to me that it requires a bit of creativity: given that a Sudoku board is a 9×9 board, I suspect I’m not the only programmer in the world who might be inclined to start with a 2d array. In a language without true 2d arrays, I might then be tempted, as Jeffries was, to then use an 81-element array and then get all wrapped around the axle, as Jeffries did, making sure I haven’t screwed up the finicky math for converting between 1d and 2d indices. And for all we know Norvig fell into the same trap and only later realized that all he really needed was the easy random access provided by a hash table. But pretty clearly Jeffries’s approach of testing the heck out of an array-based implementation wasn’t sufficient to lead him to the much better hash table-based one.

Given his choice to use a hash table, Norvig needs a list of the keys, i.e. the Cartesian product of the row labels (A-I) and the column labels (1-9). So the first bit of code he shows is a function cross which implements a Cartesian product that combines the pairs of elements by concatenation. The standard mathematical definition of the Cartesian product is:

A × B = {(a,b) | a ∈ A and b ∈ B }
                

Python’s list comprehensions let Norvig express this function in essentially the same notation as the standard mathematical definition:

def cross(A, B):
    return [a+b for a in A for b in B]
                

Could he or should he have developed this function via a test-first strategy? Dunno. Would it have been faster? Possibly, if he had any problems getting it right. But given that he’s just transcribing a mathematical definition, maybe a quick check at Python’s REPL would be sufficient to make sure he hadn’t screwed anything up.

Next he defines two variables to hold the row and column labels:

rows = 'ABCDEFGHI'
cols = '123456789'
                

Do TDD people unit test their data? I don’t know. Should they? I’m not even sure what that would mean, at least in a case like this. At any rate, he then feeds these two untested values to the cross function to produce the list of all 81 squares:

squares  = cross(rows, cols)
                

Now he’s basically done with the representation of the board. Any hash table, with the elements of squares as its keys represents a Sudoku board. Later in his code Norvig will use a hash table with strings containing the possible digits that could be put in each square as values.

However there is one other bit of work to be done: to solve a Sudoku you’re going to need to be able to map from a square to the other squares in the same row, column, and box. Having read a bit about Sudoku solving, Norvig has discovered that people use the term ‘units’ to refer to the rows, columns, and boxes and ‘peers’ to refer to all the squares that are in one of the three ‘units’ of another square. Norvig, unlike Jeffries, realized that this is better represented in data than in code and proceeds to compute the data once and for all.

First he makes a list of all the units, i.e. all rows, columns, and boxes, using list comprehensions and his cross function:

unitlist = ([cross(rows, c) for c in cols] +
            [cross(r, cols) for r in rows] +
            [cross(rs, cs) for rs in ('ABC','DEF','GHI') for cs in ('123','456','789')])
                

Now, he can use unitlist to generate a dictionary, units, that maps each square name to a list of its three units. He completely brute-forces this, linearly scanning unitlist for each square, selecting the units containing the square which itself requires a linear scan of each element of unitlist. But why write something more clever when unitlist is only 27 elements long and it’s elements are each 9 elements long and this whole computation is only going to happen once anyway? Here’s the code:

units = dict((s, [u for u in unitlist if s in u]) 
             for s in squares)
                

Once he’s got units, which will also be used in its own right later, he can compute peers, a hash table that maps from square names to the set of peer squares:

peers = dict((s, set(s2 for u in units[s] for s2 in u if s2 != s))
             for s in squares)
                

And that’s it: 7 definitions in 12 lines of code and he’s done with data representation. I’m not sure how much code Jeffries ended up with. In his fourth installment he had about 81 lines devoted to providing slightly less functionality than Norvig provided in the code we just looked at. In the fifth (and mercifully final) installment, he started adding classes and subclasses and moving things around but never presented all the code again. Safe to say it ended up quite a lot more than 12 lines; if he’s lucky it stayed under 120.

I’m not a proponent (or particularly a detractor) of TDD. If I was, I’d be pretty strongly tempted to throw Jeffries under the bus—maybe TDD isn’t quite as bad as he makes it look in this exercise. It certainly seems that, within the constraints of TDD, he could have done a much better job. Perhaps if he had stopped to think a bit about what he was doing he could have, using TDD, ended up with code as simple as Norvig’s. For instance, if he had started a bit closer to the problem domain he could have started by writing tests of his ability to map from squares to peers and units and then implemented something to provide that functionality. So maybe Norvig is right—maybe there’s not much to learn from this episode except that Fred Brooks is still right and there are still no silver bullets.

Anyway, those are some of the highlights of, and some of the context around, what the folks I interviewed for Coders had to say about testing. There are probably some other good bits I’m forgetting at the moment. Feel free to buy a copy and look for them yourself.

Last updated 2009-10-06T03:08:17Z.

Wednesday, 30 September 2009, 5:17

There's a theory in psychology called the "hedonic treadmill" which says that most people have a happiness “set point” that they return to pretty much regardless of what happens to them. Things that you might expect to make someone very happy (winning the lottery) or very sad (losing a limb) actually tend to not have a dramatic long term effect on a person’s happiness. In other words, people are pretty resilient. Or, put another way, we’re forgetful. We think how how happy we would be if we got a big raise and how much easier our lives would be with a bit more money in our pocket. And we get the raise and some things do get easier. But pretty quickly we forget how life was before and how happy we are supposed to be now. The new becomes the status quo, we find a new set of things to worry about, and we end up about as happy as we were before.

However most of us don’t get raises or experience any other big life changes often enough to get a real sense of the power of the hedonic treadmill. To get a dramatic demonstration of this phenomenon in action, all you need to do is spend two years of your life working on a book, have it published, and then obsess over the hourly updates to its Amazon Sales Rank. (The Sales Rank is the number Amazon attaches to every book it sells. Better Sales Ranks mean more sales and—more important—a book’s Sales Rank is the only readily accessible, frequently updated data an author has about how a book is selling.)

You finish your book and start paying some attention to its Sales Rank. It’s still in the high 100,000s, or even the millions. Whatever. No one but your parents has even heard of your book yet, let alone bought it. Then you get some word of mouth going on the web which gets you a few pre-orders and your Sales Rank spikes up a bit, maybe into the 2,000s. Woohoo! Party!

Then it starts drifting back down. Oh well, the book’s not even out yet, no big deal. But you dream of maybe one day breaking into the top-1000. That sounds pretty cool. Imagine, one of the top-1000 books on Amazon! They sell essentially every book in print; what’s that ten million books or so? Top-1000 would put you in the top 0.1% of all books. That’d be awesome.

Then some early reviews hit the web. Maybe they get some play on the social networking sites. Wow! Sales Rank goes shooting up—past 1,000, into the 500s. Yippee! And the book isn’t even out yet! Who knows how it’ll do once it’s actually released. Then, once again, the rank starts drifting back down. You feel a bit sad when it starts dropping but pretty soon the days of a 500s-level rank feel like a bit of a dream; anything better than 10,000 is still quite respectable. The decline continues and pretty soon it’s closer to 100,000 than 10,000, but no matter. Hitting the 500s is pretty darn good and the book still isn’t even out yet.

Then a review on a very prominent web site sends the rank rocketing back up. In a few hours it goes from greater than 100,000 to the 500s. And keeps going! Smashes the old high-water mark! Holy cannoli, it’s in the top-200! #178! Wow, that feels awesome. You never thought this could happen. And surely the book won’t have its best rank ever before it’s even out. Maybe it’s not crazy to dream of getting into the top-100, even if just for an hour.

Then the downward drift starts again. But not too fast. It’s hanging out safely in the top-2,000 for a while before dipping down into the 3,000s. And it rallies occasionally into the 1,500 to 2,000 range. Really, if it keeps on like this, this is great. An average sustained Sales Rank better than 10,000 is pretty darn respectable. And that #178 is something you can tell your grand-kids about.

Then the book is released. Whether because of a burst of word of mouth or the publisher’s PR actually working or because there are just a lot of people who wait for the book to come out before ordering, boom! the rank is moving up again. Quick jump back into the top-1,000. 800s. 200s again! Whoa! #173. Beat the old mark! And back down. That may be it. But really, nice that that first foray into the top-200 wasn’t a once-in-a-lifetime event. That’s pretty good. Now, let’s see how long it hangs out in the top-1000.

A few days, it turns out. Life in the top-1000 is pretty nice. Feels like this book might actually be a bit of a success. It would have been nice to crack the top-100 but this is good. Whoops, out of the top-1,000. Oh well. Still a long way to go before it falls out of the top-10,000. And look, it’s rallying again. 600s, 500s, 300s, 200s. Nice! Ah, #239 and then dropping again. Anyway, at least you’re back in the top-1,000 for a while. A dip out of the top-1000 then another rally back to #551.

Then much like before. Drifts down. It takes a couple days to fall out of the top-1,000 and then bounces around a bit between 1,000 and 1,500. Maybe the glory days of #173 and #178 are past but if the book just keeps steadily selling, that’s what’s important.

Then, one night before you go to bed, you see the book has been mentioned on another prominent web site. You wonder if that could possibly pop it back up into the top-200 again. That’d be nice. Maybe, just maybe, you still have a chance to crack the top-100. Something to dream about as you go to sleep.

Get up in the morning. Coffee. Check the ranks your computer has collected overnight. Holy smokes. It cracked the top-100 right after you went to bed! And it’s still there! At #22! In fact it’s been at #22 for three hours. Holy shit! Quick, email your folks! Wait, wait, new rank coming in: #16! Okay, this day is clearly going to be a total loss—you’re not going to be able to do anything until it starts going back down and you can relax. Sales Ranks are supposed to be updated every hour but that doesn’t mean they necessarily change every hour. So, a couple hours at #16. Wow—it’s holding its ground. Then, wait, wait, wait—#9. And #9 again the next hour! #8! #7! You have the #7 ranked book on all of Amazon! Only five books between you and Dan Brown’s new novel.

Finally it drops back down to #8 and hangs out there for a few hours. Then #9. Then #10. Hey, at least you’re still in the top-10. Finally you drop out of that. But you’re still in the top-25 which puts you on the first page of Amazon best sellers. So that’s cool. Anyway, you spent a day and a half in the top-10. A couple days later you drop out of the top-25 and the next day out of the top-50. But what the heck, you made it into the top-10. Who ever figured on that? At the moment you’re still in the top 100, hanging on in the 90s. You know it’ll drift back down. Will it hang out in the top-1000 for a while. Probably. Will you be sad when, inevitably, it falls out of top-1000. Probably not. Life on the hedonic treadmill.

Last updated 2009-09-30T12:41:39Z.

Monday, 28 September 2009, 15:04

Last Thursday Joel Spolsky posted an article on his blog, The Duct Tape Programmer, based on my interview with Jamie Zawinski in Coders at Work. It—as Joel’s posts often do—sparked quite a bit of commentary on the programming web, eliciting responses from Uncle Bob Martin and Tim Bray and hundreds of comments on sites like the programming reddit and hackernews.

It was probably fortunate for me that Spolsky used the title he did—praising someone while calling them a “duct-tape programmer” is provocative and the provocation probably helped drive interest which, no doubt, led to a few people buying the book. So, that’s awesome. Thanks, Joel!

On the other hand, being provocative sometimes leaves little room for nuance. Since I dragged Zawinski into this by asking him to be in the book, I thought maybe I could try to unpack some of the context for folks who haven’t yet read the interview.

The first thing to remember is that the first version of Netscape was written in 1994. When Zawinski talks about the choice to not use C++ and the problems with templates, he’s not talking about C++ and templates in 2009; he’s talking about 15 years ago, four years before the language was officially standardized and a time when different compilers didn’t necessarily implement all the nooks and crannies of the language the same way. And Netscape had to run on Windows, Unix, and Mac, so reliable portability was especially important.

The second thing to keep in mind is that the Netscape team in 1994 was under tremendous, if self-imposed, time pressure. I asked Zawinski how it was that the original Netscape team, many of whom had worked on Mosaic at the NSCA and were thus getting a second crack at the same problem, avoided falling into second-system syndrome:

Zawinski: We were so focused on deadline it was like religion. We were shipping a finished product in six months or we were going to die trying.

Seibel: How did you come up with that deadline?

Zawinski: Well, we looked around at the rest of the world and decided, if we’re not done in six months, someone’s going to beat us to it so we’re going to be done in six months.

Could they have made a different trade-off in the old “fast, good, cheap—pick two” space? Possibly. But history certainly testifies that they produced a product quickly enough that no one beat them to the punch and good enough that they changed the history of the web.

But wait! Didn’t Netscape eventually collapse under the burden of accumulated cruft? Didn’t the Mozilla project have to do a massive, ground-up rewrite because the old code base was so covered in duct tape that it was impossible to work with? Doesn’t that completely disprove Spolsky’s point?

Not clear. I haven’t done an extensive investigation of the history of Netscape’s code but I do have Zawinski’s version of events and some corroboration from Brendan Eich, who was also there at the time.

Certainly they paid a price for their rapid development:

Seibel: After this relentless pace, at some point that has to start to catch up with you in terms of the quality of the code. How did you guys deal with that?

Zawinski: Well, the way we dealt with that was badly. There’s never a time to start over and rewrite it. And it’s never a good idea to start over and rewrite it.

Yet we all know that Netscape famously was rewritten after it was spun out into the Mozilla project. So case closed, right? Not quite. What many people don’t know is that—at least according to Zawinski—the Mozilla rewrite was the second rewrite. Here’s Zawinski’s version, from the part of his Coders interview where we were talking about his work with Terry Weissman on the mail reader that was added to Netscape in version 2.0:

Zawinski: So basically [Netscape] acquired this company, Collabra, and hired this whole management structure above me and Terry. Collabra has a product that they had shipped that was similar to what we had done in a lot of ways except it was Windows-only and it had utterly failed in the marketplace.

Then they won the start-up lottery and they got acquired by Netscape. And, basically, Netscape turned over the reins of the company to this company. So rather than just taking over the mail reader they ended up taking over the entire client division. Terry and I had been working on Netscape 2.1 when the Collabra acquisition happened and then the rewrite started. Then clearly their Netscape 3.0 was going to be extremely late and our 2.1 turned into 3.0 because it was time to ship something and we needed it to be a major version.

So the 3.0 that they had begun working on became 4.0 which, as you know, is one of the biggest software disasters there has ever been. It basically killed the company. It took a long time to die, but that was it: the rewrite helmed by this company we’d acquired, who’d never accomplished much of anything, who disregarded all of our work and all of our success, went straight into second-system syndrome and brought us down.

They thought just by virtue of being here, they were bound for glory doing it their way. But when they were doing it their way, at their company, they failed. So when the people who had been successful said to them, “Look, really, don’t use C++; don’t use threads,” they said, “What are you talking about? You don’t know anything.”

Well, it was decisions like not using C++ and not using threads that made us ship the product on time. The other big thing was we always shipped all platforms simultaneously; that was another thing they thought was just stupid. “Oh, 90 percent of people are using Windows, so we’ll focus on the Windows side of things and then we’ll port it later.” Which is what many other failed companies have done. If you’re trying to ship a cross-platform product, history really shows that’s how you don’t do it. If you want it to really be cross-platform, you have to do them simultaneously. The porting thing results in a crappy product on the second platform.

Seibel: Was the 4.0 rewrite from scratch?

Zawinski: They didn’t start from scratch with a blank disk but they eventually replaced every line of code. And they used C++ from the beginning. Which I fought against so hard and, dammit, I was right. It bloated everything; it introduced all these compatibility problems because when you’re programming C++ no one can ever agree on which ten percent of the language is safe to use. There’s going to be one guy who decides, “I have to use templates.” And then you discover that there are no two compilers that implement templates the same way.

And when your background, your entire background, is writing code where multiplatform means both Windows 3.1 and Windows 95, you have no concept how big a deal that is. So it made the Unix side of things—which thankfully was no longer my problem—a disaster. It made the Mac side of things a disaster. It meant it was no longer possible to ship on low-end Windows boxes like Win16. We had to start cutting platforms out. Maybe it was time to do that, but it was a bad reason. It was unnecessary.

Brendan Eich, who was also at Netscape at this time, confirms at least part of Zawinski’s account:

There was an imperative from Netscape to make the acquisition that waved the Design Patterns book around [i.e. Collabra] feel like they were winners by using their new rendering engine, which was like My First Object-Oriented Rendering Engine. From a high level it sounded good; it used C++ and design patterns. But it had a lot of problems.

He does, however, go on to say:

But the second reason we did the big rewrite—I was in mozilla.org and I really was kind of pissed at Netscape, like Jamie, who was getting ready to quit. I thought, you know, we need to open up homesteading space to new contributors. We can’t do it with this old hairball of student code from 1994. Or my fine Unix kernel-style interpreter code.

So Zawinski and Eich differ a bit on the extent to which the Collabra folks had completely replaced the old code in 4.0. If Zawinski’s recollection is accurate and the 4.0 rewrite had really replaced all the old code, then the fact the Mozilla project felt compelled to do a big rewrite is only evidence of what a mess the first rewrite had made of things and says nothing about the quality of the pre-4.0 code. Or maybe they hadn’t quite replaced everything but had still messed things up enough that it was easier to start over. Or maybe it really was the legacy of the 1994 code—too much duct tape—that was the real problem. Ironically, if Zawinski had had his way it’d be much easier to know what really happened—he tells me in a recent email that he tried, in 1998, to get Netscape to open source both the 4.0 and 3.0 code bases but they wouldn’t go for it.

Last updated 2009-09-28T15:04:40Z.

Tuesday, 22 September 2009, 21:22

When I started work on my book Coders at Work, the first thing I had to do was come up with a list of people I wanted to interview. I came up with a lot of names myself and then put up some web pages where people could suggest names and sort them in various ways in order to show me which folks they were most interested in seeing interviewed. Thanks to some front-page play on the programming Reddit, I received quite a few suggestions and lots of people tried their hand at generating a list of fifteen or sixteen subjects.

Pretty soon the issue arose of whether I would interview any women and, if so, how many? How did it arise? Well, my mom, for one, called me to ask. And a friend pointed me to a blog post by Shelley Powers on burningbird.net criticizing the at-that-time recently released book Beautiful Code, for having only one woman among the thirty-eight authors of its thirty-three essays. (Powers’s post, unfortunately, is no longer online.)

In the end, I ended up interviewing one woman, Fran Allen, and fourteen men. Is one enough? I don’t know. Here are a few things I do know:

The issue of sexism, in the field and in the world at large, is a huge ball of worms. Why are there relatively few women programmers? Is it because all male programmers are irredeemably sexist? Because there are sex linked characteristics that make it more likely that a randomly chosen man will turn out to be a good programmer than that a randomly chosen woman will? Because girls disproportionally drop off the science and math track way back in grammar school? Or because once the field became, for whatever reason, largely male, it became less appealing for women to enter it? Take your pick. Or make up your own reason.

And does it even matter that programmers are disproportionally men? If you read the comments on sites like the programming Reddit whenever the issue comes up, there’s a strong contingent of folks who claim that the field is already an essentially perfect meritocracy and therefore if there aren’t many women programmers it’s simply because women are less interested in being programmers or are somehow less suited to it – no big deal. Obviously if that’s your point of view, you’re not going to be worried about whether there are any women in a book like Coders. However …

I was actively interested in having women in the book. I do believe there’s enough sexism left, both in the world and in the field, that a girl who finds computers fascinating and who wants to be a programmer when she grows up will face obstacles that a similarly interested and talented boy would not. One of those obstacles, I believe, is the lack of women role models. Yeah, yeah, I can hear the “perfect meritocracy” advocates now – why should a woman be a better role model than a man for an aspiring girl programmer when we’re talking about something as purely logical and rational as computer programming. They can ask that if they want. My experience is that people, starting from when we are kids, do notice gender and that the path of least resistance is to do what other people of your gender are doing. So if you’re a girl interested in programming and you see very few other girls and women programming, that’s going to be a obstacle for you to overcome.

Given that, I didn’t want to write a book that was going to be another vector for the implicit message that all programmers are men. But …

A policy of diversity for diversity’s sake is tricky to implement. Assuming you decide you want to achieve a certain level of gender diversity, your problems are just starting. No matter who I selected, man or woman, for a book like Coders, some people are going to ask, “Did they really deserve to be included in that group?” Fair enough. But if people answer that question, “Oh, he just got in because they wanted a Java programmer,” that’s unlikely to cause them to think badly of all Java programmers. But if the answer is, “She was just included because she’s a woman,” it may cause a classic “affirmative action” backlash. Plus, as either Shelly Powers or one of the commenters on her blog pointed out, it can be insulting for a woman to be invited to do something “just because” she’s a woman.

So how did it happen that I ended up with only one of my fifteen subjects being a woman?

At the end of my name gathering, which included posting a comment on the burningbird.net article asking for suggestions of names of women coders and doing my own search for likely women candidates, I had 284 names, of which ten were women. (Or nine if your brand of feminism and/or chauvinism doesn’t accept transgendered people identifying as women as “real” women.) That’s not many: 3.5%. On the other hand, it’s better than the percentage of female Beautiful Code authors (2.6%) and Turing Award winners at that time (2%). (With Barbara Liskov winning the Turing in 2008, that percentage is now 3.6%.)

So if I were to assume that my list of 284 was a good sampling of the kind of programmers I wanted to interview, then in a book of fifteen interviews, I would expect to have .53 women. Since I knew for sure I wanted at least one woman – and since it is certainly possible that despite my best efforts, my list of 284 is not an unbiased sample – I rounded up rather than down and chose Fran Allen. And I’m glad I did. Anyone who wants to claim Allen got in just because she’s a woman can come talk to me after they’ve won the Turing Award. And she provided another important bit of diversity, representing IBM from the heyday of Big Blue.

Do I have any regrets? In retrospect, I wish I had tried to interview Amy Fowler, one of the lead developers of Java’s Swing GUI toolkit. I certainly seriously considered her and she would have added not only a younger woman’s perspective to the mix but also a GUI programmer’s point of view, both of which could have made it an even better book.

Did my own sexism ever lead me astray? Possibly. Early on, Alan Kay recommended that I interview L Peter Deutsch and Dan Ingalls and I signed them up. But what about Adele Goldberg, who was also part of Kay’s group at Xerox PARC? Did some bit of sexism lurking in my heart convince me that she wasn’t really a great hacker like Deutsch and Ingalls? Or was it really, as I told myself, that I just didn’t want to overload the book with Smalltalkers? I really don’t know. I’d like to think it was purely the latter but have to admit it’s possible the former played a part too.

At any rate, I’m pleased with how the book came out and glad that I had a chance to interview Fran Allen, whose historical perspective, technical opinions, and comments on being a woman in the field were all equally interesting and valuable. I hope that if my daughter – to whom the book is dedicated – decides she wants to be a programmer when she grows up, she will find something to inspire her in Allen’s chapter and in all the other chapters of the book.

Last updated 2009-09-22T21:22:14Z.

Monday, 21 September 2009, 22:21

Whoops! Turns out that Jamie Zawinski suffered a slight brain glitch during our Coders at Work interview when recounting his time as a high-schooler working in Scott Fahlman’s CMU AI Lab. He said:

But there were some really entertaining characters in that group. Like the guy who was sort of our manager—the one keeping an eye on us—Skef Wholey, was this giant blond-haired, barbarian-looking guy. Very intimidating-looking. And he didn’t talk much. I remember a lot of times I’d be sitting there—it was kind of an open-plan cubicle kind of thing—working, doing something, writing some Lisp program. And he’d come shuffling in with his ceramic mug of beer, bare feet, and he’d just stand behind me. I’d say hi. And he’d grunt or say nothing. He’d just stand there watching me type. At some point I’d do something and he’d go, “Ptthh, wrong!” and he’d walk away. So that was kind of getting thrown in the deep end. It was like the Zen approach—the master hit me with a stick, now I must meditate.

However, as Jamie emailed to tell me yesterday, and Skef Wholey emailed me about today, Skef Wholey was not the giant, blond-haired bare-footed hacker who supervised a young Zawinski. Skef, it turns out, is 5’8” and has dark brown hair. Jamie’s giant supervisor was another guy named Rob MacLachlan.

Skef also tells me that Rob did carry a ceramic beer mug but as far as Skef recalls it held coffee, not beer. Skef also described Rob as “substantially more articulate, particularly in matters of coding and in Computer Science in general” than someone who’d just say “Ptthh, wrong!” and walk away. To be fair, Jamie did say, elsewhere in the interview, about Skef/Rob, “Well, he wasn’t completely a cave man. He would actually tell me things, too.”

My apologies to Skef for the case of mistaken identity.

Last updated 2009-09-21T22:21:24Z.

Thursday, 17 September 2009, 16:01

A number of people have emailed me to ask whether Coders at Work will be available on the Kindle. Short answer: Yes.

Longer answer: for reasons I don’t really understand, it may be a while. My publisher has, they assure me, sent the necessary files to Amazon and now Amazon has to do something – not clear to me what – before the Kindle edition is actually available. However that something can apparently take months. My editor was optimistic that Coders was selling well enough that it might be more like one month than the six months that some titles have spent in the Amazon queue but there’s no way to know.

I really don’t understand what Amazon could be possibly doing that could take 1/4 the time it took me to actually write the book but there it is. If you’re eager to get Coders on your Kindle your best bet is, I guess, to click the “I’d like to read this book on Kindle” link on the Amazon Coders at Work page. Or maybe email Amazon directly since the “I’d like to read this book on my Kindle” is ostensibly for letting the publisher know you want it and the publisher, in this case, is already is happy to oblige.

Last updated 2009-09-17T16:01:57Z.

Wednesday, 9 September 2009, 12:36

Since I last did a rollup, Coders at Work has received three more reviews on the web, bringing the grand total to nine:

• Brian Carper, briancarper.net
• David Thole, The Dark Trumpet
• Vladimir Sedach, Slashdot
• Chris Hartjes, @TheKeyboard
• Marc Hedlund, O’Reilly’s Radar
• Amit Shaw, Teleported Bits
• Tobias Svensson, return 42;
• Prakash Swaminathan, CloudKnow
• Joseph F. Miklojcik III, jfm3> _

The reviews have continued to be quite positive and the Slashdot review turned out to be quite a coup, shooting Coders's Amazon Sales rank, for one glorious hour, up to #176 across all books. While perhaps nothing can complete with the Slashdot Effect for driving an instantaneous spike in interest, I am hoping that it will help to actually get the book into peoples' hands, as should happen later this week.

Another person who got a sneak peek at the book, Andy Mulholland, the Chief Technology Officer at Capgemini, an $8-billion global consulting firm, sent me this review:

Just had a “sneak peek” at Coders at Work, a book by Peter Seibel due out in September that features interviews with 15 of the industry's most important programmers.

Through in-depth interviews with the likes of Peter Norvig, formerly VP of Search at Google and now head of Google Labs, and Brad Fitzpatrick, founder of Live Journal, as well as elder statesmen such as Donald Knuth and Ken Thompson, Peter manages to capture the thinking behind these coders’ approach to their work. The book documents an older generation of software – much of it still in use – that coexists with today's new and different types of applications. It's as though you have multiple generations of cars on the road at the same time and everything is in flux – the cars themselves, the quality of the roads, the skill of the drivers, and the very purpose for which cars are used.

This book can help programmers sort through all this complexity. It would be a good read for all those studying software today because it provides a fundamental understanding of the world they are about to enter. I highly recommend it.

I guess if everyone--from Slashdot readers to someone advising the highest levels of big-company corporate IT--likes Coders, it'll probably do okay.

Last updated 2009-09-09T12:59:05Z.

Monday, 24 August 2009, 14:42

Since its first review, Coders at Work has been reviewed five more times. Here are all the reviews to date:

• Chris Hartjes, @TheKeyboard
• Marc Hedlund, O’Reilly’s Radar
• Amit Shaw, Teleported Bits
• Tobias Svensson, return 42;
• Prakash Swaminathan, CloudKnow
• Joseph F. Miklojcik III, jfm3> _

Some highlights:

“Absolutely amazing! A page turner, just like Harry Potter for the technically minded.” —Tobias Svensson

“What books would you recommend to help a developer learn programming? For me, this book joins my short list.” —Marc Hedlund

“This book is dead sexy. When it comes out, you should definitely get a copy.” —Joseph F. Miklojcik III

“Superb book!” —Prakash Swaminathan

“Read it, because then you will know the greatest coding brains.” —Amit Shaw

Last updated 2009-08-24T14:42:54Z.

Monday, 10 August 2009, 4:45

My offer of sneak peeks at Coders at Work for would-be reviewers, has yielded its first fruit. I can only hope that all reviewers are as kind. (Though the author does accuse me of “subversive behavior” for dwelling on my subjects’ Emacs use.)

Meanwhile, I’m basically unable to function since I spend all my time waiting for the next update of my Amazon Sales Rank and checking to see whether the link to the review is being upvoted on Reddit. Ooops, gotta go; new Sales Rank coming in 48 seconds.

Last updated 2009-08-10T04:45:29Z.

Tuesday, 4 August 2009, 13:51

I just learned that my new book, Coders at Work is due to go to the printers on August 17th and should hit bookstore shelves about a month after that and, as I understand it, copies ordered from Amazon might show up even a bit earlier. I’m quite pleased with how it turned out and hope every programmer everywhere buys at least one copy!

Meanwhile I’m thinking about what’s next. The world of Lisp books seems to be exploding, with Conrad Barski’s Land of Lisp due out later this year and O’Reilly finally dropping their anti-Lisp policy to publish a book by Nick Levine. Meanwhile Practical Common Lisp continues to sell well. Perhaps a 2nd edition or sequel to Practical Common Lisp is in order. I also have several other non-Lisp book ideas bouncing around my head. We’ll see.

P.S. If you are a blogger or book reviewer interested in a sneak peek at Coders at Work, email me and I may be able to hook you up.

Last updated 2009-08-07T13:42:33Z.

Thursday, 16 July 2009, 3:30

My wife and I have what I think is a fairly equitable division of child-rearing labor: Monday mornings, and all day Wednesdays and Fridays she goes to her job as a doctor at a San Francisco clinic and I look after our now almost three-year-old daughter. Monday afternoons, Tuesdays, and Thursdays are my days to work on my writing or consulting while my wife looks after the kid. Weekends we share parenting duties.

We’ve been doing this since our daughter was three months old. Prior to that we were both home with the baby pretty much full time. We consider ourselves incredibly lucky that we’ve been able to arrange our lives this way and wouldn’t want it any other way. And our daughter has had plenty of chance to get used to the idea that although I didn’t give birth to her and have never provided her nourishment from my own body, I’m one of her parents and love her very much.

However, watching my daughter this morning for the umpteen-millionth time, say to her mom, “I don’t want you to go to work today! I want you to stay home!” something I don’t believe she’s ever said about me, it struck me that, at least as far as she is concerned, it would make a lot of sense if mom stayed home to take care of her full-time and dad went off to bring home the bacon.

My daughter, tool of the patriarchal hegemony.

Last updated 2009-07-16T03:30:42Z.

Wednesday, 20 May 2009, 15:07

It’s now almost exactly two years since I started work on my book Coders at Work and I’m finally in the endgame. Once I get all of my chapters turned in I’ll probably do some work on the website and blog a bit about behind the scenes, making of the book kind of stuff.

Last updated 2009-05-20T15:07:53Z.

Saturday, 1 November 2008, 22:59

I’ve been following the U.S. Presidential election on the excellent site www.fivethirtyeight.com which features sophisticated statistical aggregation of all the published polls done in the U.S and also at the prediction market Intrade.com. But as it comes down to the election I know I’ll need a frequently updated dashboard for watching the results. So I came up with this. (N.B. Requires Firefox 3 and at least as much screen realestate as a 15" Powerbook.)

Inspired by a page put together by Randall Munroe of xkcd fame, my dashboard periodically fetches the market price of Intrade’s state-by-state election markets, which represent the probability, as assessed by the Intrade traders, that a given candidate will win a given state. From those probabilities I compute the overall probability of various scenarios and color the map appropriate shades of blue and red. I also provide some dials and knobs (sliders) actually, to allow you to play some real-time “what if” games with the results.

There are some flaws some flaws with my statistical methodology (most notably the almost certainly erroneous assumption that the state probabilities are all independent) but it should nonetheless be a good way of tracking the results as the come in: Assuming the Intrade traders, who’ve got real money on the line, stay on the job the Intrade values will head toward certainty as exit polls and actual results become available and my dashboard will reflect that.

Last updated 2008-11-01T22:59:19Z.

Monday, 4 August 2008, 20:45

I was recently notified – somewhat to my suprise – that the Japanese publisher Ohmsha is publishing a Japanese translation of Practical Common Lisp which should now be avaliable in bookstores and on amazon.co.jp. I had known that there was a small group working on a translation but hadn’t realized they had found a publisher. My thanks to those translators and to Masako Omata at Franz who, as I understand it, did a fair bit of work to make it all happen. I got my copy in the mail the other day. Looks good though I can’t say much about the quality of the translation other than that it seems to contain quite a number of Japanese characters. I’ll be interested to hear from any Japanese readers what they think of it.

Last updated 2008-08-04T20:45:48Z.

Saturday, 8 March 2008, 14:43

This is the kind of thing, I imagine, that turns people into right-wing lunatics. Walking the dog today I saw the front page headline of the San Francico Chronicle, “Homeschoolers suffer setback: Appeals court rules parents who teach children at home must be credentialed.” Uh-oh. Our daughter is only a year and a half old so we’ve got a few years before we have to officially decide whether we’re going to home school but that’s the current plan.

Except that all the sudden that may no longer be an option unless this appeals court ruling is overturned, the legislature defies the teachers unions and changes the state’s education laws to specifically allow home schooling by uncredentialed parent-teachers, or we leave the state. Equally suddenly, I’m on the side of the right-wingers ranting about judges legislating from the bench and the nanny state trying to take over our lives. Heck, suddenly James Dobson of Focus on Family, who spent his radio show today decrying the ruling, is my ally.

We are not religous so that’s not our motivation for wanting to home school, but we are not really all that different from the homeschoolers who are. While we don’t object to the secularism of public schools—that’s one of their good points as far as I’m concerned—we object to other parts of mainstream culture: the relentless consumerism, the regimentation of academic instruction, and the emphasis on competition and working for extrinsic rewards. I’m sympathetic to the need for society (i.e. the state) to look out for the welfare of kids whose parents aren’t taking proper care of them. But to have the state tell me I have to send my daughter to the schools the state has approved and to be taught only in the way the state thinks is best makes me start thinking about holing up in a compound somewhere with too many guns and a couple years worth of canned food in the root cellar.

The quote from the Chronicle story that really killed me was from Leslie Heimov, the executive director of the Children’s Law Center of Los Angeles. She said her organization was mostly concerned that children be “in a place daily where they would be observed by people who had a duty to ensure their ongoing safety.” Uh, wouldn’t parents have a duty to ensure the safety of their children. To say nothing of looking after their education and moral development. Hmmm, I really must be turning into a crazy right-wing nutjob.

Last updated 2008-03-08T14:43:22Z.

Friday, 7 March 2008, 15:25

Once again following around after Language Log’s Geoffrey Pullum yields food for thought. Long ago Pullum wrote a blog entry entitled “More timewasting garbage, another copy-editing moron” in which he heaped scorn on the copy editors who edited Mark Pilgrim’s Dive into Python for their many grammatical incorrections. That post was the one that made me a Language Log fan since I had written a book for the same publisher and had been made batty by all the same incorrections.

John McIntyre, the assistant managing editor for the copy desk at the Baltimore Sun must have seen that post because when he recently wrote a piece for his blog about the old that vs. which usage bugaboo, after defending Fowler’s made-up rule, he tried some preemptive self-defense saying, “That will probably bring down on my head the wrath of the linguists at Language Log …, who appear to hate copy editors’ guts.” He then went on to say:

But I’m just a simple country boy from Kentucky who learned English grammar in Mrs. Jessie Perkins’ fifth- and sixth-grade classes at Elizaville Elementary School and who just tries to get by on what is reasonable and useful.

Why is it that English grammar is one of the few fields where what we learned in fifth and sixth grade is considered state of the art? I doubt the Sun’s assistant managing editor in charge of political reporting would explain the paper’s approach to election coverage by saying: “I’m just a simple country boy from Kentucky who learned about U.S. politics in Mr. Bobbie Smith’s fifth- and sixth-grade Social Studies classes at Elizaville Elementary School.”

We all understand that the way we teach politics, and just about everything else, to ten- and eleven-year-olds is simplified, if not over-simplified. But that’s usually okay since most folks who grow up to be newspaper editors or, for that matter, newspaper readers, probably go on to high-school, college, and maybe even graduate school where they are exposed to less and less simplified versions of our collective understanding of things.

Yet when it comes to grammar and the use of the English language, most folks, including professionals, seem happy to work with the version they learned when they still thought members of the opposite sex had cooties.

Last updated 2008-03-07T15:25:11Z.

Wednesday, 5 March 2008, 16:18

I see via a Geoffrey Pullum Language Log post that yet another otherwise intelligent person—this time David Gelernter, a Yale computer science professor—has been found ranting in public about the imminent destruction of the English language due to folks using they as a singular pronoun.

Pullum does his usual fine job highlighting the absurdities of this kind of rant: in this case the wild disparity between the magnitude of the social devastation allegedly being wrought and the venality of the linguistic sins, if any, being committed. Pullum—co-author of the massive and comprehensive Cambridge Grammar of the English Language—also points out that Gelernter is simply wrong about many points of grammar and historical linguistics. Pullum, however, doesn’t really take Gelernter’s argument seriously, presumably because it’s absurd and ignorant and doesn’t deserve to be. On the other hand, Gelernter’s rant is such a fine example of an anti-singular they diatribe, that it merits a closer analysis.

Gelernter’s thesis is that “the English language has become a wholly-owned subsidiary of the Academic-Industrial Complex”. In particular he claims “feminist authorities” effected a significant change to the rules of grammar such that “agreement between subject and pronoun was declared to be optional” allowing they to be used as a singular pronoun. As Pullum and regular Language Log readers of course know, this claim is deliciously ironic. The last time the Academic-Industrial Complex unilaterally changed the rules of grammar was in the 18th century, when grammarians, taking a bit too much of a cue from Latin, made up a rule that pronouns had to agree in number with their antecedents, a “rule” which, in fact, had been regularly violated by such writers as Chaucer, Shakespeare, and Jane Austen to say nothing of thousands of less notable authors and, no doubt, hundreds of thousands of plain old native English speakers.

Having made up their rule, these grammarians were then forced to choose a singular pronoun to use with indefinite antecedents (e.g. everyone, nobody) and singular nouns that could refer to a person of either gender (e.g. a person). Given the times and the fact that the grammarians were mostly men, the “natural” solution was to use he, his, and him. But one must wonder whether sentences like, “A grammarian should always keep his inkwell full”, sounded natural to an 18th-century grammarian because he really felt his was gender neutral or because he was envisioning a grammarian much like himself and all the other grammarians he knew who was, of course, also a “him”. Hard to say. I don’t have any 18th-century grammarians’ writings at hand but Gelernter gives up the game a bit with this sentence: “Who can afford to allow a virtual feminist to elbow her way like a noisy drunk into that inner mental circle where all your faculties (such as they are) are laboring to produce decent prose?” Surely that should be “elbow his way”.

Of course we’ve come a long way since the 18th century. Now that grammarians really are as likely to be women as men, maybe it’s silly to get hung up on gender-neutral he. On the other hand it’s also worth considering how recently we’ve really made progress on that front. Gelernter is full of praise for Strunk & White’s The Elements of Style (excepting editions published since E.B. White’s death, which have softened the guide’s position on the absolute correctness of the gender-neutral he and are thus “a disgrace to his memory”.) But, as Pullum pointed out in a 2004 Language Log posting, when Strunk was passing along the rule that singular they should be changed to he, women in the United States still didn’t have the vote. And when E.B. White wrote about Strunk’s “little book” in his “Letter from the East” for the July 27, 1957 New Yorker that essay—which later became the introduction to White’s revision of the guide—appeared next to an advertisement with this text: “Traveling men get juicy steaks on ‘The Executives’—United’s for-men-only nonstops to Chicago.”

Gelernter does, however, inadvertently if somewhat belligerently, get to the real point of the singular-they debate when he asks: “Why should I worry about feminist ideology while I write? Why should I worry about anyone’s ideology? Writing is a tricky business that requires one’s whole concentration, as any professional will tell you; as no doubt you know anyway.” Yes, writing is a tricky business. Largely because it’s about communicating with other human beings. Unlike computer programming, where a deterministic and strictly logical computer is the ultimate arbiter of the meaning of your creation, writing is about conveying thoughts from your mind to that of your reader, a process that is neither deterministic nor entirely logical. For better or worse, readers can be thrown for a loop by clumsy diction, abstruse vocabulary, or violation of what they consider norms, whether grammatical or social. The harsh reality of a writer’s life is that some readers will be jolted out of their concentration by a singular they while others, who would have glided right past it, will trip over a gender-neutral he.1 There’s no good way out of this mess, at least in the short term. I do believe that for Gelernter, and for many others, sentences using a singular they really do “skreak like fingernails on a blackboard.” That they do so for silly, historical reasons is no consolation. (That Gelernter considers them evidince that feminism is destroying the possibility of rational thought is, however, just stupid.) On the other hand, I’m quite certain that singular they will prevail in the long run. It was standard usage long before the 18th-century grammarians put their oar in and it is even more attractive now for people who wish to avoid implying that executives and grammarians are always men. It also follows the pattern set when the plural pronoun you drove out the singular thou.

So I do my part by using singular they in my own writing. And any writer who wants to reclaim the English language from the dead hand of the 18th-century Academic-Industrial Complex, well, they should do likewise.

Last updated 2008-03-06T05:43:46Z.

Tuesday, 27 November 2007, 6:35

Woohoo. It's all over except the interviewing. And transcribing. And editing. And reinterviewing. And more editing and then the publishing. But I've got my sixteen interviewees! I’ve just received an email from Miguel de Icaza agreeing to be interviewed, which fills out my roster. The complete list (in alphabetical order) is:

• Frances Allen
• Joe Armstrong
• Joshua Bloch
• Bernie Cosell
• Douglas Crockford
• L. Peter Deutsch
• Anders Hejlsberg
• Miguel de Icaza
• Dan Ingalls
• Simon Peyton Jones
• Alan Kay
• Donald Knuth
• Peter Norvig
• Guy Steele
• Ken Thompson
• Jamie Zawinski

There’s some more information about Coders at Work on the website and anyone should feel free to leave a comment if you have thoughts about stuff I should ask these folks.

Last updated 2007-11-27T06:35:19Z.

Wednesday, 26 September 2007, 2:34

Today I interviewed Donald Knuth for my book Coders at Work. The contents of the interview will, of course, be published in the book itself, but before the interview proper started he told me something that's a bit worrisome. According to Knuth (who may have been relating something someone told him) there are three kinds of people: people who have written no books, people who have written one book, and people who have written many books. I guess unless I stop now my fate is sealed.

Last updated 2007-09-26T02:34:34Z.

Thursday, 20 September 2007, 19:29

Things are going well on the Coders at Work front. On Tuesday I did my second interview (with Jamie Zawinski) and since I last posted about signing up Donald Knuth I have added Anders Hejlsberg, Guy Steele, and Dan Ingalls to my list of interviewees. The always-up-to-date list of who’s agreed to participate is on the Coders at Work website.

Two folks I’ve been trying to sign up for a while are John Carmack and Linus Torvalds. I emailed them both a few weeks ago and have pinged them each once since but haven’t gotten any reply. No doubt these guys get a ton of mail and mine may have gotten lost in the shuffle or been eaten by some spam filter. So I come to you, gentle readers, for help. If you happen to be close, personal friends with either of these guys and think, as I do, that their insights into the art, science, and/or craft of programming would be an interesting addition to those of Armstrong, Cosell, Deutsch, Hejlsberg, Ingalls, Peyton Jones, Kay, Knuth, Norvig, Steele, Thompson, and Zawinski, please drop them a line and ask them to get in touch with me.

Last updated 2007-09-20T19:29:31Z.

Thursday, 20 September 2007, 4:24

I wrote Practical Common Lisp because I felt that Common Lisp needed a new introductory book that could ease folks raised on other languages into Common Lisp and then show them what it’s really all about. Based on emails from readers, reviews on Amazon, word of mouth in the Lisp world, and the fact that the online version of PCL is the top hit when you Google for “lisp book”, I’ll say I succeeded tolerably well. So imagine my dismay when someone pointed out to me today the Google results for “lisp tutorial”.

The top hit is a page which apparently hasn’t been updated since around 1999 and isn’t really a tutorial anyway, so much as a large list of links including a link to the Hyperspec when it was hosted at harlequin.com.1 The next few “lisp tutorial” hits are — with all due respect — exactly the sort of dated, dry tutorials that inspired me to write Practical Common Lisp in the first place and to do a deal with Apress to allow me to keep it online even after the dead tree version was published. Practical Common Lisp doesn’t appear anywhere, as far as I can tell, in the results for “lisp tutorial”.

With that in mind I did a small bit of search engine optimization today to make sure that the phrase “Common Lisp tutorial” appears on the main page of the Practical Common Lisp web site. If you also think Lisp might be better served if PCL was at least one of the results returned to a would-be Lisper searching for a Lisp tutorial you can help out: if you have a web page where it would be reasonable to do so, consider linking to the url http://www.gigamonkeys.com/book/ with a link text of “lisp tutorial” or “common lisp tutorial”. Yes, I’m asking you to participate in a Google bombing. But it’s for a good cause. Think of the children.

Update: Based on the first couple folks I’ve seen providing links to the PCL website (thanks, guys!) I must not have made myself quite clear enough. The name of the game in a Google bombing is for everyone to use the same text for the link. If you want to play along, your HTML should look like this:

<a href="http://www.gigamonkeys.com/book/">Common Lisp tutorial</a>
                

or

<a href="http://www.gigamonkeys.com/book/">Lisp tutorial</a>
                

Last updated 2007-09-20T14:09:51Z.

Tuesday, 4 September 2007, 17:22

Today I got a phone call from Donald Knuth who has agreed to be interviewed for Coders at Work. Yipee! I’ll be interviewing him later this month. Which means I’ve got half of my sixteen interviewees signed up. The folks who have agreed so far are:

• Joe Armstrong — Inventor of Erlang
• Bernie Cosell — One of the main software developers on the original ARPANET IMPs
• Simon Peyton Jones — Co-inventor of Haskell
• Alan Kay — Inventor of Smalltalk and object oriented programming
• Donald Knuth — Author of The Art of Computer Programming and TeX.
• Peter Norvig — AIer and Lisper, Director of Research at Google
• Ken Thompson — Inventor of Unix
• Jamie Zawinski — Author of XEmacs and early Netscape/Mozilla hacker

And my spam filter on the comments page seems to be working well — it has correctly identified as ham the half dozen or so real comments that have been posted since I installed it and unerringly id’d the couple hundred spams that have been posted in the same period.

Last updated 2007-09-04T17:22:19Z.

Friday, 31 August 2007, 3:25

As of today, I’ve got seven interviewees signed up (assuming that other publisher doesn’t try to hoard Simon Peyton Jones). They are:

• Joe Armstrong — inventor of Erlang
• Bernie Cosell — one of the main software developers on the original ARPANET IMPs
• Simon Peyton Jones — co-inventor of Haskell
• Alan Kay — inventor of Smalltalk and object oriented programming
• Peter Norvig — AIer and Lisper, Director of Research at Google
• Ken Thompson — inventor of Unix
• Jamie Zawinski — author of XEmacs and early Netscape/Mozilla hacker

I haven’t done any more interviews but I’ve been working on the transcription of my first interview with Peter Norvig. It’s all coming back to me how terribly painful it is to transcribe audio. Even with my snazzy homebrew transcription software, I’ve been proceeding at a rate of about 4:1 real time to interview time. That is, to transcribe 15 minutes of interview takes me about an hour. Which means that for every two hour interview I do, it’s going to take me a full day just to prepare the raw transcript. Yes, I know you can hire people to do this but a) I don’t really have a budget for that and b) correcting a transcript can be about as much work as preparing it, particularly if the material is technical and your transcriptionist is not. The good news is that as tedious as the transcribing is, it does give me a chance to start mulling the material over in my head, getting ready to edit it or to prepare for subsequent interviews.

Finally, I got around to putting a spam filter on the Coders at Work comments page so all the stupid link-bombing spammers’ efforts should now be completely for naught. It’s been very satisfying watching it eat up the steady stream of spam comments. Of course these days the spammers are about the only comments I get so I haven’t had any live tests of my filter’s ability to recognize ham. If you have any comments about or questions for the folks I’ve signed up to interview or suggestions of who I should try to get to fill in the last ten spots, feel free to leave a comment.

Last updated 2007-08-31T03:25:32Z.

Saturday, 25 August 2007, 3:18

The interviewing has begun! Yesterday I sat with the people’s choice, Peter Norvig for our first interview session. I’ve also signed up Joe Armstrong, inventor of Erlang, and Bernie Cosell, one of the software geniuses behind the original ARPANET IMPs. Both of these guys were not only kind enough to agree to be interviewed but also invited me to stay in their homes when I travel to interview them. Simon Peyton Jones has also agreed to be interviewed assuming the publisher of another book of interviews that has already signed him up for a group interview about Haskell doesn’t object. I’ve got a half-dozen other queries out with more on the way. Hopefully soon I’ll have my full roster of sixteen interviewees signed up and can start figuring out how to stretch my rather small travel budget to get me all the places I need to go to do the interviews.

On a geekier note, after I did a practice interview with a friend and went to try out the transcription software I had downloaded, I discovered that it didn’t understand the WMA files that my digital voice recorder produces. Turns out on GNU/Linux the best way to convert WMA to MP3 involves using the program Mplayer to convert the WMA to a WAV file. While skimming through the Mplayer man page I discovered it has a “slave mode” where you can type commands at it. “Hey,” I thought, “If I can control it by typing commands at it that means I can also control it by having Emacs type commands at it.” A bit of elisp hacking later and I turned Emacs+Mplayer into a quite nice bit of transcription software. A few features:

• Auto-insertion of timestamps into the transcript at the beginning of each paragraph.
• Rewinding the audio back to the previous timestamp at the touch of a key (combo).
• Automatically backing up a few seconds when I pause.

I experimented with having it automatically pause whenever I hit the backspace key and resume when I started typing again but that didn’t turn out to be as handy as I thought it would. The nice thing, of course, about having written my own software to do this is that as I get into the work of transcribing all these interviews I’ll be able to spend an endless amount of time procrastinating by fiddling with my transcription software. Er, that is, I’ll be able to tweak it just the way I want it to maximize my transcribing efficiency. (Ironically, because of either a limitation of the WMA format or a bug in Mplayer the jumping to timestamps doesn’t work quite right when transcribing WMA files so I still end up having to convert to MP3 files.)

Last updated 2007-08-25T03:18:35Z.

Thursday, 2 August 2007, 16:38

Since I started work on my new book, Coders at Work, in mid-June, hundreds of people have suggested names of programmers I should interview and helped me sort them in a variety of ways. Thanks! Now, having digested everyone’s feedback, I’m getting ready to contact the folks I think I’d like to interview.

To prepare for this next step, I’ve done a big cleanup of the Coders at Work website and have added a way for anyone who wants to to add information about individual coders in a more structured way than leaving a comment or emailing me. For an example, take a look at Peter Norvig’s page; on the left side of the page is the information I’ve already got and on the right forms where anyone can add more. At the moment most other coders’ pages will have much less information that Norvig’s; my main task now is to rectify that, at least for the folks I’m most interested in interviewing.

With that in mind, adding information about your favorite coder is a great way to increase the chance that I end up interviewing them for the book because a) most people get more interesting the more you know about them and b) the more information I have at hand about someone, the easier it will be to prepare for an interview and with at least sixteen interviews to do, I’ve got to apply a certain amount of tactical laziness.

Last updated 2007-08-02T16:38:45Z.

Saturday, 28 July 2007, 5:34

Gary King recently blogged a question about whether a use of EVAL he had recently made in some Common Lisp code was legit. This caught my eye because I suspect this question came up while Gary was working on a project that I once worked on. Anyway, since he asked, the short answer is, “No. The old rule that if you’re using EVAL you’re doing something wrong is still true.” A slightly longer answer follows.

The problem Gary set out to solve is that he’s got a a bunch of chunks of text coming out of some sort of database and he wants to query them with logical expressions like this one:

(or 
 (and "space" "mission")
 (and (or "moon" "lunar") "landing"))
                

These expressions are obviously not Lisp but rather a query language where the logical operators AND and OR (and presumably other ones like NOT) have their usual logical meaning and literal strings should evaluate to true if the text we’re matching against testing contains the string and false if not. Thus the expression above would match text containing the words “space” and “mission” or either of “moon” or “lunar” along with “landing”. Gary considered, and rejected, writing “a simple recursive evaluator to deal with ands and ors”. Instead he wrote some code to munge around expressions like the one above into a form that he could EVAL. In other words, instead of writing his own interpreter he just munged his code into a form that Lisp could interpret for him. Which is okay. Except for the rule that if you’re using EVAL you’re doing something wrong.

What he should have done is written a compiler. Luckily Common Lisp comes with a Lisp compiler built in so all we have to do to write our own compiler is translate our source language into Lisp. Here’s how I’d do it.

First define a function that takes a query expression and returns two values, an equivalent expression with all string literals replaced with GENSYM’d variable names, and an alist of the strings and the corresponding GENSYM’d names. In this phase we also detect if the same literal string appears more than once so we only generate a single binding for each unique string.

(defun translate (tree &optional bindings)
  (typecase tree
    (cons
     (multiple-value-bind (newcar newbindings)
         (translate (car tree) bindings)
       (multiple-value-bind (newcdr newbindings2)
           (translate (cdr tree) newbindings)
         (values (cons newcar newcdr) newbindings2))))
    (symbol (values tree bindings))
    (string 
     (let ((binding (assoc tree bindings :test #'equal)))
       (cond
         (binding (values (cdr binding) bindings))
         (t 
          (let ((sym (gensym)))
            (values sym (acons tree sym bindings)))))))))
                

With this function we can translate an expression like:

(or (and "a" "b") (and (or "c" "d") (or "a" "b")))
                

Into this expression:

(OR (AND #:G1 #:G2) (AND (OR #:G3 #:G4) (OR #:G1 #:G2)))
                

and this list of bindings:

(("d" . #:G4) ("c" . #:G3) ("b" . #:G2) ("a" . #:G1))
                

Now we just need to use those two values to build up a bit of Lisp. Gary’s solution was to build up an expression that he could EVAL. But it’s better to generate a LAMBDA expression because then we can compile it. Here’s the function:1

(defun make-lambda-expression (expr)
  (multiple-value-bind (tree bindings) (translate expr)
    (let ((string (gensym)))
      `(lambda (,string) 
         (let (,@(loop for (word . sym) in bindings collect
                      `(,sym (find-word-in-string ,word ,string))))
           ,tree)))))
                

If we pass the same expression to this function we get the following lambda expression back.

(LAMBDA (#:G9)
  (LET ((#:G8 (FIND-WORD-IN-STRING "d" #:G9))
        (#:G7 (FIND-WORD-IN-STRING "c" #:G9))
        (#:G6 (FIND-WORD-IN-STRING "b" #:G9))
        (#:G5 (FIND-WORD-IN-STRING "a" #:G9)))
    (OR (AND #:G5 #:G6) (AND (OR #:G7 #:G8) (OR #:G5 #:G6)))))
                

We could use FUNCALL to evaluate this expression which would at least get us out of using EVAL.2 But the real advantage of this approach is that we can compile this expression. Since Gary said he wanted to find all the strings in his database that match a given expression, he’s probably going to be evaluating his query once per string in his database. In that case it’s probably worth it to take a small up front hit in order to speed up the execution of the query since we’re going to be executing it many times. Luckily compiling a lambda expression is about as trivial as EVALing any other expression:

(defun compile-expression (expr)
  (compile nil (make-lambda-expression expr)))
                

This function, fed a query expression, returns a compiled function that takes a single string argument and returns true if the query expression matches and false if not. On Lisps with native compilers the returned function will be compiled down to machine code just the same as if we had written it by hand in our source code. We can FUNCALL this function such as in this code that collects all the strings returned by a cursor function that match the query expression:

(defun query-strings (query database)
  (loop with predicate = (compile-expression query)
     with cursor = (string-cursor database)
     for string = (next-string cursor)
     while string
     when (funcall predicate string) collect string))
                

We can also use the query function with all of Lisp’s higher-order-functions, such as REMOVE-IF-NOT:

(remove-if-not (compile-expression query) *all-strings*)
                

Now, one could argue that there’s a whole heck of a lot of difference between using EVAL and wrapping something in a lambda expression and compiling it — in both cases you can generate, and then evaluate, arbitrary Lisp code. But there is an important difference, namely that EVAL just evaluates — it takes some data and interprets it as Lisp and gives you an answer straight away whereas compiling a lambda expression gives you a function, something that can interact with the rest of your code, as an argument to higher-order functions, and so on. Or, if you don’t buy that, at least you avoid breaking the no EVAL rule.

Update: It hit me as I was brushing my teeth that while nicely avoiding EVAL, my first solution had a big problem — because all the FIND-WORD-IN-STRING calls are done in the LET before the boolean expression is evaluated we lose all the advantage of AND and OR’s short-circuiting behavior. That is, the generated code searches for all the strings and then combines the results of all those searches. Much better (and simpler) would be to implement TRANSLATE and MAKE-LAMBDA-EXPRESSION this way:

(defun translate (tree &optional (stringvar (gensym)))
  (typecase tree
    (cons
     (cons (translate (car tree) stringvar) (translate (cdr tree) stringvar)))
    (symbol tree)
    (string `(find-word-in-string ,tree ,stringvar))))

(defun make-lambda-expression (expr)
  (let ((string (gensym)))
    `(lambda (,string) ,(translate expr string))))
                

This has the slight disadvantage that if the same literal string appears in the pattern more than once, we will potentially search for it more than once. But that’s probably much less of an issue than the problem this code fixes. Obviously if we cared to, we could generate code that caches searches once they are done to get the best of both worlds. But I’ll leave that as an exercise for the reader.

(defun find-word-in-string (word string)
  (search word string :test #'char-equal))
                        

Last updated 2007-07-28T07:07:59Z.

Thursday, 19 July 2007, 16:18

As I mentioned the other day I’ve started working on a set of general questions for my Coders at Work interviews. They are now up on the Coders at Work web site. Some of them are, no doubt, dumb and there may still be some redundancies in the list but it’s a start. As always I’m interested in what other folks think. Are there questions you’d like me to ask some or all of the programmers on my list?1 Email them to me or leave a comment on the Coders at Work comment page.

Last updated 2007-07-19T16:18:29Z.

Tuesday, 17 July 2007, 15:26

It’s been a while since I’ve posted anything about Coders at Work. Last week I put up, but did not broadly announce, a new, improved web page for sorting the nearly 300 suggested names of people to interview I’ve received. I mostly created it for my own benefit but anyone who wants to play with it can feel free to make their own sorting. (I’m mostly announcing it now because I think it’s sort of a nifty UI — much more fun to play with that the previous one — though I am also interested to see how other folks would sort the list of names I’ve got. Unfortunately, like the previous sorter, this page probably doesn’t work in IE and is only known to work for sure in Firefox.) One feature of this page that the other short list selector didn’t have, is the ability to effectively down vote names — you can put folks who you are particularly uninterested in hearing from at the bottom of the list and that tells me something.

If you think you’ve got a sorting that would make for a great book, click the Save button then “Permalink” on the next page to get a URL for your specific sorting and send it to me. I’m also still interested in new suggestions of folks to interview though as I’m getting pretty close to starting to contact folks and since I’m ultimately only going to interview on the order of sixteen people, new names have to be pretty obviously better than over 250 of the names I’ve already got to break into serious contention. As always, I’m also interested in your comments about why you think someone would (or wouldn’t) make a good interview subject.

I’ve also started serious work on a set of general questions. So far I’ve got 191 questions in 22 categories. Which is a lot — obviously I’ll have to pare that down unless I’m going to interview people for 10 or 20 hours. And that’s not even counting questions more directly tailored to the individual subjects. But it’s still better to have too many than too few so if you’ve got a question you’d like me to ask either everyone I interview or specific people, feel free to email or leave it on the Coders at Work comment page.

Update: I didn’t mention before but you can see the combined results of everyone who has submitted a sort via this new page here.

Last updated 2007-07-17T21:16:38Z.

Thursday, 12 July 2007, 16:59

Yesterday I was taking my 10-month old daughter to our parent/infant swim class at the Berkeley YMCA. She happened to be wearing a blue shirt. The woman riding down with us in in the elevator from the parking garage asked how old she was and said, “Oh, what a cute little boy.”

“Girl,” I said.

“Oh, sorry!”

“No worries. It’s the shirt. And the short hair.”

“I know, we’re all so color coded.”

As we were getting out of the elevator she said, “You know, I should be the last person in the world to do that … I teach feminist theory.”

Last updated 2007-07-12T16:59:19Z.

Saturday, 7 July 2007, 21:14

Since I put up the short list submission page1 for Coders at Work a week and a half ago, I’ve received 162 short lists of sixteen names. You can see which programmers appear on the most short lists on this page. Thank you to everyone who took the time to submit a list.

For the next step in my selection process I’ve put together a page on which I’ve lumped the potential interview subjects into various categories such as “Old school Unix hackers”, “New school Unix hackers”, “Language designers”, and so forth. My theory is that it’ll be more interesting to read interviews with different kinds of programmers than a bunch of interviews with people who’ve all done the same basic kind of work. It’s also a useful way to identify potential interviewees — I can look at each category and ask, is there someone not on this list who’d be a better representative of the category?

As usual, there are ways you can help me out, if you’re so inclined. I enumerate them on the categorization page but basically they are to send me email or leave a comment nominating someone as the best representative of a category, helping me categorize the folks I’ve already got, and suggesting interesting categories that I don’t have yet.

Last updated 2007-07-07T21:14:05Z.

Tuesday, 26 June 2007, 12:48

My previous post provoked an outpouring of suggestions of people to interview for my book Coders at Work, to have approximately three to four binary orders of magnitude more than I can possibly interview for the book. So I put together a web page with a bit of Javascript on it to help me sort through the names and build a short list of (for now) sixteen names. Since it’s a sort of an interesting exercise, and because I’m curious what other people think, I’ve put the page up on the Coders at Work website. If you want to try your hand at picking sixteen names out of the almost 256 nominees I’ve got, go for it1. I’ve also put up a comments page so people can submit suggestions and flames about the book in a more public forum than my email inbox.

Last updated 2007-06-26T12:48:38Z.

Tuesday, 19 June 2007, 14:52

I’ve put up a preliminary web site for my in-progress book Coders at Work that I mentioned the other day at www.codersatwork.com. At the moment it consists of an ever growing but still somewhat arbitrary list of potential interview subjects and a tiny bit of information about each of them. I’m going to be adding names to this list fairly quickly and then getting to work filling out details about each person while also winnowing it down to a reasonable number to actually interview. So now’d be a good time to email me suggestions of programmers who you’d like to read an interview with. Or if there are folks already on my list that you’re particularly interested in, feel free to let me know why and to point me to good sources of information about them. The more background I have going in, the better the interviews are likely to be.

Last updated 2007-06-19T14:52:39Z.

Wednesday, 13 June 2007, 3:56

In my first post to this blog I said I was working on a book about programming in groups. A few weeks ago I met with Gary Cornell the CEO of Apress and my editor on Practical Common Lisp to talk about my book idea. His reaction was, more or less, “That’s interesting but I’ve got another book you should do first.” So, as of today I’m working on a book, tentatively titled Coders at Work, which will be a collection of Q&A interviews with interesting programmers.1 It will also be a companion volume to Apress’s recently published Founders at Work by Jessica Livingston, which is a collection of interviews with founders of high-tech startups. I hope soon to have a web site set up where I’ll be putting up pages listing people I’m hoping to interview and collecting suggestions for possible subjects and questions to ask them. I’ll post here when it’s up. In the meantime, if there’s anyone you think would make an interesting interview subject or questions you think I should ask let me know.

Last updated 2007-06-13T03:56:33Z.

Tuesday, 29 May 2007, 16:54

Surely everyone involved in developing software has heard of Brooks’s Law. First presented in the eponymous chapter of Frederick P. Brooks, Jr.’s classic The Mythical Man Month, it states: “Adding manpower to a late software project makes it later.” This “law” is much beloved by software developers as a handy bucket of cold water with which to cool the ardor of overly enthusiastic managers and executives. Lately, however, I’ve been thinking about Brooks’s Law and rereading The Mythical Man Month and I’m no longer as impressed with Brooks’s analysis as I once was. This is the third in a series of posts discussing some of the reasons why. The first post in the series discussed training costs and the second talked about sequential constraints.

In addition to training costs and sequential constraints, part of the justification for Brooks’s Law is the claim that as a team expands the cost of communication between team members grows faster than total productivity of the team. As with the issue of training costs, Brooks has a point — the number of possible pairwise communications paths between a team of n people is n(n - 1)/2; that’s a simple fact of math. If you then assume that every possible pair will need to spend a certain amount of time communicating or, equivalently, that the whole team will have to get together for meetings whose total length is determined by the number of people on the team, then it’s true that the number of person-hours spent communicating will grow as the square of the number of people while total productivity, again measured in person-hours, will only increase linearly. We can all see where that’s heading — pretty soon the amount of time spent on communication will be greater than the total amount of time available to work on anything at all and nothing else will get done. But how soon?

To take a concrete example, suppose we’ve got a six person team that we’re thinking of expanding to eight; should we be concerned that the increasing communication costs will eat up any additional productivity we might get from the two extra people? We can figure it out. Suppose that each pair on our team gets together for a pure-overhead, one-hour tête à tête every week. Assuming a week is five eight-hour work days, the whole team spends 30 person-hours on communication per week out of 240 person-hours worked, leaving 210 person-hours of productive work. What happens if we expand the team to eight? Each person will now spend seven hours a week in pairwise communication and the team as a whole will spend a total 56 person-hours a week communicating. But the team will also now be able to do a total of 320 person-hours of work, leaving 264 person-hours to be spent on productive work, or 54 more hours. This calculation does demonstrate Brooks’s larger point, that a “man month” is not a useful measure of productivity — if it were, then expanding a team from six to eight, a 33% increase in size, would likewise increase productivity by 33%, not the approximately 26% we actually get. But this example doesn’t justify, by itself anyway, a blanket claim that adding people to a project will always slow it down — the eight person team can, in fact, get more done than the six person team and therefore should finish the same amount of work sooner, all other things being equal. Of course all other things are not necessarily equal — training costs can reduce the initial productivity of new team members and it’s conceivable the sequential constraints introduce a long leg that can’t be reduced by adding people. In a later post I’ll discuss whether these three factors together might be enough to justify Brooks’s Law.

So, for this team, a jump from six to eight people probably won’t add more communication costs than the productivity of the new people. But obviously Brooks’s is right that eventually quadratic growth will outpace linear growth.1 With a bit of math, we can figure out exactly when the costs of communication start outpacing gains in the ability to do useful work for a given amount of per-pair communication overhead and a given number of hours worked per week. If we have a team of n people that work h-hour weeks and in which each pair spends c hours per week on communication overhead, then the cost to the existing team members of adding one person to the team is n×c because each of the n current team members will now be part of a pair with the newcomer and will have to spend c extra time tending to that pair’s communication needs. Meanwhile the newcomer will also spend n×c hours per week on pairwise communication which, when subtracted from the h total hours they’ll work each week, gives us the amount of non-communication work per week they’ll add to the team’s total capacity. When the amount of productivity lost from the existing team members is the same as the amount of productivity added by the newcomer there’s no point in expanding the team. So we can solve this equation for n:

to get this:

With this formula we can determine that for a team with one-hour per week of per-pair overhead, working 40-hour weeks, the the biggest the team can get without loosing more productivity than it gains, is 20.

But all of these computations may be beside the point, as they’re based on the assumption that communication has to be overhead. What if we had a team of six people that spend not one, but eight hours per day on pairwise communication because they spend all their time pair programing? If we add two people to that team, there is no change in time spent per person on pairwise communication — the only change is, assuming the team rotates partners, that each person will pair with each other person less often. But the communication time obviously can’t be all overhead — the only time anything gets done is when two people are communicating.

So how can this possibly square with Brooks’s analysis? One possibility is to join the ranks of the XP skeptics and simply deny that the pair programming team could possibly get anything done. I’ve had good experiences with pair programming though so I can’t buy that. I think the problem is with Brooks’s underlying assumptions. As I’ve mentioned previously, Brooks assumes that an n-person team will partition the task of writing whatever software they need to write into n pieces, each to be written by one person. To the extent that those pieces of software need to talk to each other, so do the people writing them and this communication is extra work on top of the base amount of work required to write the software. His arguments about training costs, intercommunication, and sequential constraints are all aimed at demonstrating that a task that a single developer could complete in x months will take n developers more than x/n months because the amount of work required for n developers is no longer simply x but x plus overhead.

But there’s another possibility. What if, as I suggested in my post about Sisyphus, n people don’t have more than x work to do but less than x because n people working together and communicating a lot are much more likely to discover a better solution than any one of them working alone? In that case, time spent communicating is not extra work but a way of reducing the total amount of work done.

Last updated 2007-05-29T16:54:06Z.

Monday, 28 May 2007, 21:13

Today my wife was downstairs playing with our eight-month-old daughter, Amelia, and all I could hear was the sound of Amelia laughing, laughing, laughing. I mean, really cracking up then settling down a bit and cracking up all over again. I’m sure I’m far from the first person to have had this feeling but it gives me some small measure of hope for the human race that this little person who barely knows her own name and doesn’t know enough not to crawl off the edge of the bed, has, if nothing else, a sense of humor.

Last updated 2007-05-29T01:28:46Z.

Saturday, 26 May 2007, 23:44

I just found out that Apress has decided it’s time for a third printing of Practical Common Lisp. If I recall correctly, the first printing was 5,000 copies, the second 3,000 more. New printings are called for when the publisher thinks they’re going to run out of copies to sell to distributors so this must mean I’m not crazy to dream of someday having a 10k-copies-sold party.

This also means now would be a good time, if you’ve read the book and noticed any errors that you’ve not emailed me about, to send a note. If you put “pcl errata” in the subject it’ll make my life a bit easier. Note, however, that this is just a new printing not a new edition. For a new printing we just fix minor typos and so forth so now is not the time to tell me that there should really be a chapter about how to connect to RDBMSes or what have you.

Last updated 2007-05-26T23:44:36Z.

Tuesday, 22 May 2007, 15:56

Surely everyone involved in developing software has heard of Brooks’s Law. First presented in the eponymous chapter of Frederick P. Brooks, Jr.’s classic The Mythical Man Month, it states: “Adding manpower to a late software project makes it later.” This “law” is much beloved by software developers as a handy bucket of cold water with which to cool the ardor of overly enthusiastic managers and executives. Lately, however, I’ve been thinking about Brooks’s Law and rereading The Mythical Man Month and I’m no longer as impressed with Brooks’s analysis as I once was. This is the second in what I expect will be a series of posts discussing some of the reasons why. The first post in the series discussed training costs.

As part of his “demythologizing of the man-month” (p. 25) Brooks points out that developing software is subject to what he calls “sequential constraints”. Brooks actually makes two points about sequential constraints, but he doesn’t draw a particularly clear distinction between them in his exposition, so I’ll start by teasing them apart. They are:

1. All tasks, including software development, have some sequential constraints on their subtasks that determine the minimum time in which the whole task can be completed.
2. Even if the rest of the work can be parallelized, communication needed to coordinate work can act as a sequential constraint, putting a lower bound on the time needed to complete a task.

Point one is a simple matter of logic. Virtually every task, from harvesting a field of crops, to having a baby, has some sequential constraints on some of its subtasks that determine that certain subtasks can only be done after other subtasks are complete. It’s impossible to complete the whole task in less time than the time it takes to do the longest sequential chain. By definition, subtasks that are not sequentially constrained can be done in parallel and so more people working on them at the same time will get them done sooner than fewer people.

Tasks vary in the both the nature and extent of the sequential constraints that apply to their subtasks. Brooks gives harvesting crops as an example of a task with very few sequential constraints and bearing a child as one that nothing but a long sequentially constrained chain. It’s worth noting, however, that all real-world tasks are sequentially constrained at some level — even harvesting a field of crops, for instance, requires that someone get to the farthest corner of the field, harvest what’s there, and bring it back. No matter how many field hands you hire and no matter how minuscule the part of the field each is responsible for there’s still no way to harvest the whole crop faster than that long leg could be completed.

For practical purposes, however, Brooks is right: harvesting crops is almost entirely parallelizable and bearing a child is almost entirely not. Before we get to how communication itself can act as a sequential constraint, let’s consider another task which is more sequentially constrained than harvesting crops but less so than having a baby, namely baking a cake. Consider for instance this simple cake recipe:

1. Pre-heat the oven to 350°.
2. Prepare the cake pans — greasing, lining, and flouring.
3. Sift together flour, baking soda, and salt.
4. Cream the butter, shortening, and sugar until light and fluffy.
5. Add dry ingredients to butter/shortening/sugar mix.
6. Mix in three eggs.
7. Pour batter into cake pans.
8. Bake for 25 to 30 minutes.
9. Cool on racks for 10 minutes.
10. Remove from pans and continue cooling.

As with most recipes, there are both opportunities for parallelism and unavoidable sequential constraints. If you had a three cooks in the kitchen one of them could prepare the cake pans while another sifts together the dry ingredients and a third creams the butter, shortening, and sugar. After that, the next three steps, up to pouring the batter into the cake pans, while sequentially constrained relative to each other, could be done in parallel with the oven heating. Thereafter, everything is sequentially constrained. No matter how many cooks you have, you have to heat the oven before you bake the cake and bake the cake before it can cool. Thus there’s no way to decrease the total time it takes to make a cake below about 45 minutes.

Now let’s consider how software development is sequentially constrained. As anyone who has written software knows, there are sequential constraints but where do they come from? There are certainly no physical constraints such as the one that keeps a baker from pouring batter into cake pans before the batter has been made. If, somehow, we knew at the beginning of a development project all the lines of code that needed to be written, we could type them in any order we wanted — the software would work just as well in the end. But the notion that we could know in advance all the code that needs to be written and type it like we were taking dictation is just crazy. Programming isn’t primarily a typing problem, it’s a thinking problem. And thoughts need to be thought in the proper order.

In fact, the only way to figure out how a software system ultimately fits together is to build it. In order to know, in detail, how part X is going to work we need to know how part Y, with which it interacts, is going to work. And the only way to know how Y is going to work is to build it. It may be that we can completely build X and then build Y or we may need to alternate — build a bit of X in order to develop enough information to build a bit of Y from which we learn enough to build another bit of X, and so on. It might also be equally possible to start by building X and then build Y or to start with Y and then build X. But however we do it, the flow of information about the system we are building are the inherent sequential constraints we operate under. Note that this has nothing to do with communication — even if the system were being built by a single developer these constraints would still constrain the order in which various parts of the system could be built.

Now, keeping in mind these inherent sequential constraints, let’s consider Brooks’s second point, that the need to communicate can itself act as a sequential constraint. This is the software development equivalent of Ahmdal’s Law from parallel computing which says that no matter how much you can parallelize a computation, it’ll never complete faster than the time it takes to combine the results of all the parallel computations. In both software development and parallel computing this is because communication is inherently sequential. If ten people — or ten CPUs — each have six minutes worth of information to convey to each other, it’s going to take at least an hour of elapsed time no matter how you slice it; ultimately each person is going to have to spend six minutes “transmitting” their information and fifty-four minutes “receiving” information from the other nine people.

To see how this effect plays out, imagine we have an idealized software development task whose coding can be partitioned among however many developers we like but for every ten hours a developer is going to spend coding, they need to spend one hour writing down what they’re going to do for the benefit of the rest of the team and everybody has to read everyone else’s notes. In other words, before each ten hours’ worth of coding, a developer spends an hour writing an email about what they are about to implement and sends it to all the other developers. Then they have to read the other developers’ emails, spending an hour to absorb each one. After all that communication, the developers can each code for ten hours. For simplicity, we’ll assume that even a developer working alone would spend the hour writing notes for themself documenting what they plan to do in the next ten hours.

Suppose the total coding time needed to develop the system is 100 person-hours. A single developer could do it in 110 hours, ten chunks of an hour of note writing followed by ten hours of coding. Two developers could do it in five chunks of work with each each chunk consisting of twelve hours of work: an hour writing notes, an hour reading the other developer’s notes, and ten hours coding. Thus for the team of two, the total elapsed time would be 60 hours, of which 10 would have been spent on communication. Five developers could complete the project in only two chunks but each chunk would be fifteen hours: one hour writing, four reading, and ten coding. Thus their elapsed time would be 30 hours with 10 hours spent on communication. Ten developers would be done in 20 hours elapsed time — ten hours of development after ten hours of communication.

Even if we could scale down the communication cost proportionally, so less than ten hours of individual work requires a proportionally smaller amount of email writing and reading, the elapsed communication time still stays at ten hours: twenty developers would spend a half-hour writing their emails and nine and a half hours reading nineteen emails before coding for five hours. Indeed, as the number of developers approaches infinity, the amount of time spent coding approaches zero as does the amount of time each developer spends writing their own email while the amount of time spent reading the infinite number of infinitesimally short emails from other developers approaches ten hours and the project as a whole still takes a minimum of ten hours to complete. Thus even when there are no other sequential constraints — when we assume that an infinite number of developers can each be given an infinitesimally small part of the project to work on in isolation — communication remains the one activity that must be performed sequentially.

In real software projects, of course, things are more complicated. The inherent sequential constraints — those that would affect a single developer working alone — interact with communication induced constraints in all sorts of complicated ways. For one thing, if we assume — as Brooks seems to — that the overall task is partitioned into subtasks, each to be developed by a single developer, then the way we do the partitioning can have dramatic affects on the amount of communication needed. If we split the system at its natural joints, then communication will be minimized — if subsystems are naturally decoupled then developers can work on their bit for a while, developing lots of information about how their part of the system works, which only they need to know, and just a little bit of information that they need to share with other developers. On the other hand, if the partitioning is poor, each developer’s part of the system will depend on many details of other parts and the developers will either need to communicate much more often or, more likely, will all go off in their own directions for a bit too long and then discover, when they compare notes, that they need to backtrack and rework things in order to make everything fit together.1

Another issue, which Brooks doesn’t mention, is that the need to communicate can stall productive work. One of the idealized aspects of the hypothetical project above is that the developers work in perfect lockstep — everyone communicates and then works for exactly ten hours and the cycle repeats. At no point is anyone stalled waiting for someone else. In real projects, some subtasks will be bigger than others leaving developers whose pieces happens to be smaller waiting after they’ve finished their work to communicate with developers whose pieces are larger. Every hour that they spend waiting is an hour that gets added to the total number of person-hours it takes to complete the project.

That all said, there’s nothing that says the only way to divide up a task is by partitioning it into pieces that are each implemented by a single developer. In fact there are all sorts of reasons, which I’ll talk about in a later post, that that might be a bad idea. For now, let’s just note that if we could avoid a strict upfront partitioning, and could let developers share ownership of the system as a whole, working together frequently and sharing ideas about how it all fits together, they could probably much more closely emulate the order of development that we would see if we watched a single developer build the whole system, constrained only by the inherent constraints of needing to build enough of X in order to know enough to build Y and discovering, as they go along, enough bits that can be naturally carved off and done in parallel to keep everybody busy.2

So how does all this relate to Brook’s Law? In the concluding paragraph of the chapter, right after he has stated his Law, Brooks goes on to say:

The number of months of a project depends upon its sequential constraints. The maximum number of men depends upon the number of independent subtasks. From these two quantities one can derive schedules using fewer men and more months. ... One cannot, however, get workable schedules using more men and fewer months. (pp. 25-6)

The last sentence is only true if the number of workers currently on the project is sufficient to take advantage of all the opportunities for parallelism. For instance, suppose we have a project consisting of forty individual tasks, each of which will take a weeks’s worth of work by one person. Now suppose ten of those tasks are inherently sequentially constrained while the other thirty tasks can be done at any time, in any order. Because of the ten sequentially constrained tasks, the project can’t be completed in any less than ten calendar weeks. But suppose the project has been assigned to a two-person team. It will take them twenty weeks to do the whole project, ten weeks longer than the minimum. Clearly in this case, we can get “workable schedules using more men and fewer months” by adding one or two people to the team. A team of three would finish in a bit over thirteen weeks and four would finish in the minimum time of ten weeks. To say that we can’t reduce calendar time because of sequential constraints would only be correct if we had originally assigned the project to a four person team.

In general, given that Brooks’s Law is talking about late projects, that is, ones we badly underestimated in the first place, what’s the likelihood that our estimate of how many people we needed was exactly right? The real question, if we’re concerned about sequential constraints, is whether or not there’s work that could be done in parallel. Sometimes there is and sometimes there isn’t and assuming that there never is is just as foolish as assuming that there always is.

Last updated 2007-05-22T15:56:21Z.

Friday, 18 May 2007, 3:02

Surely everyone involved in developing software has heard of Brooks’s Law. First presented in the eponymous chapter of Frederick P. Brooks, Jr.’s classic The Mythical Man Month, it states: “Adding manpower to a late software project makes it later.” This “law” is much beloved by software developers as a handy bucket of cold water with which to cool the ardor of overly enthusiastic managers and executives. Lately, however, I’ve been thinking about Brooks’s Law and rereading The Mythical Man Month and I’m no longer as impressed with Brooks’s analysis as I once was. This is the first in what I expect will be a series of posts discussing some of the reasons why.

When Brooks says that adding manpower makes a late project later, he doesn’t specify what he means by later. Later than it already is? Almost certainly, but so what? Later than your new wildly optimistic estimate? Probably, but again not all that interesting. The slightly paradoxical interpretation that makes Brooks’s Law such a perennial on amusing quotation lists is: later than it would have been if you had just left well enough alone.

Of the various reasons Brooks gives in the chapter “The Mythical Man Month” for projects running out of calendar time, the only one that has specifically to do with adding staff to an existing project is the cost of training the added staff. There are other costs associated with having a bigger team that such as potentially increased intercommunication costs and the need to repartition tasks. I’ll discuss those costs in later posts but for now I’m concerned only with whether Brooks’s own analysis of the costs of training holds water.

If we were to take Brooks’s Law as literally true, then we would have to believe that the costs of training new staff will always be higher than any capacity for productive work they might eventually develop. That seems unlikely. However, Brooks’s Law only refers to “late” projects so perhaps there’s something about being late that makes it true. Unfortunately, he doesn’t define “late” any more than he defines “later” so if we want to apply Brooks’s Law wisely we’re on our own — we need to ask, when can we get more done by adding staff than by not?

Much more often than Brooks lets on, it seems. In the section “Regenerative Schedule Disaster” Brooks uses a hypothetical project, originally estimated to be twelve person-months of effort and assigned to a three person team, to demonstrate how training costs affect our ability to speed up a project. In his scenario the project has been divided into four milestones, each of which should be completed in one calendar month by the team of three, i.e. three person-months per milestone. Unfortunately it takes the team two calendar months, or six person-months, to finish the first milestone, so there are only two months left to complete the remaining three milestones. Brooks then considers two sub-scenarios — one where only the first milestone was mis-estimated, in which case there are nine person-months worth of work left and two months in which to do it, and another where the underestimation was systematic so the three remaining milestones are all, like the first, six person-months of work leaving eighteen person-months of work. The question he then poses is, what happens if a manager attempts to get the project finished in the remaining two calendar months by adding staff.

In the first sub-scenario, a manager who ignores training costs would calculate that they need four and a half people to do nine person-months of work in two months. Rounding up to five, subtracting the three they’ve already got, and they add two people. In the second scenario, eighteen divided by two is nine, subtract the three they’ve got, and they’d need to grow by six. Brooks then analyzes the first sub-scenario, making the rather conservative assumption that it’ll take one month of full-time work by one of the existing team members to train the two newcomers before they’ll be able to do any work. Under that assumption, only two people will do productive work during the third month so only two more person-months of actual work will be done, leaving seven. In the fourth, and final, month, the new people will start contributing and the trainer can get back to real work but it’s too late — they’ll get five person-months worth of work done but with seven left to do the schedule is blown.

But there’s another way to look at it. With the two newcomers, the team managed to complete a total of thirteen person-months worth of actual (non-training) work, or almost 87% of the originally planned functionality (assuming the revised estimate of fifteen person-months for the whole project is correct.) What would have happened if they had heeded Brooks’s Law and just kept going with the original three-person team? They’d have completed only twelve person-months, or 80% of the originally planned effort. Or, if it’s more important to deliver 100% of the functionality as soon as possible, the original team would have needed another month, blowing the schedule by 25% while the augmented team would only need an additional two-fifths of a month, or about 10% over the original schedule.

In Brooks’s second sub-scenario, where the actual project size is assumed to be twenty-four person-months, the benefits of adding staff are even more pronounced. Assuming the same one-month of full-time training, the augmented team finishes almost 71% of the originally planned effort in four months compared to only 50% by the original team. Or they can finish the whole project in a bit less than five months total, extending the original schedule by about 20%, compared to the 100% by which to the original team would blow the original schedule.

The problem is not that adding staff to the project didn’t help; it’s that it didn’t help quite enough. You might ask, why not account for the training costs when figuring out how many new staff are needed? Brooks briefly considers that idea and rejects it on the grounds that the seven person team needed in the first sub-scenario to finish the remaining seven person-months worth of work after training would be too different in kind from a five person team for it to be feasible. That may be true but the question remains, what’s the alternative? Brooks considers attempts to finish the project on the original four-month schedule “disastrous” and recommends that we should instead “reschedule” or “trim the task”. Both of those are probably wise strategies but even with Brooks’s conservative assumptions about training time, the expanded team would still get more done needing either less of a schedule slip or less trimming of functionality.

At any rate, it’s not the case, in either sub-scenario, that training costs on their own would cause the project to finish later with additional staff than it would have without. Brooks does, however, make one important point when he says:

Notice by the end of the third month, things look very black. The [second] milestone has not been reached in spite of all the managerial effort. The temptation is very strong to repeat the cycle, adding yet more manpower. Therein lies madness. (pp. 24-5)

It is important not to lose one’s nerve. If you’ve already used up two months of a four-month schedule, it’s going to be queasy-making to reduce your productivity by a third for another whole month. If you do, you’ve got to stick with it to reap the benefits as your new workers get up to speed. It also suggests two bits of tactics. One: make sure you add enough new staff. If you’re going to take the hit of losing the output of one or more of your currently productive workers to training you want to make sure you get as big a return on that investment as possible — add as many people as you can afford and as you think can be trained in a reasonable amount of time. Second, make sure you invest enough in training. In his own reappraisal of Brooks’s Law Steve McConnell called Brooks’s assumptions about training costs “absurdly conservative” and they may be. But notice that even with those conservative assumptions the investment can still pay off quickly. It can be tempting to try to cheat, adding staff without explicit training, hoping they’ll somehow get up to speed on their own. If it works, great, but more likely they’ll just nibble away at the productivity of the current staff without ever becoming productive enough to offset the cost. Better to plan conservatively and then end the training ahead of schedule if they’re ready to get to fully productive work sooner than planned.

Last updated 2007-05-18T16:46:31Z.

Friday, 11 May 2007, 3:45

As all software developers know, nine women can’t have a baby in a month. Or in Fred Brooks’s more elegant phrasing: “The bearing of a child takes nine months, no matter how many women are assigned.” (The Mythical Man Month, p. 17) The point, of course, is that some tasks are, as Brooks would say, “sequentially constrained”. They’re going to take a certain amount of time no matter what — the time can’t be reduced by having more people work on them.

On the other hand, is it actually the case that one woman can have a baby in nine months? Suppose we have just been put in charge of Project New Baby that must produce a brand new baby in nine months. How should we staff the project. Easy enough — nine women can’t have a baby in a month, right? No point in overstaffing so we’ve just got to find a couple, make sure they’re both fertile and interested in having a kid, and we’re good to go. But wait a sec’, what’s the chance they’ll miss our nine month deadline by more than a month? Pretty high, it turns out.

Typically a couple trying to get pregnant has about a 16% chance in any given month. Once they’ve conceived, there’s, sadly, about a 15-20% chance of miscarriage, usually within the first three months. So the chance our couple will produce a baby nine months from now is only .16 × .85 or 13.6%. If we wanted to we could compute the average time we should expect it to take for one couple to have a baby, using math similar to that in an earlier post. But suppose the deadline is hard — we really, really need to finish Project New Baby in nine months — is there anything we can do?

Sure. Throw bodies at it. While a single couple has a 86.4% chance of missing our deadline, if we had two couples, the chance that they’d both miss it is only .864² or about 74.6%. With three couples, the chance of blowing it is down to 64.4%. To figure out how many couples we need to have a P chance of hitting our deadline, just plug P into this formula:

Of course, this could get expensive if we need to be really certain of hitting that deadline — to have a 90% chance of hitting it we’d need sixteen couples. But depending on how important Project New Baby and it’s deadline are, it might be worth it.

So what in software is like making babies? Let’s take a look at how Brooks himself tied making babies to making software:

The bearing of a child takes nine months, no matter how many women are assigned. Many software tasks have this characteristic because of the sequential nature of debugging. (p. 17)

Unfortunately, I haven’t been able to find anywhere where he explains what he means by “the sequential nature of debugging” but I can see how debugging is like having a baby. And not that they both can be incredibly painful and that you have a great feeling of relief when you’re done. The similarity that I see is that the time it takes to find a bug has a large random component, like trying to conceive a child. Basically when you’re looking for a bug, there’s some probability p that you’ll find the bug for each unit of time that you spend looking, just like a couple has a 16% chance of getting pregnant for each month they spend trying. If you’re a skillful debugger and know your code really well p will be higher but there’s always a random element — if you go down the wrong path it can take you a while to realize it and all that time is lost whereas if you had tried a different path first, you might have found the bug right away. This is why it’s almost useless to try to estimate how long it will take to find a bug. You could find it in the next five minutes or five weeks from now. Once you find the bug of course you also have to fix it but that tends to be less random — unlike a pregnancy, which always lasts about nine months after conception, different bugs will require more or less work to fix, but once you’ve found it you can usually characterize how big a job it’ll be. And for many, if not most, bugs finding them is the hard part — once you’ve well and truly tracked them down, the fix is often trivial.

All of which suggests we can use the same technique to speeding up debugging as we did on Project New Baby — throw bodies at it. Suppose we’re ten days from the end of a release and there’s one last serious bug to be tracked down. Suppose my chance of finding it is 10% per day. The chance that I won’t find it in the next ten days is (1 − .1)¹⁰ or about 35%. But if there’s someone else who can also look for it — say a pair programming partner — who also has a 10% chance of finding it per day, and we both work at it separately. Then the chance that the bug will remain at large by the end of the release drops to 12%. If we can throw even more developers at it, then the chances of the bug escaping drop even more: 4% chance with three developers, 1% with four, 0.5% with five.

Obviously, to be able to take advantage of this strategy requires having multiple developers with enough familiarity with the code to be able to pitch in. Which seems to me a strong argument for practices such as pair programming and collective code ownership. An interesting side question is whether, if you do have developers to throw at debugging in this way, it is better for them to work independently or should they pair up for the debugging on the grounds that two heads are better than one?

Last updated 2007-05-11T03:57:55Z.

Wednesday, 9 May 2007, 4:00

While working on another blog entry (still in progress) about Brooks’s Law, I got to thinking about pair programming and how it’s possible that two people working together, sitting at one computer, can be more productive than the same two people working on their own and combining their work. I certainly believe they can, based on my own experiences with pair programming. But after immersing myself in Brooksian notions of how communication costs quickly eat up all available productivity it seems a bit of a paradox. To the extent that writing software is like carrying rocks up a hill — and doesn’t it often feel that way? — here’s an explanation.

Suppose you have a hundred heavy rocks that you need to carry up a hill. They’re not so heavy that you can’t do it but they’re heavy enough that moderately often you’ll lose your grip and the rock will roll back to the bottom of the hill. Let’s say on each attempt to carry a rock up the hill there’s a 70% chance you’ll lose your grip. Assume that when you don’t drop the rock it takes five minutes to carry it up the hill and a minute to walk back down. First question: how long will it take you to get all the rocks to the top of the hill? Obviously in practice it depends on how often that 70% chance of the dropping the rock actually bites you, but we can figure out an expected value. If the drops are randomly distributed — sometimes near the bottom of the hill and sometimes near the top — you’ll lose an average of three minutes per drop. But once you drop a rock you have to start all over again with it and there’s a chance you’ll drop it again. Thus the amount of time you should expect to spend on each rock is six minutes plus the sum of this infinite series:

Add that seven minutes to the six minutes to get it to the top of the hill without dropping and we get an average of thirteen minutes per rock, or 1,300 minutes for all one hundred rocks.

Now suppose you had a partner. Assuming there’s room for two people to carry rocks at the same time, one way to reduce the time it takes to get all the rocks to the top of the hill would be to simply each carry fifty rocks — the 1,300 minutes would be cut in half, to 650 minutes. But there’s another possibility — since the rocks are just a bit too heavy for one person to manage 100% reliably perhaps the two of you working together would be strong enough to never drop a rock. In that case, you could carry all hundred rocks up without dropping any and the whole job would take only 600 minutes, even better than splitting the work.

Of course if the chance of one person dropping a rock was lower, then working separately might be a better bet. In fact we can figure out exactly what probability makes it better to work separately or together by solving this inequality for p:

The numerator of the left hand side represents the expected time it’ll take for one person to get one rock to the top if it takes x minutes with no drops. We divide by two to account for the fact that there are two people working at it. The right hand side represents the time taken with both folks working together and never dropping a rock. After some algebra the xs all go away and it turns out that when the probability of dropping a rock is greater than ⅔ it’s better to pair up than to work separately.

Now, a ⅔ chance may seem fairly high but it’s worth thinking about where that probability comes from. Let’s consider how a ⅔ chance of dropping the rock over five minutes relates to the chance that we’ll drop it in any single minute. To back out the per-minute chance of dropping, given the total probability of dropping and the number of minutes, we start by recognizing that the probability of dropping is equivalent to one minus the probability of not dropping. And to not drop for five minutes we need to not drop for one minute, five times in a row. More generally, to not drop for m minutes, we need to not drop for one minute, m times in a row. If h is the probability of holding (i.e. not dropping) for one minute, and the probability of holding in any one minute is independent of any other minute (i.e. dropping is more or less random and not the result of fatigue), then the combined probability of m minutes is h^m. Thus if D is the probability that we’ll drop a rock any time in m, then we can figure out h, the probability that we can hold a rock for a minute, and from there, trivially, d, the probability that we’ll drop it in any one minute, for a given D and number of minutes m as shown here:

Plugging our ⅔ chance and 5 minutes into this formula we find that that a ⅔ chance of dropping over five minutes is equivalent to about a 20% chance of dropping in any single minute. If we want to find out the probability that we’ll drop a rock over a m minute trip, given d, we can use this formula:

Or perhaps more to the point we can solve this inequality:

to determine the relationship between d and m that determines when the total probability of dropping is greater than the ⅔ chance that makes it worthwhile to pair up rather than working separately:

With this formula we can see that if it only took us three minutes to climb the hill, we could live with up to a 31% chance of dropping per minute before pairing would make sense. But if it took us 20 minutes, then we’d do well to pair up even if every minute we had a 95% chance of keeping our grip.

So is developing software like carrying rocks? I’d argue that in many ways it is. Programming requires keeping a bunch of things in mind and if you lose your mental grip on any of them for a moment you either have to backtrack to re-figure out how things fit together or, worse yet, you proceed with a faulty understanding and introduce a bug which later requires a lot of time to track down and fix. In fact programming is in some ways worse than carrying rocks because the cost of a momentary slip of concentration can be much more than simply the equivalent of a rock rolling back to where you started. A bug that you create a few hours into a programming session may take many hours or even days to track down and fix. Luckily pairing can help there too — while one partner is focusing their mind on the next thing the other partner’s mind may linger for a moment and have a “Wait a sec’” moment that catches a bug before it gets too far away, the equivalent of catching a dropped rock before it rolls all the way back down the hill. Or when bugs do get in, a pair can often find them faster than a single programmer, much the way two people would be able to find a dropped rock if it didn’t just roll back to the bottom of the hill but bounced off in some random direction into thick weeds.

Last updated 2007-05-09T04:00:26Z.

Thursday, 26 April 2007, 21:50

A few weeks ago I was in the midst of reading Steve McConnell’s Software Estimation: Demystifying the Black Art when I had the conversation with my friend Marc that I have written about previously. Marc, as I mentioned before, is the founder of and Chief Product Officer at a small startup called Wesabe. When I told him what I was reading, he asked, “Do you believe it?”

“Sure,” I said. By which I meant nothing in the book had struck me as patently bogus. A lot of it is good sense about the limits of estimation, the relation of estimation to planning, and why estimation is so hard. Parts II and III of the book present specific estimation techniques that, assuming one had the relevant inputs and historical data, seem likely to be capable of producing fairly accurate estimates. Now, the descriptions of some of these techniques made me think — wow, if that’s what it takes to produce good estimates, no wonder we all muck along with crappy ones. But it did for the most part seem like the kind of thing we Serious Software Professionals™ should be doing.

Marc — it turns out — is far more skeptical about the whole enterprise of software estimation. He tells me that at Wesabe they never make schedule estimates. He manages his developers, as he has explained in a blog entry, by trying to get them excited enough about the features he thinks should be added to Wesabe that they decide to go ahead and add them. Or they may get excited about their own ideas and add them instead. Marc retains final authority over what gets added to the product and a Wesabe developer who consistently gets excited about developing things that Marc refuses to allow into the product should probably make sure their resume is up to date. But he never asks them for estimates. He does encourage his developers to spend most of their time working on things that they can finish quickly and get into the product, but that seems to be as much about what he thinks most likely to keep his developers happy as anything else. When they need to, Wesabe developers will tackle bigger projects, still without estimating how long they will take. Marc’s point of view, I take it, is that the only reason to do these big projects is because you have to, and if you have to, it doesn’t really matter how long it’s going to take.

Now, Marc’s a smart guy and he’s been managing software developers for as long as I’ve known him (more than a decade) and I know he thinks a lot about how he does what he does. On the other hand, Steve McConnell’s also a smart guy whose books I’ve been a big fan of for about as long. So, how to reconcile these two points of view? Is Marc’s approach only tenable in a startup? Or maybe McConnell’s approach to estimation is only worthwhile in big organizations, that are doing more or less the same thing over and over again.

So I returned to reading Software Estimation with a new question in mind: Why estimate? The nearest McConnell comes to an answer is in section 1.7 “Estimation’s Real Purpose”, where he gives this definition of a good estimate:

A good estimate is an estimate that provides a clear enough view of the project reality to allow the project leadership to make good decisions about how to control the project to hit its targets.

I think the key word in McConnell’s definition is “targets”. The reason Marc can get away with not estimating is because he’s found a way to manage without setting targets. So the question “Why estimate?” is better expressed as “Why set targets?”

Sometimes we set targets in order to convince others, or ourselves, that something can be done. We may set targets to inspire ourselves to do more, though it’s not clear that’s a winning move, and even less so when managers set a target to “inspire” the folks who work for them. (See DeMarco and Lister’s Peopleware, chapter 3 and the discussion of Spanish Theory managers.) We may also set targets to give ourselves a feeling of control over the future, illusory though that feeling may be. After the fact, a target hit or missed can tell us whether or not we did what we set out to do. However if we missed a target, we can’t know whether that’s because the target was unrealistic or because we didn’t perform as well as we should have. Setting and hitting targets does make it look like we know what we’re doing but we need to keep in mind that targets rarely encompass all the things we care about — it’s much easier to set a target date for delivering software than a target for how much users will love it.

If the goal is simply to develop as much software as we can per unit time, estimates (and thus targets), may be a bad idea. In chapter 4 of Peopleware, DeMarco and Lister discuss a study done in 1985 by researchers at the University of New South Wales. According to Peopleware the study analyzed 103 actual industrial programming projects and assigned each project a value on a “weighted metric of productivity”. They then compared the average productivity scores of projects grouped by how the projects’ estimates were arrived at. They found that, as folks had long suspected, that programmers are more productive when working against their own estimates as opposed to estimates created by their boss or even estimates created jointly with their boss, averaging 8.0 on the productivity metric for programmer-estimated projects vs 6.6 and 7.8 for boss-estimated and jointly-estimated. The study also found that on projects where estimates were made by third-party system analysts the average productivity was even higher, 9.5. This last result was a bit of a surprise, ruling out the theory that programmers are more productive when trying to meet their own own estimates because they have more vested in them. But the real surprise was that the highest average productivity, with a score of 12.0, was on those projects that didn’t estimate at all.

There is, however, one other reason to estimate: to coordinate our work with others. The marketing department would like the developers to estimate what features will be included in the next release so they can get to work writing promotional materials. Or one developer wants to know when a feature another developer is working on will be ready so he can plan his own work that depends on it. Note however, that in cases like this, estimates are really just a tool for communication. Marketing needs to know what’s going to end up in the release and the developers, by virtue of being the ones building it, have the information and somehow that information has to be communicated from the developers to the marketers. But there are lots of ways that could happen. In a small company it might happen for free — everyone knows what everyone else is working on so the marketers will have as good an idea as anyone what’s actually going to be ready for the release. If water-cooler conversations are insufficient, then marketing and development could meet to talk about it on a regular basis. Or the developers could maintain an internal wiki about what’s going on in development. Some of these methods may work better than others but it’s not a given that using estimates is always the best way.

To decide whether estimates are a good way to communicate, we need a way to compare different methods of communication. I’d argue that all methods of communication can be modeled, for our present purposes, with the following five parameters:

• Preparation time required
• Communication time required
• Rate at which information is conveyed
• Rate at which misinformation is conveyed
• Other benefits and costs

The idea is that communication happens in this pattern: one or more people spend some amount of time preparing to communicate. This would include activities such as thinking, writing, estimating, etc. Then the communication proper happens, which takes some time. This may require time from both the sender and the receiver (conversations, meetings, presentations, etc.) or only the receiver (reading an email, looking at a web site).

After the communication is complete, some amount of information has been conveyed and also, sadly, some amount of misinformation. The misinformation may arise from faulty preparation, from misstatements by the sender, or from misunderstandings by the receiver. Obviously different methods of communication will be able to convey more or less information in a given amount of time and will be more or less prone to miscommunication.

Finally, different methods of communication can have benefits beyond the information conveyed and costs other than the time spent and the misinformation conveyed. For instance, chatting around the water-cooler may build team cohesion while highly contentious meetings may have the opposite effect. Another important kind of benefit is that the preparation and communication phases may itself improve the communicators’ understanding of what they are communicating about. For example, writing clearly on any topic invariably forces the writer to clarify their own thoughts and the give and take in a well-run meeting can help bring out and refine what were previously only amorphous notions.

Now we can look at estimation as a communication tool according to this model. The preparation time for good estimates is fairly high. This is why so many developers try to avoid estimating or give it short shrift. The communication time is low — since an estimate distills things to the essence of “these features by this date” or “this much effort for that much functionality” it can be quickly conveyed. However, exactly because estimates do distill things, they are a low bandwidth form of communication. Without a lot of other communication about what exactly the features are going to look like, a list of features and the dates when they will be done, leaves out a lot. Worse yet, estimates are notoriously prone to conveying misinformation, either because the estimate is inaccurate or because an accurate estimate is misunderstood. McConnell devotes a whole chapter, “Estimate Presentation Styles”, to discussing how to present estimates so they will be less likely to be misunderstood, but no presentation style can help the poor estimator who’s giving an estimate to an audience that hears “2 to 4 weeks” as “2 weeks”.

When it comes to other costs and benefits, it’s hard to say. If we assume for the sake of argument that it is possible to make good estimates, does the very act of preparing the estimate have its own benefits? Certainly, making a good estimate requires identifying all the work to be done, so a team that has gone through the exercise of estimating may identify all the work that actually needs to be done sooner than a team that hasn’t, which may allow the work to be done more efficiently. Another potential consideration is that in a company where non-developers expect developers to be able to provide reliable estimates, then meeting those expectations has the social benefit of giving the rest of the company confidence in their development team.

On the other hand, there’s no guarantee of obtaining those benefits. Estimating badly certainly has bad ancillary costs. Leaving aside the consequences in terms of the misinformation it generates, it’s just no fun to estimate when you know your estimates are bad. Either you feel guilty because you believe it’s possible to do well and that you should be better at it or you think it’s actually impossible and therefore whoever is asking you for the estimate is wasting your time. And if good estimates can increase the company’s confidence in their developers, bad estimating can destroy it. Or, worse yet, developers can be unfairly deemed unreliable because the rest of the company expects more precision in estimation than is actually possible. According to McConnell, at the beginning of a project even the best estimators can be up to 4x off in either direction. Which means unless folks can accept an initial estimate of “from three months to four years” and a promise to refine it over time, developers are bound to disappoint.

So we have: High preparation cost. Low communication costs. Low bandwidth with high potential for misinformation. Ambiguous ancillary costs and benefits. Clearly, if you’re going to estimate, you’d better do it well and accept the costs that entails. But if you doubt it’s possible to do well, either in principle, or in your particular situation, then it can be useful to think about other ways to make sure the necessary communication happens.

Last updated 2007-04-27T20:34:57Z.

Thursday, 12 April 2007, 20:02

The other day I was talking to my friend Marc about software development. Marc is founder and Chief Product Officer of Wesabe a startup that, “makes it easy to better understand how you spend your money and links you to a community of people dedicated to helping each other make better financial decisions.” Wesabe is one of these new-fangled companies whose whole product is their web site, which they update, according to Marc, every day. The frequency of their updates came up when I was using version control as an example of something that developers need to understand in order to work well in groups — not just the mechanics of a particular version control system, but the how and why of things like release branches. Marc suggested that my thinking was way out of date — that because everyone is delivering their software on the web these days, release branches were an anachronism. He’s certainly right that if you release your software by updating your own web site then there’s no reason to make release branches — the point of release branches is to allow you to go back and fix bugs in old releases and if you’re delivering your software as a service there’s only one release and no direction to go but forward.

My question is: what percentage of software startups these days are planning to deliver their software as a service vs “traditional” delivery of software to be run by customers? Is Marc really right that the software-as-service model is so prevalent now that a book aimed at developers who want to learn the things they need to know to be productive members of a programming team can ignore older styles of delivering software?

Another question is, even if Marc is right that everyone is delivering their software as a service, to what extent they will be able to avoid having to support older versions of their software? It’s one thing for small companies, still in early days of fleshing out their feature sets, to continually upgrade and force their users to come along for the ride. But as they get more users and a more mature feature set, I suspect they will discover that not all their users will be willing to accept upgrades willy nilly. Presumably this will be less of an issue for companies whose real value is the social network of the users as opposed to the value being in the raw functionality of the software itself. In the former case the whole point is to have exactly one instance of the software, shared by all users while in the latter case, if the users are depending on particular functionality, they may not want it to change, even if the change is ostensibly an “upgrade”.

Have an opinion? Let me know.

Update: My former boss Bob Pasker, now a Venture Advisor and CTO-in-Residence at Azure Capital Partners, gives me this breakdown of the software delivery mechanism of companies he’s seen recently in his work at Azure:

• 50% delivered over the web, either as HTML or flash
• 25% require an install, either on server or on desktop
• 25% other (mobile, email, hardware, or both)

Last updated 2007-04-13T05:18:29Z.

Tuesday, 10 April 2007, 23:25

In my first entry on this blog I mentioned the Gigamonkey Four-Step Algorithm for Writing a Book. In the past week, I’ve twice had occasion to explain this algorithm to someone. Rather than wait for a third occasion, I figured I’d write it down. Here it is.

Step 1 Write the index. Write down, in no particular order, every word, phrase, name, and concept that you would expect to appear in a well-prepared index of your book, assuming the darn thing was already written.

Step 2 Write a hierarchical outline. This outline should contain, somewhere, all the items from your index. Follow the rule from junior-high essay writing that you can only create a sub-topic under a topic if it will have at least one sibling. But don’t worry if your outline gets ridiculously deeply nested. If you’re a forest-then-trees kind of person you can build your outline top down — define the top-level topics you want to talk about, then split those topics into sub-topics and those sub-topics into sub-sub-topics, and so on until you get down to a granularity where you can start inserting items from the index. Or you can work bottom-up — start directly from your index, lumping related items together, then lumping the lumps into bigger lumps. I found that alternating between top-down and bottom-up worked best for me.

Step 3 Write a a flattened outline. Books are a linear medium. While it can be immensely useful to organize your thoughts hierarchically, you are ultimately going to have to guide your reader from A to Z along some linear path. In his essay The Cognitive Style of PowerPoint Edward Tufte observes:

People have communicated about complex matters for centuries without hierarchical bullet lists. Richard Feynman wrote about much of physics — mechanics, optics, thermodynamics, quantum behavior — in a 600-page book with only 2 levels: chapters and headings within chapters. (Emphasis in original.)

Following Feynman’s lead, in this step you should produce on outline with no more than three-levels: chapters, sections, and paragraphs. Each paragraph-level outline item should be — again shades of junior high — a topic sentence for that paragraph. This will let you judge whether it’s actually going to be possible to move from paragraph to paragraph in a reasonable way — can you imagine a transition that’s going to get you from the topic of one paragraph to the topic of the next?

Note that there’s more to this step than simply flattening the outline from step two. The step two outline is a taxonomy of all the things you want to talk about, but it’s unlikely your readers are ready to digest a whole taxonomy in one go. You’ll likely find that in order to provide a comprehensible linear order for your reader, you’ll need to split certain taxonomic topics across different chapters or sections. For example, when writing Practical Common Lisp, my hierarchical outline contained top-level sections on functions, variables, and macros, three of the main elements of Common Lisp. Initially I thought each of these top-level categories would map to a chapter. However, it turned out it was impossible to write about all aspects of functions without at least some discussion of variables. Yet, there were also aspects of variables that could only be meaningfully discussed after discussing functions. And macros were similarly intertwined with the other two topics. Eventually I figured out that what I needed to do was to write a chapter in which I would discuss the basics — but just the basics — of functions, variables, and macros before moving onto dedicated chapters for each topic. It also turned out that it was useful to split my discussion of macros into two chapters. Thus the three top-level topics from my hierarchical outline had to be split: part of each became a section of chapter four and the remainder became the root of their own chapters except for the macros section which was split into two chapters. In other words, this is the step where you think about the structure of your book, whether you can march straight from A to Z or whether you need to occasionally circle back to clarify things that you couldn’t give full justice to the first time you mentioned them.

Step 4 Write the book. In theory, all that remains is to work through your step three outline, fleshing out each paragraph-level topic sentence with the full paragraph and providing a transition to the next paragraph. In practice, you’ll probably have to frequently pop back to step three if not all the way back to steps one and two. During the actual writing of Practical Common Lisp, I was constantly discovering things that, had I explained them already, would make whatever paragraph, section, or chapter I was currently working on completely straight-forward to write. But then I’d have to figure out how to work those things in somewhere earlier. And often when I actually tried to write those earlier sections, I’d discover some other concepts that I’d really need to introduce before I could write them. I’d know I was having a particularly bad day when I’d discover that I’d looped — that the thing I needed to discuss first, in order to be able to explain everything else I had pushed on the stack, was the very thing I had started with before pushing all these other things on the stack. At that point I’d have to go back to step three and re-outline the relevant sections in light of my new understanding of how things actually fit together. Maybe I could have avoided some of those bad days by better up-front outlining but I suspect not; it was only by actually getting down to the nitty gritty of writing paragraphs and wrestling with how to explain each thing that I could discover the flaws in my more general plan of attack. It was painful at times but I don’t think there was any way to avoid it.

Step four is also obviously the step when you deal with crafting your deathless prose. I don’t have a lot to say about that as good prose style seems more a matter for heuristics than algorithms. That said, if you’ve actually organized your material into some rational, linear order, and you can manage to say what you mean at the sentence and paragraph level, you’ll be doing a lot better than a lot of authors. I did find it useful to print my chapters on actual paper in a reasonable-size font with nice wide margins and double spacing between the lines and to sit down to edit them, away from the computer, with a red pen in hand.

But what do I know? I’ve only written one book. Check back in a few years when I’ve finished my next book and I’ll probably have a whole new algorithm. But for now, as far as I know, that’s how to write a book.

Last updated 2007-04-10T23:25:23Z.

Tuesday, 10 April 2007, 19:01

Hello. Welcome to my blog. If you know who I am you’re probably either my mother or have read my book Practical Common Lisp. Or possibly you’ve either seen the video of a talk I gave at Google’s New York office last May or heard the IT Conversations interview I did with Phil Windley of Technometria. Any of which — except in the case where you’re my mother — might lead you to expect that this blog is going to be about Lisp. Which, though I’ll probably have occasional things to say about Lisp, it mostly won’t. But I’m hoping you might stick around anyway. (Yeah, Mom, I know you will.)

So what can you expect if you do? You’ll get to see me exploring ideas that I hope will eventually turn into another book. Since Practical Common Lisp came out in April 2005, I’ve been consulting — mostly Lisp work, some not. I wound all that down last September in anticipation of the arrival of a baby, my daughter Amelia. Now, much as my wife’s memories of the pain of childbirth are fading, my own memories of the pain of writing a book have faded from visceral to merely intellectual. I remember saying, “I’m never ever going to do this again,” but I can no longer summon up the feeling that led me to say it. Which must mean it’s time to write another one.

At a recent get together of the Bay Area Lispniks I asked the folks at my table what computer-related book they felt was missing. One fellow suggested a book on the programming language D. Hmmmm, I think I’ve already done my bit trying to promote a deserving but unappreciated language to a wider audience. (And I don’t really know enough about D to say whether it’s really deserving. Though any language that starts from the premise that C++ needs to replaced is at least starting out on the right foot.) Another guy suggested a book on “programming for the data center” — i.e. how to write programs that are going to run on a Google-style room full of computers. That sounded a bit more promising but not something that I have any particular expertise in; someone at Google should write it. Then, speaking of Google, Peter Norvig suggested a book about how to program in groups. By which he meant a book that explains all the stuff that a programmer needs to know to be an effective member of a team; the stuff that a lone-hacker, either self-taught or fresh out of a comp sci program, typically has to learn on the job. Norvig claimed — and I tend to agree, based on my own bookshelf — that there isn’t really a good book on that topic. There are books about how to be a better programmer, as an individual. And there are books about project management from which an enterprising developer could back out some lessons about how to be an effective contributor to a team. But nothing that I’ve seen aimed at helping an individual developer develop the skills that are specific to working in the context of a development team.

This suggestion immediately pricked my interest: I’ve always been interested in the question of “what is the best way to develop software?” One way to look at my interest in Lisp is that it is part of an answer to this bigger question. But as much as I love Lisp, I recognize it’s not a complete answer — most software shops would probably get more bang for their buck from adopting techniques that allow groups of people to work together most effectively than they would from switching to a better programming language, even Lisp. So now I’m excited to start writing about that.

My basic plan is to talk to everyone I know who’s involved in developing software and anyone they’ll introduce me to and to read or reread anything and everything relevant I can get my hands on. Then I’ll be ready to apply the Gigamonkey Four-Step Algorithm for Writing a Book that I developed while writing Practical Common Lisp. If you have suggestions of people I should talk to or things I should read please drop me a note. Since all that talking and reading will take a while, it’ll probably be a while before I can get down to the writing phase proper. In the meantime I plan to use this blog as a place to think out loud and to try to keep up my writing chops. And if the stuff I write here helps either to drum up advance interest in the book or to attract folks interested in hiring me as a consultant, that’d be fine too. (Unlike when I wrote Practical Common Lisp, when I spent two years without working other than on the book, this time around I plan to try to combine writing with consulting, both because I’ve got a new mouth to feed and because the right kind of consulting gigs will actually give me more material for the book. See the section “Gigamonkeys Consulting” in the sidebar if you’re interested in hiring me.)

Last updated 2007-04-10T23:25:23Z.