tin-foil-hats
Tin Foil Hats and Rubber Ducks
It is with the deepest regret that I must announce that my blog will be hopelessly boring from now on. (Well, depending on how you felt about it already, you could say it'll simply continue to be boring.) Either way, allow me to set your expectations as low as they can possibly go.
Most of my blogging in the past year happened because I did something a bit odd, and it caused me to go temporarily insane. Made some of my blogs mildly interesting, possibly. That, and the wine. That helped too. But mostly my blogging was just my account of me chasing my own tail, desperately trying to make sense of a difficult, self-imposed puzzle.
What was the puzzle, how did I go crazy, and why are my blogs going to be unutterably boring from now on? Allow me to tell you in the most boring way imaginable!
Pretend for the moment that your boss has asked you to write down a list of all the factors that contribute to being able to develop large-scale, high-quality, robust software systems. (This didn't actually happen, but I figured I could overtly manipulate you by bringing your boss into it.)
If you think at all like I do, you'd list a whole bunch of stuff: skilled engineers, quality tools, a good operating-system and language platform, good 3rd-party databases/webservers/middleware/etc., a high-quality recruiting program, good facilities and computing equipment, and so on.
We do this kind of thing, at least mentally, all the time. Some of the new dev-centers are making checklists like that explicitly, in fact.
Well, here's the strange, insanity-inducing twist I added. I started with that semi-hypothetical list, and I jammed my thumb down hard on two factors that we don't normally think of as "constraints" — not really even soft constraints. One was "number of engineers", and one was "lines of code" (or more precisely, "code weight", which can be loosely approximated by lines of code, although tokens or expressions might be a better approximation.)
And then some really weird stuff happened.
Growing (like) Crazy
Normally, for these two software engineering factors (SDEs and "source lines of code", or SLOC for short), we assume both values tend towards infinity. It's axiomatic. If you want more features, you add more engineers, and they'll add more lines of code. These assumptions are built into our software cultural consciousness, industry-wide.
As far as engineers go: more is always considered better. There are a few constraints imposed by finance, but we don't often bump up against them; being able to hire good people is usually the first stumbling block you encounter. Sometimes it's the only constraint. You have plenty of work in your queue, and open seats, and headcount, but those dang Bar Raisers keep getting in your way. The experienced hiring managers know all the tricks for getting around that little problem, but even so, hiring people is usually the hardest part of expanding your org.
Generally speaking, you don't want any constraints on the number of engineers you can hire. That's why so many companies are offshoring their development. They can get more developer for their buck. Offshore, you can buy bulk engineer by the six-pack, or by the case, but a full keg of engineer is definitely your best value.
In the productivity game, what you really want is to remove all obstacles to obtaining more engineers. For instance, you can do what Google does, and give each manager 200 direct reports, rather than the classic 7 or 8, so your management overhead grows with the log base-200 of the number of engineers, rather than the log base-8. That's the kind of clever thinking that comes from hiring math-y Ph.D.s, you know.
You can also use open-source software (e.g. Subversion instead of Perforce), thereby eliminating the "tax on expansion" imposed by per-developer software licenses. You can make cubes smaller, temporarily eliminating facilities as the hard constraint on physical expansion. Then you set up massive recruiting pipelines, you start up your training programs, and you direct all your efforts towards getting more SDEs in the door.
Because more engineers is always better. That's how you get more stuff done.
And as for source lines of code, well, hell, we don't even look at that. It's not in anyone's metrics deck; it's at best a matter of passing interest in hallway conversations. "Did you hear we added 9 million more lines in Q4?" "Naw, weird." "Yeah. Hey, how's your new service going?" "Bad." "Yeah, mine too. We need more people."
When we try to improve productivity, we're doing a balancing act. We let all the constraints slither around on the paper, attempting to find a harmonic balance that optimizes for maximal expansion. ("If we go with 5 engineers per cube, we'll need to add more air-holes to the building exterior, which would be costly. Maybe we can hire smaller SDEs...") Slither, slither.
That's what we do. You know I'm right, too.
What I did was take a hammer firmly in hand, and pound two large spikes into the constraints list, permanently fixing #SDEs and #SLOC to small, hardwired values. You can grow features all you want, but not your systems or staff. I wanted to find a solution to the simultaneous equations in the context of (nearly) zero growth.
And over the next few months, I went insane.
I actually said earlier that I went "temporarily insane", but what really happened was more like a transition from an amateur insane person, unfamiliar with the ins, outs, and social consequences of being a total nut case, to a seasoned pro, wearing a rubber ducky and tin foil on my head.
It was the transition that made my blogs interesting, I think. To me, anyway.
Eventually, the constraints stopped slithering, and I stood there with my inflatable ducky life preserver around my trenchcoat, and my shiny tinfoil sailboat on my head, staring at where the constraints had landed. And I became — sort of content, although tinged with a lot of forward-looking enthusiasm, because now I know exactly where I'm headed. No more tortured soul, and thus, no more interesting blog rants.
Sorry.
For what it's worth, I am a much happier person now than I was during my frantic search of that oddball constraint space. I just can't talk to most programmers anymore, because I now officially make them Really Uncomfortable, and they Look At Me Funny.
Thankfully, some of my good friends at Amazon happened to go nuts in the same way at the same time, so I'm not totally alone in the Tinfoil Society. They all saw the martian radio waves too, and they made similar-looking hats.
And, as it happens, we have quite a few brilliant engineers wandering the halls of Amazon who actually wear tinfoil undershirts, but keep them well-hidden, so as not to be perceived as insane by the general population. Never fear; I will protect their identities.
Martian Radio Waves
So how did a perfectly normal, ordinary, garden-variety sane person like me go crazy? And what exactly is the tinfoil hat for?
If you've ever read one of my blogs, you already know the hat is Lisp, and the rubber ducky is Emacs. And because in all likelihood you're not crazy, you felt a minor wave of revulsion when you read that last sentence. Sort of like how we felt when we accidentally went out the back exit in the old Columbia Building downtown. You old-timers know what I'm talking about.
How did I wind up rejecting polite society and succumbing to the Dark Side?
Well, if you constrain #SDEs to zero, and #SLOC to "manageable", and (incidentally) you also constrain the number of mutually incompatible, barely-interoperable languages, frameworks, platforms, technologies and protocols to some value smaller than the 200 or so we use at Amazon — let's call it "under 10 total", for sake of argument — then you need to find something else, something that you didn't previously think of as a variable, and you need to make it variable.
I never said why I wanted to constrain #SDEs or #SLOC. I'm not going to try to justify it, or convince you that they're good things to constrain. Too much effort. In fact, I doubt I could convince you. Let's just chalk it up to brain damage — mine. I believe, with deep-seated and un-shakeable zeal, that any large company (Yahoo, say, for sake of example) has a system that is 100 times larger than it needs to be, and they have 100 times more engineers than they need, and they've spent 5 to 10 times as long building it as they had to.
Put another way, I started to believe that SDEs can be (or at least, should be) orders of magnitude more productive.
I just didn't know how. So I set out to see how it could be done at a micro-level, fixing #SDEs to 1. Me.
I tried typing faster. But I already type pretty fast.
I tried automated complexity-managing tools like ANTLR, GCC/XML, etc. They're fun, sure, but they don't really help much. They just tell you how messed up your system is. I already knew that.
I tried working a lot of hours. There are only so many hours available, though. Time is a constraint you can't mess with much.
I tried agile development practices, design patterns, and other latest-and-greatest, state-of-the-art, whiz-bang modern software engineering techniques. They're nice, but it turns out that none of them constrains #SDEs or #SLOC. You may have smaller teams, but you can have as many small teams as you want. And refactoring makes your code base larger, or so says Fowler — and he's right. So none of these things helped at all with my strange new constraints.
I hit the textbooks, and finally learned all the things I was supposed to have learned in my comp sci degree. And that's when I started noticing certain new possibilities, and first felt the martian radio waves being emitted from, oddly enough, Sun Microsystems, IBM, and a few other big companies. These radio waves had been messing with my head, telling me that I liked coffee a lot. That's when I started wearing the tin foil.
You Can't Sell Lisp
I'm not going to try to sell you on Lisp. It can't be done. You have to find it yourself, and when you do, many of your friends will become rather... distant, as if you'd inexplicably stopped bathing.
I'll mention in passing that Lisp looks really weird, sort of like you smacked cartoon source-code with a two-by-four, and now it's surrounded by vibrations. You can almost hear the ringing sound. But eventually it starts to look like big, juicy bunches of grapes on a lovely old vine. Big, ripe, colorful, bunches of grapes, for crafting the finest wines. It takes a while, but it does become very appealing to look at. You get that nice uplift, a quickening of your pulse, when you come back to Lisp after staring at more complex syntaxes for long enough.
That uplift is to some extent true of any language you're highly familiar with, I suppose — but I still prefer looking at some alphabets more than others. Not all written (and spoken) natural languages really are equally melodious.
The comparison to sounds is apt: I've come to realize that Lisp is the classical music of programming languages. I went more or less directly from heavy metal to classical music as a teenager, and I've been listening almost exclusively to classical music for close to 20 years. So I sometimes forget that there are many other kinds of music, and I'm in a relative minority. Personal taste plays a major role, and you can't explain to someone how nice classical music is. They'll either listen to it enough to start liking it on their own, or they never will.
Classical music fans are often kind of snobby, come to think of it. I don't think I am, though, so perhaps I can attempt a non-snobby approach to Lisp. Let me put it this way: I didn't expect the answer to be Lisp, I didn't want it to be Lisp, and I probably would have been (and might still be) happier if it had turned out to be, say, Ruby or Haskell or something.
That's in no small part because Lisp is old. We tend to believe that only new things are valuable. New books are more interesting than old books, right? I've only started actually reading the classics (e.g. Aristotle, Hobbes, Dickens, Dumas, any dead person, really) in the past few years, and they're wonderful. But I never thought so before. I had to read a lot of sci-fi, fantasy, horror, and other stuff before it started to have a "sameness" to it that bored me (with rare exceptions that themselves will, I think, become classics in time).
New music, new books, new ideas, and above all, new technologies. We think everything has to be new. Look at almost any old language: Fortran, Ada, Pascal, Awk — they all suck, don't they? Old operating systems suck, old software applications suck. Brings to mind floppy drives, VGA displays, all that good old stuff. So of course we intuitively believe that Lisp, being even older, must also suck. And it must be highly impractical, too (a nice reassuring thought, on the off chance that it doesn't, in fact, suck as bad as Pascal. You're still exempt from needing to bother about it.)
Old algorithms don't suck, unless perhaps you count Bubble Sort. Generally, the more tried-and-true an algorithm or data structure is (DFS, BFS, quicksort, binary search, hashing, etc.), the more confidence you have in it. Ideas are fundamental and timeless, but technologies are always replaced.
Lisp straddles the line between a fundamental idea and a technology — which is another reason it took me so long to settle on it. And in many ways I'm still unhappy with it. But I believe it's the best thing out there, and will continue to be so for (at least) my lifetime. (My bet on Emacs fifteen years ago is still paying off.) So I'm going to wear my inflatable duck raft with pride from now on, which I'm sure will bore any non-duck, non-tinfoil wearers to tears.
Again: sorry. Really. But I'm getting old and impatient, so I'm going to stop trying to drag my old Java/Perl/C++ comrades-in-arms along with me. I'm stepping into a new, rubber-duck-filled world, and it's pretty big (almost overwhelmingly so), so it's going to keep me busy enough. I'll be mostly blogging about functional languages, and primarily about Lisp, since even though I wanted one of the newer functional languages (e.g. OCaml, Haskell, SML) to work out for production work, they're just not there yet, and I don't have time to waste waiting around for them.
Plus Lisp people are never jealous of other languages, because Lisp easily incorporates any features someone else cleverly invents. The only real reason to keep an eye on Haskell is so you can apply its ideas directly in your Lisp code.
It's not that I'm going to stop writing blogs. I'll certainly have a lot to write about. Lisp is really, really huge. And, sadly, it's not anywhere near as well-packaged as Java — although, to be sure, Java is getting pretty darn big, too, and I may have forgotten how overwhelming Java is to people who are just coming up to speed on it. Not the language, mind you, but the tools, technologies, resources, gotchas, best practices, etc. Sure, it's all documented, but you still have to go read it all. Same goes for XML. It's probably the case for anything that's truly production-quality.
So I've got my work cut out for me. Or should I say, we have our work cut out for us?
Calling All Foil-Wearers
Yesterday I passed an Amazon engineer in the hall — a person I'm acquainted with, but don't know his background. You can just tell he's smart, and he's got a reputation for being really solid. On a whim, I asked him: "Are you a Lisp person or a Smalltalk person?" He stared at me in momentary surprise, and said: "Lisp. How did you know to even ask the question?"
I told him that I knew he was smart, and that most of the smart people I know turn out to be one or the other. He proceeded to tell me about his first job, developing an expert system in Lisp, and he said it was still his favorite language.
He's not on our lisp-hackers mailing list, and he doesn't talk about his Lisp background. He just wears his tin foil and quietly goes about doing his regular job (extremely well, or so I've heard), which has nothing to do with Lisp.
There are a lot of folks like him here. It's odd, but ever since I got out of school — heck, even back when I was in school — I've noticed that a seemingly disproportionate number of the truly amazing gurus are Lisp folks. So if you set your net to catch a bunch of smart fish, it's a sure bet that some of them will be closet Lisp hackers.
Anyway, I'm going to dial up the traffic on our lisp list a bit, as I try to come up to speed on how this crazy Common Lisp thing works. And I'll keep a running document, some in my blog, of what Common Lisp looks like to an ex-Java programmer.
Yep. Boring.
For those of you who are now officially departing the crazy train ride of "Stevey Unwittingly and Unwillingly Finding Lisp", it's been nice. Thank you, please come again!
For those of you who have kept your Loony Badge carefully wrapped in tin foil during your stay here, feel free to come say hi sometime. I get the distinct impression there's a revolution about to happen. Could be fun.
(Published March 11, 2005)
Anyhow ...
I'll probably keep following along the ride. I finally committed to Emacs a couple of weeks ago, even going so far as to make "vim" an alias for "xemacs -nw" in my rc files. That way I don't have to worry about my finger habits betraying me ;-)
Ruby is doing a lot to satisfy my conventional OO tastes, but the other side of my brain has been wandering into Rebol a whole heck of a lot. Funny thing is, Rebol is starting to look like Lisp with some cool macros added. In fact, everything is starting to look like Lisp. Maybe it's something in the coffee. I am starting to understand the idea that Lisp is part technology and part ... well ... idea.
I've had my tinfoil for a while, but I didn't know it was a hat until now. Apparently, I had been wearing as underwear.
Posted by: Brian W. at March 11, 2005 02:19 AM
Ha, terrific post!
>> I've come to realize that Lisp is the classical music of programming languages.
This is a great metaphor, IMO.
Hey, here's a cool-looking lisp that I stumbled across. I've only glanced at the docs but it looks clean, small (20KB footprint) but surprisingly featureful. It has an interesting macro system too. And even TCP/IP support :)
http://www.oliotalo.fi/hedgehog/
I know it's not appropriate for our purposes, but the nice thing about lisp dialects is that if you see a neat feature you like, you can almost certainly implement it yourself in the lisp you do intend to use.
Chris
Posted by: Chris N. at March 11, 2005 06:44 PM
Also have you seen this?
http://dirkgerrits.com/erlisp/
Very early stages yet, but could end up as the best of both worlds.
I loved the 'Concurrent Programming In Erlang' free-download book. The conceptual model, the language, and the runtime environment all sound wonderful. The book itself is one of the simplest, clearest descriptions of a language I've ever read. But in a production system, I think you may need to be able to alter or enhance the runtime (which of course the authors can, as they also build the runtime...).
Chris
Posted by: Chris N. at March 11, 2005 08:56 PM
Hrmm, apparently I've been Outside the Asylum for quite some time, then. I wasn't aware that the common perception was that more programmers and more SLOC were good. I just assumed that more SLOC was a result that was tolerated (but that I tried to explain didn't have to be), and more engineers was a result of there being so many SLOC (one of those cycle things that's vicious).
I guess I've been fighting the wrong fight. Here I am trying to show that you can get rid of SLOC and programmers, but I'm showing it to people who don't even see why you would want to in the first place.
Count me in for the revolution.
/me sharpens his parentheses.
Posted by: Greg P. at March 14, 2005 11:01 PM
Chris,
If you want erlang style concurrency in a lisp dialect you need to check out Gambit Scheme. Erlang has actually been implemented on top of it in the past.
Posted by: Eric M. at March 15, 2005 02:17 AM
heh you got to love the 'i wont try to sell you lisp' rhetoric :), yet you haven't failed to mention it in all your blogs (I dont mind, but it's just so ironic that you'd say that :).
who has done performance studies that measure productivity between fluent Java (or any other language, let's say) and Lisp programmers? What were the gains by using one over the other for the same type of task? Just curious. Lisp has my interest, and even though I'm not apt at it, I'd be willing to learn it if I can be convinced it has something in it for me that will allow me to stay lazy - or lazier than I am. :).
Posted by: John Doe at March 31, 2005 11:26 AM