Glenn Vanderburg on Metaprogramming

After having finished my second major deployment in as many weeks (more on that later), I'm finally catching up on my reading (and listening) around the net. Today I heard Glenn Vanderburg on the Rails podcast, and early in the interview he succinctly states what it is that makes programming in Ruby in general, and Rails in particular, so productive. I'll transcribe the relevant bit here, for those of you who don't want to listen to the whole interview:

Pretty high on the list of things that every programmer should read is this old paper from the 70's by Fred Brooks, called No Silver Bullet. In that paper, Brooks makes the claim that we're very unlikely to see any one new development in software that yields a consistent 10-fold productivity increase.
[...]
I think the jury's still out on that. But the really interesting part of the paper to me is Brook's supporting argument, and it's really unfortunate that all the debates on silver bullets over the years have overshadowed that.

He points out that in every peice of software there are two kinds of complexity. There's the "essential complexity" that's inherent in the problem you're trying to solve, and it's really impossible to imagine any solution to the problem that wouldn't be at least that complex, just because that's what the problem involves. But then there's what Brooks calls the "accidental complexity" (I prefer the word "incidental") that's required to adapt your general-purpose computing platform and language and so forth so that it's suitable for solving that problem. Things like infrastructure libraries, database interactions, data structure manipulation, and other things like that all constitute that incidental complexity. They really have nothing to do with what a domain-expert would describe to you when they're trying to solve a problem.

In most real-world systems, that incidental stuff is all sort of mixed up with the essential stuff. You'll have faily low-level programming mechanics wrapped up, in, and around the essential logic that's a part of the business problem. I think that actually introduces a third kind of complexity, because the dissonance between the essential and the incidental complexity makes it hard to tell which is which, for one thing, and it's also difficult to understand one in the context of the other.

So all that incidental complexity, plus the the dissonance from having the two things mixed together like that, has a lot to do with why many systems are so much more difficult to maintain than they were to develop in the first place.

So that's the problem, and it's been a problem for software forever. The best designed systems have the two kinds of complexity well-separated, and domain-specific languages are the best way I know of to really draw that boundary and really keep them separated. [...] One of the greatest strengths of Ruby is its fantastic facilities for extending the language with domain-specific constructs like that.

I was familiar with No Silver Bullet and the incidental complexity argument, but Glenn's observation of the dissonance between code that addresses the essential complexity and code that addresses the incidental complexity was new to me. It made me realize why I've found Ruby on Rails to be so much more sustainably productive than the Perl, Java, and even Python platforms I'd done so much work on in the past.

Rails provides a suite of great domain-specific constructs for building database-driven web applications, but then it goes one step further and encourages you to extend the language for the specific domain you're working in. Which means that by using Ruby on Rails you win twice. First, the incidental complexity is reduced because Rails covers a lot of it for you. Second, the dissonance between the essential complexity and incidental complexity is kept to a minimum by the ease with which you can separate the two. The resulting improvement in maintainability is what's encouraged me to use Rails for every new application I've built for over a year now.

I won't go so far as to suggest that Ruby is the best bet for all programming tasks, or even that Rails is the best platform for all websites, but I have observed that Ruby on Rails makes a great platform for a surprisingly large variety of web development projects.