The Pragmatic Programmers Interviewby chromatic
06/24/2004 Andy Hunt and Dave Thomas are The Pragmatic Programmers, two experienced and intelligent software developers with impressive experience, including the authoring of the popular The Pragmatic Programmer and the well-regarded Programming Ruby. Recently, they launched their own small publishing company to produce books on agile and pragmatic software development. Andy and Dave recently agreed to an interview with the O'Reilly Network.
http://www.onlamp.com/pub/a/onlamp/2004/06/24/pragmatic_programmers.html
O'Reilly Network: In the five years since the release of The Pragmatic Programmer, it's had ten printings in English, with translations in five languages. Since then, agile programming techniques have started to see mainstream acceptance, especially in areas such as test-driven development and refactoring. What's the next step to improving software development?
Andy Hunt: I think the most important thing we need to do next is cut the clutter. Simple applications should be simple to write, easy to install and maintain. That used to be the case, but the current state of the world with web applications, J2EE, and so on is quite a bit more complicated than it needs to be.
Dave Thomas: Interestingly, we're seeing definite signs of pushback against the complexity. The "I'm mad as hell and I'm not going to take it any more" meme is starting to spread among developers, but it's a slow business. To some extent, the reluctance to change is only natural -- after all, folks have spent years learning the fine arcane details of J2EE and all the associated frameworks, and now they're being told that all that detail is too much. But at the same time, they're seeing the benefits of using tools such as Spring, Hibernate, Pico, and so on. We're even seeing folks abandon the J2EE world entirely in favor of lighter-weight solutions when appropriate. This wouldn't have happened two years ago, where a "Hello, World!" web app still needed 17 deployment descriptors to install.
Andy: That said, the whole environment is still much more complicated than necessary, and that's costing us all dearly.
In order to improve software development, and keep our jobs, we need to provide greater value to the organizations that pay us. Specializing in arcane bits of technology is not as useful as it used to be, and anyone can do that. So we need to move beyond worrying about CGI parameters and think more about business value, usability, and so on.
Dave: Think about the kind of work that can be effectively outsourced (where "effectively" is used in the context of some manager's opinion). Can they ship stuff offshore that can be specified down to some fine level of detail? Yup! Can they send repetitive, rule-based, highly constrained stuff overseas? You bet! The stuff that will stay is the stuff that involves more intuition, and more interaction. To get job security, developers need to position themselves as highly effective business-value generators, working with the rest of the company to solve common goals. If you sit in your cube waiting for a spec to be thrown over the wall, then you may be in for a wait -- that spec might be in an envelope on its way to Bangalore.
ORN: Ward Cunningham once told me "With better IDEs, static languages such as Java are almost as easy to program in as dynamic languages." You're both fans of the Ruby language, especially when compared to more static languages. Do you think that the potential shift to writing business applications in slightly more dynamic languages such as Java and C# (slightly in the sense that they're still rather static, dynamic in the sense that they handle memory management for you and have improved reflection and introspection) is good for programmer productivity and ease, or do those languages not go far enough? If not, how much further should they go?
Andy: I don't think they go far enough. Memory management, for instance, is a great convenience; it wasn't that long ago when we had to use program overlays by hand in order to fit a large program into 64K of RAM or so. So while we've made some good advances, many popular languages and environments still inadvertently encourage tight coupling and rigid code, and we need to get beyond that.
Dave: Let me take that further. Java and C# are not really dynamic in any meaningful way. Memory management is a small part of the picture, but the real gains in these more dynamic languages come from different areas, particularly from a flexible type model and from the ability to metaprogram.
Ultimately, it comes down to ease of expression. If I can express myself in code at a level closer to the problem domain, then I'm going to be more effective, and my code is likely to be easier to maintain and extend. Paul Graham makes a big deal out of the way Lisp helped him while building the software that became Yahoo Stores, and he's right. These languages, applied properly, are a strategic advantage. I know some companies are using them with great success. And you know -- they're keeping quiet about it.
ORN: A recent weblog entitled Why Do We Need Publishers? pointed out that print-on-demand (POD) makes small print runs more affordable and more profitable than cultivating a relationship with a professional publisher. You've been handling much of the production work at least since Programming Ruby, going as far as producing camera-ready copy. How do your experiences match the ideas in this weblog?
Andy: Actually, we started way back on our first book, The Pragmatic Programmer. We did all of the typesetting, figures, indexing, you name it. For every book we've written to date, we've given the publisher or printer a ready-to-print PDF or PostScript file. That's why it was an easy decision for us to start our own publishing company last fall.
Dave: The issue though isn't really to do with printing itself. The basis of publishing is the delivery of high-quality content, and the printing technology (print-on-demand, conventional printing, or PDF delivery) are really just mechanisms.
Andy: Are publishers still needed? Good ones, yes. The problem the industry has right now is that quite a few publishers will publish any piece of crap and hope someone buys it. They've abdicated the important role of "publisher as editorial filter." Let's face it, there are more books being printed today than you can count. In order to decide what's worth reading and what's not, you look to the publisher's reputation. You want someone you can trust to sift through the chaff and produce books that you can rely on.
Dave: That's been hammered home to us by the authors writing for The Pragmatic Bookshelf. When we first started writing books, I was expecting an editor to work alongside me, helping me with my writing style and nudging the content in the right direction. However, that wasn't really the case -- editors dipped into book content when milestones come along, but otherwise left authors on their own.
We tried to do things differently. We offer authors different levels of participation, from hands-off (the traditional model) to something closer to a partnership, where we'll work with content as they produce it. When Mike Clark was writing Pragmatic Automation, I probably averaged an hour a day on the phone with him in the last month. We both thoroughly enjoyed it, and the book is great! I'm not saying that I contributed much to the book itself; I think the important part was being a sounding board for Mike's ideas.
That kind of involvement is the value that a publisher should be adding. Whether we then use POD or conventional technologies is really a fairly unimportant decision.
Andy: Right. And the publisher's value is reflected in their reputation. O'Reilly (the distributor of our books), for instance, has an excellent reputation for no-nonsense technical reference books. Other publishers have their reputations in various other niches. Our Pragmatic Bookshelf has already started to gain a reputation for producing books that you can use, right out of the gate, to make your daily work and project better, regardless of technology, operating system, or methodology.
So regardless of how books end up being distributed -- whether it's on paper, or e-ink, or on screen, or in a mid-air hologram -- you still need people you can trust to help sort out the excellent from the mediocre. That's what a good publisher does.
ORN: I've personally had a great deal of success with two of your better-known ideas, the DRY principle and the broken windows story. (The idea about automating everything that can and should be automated is also quite handy.) While I'm sure it's gratifying to see things you talk about gain wider acceptance (and humbling, in the cases where other smart people have said them, but your version is the one that people remember), are there any practices you really like that haven't caught on yet?
Andy: Well, most thoughtful developers have adopted these approaches, or have always done them under a different name. But many still have not. I'd like to see everyone get on board so we can move on, past the easy stuff, and start talking about more difficult problems of architecture and design, language support away from bad habits, more elegant solutions to common problems, and so on.
Despite DRY, decoupling and writing code to an appropriate level of abstraction are still huge issues with many teams.
Dave: I'd agree with Andy there. I'd also add that I'd like to see more developers taking on the idea of continuous learning. This is the basis of our Pragmatic Investment plan (outlined in the talk "How To Keep Your Job"). I think it's vitally important, particularly as the industry is changing, that every developer has a plan for how they will grow their own value, and then puts aside enough time each week to execute that plan.
ORN: Alternately, am I hanging out with smart and disciplined people, while the programmers in the trenches are still tracking dependencies by hand? Is that a danger for people writing about programming? How do you avoid that trouble?
Dave: There's nothing worse than a software consultant who can't code. Andy and I work hard to keep our hands in, coding on client projects and writing substantial projects for ourselves. That's a key part of being pragmatic: we need to know what works, and not just what the books say should work. Part of the trick for introducing practices is understanding that people aren't particularly disciplined. That's why we try to arrange things so that the best way to do something is also the easiest -- people will just tend to do it naturally. Automation plays a key role here: if you can do it the right way by pushing a button, or the wrong way by pushing 10 buttons, copying a file, emailing three friends, and whistling "Rule Britannia," most folks will choose the one-button right way.
Andy: When Dave and I go out and speak at conferences and such, we're seeing a lot of smart, disciplined people and responding to their questions and their concerns. But there's a great many other folks who don't go to the conferences, don't read our columns, and who we only get to see when we're in the field. So to avoid getting locked in the ivory tower, we try hard to stay in touch, to stay in the field with real practitioners facing real problems.
As an industry, we're all still figuring this out. There is no "one right way" to build software, and we can all learn from each other to make our jobs easier. But as a side note, not everyone is interested in helping all programmers. At a recent conference, a Capability Maturity Model (CMM) expert told me -- during a panel discussion in front of a live audience -- that I should "fire my clients," because they weren't CMM Level 3 and didn't yet embrace the common Starter Kit best practices of Version Control, Unit Testing, and Automation.
Now some 40 percent of teams in the U.S. don't use version control at all. 76 percent don't unit test, and 70 percent don't have a daily build. That's a lot of "clients to fire." And even if we did, that wouldn't fix the problem, nor make them go away. You know there are folks on those teams who have heard of and want to try these practices, but maybe their teammates or bosses don't agree. We can help: our publishing, our articles, our workshops can help everyone, whether just starting out or fine-tuning a well-oiled team. We will not abandon any of our readers, no matter what these "process purists" say.
As development gets more and more difficult, we'll continue to try and help each programmer stay on top of their game any way we can.
ORN: Where did you find your statistics that 40 percent of projects don't use source control? That's really scary.
Andy: Originally in:
Zeichick, Alan.
"Debuggers, Source Control Keys to Quality."
Software Development Times, March 1, 2002
And echoed in:
Cusumano et al,
"A Global Survey of Software Development Practices."
MIT Sloan School of Mgmt, paper 178, June 2003.
Which has other gems such as 76 percent of U.S. teams don't run a daily build. Note also that 100 percent of teams in India do. (I'm being a little fast and loose with these numbers, as the statistics vary depending on when daily builds are adopted in the project's lifecycle, but you get the gist of it.)
ORN: Given those figures, it seems that there's a good opportunity for developers to begin to change their organizations. (I think of Jim Shore's ChangeYourOrganizationDiary on the Portland Pattern Repository -- he ran into many, many obstacles.) How can people start to make changes for the better in their organizations?
Andy: Cattle prod?
Being an effective change agent is a difficult business. Probably the most realistic approach is to "be the change you want to see." In other words, adopt as much of this stuff as you can personally, or in a small sub-team. Success will eventually be noticed, and others will begin to emulate you. It's a long, slow process, and the organizational antibodies (akin to white cells in the blood) will come after you at some point. But that's really a topic for a whole separate interview or article.
Dave: That's right. There are three steps. The first is to know where you want to end up. The second is to live it, and encourage others to live it. The third is to know that you're not going to get there!
I'm not being negative here: this is just agility in action. Any plan is obsolete once you start implementing it, and that goes double when you're trying to instigate change. We need to dream of better ways, but we also need to ensure that we have our feet firmly on the ground when we try to push organizations in different directions.
Organizations aren't rational, any more than people are. But they will still listen if you have a good story to tell. That story has to be about value and risk. Explain how agile processes can reduce risk. Explain how lightweight approaches can earn value faster. And explain how they should outsource the mundane stuff, and leave their talented pool of in-house developers free to work on the next revolutionary change to the company's business.
chromatic is the technical editor of the O'Reilly Network.
