I read Mike Clark's Wag your EJB comments and James Duncan Davidson's Listen to the EJBs go Pop response shortly after last week's Pragmatic Practitioners meeting here in Dallas. And something clicked. (It should have been obvious from the beginning, I suppose, but brains work in funny ways.) Mike's comments are about how to deal with EJBs, and ask "where did we go wrong?" Duncan tried to answer that question in the context of EJB history. My goal here is to try to draw lessons that we can apply to other technology choices.

The meeting Tuesday night was a discussion of tools, and Dave started it off with the following intentionally provocative statement: "It is impossible to do quality software engineering with an IDE." Of course, there were a few people who took issue with that, but we quickly converged on something we could all agree on: while IDEs offer a lot of useful tools, you should never allow the IDE to shape your project. The project should assume its natural shape, and the tools should support that. We also agreed that the fastest way to go down the wrong path in that regard is to let the IDE handle builds.

The problem, of course, is that IDEs tend to impose a particular view of project structure. And more often than not, they only support particular compilation tools. So if you want to build part of your system using another tool, or a custom code generator, that part of your build can't be automated by the IDE. So you have to make a choice between automation and the natural technologies for developing your system. You can't win unless you take the build process away from the IDE and use a more general-purpose tool.

There's always a tradeoff between special-purpose and general-purpose tools. Special-purpose tools can make life a lot easier when your needs match what they were designed for, but they aren't easily adapted to other situations. General-purpose tools are never quite as easy as a specialized tool at its best, but they're life savers when you're thrown a curve ball.

I think the choice between specialization and generality is particularly important for software development. Part of this is because even software projects that seem superficially similar tend to differ in many surprising respects. (In other words, software projects nearly always throw you a curve ball.) Additionally, there is the problem that we still don't understand software development very well in any fundamental sense, so folks building specialized tools often make mistakes when deciding where to restrict flexibility. (Another of Duncan's recent blog entries aims straight at this, and scores a bullseye -- I don't know how many copies of that Gosling paper I've printed and given to clients over the past few years.)

A great example of the way seemingly benign restrictions can bite you later is make. That program is shockingly general, at least if you measure that by the number of uses to which it's been put that the designer never envisioned. But it has one unfortunate limitation -- a bias toward operations within a single directory -- that renders it a poor choice for a task very close to its design center: building Java programs.

As Duncan points out, EJBs were designed for very narrow problems, and might be very good for those problems. (I'm not so sure even that's true, but that's another argument.) But they enforce a very narrow, specialized programming model. No threads. No ... well, no lots of things. You give up a lot to program in an EJB container, and that might be worth it if your problems are close enough to EJB's strengths that you really get a lot in return. But as Mike points out, most systems are warped by EJBs, without seeing any compensating benefit.

It's instructive to contrast EJBs with the other core element of J2EE: servlets. Within the externally imposed constraints of the request/response style of HTTP, they are nearly as general-purpose as you can get. "Here's the request: deal with it and get back to me with a response." There are a lot of other things that servlet containers can do for a servlet, but the servlet has to ask. And by comparison with EJBs, servlets are broadly applicable and extremely successful. (After that exercise, if you want to get depressed, think about the technology that was sacrificed on the EJB altar: Jini.)

When looking at a software development technology, favor generality over specialized tools that claim to make things easy. Unless the tool is aimed at a very well understood domain, and your system is squarely within that domain, chances are you'll regret going the special-purpose route.

I'm not vain enough to think that everyone who comes across this weblog is going to find all of my thoughts fascinating. So I've been giving thought to why I do this. The old "a writer has to write" thing seems like a cop-out, and besides, if that were the answer, why've I never been interested in creating a private journal?

Part of the answer can be seen in the way I started blogging. This entry flowed from my fingertips one morning, almost fully formed, and I thought it was fairly good ... but so far as I could tell it wasn't sellable. Many of the things I write are that way: they're very small pieces that would be difficult to turn into full-fledged articles (and I wouldn't have time to do it anyway). Thinking about that led me to approach O'Reilly about a blogging spot on their site.

Another part of the answer lies in the heading on my "blogroll" over in the right margin. A few weeks ago, discussing blogging with Mike Clark, I described the blogging community as "like Usenet, but with only the interesting people." I like that phrase, and it's a pretty apt description. I recall that even in the heyday of Usenet (i.e., before AOL and spam) I tended to gravitate toward posts by people who were interesting. (Of course, dull people blog, too. But the point is that you can choose which blogs you read.)

Communities like that are important. So important that people have to keep reinventing them. After Usenet degraded, mailing lists experienced something of a resurgence, as did IRC. For a while, in the software development world, Ward's Wiki filled the void. These days, much of that community seems to be rebuilding itself using weblogs.

All of these things are just different ways of extending our communities of interaction, finding other people to be a part of them, and increasing the opportunities for sharing ideas. Five years in a row now, at JavaOne, Duncan and I have made it a point to have at least one extended meal together. It's a great opportunity to share ideas (baked or not), things we've learned, things we're working on, and things we're thinking about. Each of us always comes away with a head spinning with new things. Dave and I have lunch every two or three weeks, for similar reasons. Greg Vaughn and I regularly have extended IM chats that help keep us energized with new ideas. Dave, Chris Morris, Joe Tatem, and I recently organized the Dallas Pragmatic Practitioners, and it's the best "user group" I've ever been a part of -- largely because there's so much opportunity for informal discussion, where you learn so much from the ideas and experiences of others. (I can think of several other people or groups I could put in this list, but I think you've got the idea.)

Blogging is one of the ways I contribute to this little "idea ecology" that I'm a part of.

Here's an immediate example: last Thursday, Mike Clark wrote some interesting things about EJBs, and Duncan responded. The combination of those two posts resonated with a discussion we'd had at the Dallas Practitioners meeting two nights prior. (Synergy!) Until today, I've been too busy to write my own response knitting the three together, but you'll see it next.