As discriminatory as this is going to sound, this one is for the old-timers. If you started programming after the turn of the milennium, I don’t know if you’re going to be able to follow the trend of this post—not out of any serious deficiency on your part, hardly that. But I think this is something only the old-timers are going to identify with. (And thus, do I alienate probably 80% of my readership, but so be it.)

Is it me, or is programming just less interesting today than it was two decades ago?

By all means, shake your smartphones and other mobile devices at me and say, “Dude, how can you say that?”, but in many ways programming for Android and iOS reminds me of programming for Windows and Mac OS two decades ago. HTML 5 and JavaScript remind me of ten years ago, the first time HTML and JavaScript came around. The discussions around programming languages remind me of the discussions around C++. The discussions around NoSQL remind me of the arguments both for and against relational databases. It all feels like we’ve been here before, with only the names having changed.

Don’t get me wrong—if any of you comment on the differences between HTML 5 now and HTML 3.2 then, or the degree of the various browser companies agreeing to the standard today against the “browser wars” of a decade ago, I’ll agree with you. This isn’t so much of a rational and logical discussion as it is an emotive and intuitive one. It just feels similar.

To be honest, I get this sense that across the entire industry right now, there’s a sort of malaise, a general sort of “Bah, nothing really all that new is going on anymore”. NoSQL is re-introducing storage ideas that had been around before but were discarded (perhaps injudiciously and too quickly) in favor of the relational model. Functional languages have obviously been in place since the 50’s (in Lisp). And so on.

More importantly, look at the Java community: what truly innovative ideas have emerged here in the last five years? Every new open-source project or commercial endeavor either seems to be a refinement of an idea before it (how many different times are we going to create a new Web framework, guys?) or an attempt to leverage an idea coming from somewhere else (be it from .NET or from Ruby or from JavaScript or….). With the upcoming .NET 4.5 release and Windows 8, Microsoft is holding out very little “new and exciting” bits for the community to invest emotionally in: we hear about “async” in C# 5 (something that F# has had already, thank you), and of course there is WinRT (another platform or virtual machine… sort of), and… well, honestly, didn’t we just do this a decade ago? Where is the WCFs, the WPFs, the Silverlights, the things that would get us fired up? Hell, even a new approach to data access might stir some excitement. Node.js feels like an attempt to reinvent the app server, but if you look back far enough you see that the app server itself was reinvented once (in the Java world) in Spring and other lightweight frameworks, and before that by people who actually thought to write their own web servers in straight Java. (And, for the record, the whole event-driven I/O thing is something that’s been done in both Java and .NET a long time before now.)

And as much as this is going to probably just throw fat on the fire, all the excitement around JavaScript as a language reminds me of the excitement about Ruby as a language. Does nobody remember that Sun did this once already, with Phobos? Or that Netscape did this with LiveScript? JavaScript on the server end is not new, folks. It’s just new to the people who’d never seen it before.

In years past, there has always seemed to be something deeper, something more exciting and more innovative that drives the industry in strange ways. Artificial Intelligence was one such thing: the search to try and bring computers to a state of human-like sentience drove a lot of interesting ideas and concepts forward, but over the last decade or two, AI seems to have lost almost all of its luster and momentum. User interfaces—specifically, GUIs—were another force for a while, until GUIs got to the point where they were so common and so deeply rooted in their chosen pasts (the single-button of the Mac, the menubar-per-window of Windows, etc) that they left themselves so little room for maneuver. At least this is one area where Microsoft is (maybe) putting the fatted sacred cow to the butcher’s knife, with their Metro UI moves in Windows 8… but only up to a point.

Maybe I’m just old and tired and should hang up my keyboard and go take up farming, then go retire to my front porch’s rocking chair and practice my Hey you kids! Getoffamylawn! or something. But before you dismiss me entirely, do me a favor and tell me: what gets you excited these days? If you’ve been programming for twenty years, what about the industry today gets your blood moving and your mind sharpened?

Well, friends, another year has come and gone, and it's time for me to put my crystal ball into place and see what the upcoming year has for us. But, of course, in the long-standing tradition of these predictions, I also need to put my spectacles on (I did turn 40 last year, after all) and have a look at how well I did in this same activity twelve months ago.

Let's see what unbelievable gobs of hooey I slung last year came even remotely to pass. For 2011, I said....

• THEN: Android’s penetration into the mobile space is going to rise, then plateau around the middle of the year. Android phones, collectively, have outpaced iPhone sales. That’s a pretty significant statistic—and it means that there’s fewer customers buying smartphones in the coming year. More importantly, the first generation of Android slates (including the Galaxy Tab, which I own), are less-than-sublime, and not really an “iPad Killer” device by any stretch of the imagination. And I think that will slow down people buying Android slates and phones, particularly since Google has all but promised that Android releases will start slowing down.
  • NOW: Well, I think I get a point for saying that Android's penetration will rise... but then I lose it for suggesting that it would slow down. Wow, was I wrong on that. Once Amazon put the Kindle Fire out, suddenly for the first time Android tablets began to appear in peoples' hands in record numbers. The drawback here is that most people using the Fire don't realize it's an Android tablet, which certainly hurts Google's brand-awareness (not that Amazon really seems to mind), but the upshot is simple: people are still buying devices, even though they may already own one. Which amazes me.
• THEN: Windows Phone 7 penetration into the mobile space will appear huge, then slow down towards the middle of the year. Microsoft is getting some pretty decent numbers now, from what I can piece together, and I think that’s largely the “I love Microsoft” crowd buying in. But it’s a pretty crowded place right now with Android and iPhone, and I’m not sure if the much-easier Office and/or Exchange integration is enough to woo consumers (who care about Office) or business types (who care about Exchange) away from their Androids and iPhones.
  • NOW: Despite the catastrophic implosion of RIM (thus creating a huge market of people looking to trade their Blackberrys in for other mobile phones, ones which won't all go down when a RIM server implodes), WP7 has definitely not emerged as the "third player" in the mobile space; or, perhaps more precisely, they feel like a distant third, rather than a creditable alternative to the other two. In fact, more and more it just feels like this is a two-horse race and Microsoft is in it still because they're willing to throw loss after loss to stay in it. (For what reason, I'm not sure--it's not clear to me that they can ever reach a point of profitability here, even once Nokia makes the transition to WP7, which is supposedly going to take years. On the order of a half-decade or so.) Even living here in Redmon, where I would expect the WP7 concentration to be much, much higher than anywhere else in the world, it's still more common to see iPhones and 'droids in peoples' hands than it is to see WP7 phones.
• THEN: Android, iOS and/or Windows Phone 7 becomes a developer requirement. Developers, if you haven’t taken the time to learn how to program one of these three platforms, you are electing to remove yourself from a growing market that desperately wants people with these skills. I see the “mobile native app development” space as every bit as hot as the “Internet/Web development” space was back in 2000. If you don’t have a device, buy one. If you have a device, get the tools—in all three cases they’re free downloads—and start writing stupid little apps that nobody cares about, so you can have some skills on the platform when somebody cares about it.
  • NOW: Wow, yes. Right now, if you are a developer and you haven't spent at least a little time learning mobile development, you are excluding yourself from a development "boom" that rivals the one around Web sites in the mid-90's. Seriously: remember when everybody had to have a website? That's the mentality right now with a ton of different companies--"we have to have a mobile app!" "But we sell condom lubricant!" "Doesn't matter! We need a mobile app! Build us something! Go go go go go!"
• THEN: The Windows 7 slates will suck. This isn’t a prediction, this is established fact. I played with an “ExoPC” 10” form factor slate running Windows 7 (Dell I think was the manufacturer), and it was a horrible experience. Windows 7, like most OSes, really expects a keyboard to be present, and a slate doesn’t have one—so the OS was hacked to put a “keyboard” button at the top of the screen that would slide out to let you touch-type on the slate. I tried to fire up Notepad and type out a haiku, and it was an unbelievably awkward process. Android and iOS clearly own the slate market for the forseeable future, and if Dell has any brains in its corporate head, it will phone up Google tomorrow and start talking about putting Android on that hardware.
  • NOW: Yeah, that was something of a "gimme" point (but I'll take it). Windows7 on a slate was a Bad Idea, and I'm pretty sure the sales reflect that. Conduct your own anecdotal poll: see if you can find a store somewhere in your town or city that will actually sell you a Windows7 slate. Can't find one? I can--it's the Microsoft store in town, and I'm not entirely sure they still stock them. Certainly our local Best Buy doesn't.
• THEN: DSLs mostly disappear from the buzz. I still see no strawman (no “pet store” equivalent), and none of the traditional builders-of-strawmen (Microsoft, Oracle, etc) appear interested in DSLs much anymore, so I think 2010 will mark the last year that we spent any time talking about the concept.
  • NOW: I'm going to claim a point here, too. DSLs have pretty much left us hanging. Without a strawman for developers to "get", the DSL movement has more or less largely died out. I still sometimes hear people refer to something that isn't a programming language but does something technical as a "DSL" ("That shipping label? That's a DSL!"), and that just tells me that the concept never really took root.
• THEN: Facebook becomes more of a developer requirement than before. I don’t like Mark Zuckerburg. I don’t like Facebook’s privacy policies. I don’t particularly like the way Facebook approaches the Facebook Connect experience. But Facebook owns enough people to be the fourth-largest nation on the planet, and probably commands an economy of roughly that size to boot. If your app is aimed at the Facebook demographic (that is, everybody who’s not on Twitter), you have to know how to reach these people, and that means developing at least some part of your system to integrate with it.
  • NOW: Facebook, if anything, has become more important through 2011, particularly for startups looking to get some exposure and recognition. Facebook continues to screw with their user experience, though, and they keep screwing with their security policies, and as "big" a presence as they have, it's not invulnerable, and if they're not careful, they're going to find themselves on the other side of the relevance curve.
• THEN: Twitter becomes more of a developer requirement, too. Anybody who’s not on Facebook is on Twitter. Or dead. So to reach the other half of the online community, you have to know how to connect out with Twitter.
  • NOW: Twitter's impact has become deeper, but more muted in some ways--people don't think of Twitter as a "new" channel, but one that they've come to expect and get used to. At the same time, how Twitter is supposed to factor into different applications isn't always clear, which hinders Twitter's acceptance and "must-have"-ness. Of course, Twitter could care less, it seems, though it still confuses me how they actually make money.
• THEN: XMPP becomes more of a developer requirement. XMPP hasn’t crossed a lot of people’s radar screen before, but Facebook decided to adopt it as their chat system communication protocol, and Google’s already been using it, and suddenly there’s a whole lotta traffic going over XMPP. More importantly, it offers a two-way communication experience that is in some scenarios vastly better than what HTTP offers, yet running in a very “Internet-friendly” way just as HTTP does. I suspect that XMPP is going to start cropping up in a number of places as a useful alternative and/or complement to using HTTP.
  • NOW: Well, unfortunately, XMPP still hides underneath other names and still doesn't come to mind when people are thinking about communication, leaving this one way unfulfilled. *sigh* Maybe someday we will learn that not everything has to go over HTTP, but it didn't happen in 2011.
• THEN: “Gamification” starts making serious inroads into non-gaming systems. Maybe it’s just because I’ve been talking more about gaming, game design, and game implementation last year, but all of a sudden “gamification”—the process of putting game-like concepts into non-game applications—is cresting in a big way. FourSquare, Yelp, Gowalla, suddenly all these systems are offering achievement badges and scoring systems for people who want to play in their worlds. How long is it before a developer is pulled into a meeting and told that “we need to put achievement badges into the call-center support application”? Or the online e-commerce portal? It’ll start either this year or next.
  • NOW: Gamification is emerging, but slowly and under the radar. It's certainly not as strong as I thought it would be, but gamification concepts are sneaking their way into a variety of different scenarios (beyond games themselves). Probably can't claim a point here, no.
• THEN: Functional languages will hit a make-or-break point. I know, I said it last year. But the buzz keeps growing, and when that happens, it usually means that it’s either going to reach a critical mass and explode, or it’s going to implode—and the longer the buzz grows, the faster it explodes or implodes, accordingly. My personal guess is that the “F/O hybrids”—F#, Scala, etc—will continue to grow until they explode, particularly since the suggested v.Next changes to both Java and C# have to be done as language changes, whereas futures for F# frequently are either built as libraries masquerading as syntax (such as asynchronous workflows, introduced in 2.0) or as back-end library hooks that anybody can plug in (such as type providers, introduced at PDC a few months ago), neither of which require any language revs—and no concerns about backwards compatibility with existing code. This makes the F/O hybrids vastly more flexible and stable. In fact, I suspect that within five years or so, we’ll start seeing a gradual shift away from pure O-O systems, into systems that use a lot more functional concepts—and that will propel the F/O languages into the center of the developer mindshare.
  • NOW: More than any of my other predictions (or subjects of interest), functional languages stump me the most. On the one hand, there doesn't seem to be a drop-off of interest in the subject, based on a variety of anecdotal evidence (books, articles, etc), but on the other hand, they don't seem to be crossing over into the "mainstream" programming worlds, either. At best, we can say that they are entering the mindset of senior programmers and/or project leads and/or architects, but certainly they don't seem to be turning in to the "go-to" language for projects being done in 2011.
• THEN: The Microsoft Kinect will lose its shine. I hate to say it, but I just don’t see where the excitement is coming from. Remember when the Wii nunchucks were the most amazing thing anybody had ever seen? Frankly, after a slew of initial releases for the Wii that made use of them in interesting ways, the buzz has dropped off, and more importantly, the nunchucks turned out to be just another way to move an arrow around on the screen—in other words, we haven’t found particularly novel and interesting/game-changing ways to use the things. That’s what I think will happen with the Kinect. Sure, it’s really freakin’ cool that you can use your body as the controller—but how precise is it, how quickly can it react to my body movements, and most of all, what new user interface metaphors are people going to have to come up with in order to avoid the “me-too” dancing-game clones that are charging down the pipeline right now?
  • NOW: Kinect still makes for a great Christmas or birthday present, but nobody seems to be all that amazed by the idea anymore. Certainly we aren't seeing a huge surge in using Kinect as a general user interface device, at least not yet. Maybe it needed more time for people to develop those new metaphors, but at the same time, I would've expected at least a few more games to make use of it, and I haven't seen any this past year.
• THEN: There will be no clear victor in the Silverlight-vs-HTML5 war. And make no mistake about it, a war is brewing. Microsoft, I think, finds itself in the inenviable position of having two very clearly useful technologies, each one’s “sphere of utility” (meaning, the range of answers to the “where would I use it?” question) very clearly overlapping. It’s sort of like being a football team with both Brett Favre and Tom Brady on your roster—both of them are superstars, but you know, deep down, that you have to cut one, because you can’t devote the same degree of time and energy to both. Microsoft is going to take most of 2011 and probably part of 2012 trying to support both, making a mess of it, offering up conflicting rationale and reasoning, in the end achieving nothing but confusing developers and harming their relationship with the Microsoft developer community in the process. Personally, I think Microsoft has no choice but to get behind HTML 5, but I like a lot of the features of Silverlight and think that it has a lot of mojo that HTML 5 lacks, and would actually be in favor of Microsoft keeping both—so long as they make it very clear to the developer community when and where each should be used. In other words, the executives in charge of each should be locked into a room and not allowed out until they’ve hammered out a business strategy that is then printed and handed out to every developer within a 3-continent radius of Redmond. (Chances of this happening: .01%)
  • NOW: Well, this was accurate all the way up until the last couple of months, when Microsoft made it fairly clear that Silverlight was being effectively "put behind" HTML 5, despite shipping another version of Silverlight. In the meantime, though, they've tried to support both (and some Silverlighters tell me that the Silverlight team is still looking forward to continuing supporting it, though I'm not sure at this point what is rumor and what is fact anymore), and yes, they confused the hell out of everybody. I'm surprised they pulled the trigger on it in 2011, though--I expected it to go a version or two more before they finally pulled the rug out.
• THEN: Apple starts feeling the pressure to deliver a developer experience that isn’t mired in mid-90’s metaphor. Don’t look now, Apple, but a lot of software developers are coming to your platform from Java and .NET, and they’re bringing their expectations for what and how a developer IDE should look like, perform, and do, with them. Xcode is not a modern IDE, all the Apple fan-boy love for it notwithstanding, and this means that a few things will happen:
  • Eclipse gets an iOS plugin. Yes, I know, it wouldn’t work (for the most part) on a Windows-based Eclipse installation, but if Eclipse can have a native C/C++ developer experience, then there’s no reason why a Mac Eclipse install couldn’t have an Objective-C plugin, and that opens up the idea of using Eclipse to write iOS and/or native Mac apps (which will be critical when the Mac App Store debuts somewhere in 2011 or 2012).
  • Rumors will abound about Microsoft bringing Visual Studio to the Mac. Silverlight already runs on the Mac; why not bring the native development experience there? I’m not saying they’ll actually do it, and certainly not in 2011, but the rumors, they will be flyin….
  • Other third-party alternatives to Xcode will emerge and/or grow. MonoTouch is just one example. There’s opportunity here, just as the fledgling Java IDE market looked back in ‘96, and people will come to fill it.
  • NOW: Xcode 4 is "better", but it's still not what I would call comparable to the Microsoft Visual Studio or JetBrains IDEA experience. LLVM is definitely a better platform for the company's development efforts, long-term, and it's encouraging that they're investing so heavily into it, but I still wish the overall development experience was stronger. Meanwhile, though, no Eclipse plugin has emerged (that I'm aware of), which surprised me, and neither did we see Microsoft trying to step into that world, which doesn't surprise me, but disappoints me just a little. I realize that Microsoft's developer tools are generally designed to support the Windows operating system first, but Microsoft has to cut loose from that perspective if they're going to survive as a company. More on that later.
• THEN: NoSQL buzz grows. The NoSQL movement, which sort of got started last year, will reach significant states of buzz this year. NoSQL databases have a lot to offer, particularly in areas that relational databases are weak, such as hierarchical kinds of storage requirements, for example. That buzz will reach a fever pitch this year, and the relational database moguls (Microsoft, Oracle, IBM) will start to fight back.
  • NOW: Well, the buzz certainly grew, and it surprised me that the big storage guys (Microsoft, IBM, Oracle) didn't do more to address it; I was expecting features to emerge in their database products to address some of the features present in MongoDB or CouchDB or some of the others, such as "schemaless" or map/reduce-style queries. Even just incorporating JavaScript into the engine somewhere would've generated a reaction.

Overall, it appears I'm running at about my usual 50/50 levels of prognostication. So be it. Let's see what the ol' crystal ball has in mind for 2012:

• Lisps will be the languages to watch. With Clojure leading the way, Lisps (that is, languages that are more or less loosely based on Common Lisp or one of its variants) are slowly clawing their way back into the limelight. Lisps are both functional languages as well as dynamic languages, which gives them a significant reason for interest. Clojure runs on top of the JVM, which makes it highly interoperable with other JVM languages/systems, and Clojure/CLR is the version of Clojure for the CLR platform, though there seems to be less interest in it in the .NET world (which is a mistake, if you ask me).
• Functional languages will.... I have no idea. As I said above, I'm kind of stymied on the whole functional-language thing and their future. I keep thinking they will either "take off" or "drop off", and they keep tacking to the middle, doing neither, just sort of hanging in there as a concept for programmers to take and run with. Mind you, I like functional languages, and I want to see them become mainstream, or at least more so, but I keep wondering if the mainstream programming public is ready to accept the ideas and concepts hiding therein. So this year, let's try something different: I predict that they will remain exactly where they are, neither "done" nor "accepted", but continue next year to sort of hang out in the middle.
• F#'s type providers will show up in C# v.Next. This one is actually a "gimme", if you look across the history of F# and C#: for almost every version of F# v."N", features from that version show up in C# v."N+1". More importantly, F# 3.0's type provider feature is an amazing idea, and one that I think will open up language research in some very interesting ways. (Not sure what F#'s type providers are or what they'll do for you? Check out Don Syme's talk on it at BUILD last year.)
• Windows8 will generate a lot of chatter. As 2012 progresses, Microsoft will try to force a lot of buzz around it by keeping things under wraps until various points in the year that feel strategic (TechEd, BUILD, etc). In doing so, though, they will annoy a number of people by not talking about them more openly or transparently. What's more....
• Windows8 ("Metro")-style apps won't impress at first. The more I think about it, the more I'm becoming convinced that Metro-style apps on a desktop machine are going to collectively underwhelm. The UI simply isn't designed for keyboard-and-mouse kinds of interaction, and that's going to be the hardware setup that most people first experience Windows8 on--contrary to what (I think) Microsoft thinks, people do not just have tablets laying around waiting for Windows 8 to be installed on it, nor are they going to buy a Windows8 tablet just to try it out, at least not until it's gathered some mojo behind it. Microsoft is going to have to finesse the messaging here very, very finely, and that's not something they've shown themselves to be particularly good at over the last half-decade.
• Scala will get bigger, thanks to Heroku. With the adoption of Scala and Play for their Java apps, Heroku is going to make Scala look attractive as a development platform, and the adoption of Play by Typesafe (the same people who brought you Akka) means that these four--Heroku, Scala, Play and Akka--will combine into a very compelling and interesting platform. I'm looking forward to seeing what comes of that.
• Cloud will continue to whip up a lot of air. For all the hype and money spent on it, it doesn't really seem like cloud is gathering commensurate amounts of traction, across all the various cloud providers with the possible exception of Amazon's cloud system. But, as the different cloud platforms start to diversify their platform technology (Microsoft seems to be leading the way here, ironically, with the introduction of Java, Hadoop and some limited NoSQL bits into their Azure offerings), and as we start to get more experience with the pricing and costs of cloud, 2012 might be the year that we start to see mainstream cloud adoption, beyond "just" the usage patterns we've seen so far (as a backing server for mobile apps and as an easy way to spin up startups).
• Android tablets will start to gain momentum. Amazon's Kindle Fire has hit the market strong, definitely better than any other Android-based tablet before it. The Nooq (the Kindle's principal competitor, at least in the e-reader world) is also an Android tablet, which means that right now, consumers can get into the Android tablet world for far, far less than what an iPad costs. Apple rumors suggest that they may have a 7" form factor tablet that will price competitively (in the $200/$300 range), but that's just rumor right now, and Apple has never shown an interest in that form factor, which means the 7" world will remain exclusively Android's (at least for now), and that's a nice form factor for a lot of things. This translates well into more sales of Android tablets in general, I think.
• Apple will release an iPad 3, and it will be "more of the same". Trying to predict Apple is generally a lost cause, particularly when it comes to their vaunted iOS lines, but somewhere around the middle of the year would be ripe for a new iPad, at the very least. (With the iPhone 4S out a few months ago, it's hard to imagine they'd cannibalize those sales by releasing a new iPhone, until the end of the year at the earliest.) Frankly, though, I don't expect the iPad 3 to be all that big of a boost, just a faster processor, more storage, and probably about the same size. Probably the only thing I'd want added to the iPad would be a USB port, but that conflicts with the Apple desire to present the iPad as a "device", rather than as a "computer". (USB ports smack of "computers", not self-contained "devices".)
• Apple will get hauled in front of the US government for... something. Apple's recent foray in the legal world, effectively informing Samsung that they can't make square phones and offering advice as to what will avoid future litigation, smacks of such hubris and arrogance, it makes Microsoft look like a Pollyanna Pushover by comparison. It is pretty much a given, it seems to me, that a confrontation in the legal halls is not far removed, either with the US or with the EU, over anti-cometitive behavior. (And if this kind of behavior continues, and there is no legal action, it'll be pretty apparent that Apple has a pretty good set of US Congressmen and Senators in their pocket, something they probably learned from watching Microsoft and IBM slug it out rather than just buy them off.)
• IBM will be entirely irrelevant again. Look, IBM's main contribution to the Java world is/was Eclipse, and to a much lesser degree, Harmony. With Eclipse more or less "done" (aside from all the work on plugins being done by third parties), and with IBM abandoning Harmony in favor of OpenJDK, IBM more or less removes themselves from the game, as far as developers are concerned. Which shouldn't really be surprising--they've been more or less irrelevant pretty much ever since the mid-2000s or so.
• Oracle will "screw it up" at least once. Right now, the Java community is poised, like a starving vulture, waiting for Oracle to do something else that demonstrates and befits their Evil Emperor status. The community has already been quick (far too quick, if you ask me) to highlight Oracle's supposed missteps, such as the JVM-crashing bug (which has already been fixed in the _u1 release of Java7, which garnered no attention from the various Java news sites) and the debacle around Hudson/Jenkins/whatever-the-heck-we-need-to-call-it-this-week. I'll grant you, the Hudson/Jenkins debacle was deserving of ire, but Oracle is hardly the Evil Emperor the community makes them out to be--at least, so far. (I'll admit it, though, I'm a touch biased, both because Brian Goetz is a friend of mine and because Oracle TechNet has asked me to write a column for them next year. Still, in the spirit of "innocent until proven guilty"....)
• VMWare/SpringSource will start pushing their cloud solution in a major way. Companies like Microsoft and Google are pushing cloud solutions because Software-as-a-Service is a reoccurring revenue model, generating revenue even in years when the product hasn't incremented. VMWare, being a product company, is in the same boat--the only time they make money is when they sell a new copy of their product, unless they can start pushing their virtualization story onto hardware on behalf of clients--a.k.a. "the cloud". With SpringSource as the software stack, VMWare has a more-or-less complete cloud play, so it's surprising that they didn't push it harder in 2011; I suspect they'll start cramming it down everybody's throats in 2012. Expect to see Rod Johnson talking a lot about the cloud as a result.
• JavaScript hype will continue to grow, and by years' end will be at near-backlash levels. JavaScript (more properly known as ECMAScript, not that anyone seems to care but me) is gaining all kinds of steam as a mainstream development language (as opposed to just-a-browser language), particularly with the release of NodeJS. That hype will continue to escalate, and by the end of the year we may start to see a backlash against it. (Speaking personally, NodeJS is an interesting solution, but suggesting that it will replace your Tomcat or IIS server is a bit far-fetched; event-driven I/O is something both of those servers have been doing for years, and the rest of it is "just" a language discussion. We could pretty easily use JavaScript as the development language inside both servers, as Sun demonstrated years ago with their "Phobos" project--not that anybody really cared back then.)
• NoSQL buzz will continue to grow, and by years' end will start to generate a backlash. More and more companies are jumping into NoSQL-based solutions, and this trend will continue to accelerate, until some extremely public failure will start to generate a backlash against it. (This seems to be a pattern that shows up with a lot of technologies, so it seems entirely realistic that it'll happen here, too.) Mind you, I don't mean to suggest that the backlash will be factual or correct--usually these sorts of things come from misuing the tool, not from any intrinsic failure in it--but it'll generate some bad press.
• Ted will thoroughly rock the house during his CodeMash keynote. Yeah, OK, that's more of a fervent wish than a prediction, but hey, keep a positive attitude and all that, right?
• Ted will continue to enjoy his time working for Neudesic. So far, it's been great working for these guys, and I'm looking forward to a great 2012 with them. (Hopefully this will be a prediction I get to tack on for many years to come, too.)

I hope that all of you have enjoyed reading these, and I wish you and yours a very merry, happy, profitable and fulfilling 2012. Thanks for reading.

Many of you have undoubtedly noticed that my blogging has dropped off precipitously over the last half-year. The reason for that is multifold, ranging from the usual “I just don’t seem to have the time for it” rationale, up through the realization that I have a couple of regular (paid) columns (one with CoDe Magazine, one with MSDN) that consume a lot of my ideas that would otherwise go into the blog.

But most of all, the main reason I’m finding it harder these days to blog is that as of July of this year, I have joined forces with Neudesic, LLC, as a full-time employee, working as an Architectural Consultant for them.

Neudesic is a Microsoft partner (as a matter of fact, as I understand it we were Microsoft’s Partner of the Year not too long ago), with several different technology practices, including a Mobile practice, a User Experience practice, a Connected Systems practice, and a Custom Application Development practice, among others. The company is (as of this writing) about 400 consultants strong, with a number of Microsoft MVPs and Regional Directors on staff, including a personal friend of mine, Simon Guest, who heads up the Mobile Practice, and another friend, Rick Garibay, who is the Practice Director for Connected Systems. And that doesn’t include the other friends I have within the company, as well as the people within the company who are quickly becoming new friends. I’m even more tickled that I was instrumental in bringing Steven “Doc” List in, to bring his agile experience and perspective to our projects nationwide. (Plus I just like working with Doc.)

It’s been a great partnership so far: they ask me to continue doing the speaking and writing that I love to do, bringing fame and glory (I hope!) to the Neudesic name, and in turn I get to jump in on a variety of different projects as an architect and mentor. The people I’m working with are great, top-notch technology experts and just some of the nicest people I’ve met. Plus, yes, it’s nice to draw a regular bimonthly paycheck and benefits after being an independent for a decade or so.

The fact that they’re principally a .NET shop may lead some to conclude that this is my farewell letter to the Java community, but in fact the opposite is the case. I’m actively engaged with our Mobile practice around Android (and iOS) development, and I’m subtly and covertly (sssh! Don’t tell the partners!) trying to subvert the company into expanding our technology practices into the Java (and Ruby/Rails) space.

With the coming new year, I think one of my upcoming responsibilities will be to blog more, so don’t be too surprised if you start to see more activity on a more regular basis here. But in the meantime, I’m working on my end-of-year predictions and retrospective, so keep an eye out for that in the next few days.

(Oh, and that link that appears across the bottom of my blog posts? Someday I’m going to remember how to change the text for that in the blog engine and modify it to read something more Neudesic-centric. But for now, it’ll work.)

Recently I got an email from Bohdan Zograf, who offered:

Hi!

I'm willing to translate publication located at http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx to the Belorussian language (my mother tongue). What I'm asking for is your written permission, so you don't mind after I'll post the translation to my blog.

I agreed, and next thing I know, I get the next email that it’s done. If your mother tongue is Belorussian, then I invite you to read the article in its translated form at http://www.moneyaisle.com/worldwide/the-vietnam-of-computer-science-be.

Thanks, Bohdan!

Long-time readers of this blog know what’s coming next: it’s time for Ted to prognosticate on what the coming year of tech will bring us. But I believe strongly in accountability, even in my offered-up-for-free predictions, so one of the traditions of this space is to go back and revisit my predictions from this time last year. So, without further ado, let’s look back at Ted’s 2010 predictions, and see how things played out; 2010 predictions are prefixed with “THEN”, and my thoughts on my predictions are prefixed with “NOW”:

For 2010, I predicted....

• THEN: ... I will offer 3- and 4-day training classes on F# and Scala, among other things. OK, that's not fair—yes, I have the materials, I just need to work out locations and times. Contact me if you're interested in a private class, by the way.
• NOW: Well, I offered them… I just didn’t do much to advertise them or sell them. I got plenty busy just with the other things I had going on. Besides, this and the next prediction were pretty much all advertisement anyway, so I don’t know if anybody really counts these two.
• THEN: ... I will publish two books, one on F# and one on Scala. OK, OK, another plug. Or, rather, more of a resolution. One will be the "Professional F#" I'm doing for Wiley/Wrox, the other isn't yet finalized. But it'll either be published through a publisher, or self-published, by JavaOne 2010.
• NOW: “Professional F# 2.0” shipped in Q3 of 2010; the other Scala book I decided not to pursue—too much stuff going on to really put the necessary time into it. (Cue sad trombone.)
• THEN: ... DSLs will either "succeed" this year, or begin the short slide into the dustbin of obscure programming ideas. Domain-specific language advocates have to put up some kind of strawman for developers to learn from and poke at, or the whole concept will just fade away. Martin's book will help, if it ships this year, but even that might not be enough to generate interest if it doesn't have some kind of large-scale applicability in it. Patterns and refactoring and enterprise containers all had a huge advantage in that developers could see pretty easily what the problem was they solved; DSLs haven't made that clear yet.
• NOW: To be honest, this one is hard to call. Martin Fowler published his DSL book, which many people consider to be a good sign of what’s happening in the world, but really, the DSL buzz seems to have dropped off significantly. The strawman hasn’t appeared in any meaningful public way (I still don’t see an example being offered up from anybody), and that leads me to believe that the fading-away has started.
• THEN: ... functional languages will start to see a backlash. I hate to say it, but "getting" the functional mindset is hard, and there's precious few resources that are making it easy for mainstream (read: O-O) developers make that adjustment, far fewer than there was during the procedural-to-object shift. If the functional community doesn't want to become mainstream, then mainstream developers will find ways to take functional's most compelling gateway use-case (parallel/concurrent programming) and find a way to "git 'er done" in the traditional O-O approach, probably through software transactional memory, and functional languages like Haskell and Erlang will be relegated to the "What Might Have Been" of computer science history. Not sure what I mean? Try this: walk into a functional language forum, and ask what a monad is. Nobody yet has been able to produce an answer that doesn't involve math theory, or that does involve a practical domain-object-based example. In fact, nobody has really said why (or if) monads are even still useful. Or catamorphisms. Or any of the other dime-store words that the functional community likes to toss around.
• NOW: I think I have to admit that this hasn’t happened—at least, there’s been no backlash that I’ve seen. In fact, what’s interesting is that there’s been some movement to bring those functional concepts—including monads, which surprised me completely—into other languages like C# or Java for discussion and use. That being said, though, I don’t see Haskell and Erlang taking center stage as application languages—instead, I see them taking supporting-cast kinds of roles building other infrastructure that applications in turn make use of, a la CouchDB (written in Erlang). Monads still remain a mostly-opaque subject for most developers, however, and it’s still unclear if monads are something that people should think about applying in code, or if they are one of those “in theory” kinds of concepts. (You know, one of those ideas that change your brain forever, but you never actually use directly in code.)
• THEN: ... Visual Studio 2010 will ship on time, and be one of the buggiest and/or slowest releases in its history. I hate to make this prediction, because I really don't want to be right, but there's just so much happening in the Visual Studio refactoring effort that it makes me incredibly nervous. Widespread adoption of VS2010 will wait until SP1 at the earliest. In fact....
• NOW: Wow, did I get a few people here in Redmond annoyed with me about that one. And, as it turned out, I was pretty off-base about its stability. (It shipped pretty close if not exactly on the ship date Microsoft promised, as I recall, though I admit I wasn’t paying too much attention to it.)  I’ve been using VS 2010 for a lot of .NET work in the last six months, and I’ve yet (knock on wood) to have it crash on me. /bow Visual Studio team.
• THEN: ... Visual Studio 2010 SP 1 will ship within three months of the final product. Microsoft knows that people wait until SP 1 to think about upgrading, so they'll just plan for an eager SP 1 release, and hope that managers will be too hung over from the New Year (still) to notice that the necessary shakeout time hasn't happened.
• NOW: Uh…. nope. In fact, SP 1 has just reached a beta/CTP state. As for managers being too hung over, well…
• THEN: ... Apple will ship a tablet with multi-touch on it, and it will flop horribly. Not sure why I think this, but I just don't think the multi-touch paradigm that Apple has cooked up for the iPhone will carry over to a tablet/laptop device. That won't stop them from shipping it, and it won't stop Apple fan-boiz from buying it, but that's about where the interest will end.
• NOW: Oh, WOW did I come so close and yet missed the mark by a mile. Of course, the “tablet” that Apple shipped was the iPad, and it did pretty much everything except flop horribly. Apple fan-boys bought it… and then about 24 hours later, so did everybody else. My mom got one, for crying out loud. And folks, the iPad—along with the whole “slate” concept—is pretty clearly here to stay.
• THEN: ... JDK 7 closures will be debated for a few weeks, then become a fait accompli as the Java community shrugs its collective shoulders. Frankly, I think the Java community has exhausted its interest in debating new language features for Java. Recent college grads and open-source groups with an axe to grind will continue to try and make an issue out of this, but I think the overall Java community just... doesn't... care. They just want to see JDK 7 ship someday.
• NOW: Pretty close—except that closures won’t ship as part of JDK 7, largely due to the Oracle acquisition in the middle of the year here. And I was spot-on vis-à-vis the “they want to see JDK 7 ship someday”; when given the chance to wait for a year or so for a Java-with-closures to ship, the community overwhelmingly voted to get something sooner rather than later.
• THEN: ... Scala either "pops" in 2010, or begins to fall apart. By "pops", I mean reaches a critical mass of developers interested in using it, enough to convince somebody to create a company around it, a la G2One.
• NOW: … and by “somebody”, it turns out I meant Martin Odersky. Scala is pretty clearly a hot topic in the Java space, its buzz being disturbed only by Clojure. Scala and/or Clojure, plus Groovy, makes a really compelling JVM-based stack.
• THEN: ... Oracle is going to make a serious "cloud" play, probably by offering an Oracle-hosted version of Azure or AppEngine. Oracle loves the enterprise space too much, and derives too much money from it, to not at least appear to have some kind of offering here. Now that they own Java, they'll marry it up against OpenSolaris, the Oracle database, and throw the whole thing into a series of server centers all over the continent, and call it "Oracle 12c" (c for Cloud, of course) or something.
• NOW: Oracle made a play, but it was to continue to enhance Java, not build a cloud space. It surprises me that they haven’t made a more forceful move in this space, but I suspect that a huge amount of time and energy went into folding Sun into their corporate environment.
• THEN: ... Spring development will slow to a crawl and start to take a left turn toward cloud ideas. VMWare bought SpringSource for a reason, and I believe it's entirely centered around VMWare's movement into the cloud space—they want to be more than "just" a virtualization tool. Spring + Groovy makes a compelling development stack, particularly if VMWare does some interesting hooks-n-hacks to make Spring a virtualization environment in its own right somehow. But from a practical perspective, any community-driven development against Spring is all but basically dead. The source may be downloadable later, like the VMWare Player code is, but making contributions back? Fuhgeddabowdit.
• NOW: The Spring One show definitely played up Cloud stuff, and springsource.com seems to be emphasizing cloud more in a couple of subtle ways. Not sure if I call this one a win or not for me, though.
• THEN: ... the explosion of e-book readers brings the Kindle 2009 edition way down to size. The era of the e-book reader is here, and honestly, while I'm glad I have a Kindle, I'm expecting that I'll be dusting it off a shelf in a few years. Kinda like I do with my iPods from a few years ago.
• NOW: Honestly, can’t say that I’m using my Kindle a lot, but I am reading using the Kindle app on non-Kindle hardware more than I thought I would be. That said, I am eyeing the new Kindle hardware generation with an acquisitive eye…
• THEN: ... "social networking" becomes the "Web 2.0" of 2010. In other words, using the term will basically identify you as a tech wannabe and clearly out of touch with the bleeding edge.
• NOW: Um…. yeah.
• THEN: ... Facebook becomes a developer platform requirement. I don't pretend to know anything about Facebook—I'm not even on it, which amazes my family to no end—but clearly Facebook is one of those mechanisms by which people reach each other, and before long, it'll start showing up as a developer requirement for companies looking to hire. If you're looking to build out your resume to make yourself attractive to companies in 2010, mad Facebook skillz might not be a bad investment.
• NOW: I’m on Facebook, I’ve written some code for it, and given how much the startup scene loves the “Like” button, I think developers who knew Facebook in 2010 did pretty well for themselves.
• THEN: ... Nintendo releases an open SDK for building games for its next-gen DS-based device. With the spectacular success of games on the iPhone, Nintendo clearly must see that they're missing a huge opportunity every day developers can't write games for the Nintendo DS that are easily downloadable to the device for playing. Nintendo is not stupid—if they don't open up the SDK and promote "casual" games like those on the iPhone and those that can now be downloaded to the Zune or the XBox, they risk being marginalized out of existence.
• NOW: Um… yeah. Maybe this was me just being hopeful.

In general, it looks like I was more right than wrong, which is not a bad record to have. Of course, a couple of those “wrong”s were “giving up the big play” kind of wrongs, so while I may have a winning record, I still may have a defense that’s given up too many points to be taken seriously. *shrug* Oh, well.

What portends for 2011?

• Android’s penetration into the mobile space is going to rise, then plateau around the middle of the year. Android phones, collectively, have outpaced iPhone sales. That’s a pretty significant statistic—and it means that there’s fewer customers buying smartphones in the coming year. More importantly, the first generation of Android slates (including the Galaxy Tab, which I own), are less-than-sublime, and not really an “iPad Killer” device by any stretch of the imagination. And I think that will slow down people buying Android slates and phones, particularly since Google has all but promised that Android releases will start slowing down.
• Windows Phone 7 penetration into the mobile space will appear huge, then slow down towards the middle of the year. Microsoft is getting some pretty decent numbers now, from what I can piece together, and I think that’s largely the “I love Microsoft” crowd buying in. But it’s a pretty crowded place right now with Android and iPhone, and I’m not sure if the much-easier Office and/or Exchange integration is enough to woo consumers (who care about Office) or business types (who care about Exchange) away from their Androids and iPhones.
• Android, iOS and/or Windows Phone 7 becomes a developer requirement. Developers, if you haven’t taken the time to learn how to program one of these three platforms, you are electing to remove yourself from a growing market that desperately wants people with these skills. I see the “mobile native app development” space as every bit as hot as the “Internet/Web development” space was back in 2000. If you don’t have a device, buy one. If you have a device, get the tools—in all three cases they’re free downloads—and start writing stupid little apps that nobody cares about, so you can have some skills on the platform when somebody cares about it.
• The Windows 7 slates will suck. This isn’t a prediction, this is established fact. I played with an “ExoPC” 10” form factor slate running Windows 7 (Dell I think was the manufacturer), and it was a horrible experience. Windows 7, like most OSes, really expects a keyboard to be present, and a slate doesn’t have one—so the OS was hacked to put a “keyboard” button at the top of the screen that would slide out to let you touch-type on the slate. I tried to fire up Notepad and type out a haiku, and it was an unbelievably awkward process. Android and iOS clearly own the slate market for the forseeable future, and if Dell has any brains in its corporate head, it will phone up Google tomorrow and start talking about putting Android on that hardware.
• DSLs mostly disappear from the buzz. I still see no strawman (no “pet store” equivalent), and none of the traditional builders-of-strawmen (Microsoft, Oracle, etc) appear interested in DSLs much anymore, so I think 2010 will mark the last year that we spent any time talking about the concept.
• Facebook becomes more of a developer requirement than before. I don’t like Mark Zuckerburg. I don’t like Facebook’s privacy policies. I don’t particularly like the way Facebook approaches the Facebook Connect experience. But Facebook owns enough people to be the fourth-largest nation on the planet, and probably commands an economy of roughly that size to boot. If your app is aimed at the Facebook demographic (that is, everybody who’s not on Twitter), you have to know how to reach these people, and that means developing at least some part of your system to integrate with it.
• Twitter becomes more of a developer requirement, too. Anybody who’s not on Facebook is on Twitter. Or dead. So to reach the other half of the online community, you have to know how to connect out with Twitter.
• XMPP becomes more of a developer requirement. XMPP hasn’t crossed a lot of people’s radar screen before, but Facebook decided to adopt it as their chat system communication protocol, and Google’s already been using it, and suddenly there’s a whole lotta traffic going over XMPP. More importantly, it offers a two-way communication experience that is in some scenarios vastly better than what HTTP offers, yet running in a very “Internet-friendly” way just as HTTP does. I suspect that XMPP is going to start cropping up in a number of places as a useful alternative and/or complement to using HTTP.
• “Gamification” starts making serious inroads into non-gaming systems. Maybe it’s just because I’ve been talking more about gaming, game design, and game implementation last year, but all of a sudden “gamification”—the process of putting game-like concepts into non-game applications—is cresting in a big way. FourSquare, Yelp, Gowalla, suddenly all these systems are offering achievement badges and scoring systems for people who want to play in their worlds. How long is it before a developer is pulled into a meeting and told that “we need to put achievement badges into the call-center support application”? Or the online e-commerce portal? It’ll start either this year or next.
• Functional languages will hit a make-or-break point. I know, I said it last year. But the buzz keeps growing, and when that happens, it usually means that it’s either going to reach a critical mass and explode, or it’s going to implode—and the longer the buzz grows, the faster it explodes or implodes, accordingly. My personal guess is that the “F/O hybrids”—F#, Scala, etc—will continue to grow until they explode, particularly since the suggested v.Next changes to both Java and C# have to be done as language changes, whereas futures for F# frequently are either built as libraries masquerading as syntax (such as asynchronous workflows, introduced in 2.0) or as back-end library hooks that anybody can plug in (such as type providers, introduced at PDC a few months ago), neither of which require any language revs—and no concerns about backwards compatibility with existing code. This makes the F/O hybrids vastly more flexible and stable. In fact, I suspect that within five years or so, we’ll start seeing a gradual shift away from pure O-O systems, into systems that use a lot more functional concepts—and that will propel the F/O languages into the center of the developer mindshare.
• The Microsoft Kinect will lose its shine. I hate to say it, but I just don’t see where the excitement is coming from. Remember when the Wii nunchucks were the most amazing thing anybody had ever seen? Frankly, after a slew of initial releases for the Wii that made use of them in interesting ways, the buzz has dropped off, and more importantly, the nunchucks turned out to be just another way to move an arrow around on the screen—in other words, we haven’t found particularly novel and interesting/game-changing ways to use the things. That’s what I think will happen with the Kinect. Sure, it’s really freakin’ cool that you can use your body as the controller—but how precise is it, how quickly can it react to my body movements, and most of all, what new user interface metaphors are people going to have to come up with in order to avoid the “me-too” dancing-game clones that are charging down the pipeline right now?
• There will be no clear victor in the Silverlight-vs-HTML5 war. And make no mistake about it, a war is brewing. Microsoft, I think, finds itself in the inenviable position of having two very clearly useful technologies, each one’s “sphere of utility” (meaning, the range of answers to the “where would I use it?” question) very clearly overlapping. It’s sort of like being a football team with both Brett Favre and Tom Brady on your roster—both of them are superstars, but you know, deep down, that you have to cut one, because you can’t devote the same degree of time and energy to both. Microsoft is going to take most of 2011 and probably part of 2012 trying to support both, making a mess of it, offering up conflicting rationale and reasoning, in the end achieving nothing but confusing developers and harming their relationship with the Microsoft developer community in the process. Personally, I think Microsoft has no choice but to get behind HTML 5, but I like a lot of the features of Silverlight and think that it has a lot of mojo that HTML 5 lacks, and would actually be in favor of Microsoft keeping both—so long as they make it very clear to the developer community when and where each should be used. In other words, the executives in charge of each should be locked into a room and not allowed out until they’ve hammered out a business strategy that is then printed and handed out to every developer within a 3-continent radius of Redmond. (Chances of this happening: .01%)
• Apple starts feeling the pressure to deliver a developer experience that isn’t mired in mid-90’s metaphor. Don’t look now, Apple, but a lot of software developers are coming to your platform from Java and .NET, and they’re bringing their expectations for what and how a developer IDE should look like, perform, and do, with them. Xcode is not a modern IDE, all the Apple fan-boy love for it notwithstanding, and this means that a few things will happen:
• Eclipse gets an iOS plugin. Yes, I know, it wouldn’t work (for the most part) on a Windows-based Eclipse installation, but if Eclipse can have a native C/C++ developer experience, then there’s no reason why a Mac Eclipse install couldn’t have an Objective-C plugin, and that opens up the idea of using Eclipse to write iOS and/or native Mac apps (which will be critical when the Mac App Store debuts somewhere in 2011 or 2012).
• Rumors will abound about Microsoft bringing Visual Studio to the Mac. Silverlight already runs on the Mac; why not bring the native development experience there? I’m not saying they’ll actually do it, and certainly not in 2011, but the rumors, they will be flyin….
• Other third-party alternatives to Xcode will emerge and/or grow. MonoTouch is just one example. There’s opportunity here, just as the fledgling Java IDE market looked back in ‘96, and people will come to fill it.
• NoSQL buzz grows. The NoSQL movement, which sort of got started last year, will reach significant states of buzz this year. NoSQL databases have a lot to offer, particularly in areas that relational databases are weak, such as hierarchical kinds of storage requirements, for example. That buzz will reach a fever pitch this year, and the relational database moguls (Microsoft, Oracle, IBM) will start to fight back.

I could probably go on making a few more, but I think these are enough to get me into trouble for the year.

To all of you who’ve been readers of this blog for the past year, I thank you—blog-gathered statistics tell me that I get, on average, about 7,000 hits a day, which just stuns me—and it is a New Years’ Resolution that I blog more and give you even more reason to stick around. Happy New Year, and may your 2011 be just as peaceful, prosperous, and eventful as you want it to be.

Startup Weekend came to Redmond this weekend, and as I write this it is all of three hours over. In the spirit of capturing post-mortem thoughts as quickly as possible, I thought I’d blog my reactions and thoughts from it, both as a reference for myself for the next one, and as a guide/warning/data point for others considering doing it.

A few weeks ago, emails started crossing the Seattle Tech Startup mailing list about this thing called “Startup Weekend”. I didn’t do a whole lot of research around it—just glanced at the website, and asked a few questions of the organizer in an email. Specifically, I wanted to know that as a tech guy, with no specific startup ideas, I would find something to do. I was reassured immediately that, in fact, as a tech guy, I would be “heavily recruited” by others at the event who were business types.

First takeaway: I can’t speak for all the events, this being my first, but it was a surprise, nay, a shock, to me just how many “business” and/or “marketing” types were at this event. I seriously expected that tech folks would outnumber the non-tech folks by a substantial margin, but it was exactly the opposite, probably on the order of 2 to 1. As a developer, I was definitely being courted, rather than hunting for a team to find a way to make room for me. It was refreshing, exciting and a little overwhelming at the same time.

The format of the event is interesting: anybody can pitch an idea, then everybody in the room is free to “attach” themselves to that idea to form a team to implement it somehow, sort of a “Law of Two Feet” applied to team-building.

Second takeaway: Have a pretty clear idea of what you want to do here. The ideas initially all sound pretty good, and choosing between them can actually be quite painful and difficult. Have a clear goal for yourself what you want out of the weekend—to socialize, to stretch yourself, to build a business, whatever. Mine were (1) just to be here and experience the event, (2) to socialize and network more deeply with the startup scene, (3) to hack on some code and try to ship something, and (4) to learn some new tech that I hadn’t had the chance to use beyond a “Hello World” demo before. There was always the chance I wouldn’t get any of those things, in which case I accepted a consolation prize of simply watching how the event was structured and run, since it operates in many ways on the same basic concept that  GiveCamp does, which is something I want to see done in Seattle sooner rather than later. So just by going and watching the event as a uninvolved observer was worth the price of admission, so once I’d walked through the door, I’d already met my #1 win condition.

I realized as I was choosing which team to join that I didn’t want to be paired alone with the project-pitching person (whomever that would be), since I had no idea how this event worked or what we were going for, so I deliberately turned away from a few projects that sounded interesting. I ended up as part of a team that was pretty well spread-out in terms of skillsets/interests (Chris, developer and “original idea guy”, Libby, business development, Maizer, also business development, Mohammed, small businessman, and Aaron, graphic designer), working on an idea around “social bar gaming”. In other words, we had a nebulous fuzzy idea about using games on a mobile device to help people in bars connect to other people in bars via some kind of “scavenger hunt” or similar social-engagement game. I had suggested that maybe one feature or idea would be to help groups of hard-drinking souls chart their path between bars (something like a Traveling Saleman’s Problem meets a PubCrawl), and Chris thought that was definitely something to consider. We laid out a brief idea of what it was we wanted to build, then parted ways Friday night about midnight or so, except for Chris and myself, who headed out to Denny’s to mull over some technology ideas for a while, until about 3 AM in the morning.

Third takeaway: Hoard the nighttime hours on Friday, to spend them working on the app the rest of the weekend. Even though you’re full of energy Friday night, rarin’ to go, bank it because you’ll need to be well-rested for the marathon that is Saturday and Sunday.

Chris and I briefly discussed the technology approaches we could use, and settled in on using Azure for the backplane, mostly because I felt it would be the quickest way to get us provisioned on a server, and it was an excuse for me to play with Azure, something I haven’t had much of a chance to do beyond simple demos. We also thought that having some kind of Facebook integration would be a good idea, depending on what we actually wanted to do with the idea. I thought to myself, “OK, so this is going to be interesting for me—I’m going to be actually ‘stretching’ on three things simultaneously: Azure, Facebook, and whatever Web framework we use to build this”, since I haven’t done much Web work in .NET in many, many years, and don’t consider myself “up to speed” on either ASP.NET or ASP.NET MVC. Chris was a “front to middle tier” guy, though, so I figured I’d focus on the Azure back-end parts—storage, queueing, etc—and maybe the Facebook integration, and we’d be good.

By Saturday morning, thanks to a few other things I had to do after Chris left, I got there a bit late—about 10:30—fully expecting that the team had a pretty clear app vision laid out and ready for Chris and I to execute on. Alas, not quite—we were still sort of thrashing on what exactly we wanted to build—specifically, we kept bouncing back and forth between what the game would be and how it would be monetized. If we wanted to sell to bars as a way to get more bodies in the door, then we needed some kind of “check-in” game where people earned points for bringing friends to the bar. Or we could sell to bars by creating a game that was a kind of “scavenger hunt”, forcing patrons to discover things about the bar or about new drinks the bar sells, and so on. But we also wanted a game that was intrinsically social, forcing peoples’ eyes away from the screens and up towards the other patrons—otherwise why play the game?

Aaron, a two-time veteran of Startup Weekend, suggested that we finalize our vision by 11 AM so we could start hacking. By 11 AM, we had a vision… until about an hour later, when I realized that Libby, Chris, Maizer, and Mohammed were changing the game to suit new monetization ideas. We set another deadline for 2 PM, at which point we had a vision…. until about an hour later again, when I looked up again and found them discussing again what kind of game we wanted to build. In the end, it wasn’t until 7 or 8 PM Saturday when we finally nailed down some kind of game app idea—and then only because Aaron came out of his shell a little and politely yelled at the group for wasting all of our time.

Fourth takeaway: Know what’s clear and unclear about your vision/idea. I think we didn’t realize how nebulous our ideas were until we started trying to put game mechanics around it, and that was what led to all the thrashing on ideas.

Fifth takeaway: Put somebody in charge. Have a dictator in place. Yes, everybody wants to be polite and yes, choosing a leader can be a bit uncomfortable, but having that final unambiguous deciding vote—a leader—who can make decisions and isn’t afraid to do so would have saved us a lot of headache and gotten us much more quickly down the path. Libby said it best in our little post-mortem at the bar afterwards: Don’t you dare leave Friday night until everybody is 100% clear on what you’re building.

Meanwhile, on the technical front, another warm front was meeting another cold front and developing into a storm. When we’d decided to use Azure, I had suggested it over Google App Engine because Chris had said he’d done some development with it before, so I figured he was comfortable with it and ready to go. As we started pulling out laptops to start working, though, Chris mentioned that he needed to spin up a virtual machine with Windows 7, Visual Studio, and the Azure tools in it. No worries—I needed some time to read up on Azure provisioning, data storage, and so on.

Unfortunately, setting up the VM took until about 8 PM Saturday night, meaning we lost 11 of our 15 hours (9 AM to midnight) for that day.

Sixth takeaway: Have your tools ready to go before you get there. Find a hosting provider—come on, everybody is going to need a hosting provider, even if you build a mobile app—and have a virtual machine or laptop configured with every dev tool you can think of, ready to go. Getting stuff downloaded and installed is burning a very precious commodity that you don’t have nearly enough of: time.

Seventh takeaway: Be clear about your personal motivation/win conditions for the weekend. Yes, I wanted to explore a new tech, but Chris took that to mean that I wasn’t going to succeed if we abandoned Azure, and as a result, we burned close to 50% of our development cycles prepping a VM just so I could put Azure on my resume. I would’ve happily redacted that line on my resume in exchange for getting us up and running by 11 AM Saturday morning, particularly because it became clear to me that others in the group were running with win conditions of “spin up a legitimate business startup idea”, and I had already met most of my win conditions for the weekend by this point. I should’ve mentioned this much earlier, but didn’t realize what was happening until a chance comment Chris made in passing Saturday night when we left for the night.

Sunday I got in about noonish, owing to a long day, short night, and forgotten cell phone (alarm clock) in the car. By the time I got there, tempers were starting to flare because we were clearly well behind the curve. Chris had been up all night working on HTML forms for the game, Aaron had been up all night creating some (amazing!) graphics for the game, I had been up a significant part of the night diving into Facebook APIs, and I think we all sensed that this was in real danger of falling apart. Unfortunately, we couldn’t even split the work between Chris and I, because we had (foolishly) not bothered to get some kind of source-control server going for the code so we could work in parallel.

See the sixth takeaway. It applies to source-control servers, too. And source-control clients, while we’re at it.

We were slotted to present our app and business idea first, as it turned out, which was my preference—I figured that if we went first, we might set a high bar that other groups would have a hard time matching. (That turned out to be a really false hope—the other groups’ work was amazing.) The group asked me to make the pitch, which was fine with me—when have I ever turned down the chance to work a crowd?

But our big concern was the demo—we’d originally called for a “feature freeze” at 4PM, so we would have time to put the app on the server and test it, but by 4:15 Chris was still stitching pages together and putting images on pages. In fact, the push to the Azure server for v0.1 of our app happened about about 5:15, a full 30 seconds before I started the pitch.

The pitch itself was deliberately simple: we put Libby on a bar stool facing the crowd, Mohammed standing against a wall, and said, “Ever been in a bar, wanting to strike up a conversation with that cute girl at the far table? With Pubbn, we give you an excuse—a social scavenger hunt—to strike up a conversation with her, or earn some points, or win a discount from the bar, or more. We want to take the usual social networking premise—pushing socialization into the network—and instead flip it on its ear—using the network to make it easier to socialize.” It was a nice pitch, but I forgot to tell people to download the app and try it during the demo, which left some people thinking we never actually finished anything. ACK.

Pubbn, by the way, was our app name, derived (obviously) from “going pubbing”, as in, going out to drink and socialize. I loved this name. It’s up at http://www.pubbn.com, but I’ll warn you now, it’s a static mockup and a far cry from what we wanted to end up with—in fact, I’ll go out on a limb and say that of all the projects, ours was by far the weakest technical achievement, and I lay the blame for that at my feet. I should’ve stepped up and taken more firm control of the development so Chris could focus more on the overall picture.

The eventual winners for the weekend were “Doodle-A-Doodle”, a fantastic learn-to-draw app for kids on the iPad; “Hold It!”, a game about standing in line in the mens’ room; and “CamBadge”, a brilliant little iPhone app for creating a conference badge on your phone, hanging your phone around your neck, and snapping a picture of the person standing in front of you with a single touch to the screen (assuming, of course, you have an iPhone 4 with its front-facing camera).

“CamBadge” was one of the apps I thought about working on, but passed on it because it didn’t seem challenging enough technologically. Clearly that was a foolish choice from a business perspective, but this is why knowing what your win conditions for the weekend are so important—I didn’t necessarily want to build a new business out of this weekend, and, to me, the more important “win” was to make a social connection with the people who looked like good folks to know in this space—and the “CamBadge” principal, Adam, clearly fit that bill. Drinking with him was far more important—to me—than building an app with him. Next Startup Weekend, my win conditions might be different, and if so, I’d make an entirely different decision.

In the end, Startup Weekend was a blast, and something I thoroughly recommend every developer who’s ever thought of going independent do. The cost is well, well worth the experience, and if you fail miserably, well, better to do that here, with so little invested, than to fail later in a “real” startup with millions attached.

By the way, Startup Weekend Redmond was/is #swred on Twitter, if you want to see the buzz that came out of it. Particularly good reading are the Tweets starting at about 5 PM tonight, because that’s when the presentations started.

Hey, anybody who’s got significant VMWare mojo, help out a bro?

I’ve got a Win7 VM (one of many) that appears to be exhibiting weird disk behavior—the vmdk, a growable single-file VMDK, is almost precisely twice the used space. It’s a 120GB growable disk, and the Win7 guest reports about 35GB used, but the VMDK takes about 70GB on host disk. CHKDSK inside Windows says everything’s good, and the VMWare “Disk Cleanup” doesn’t change anything, either. It doesn’t seem to be a Windows7 thing, because I’ve got a half-dozen other Win7 VMs that operate… well, normally (by which I mean, 30GB used in the VMDK means 30GB used on disk). It’s a VMWare Fusion host, if that makes any difference. Any other details that might be relevant, let me know and I’ll post.

Anybody got any ideas what the heck is going on inside this disk?

By now, the Twitter messages have spread, and the word is out: at Uberconf this year, I did a session ("Pragmatic Architecture"), which I've done at other venues before, but this time we made it into a 180-minute workshop instead of a 90-minute session, and the workshop included breaking the room up into small (10-ish, which was still a teensy bit too big) groups and giving each one an "architectural kata" to work on.

The architectural kata is a take on PragDave's coding kata, except taken to a higher level: the architectural kata is an exercise in which the group seeks to create an architecture to solve the problem presented. The inspiration for this came from Frederick Brooks' latest book, The Design of Design, in which he points out that the only way to get great designers is to get them to design. The corollary, of course, is that in order to create great architects, we have to get them to architect. But few architects get a chance to architect a system more than a half-dozen times or so over the lifetime of a career, and that's only for those who are fortunate to be given the opportunity to architect in the first place. Of course, the problem here is, you have to be an architect in order to get hired as an architect, but if you're not an architect, then how can you architect in order to become an architect?

Um... hang on, let me make sure I wrote that right.

Anyway, the "rules" around the kata (which makes it more difficult to consume the kata but makes the scenario more realistic, IMHO):

• you may ask the instructor questions about the project
• you must be prepared to present a rough architectural vision of the project and defend questions about it
• you must be prepared to ask questions of other participants' presentations
• you may safely make assumptions about technologies you don't know well as long as those assumptions are clearly defined and spelled out
• you may not assume you have hiring/firing authority over the development team
• any technology is fair game (but you must justify its use)
• any other rules, you may ask about

The groups were given 30 minutes in which to formulate some ideas, and then three of them were given a few minutes to present their ideas and defend it against some questions from the crowd.

An example kata is below:

Architectural Kata #5: I'll have the BLT

a national sandwich shop wants to enable "fax in your order" but over the Internet instead

users: millions+

requirements: users will place their order, then be given a time to pick up their sandwich and directions to the shop (which must integrate with Google Maps); if the shop offers a delivery service, dispatch the driver with the sandwich to the user; mobile-device accessibility; offer national daily promotionals/specials; offer local daily promotionals/specials; accept payment online or in person/on delivery

As you can tell, it's vague in some ways, and this is somewhat deliberate—as one group discovered, part of the architect's job is to ask questions of the project champion (me), and they didn't, and felt like they failed pretty miserably. (In their defense, the kata they drew—randomly—was pretty much universally thought to be the hardest of the lot.) But overall, the exercise was well-received, lots of people found it a great opportunity to try being an architect, and even the team that failed felt that it was a valuable exercise.

I'm definitely going to do more of these, and refine the whole thing a little. (Thanks to everyone who participated and gave me great feedback on how to make it better.) If you're interested in having it done as a practice exercise for your development team before the start of a big project, ping me. I think this would be a *great* exercise to do during a user group meeting, too.

Code Katas are small, relatively simple exercises designed to give you a problem to try and solve. I like to use them as a way to get my feet wet and help write something more interesting than "Hello World" but less complicated than "The Internet's Next Killer App".

This one is from the UVa online programming contest judge system, which I discovered after picking up the book Programming Challenges, which is highly recommended as a source of code katas, by the way. Much of the advice parts of the book can be skimmed or ignored by the long-time professional developer, but it's still worth a read, since it can be an interesting source of ideas and approaches when solving real-world scenarios.

Problem: You work for a manufacturing company, and they have just received their newest piece of super-modern hardware, a highly efficient assembly-line mechanized pneumatic item manipulator, also known in some circles as a "robotic arm". It is driven by a series of commands, and your job is to write the software to drive the arm. The initial test will be to have the arm move a series of blocks around.

Context: The test begins with n number of blocks, laid out sequentially next to each other, each block with a number on it. (You may safely assume that n never exceeds 25.) So, if n is 4, then the blocks are laid out (starting from 0) as:

0: 0

1: 1

2: 2

3: 3

The display output here is the block-numbered "slot", then a colon, then the block(s) that are stacked in that slot, lowest to highest in left to right order. Thus, in the following display:

0:

1:

2: 0 1 2 3

3:

The 3 block is stacked on top of the 2 block is stacked on top of the 1 block is stacked on top of the 0 block, all in slot 2. This can be shortened to the representation [0:, 1:, 2: 0 1 2 3, 3:] for conciseness.

The arm understands a number of different commands, as well as an optic sensor. (Yeah, the guys who created the arm were good enough to write code that knows how to read the number off a block, but not to actually drive the arm. Go figure.) The commands are as follows, where a and b are valid block numbers (meaning they are between 0 and n-1):

• "move a onto b" This command orders the arm to find block a, and return any blocks stacked on top of it to their original position. Do the same for block b, then stack block a on top of b.
• "move a over b" This command orders the arm to find block a, and return any blocks stacked on top of it to their original position. Then stack block a on top of the stack of blocks containing b.
• "pile a onto b" This command orders the arm to find the stack of blocks containing block b, and return any blocks stacked on top of it to their original position. Then the arm must find the stack of blocks containing block a, and take the stack of blocks starting from a on upwards (in other words, don't do anything with any blocks on top of a) and put that stack on top of block b.
• "pile a over b" This command orders the arm to find the stack of blocks containing block a and take the stack of blocks starting from a on upwards (in other words, don't do anything with any blocks on top of a) and put that stack on top of the stack of blocks containing block b (in other words, don't do anything with the stack of blocks containing b, either).
• "quit" This command tells the arm to shut down (and thus terminates the simulation).

Note that if the input command sequence accidentally offers a command where a and b are the same value, that command is illegal and should be ignored.

As an example, then, if we have 4 blocks in the state [0: 0, 1: 1, 2: 2, 3: 3], and run a "move 2 onto 3", we get [0: 0, 1: 1, 2:, 3: 3 2]. If we then run a "pile 3 over 1", we should end up with [0: 0, 1: 1 3 2, 2:, 3:]. And so on.

Input: n = 10. Run these commands:

1. move 9 onto 1
2. move 8 over 1
3. move 7 over 1
4. move 6 over 1
5. pile 8 over 6
6. pile 8 over 5
7. move 2 over 1
8. move 4 over 9
9. quit

The result should be [0: 0, 1: 1 9 2 4, 2:, 3: 3, 4:, 5: 5 8 7 6, 6:, 7:, 8:, 9:]

Challenges:

• Implement the Towers of Hanoi (or as close to it as you can get) using this system.
• Add an optimizer to the arm, in essence reading in the entire program (up to "quit"), finding shorter paths and/or different commands to achieve the same result.
• Add a visual component to the simulation, displaying the arm as it moves over each block and moves blocks around.
• Add another robotic arm, and allow commands to be given simultaneously. This will require some thought—does each arm execute a complete command before allowing the other arm to execute (which reduces the performance having two arms might offer), or can each arm act entirely independently? The two (or more) arms will probably need separate command streams, but you might try running them with one command stream just for grins. Note that deciding how to synchronized the arms so they don't conflict with one another will probably require adding some kind of synchronization instructions into the stream as well.

Code Katas are small, relatively simple exercises designed to give you a problem to try and solve. I like to use them as a way to get my feet wet and help write something more interesting than "Hello World" but less complicated than "The Internet's Next Killer App".

Rick Minerich mentioned this one on his blog already, but here is the original "problem"/challenge as it was presented to me and which I in turn shot to him over a Twitter DM:

I have a list, say something like [4, 4, 4, 4, 2, 2, 2, 3, 3, 2, 2, 2, 2, 1, 1, 1, 5, 5], which consists of varying repetitions of integers. (We can assume that it's always numbers, and the use of the term "list" here is generic—it could be a list, array, or some other collection class, your choice.) The goal is to take this list of numbers, and "compress" it down into a (theoretically smaller) list of numbers in pairs, where the first of the pair is the occurrence number of the value, which is the second number. So, since the list above has four 4's, followed by three 2's, two 3's, four 2's, three 1's and two 5's, it should compress into [4, 4, 3, 2, 2, 3, 3, 1, 2, 5].

Update: Typo! It should compress into [4, 4, 3, 2, 2, 3, 4, 2, 3, 1, 2, 5], not [4, 4, 3, 2, 2, 3, 3, 1, 2, 5]. Sorry!

Using your functional language of choice, implement a solution. (No looking at Rick's solution first, by the way—that's cheating!) Feel free to post proposed solutions here as comments, by the way.

This is a pretty easy challenge, but I wanted to try and solve it in a functional mindset, which the challenger had never seen before. I also thought it made for an interesting challenge for people who've never programming in functional languages before, because it requires a very different approach than the imperative solution.

Extensions to the kata (a.k.a. "extra credit"):

• How does the implementation change (if any) to generalize it to a list of any particular type? (Assume the list is of homogenous type—always strings, always ints, always whatever.)
• How does the implementation change (if any) to generalize it to a list of any type? (In other words, a list of strings, ints, Dates, whatever, mixed together within the list: [1, 1, "one", "one", "one", ...] .)
• How does the implementation change (if any) to generate a list of two-item tuples (the first being the occurence, the second being the value) as the result instead? Are there significant advantages to this?
• How does the implementation change (if any) to parallelize/multi-thread it? For your particular language how many elements have to be in the list before doing so yields a significant payoff?

By the way, some of the extension questions make the Kata somewhat interesting even for the imperative/O-O developer; have at, and let me know what you think.

A couple of days ago, a buddy of mine, Scott Hanselman, wrote a nice little intro to the "dynamic" type in C# 4.0. In particular, I like (though don't necessarily 100% agree with) his one-sentence summation of dynamic as "There's no way for you or I to know the type of this now, compiler, so let's hope that the runtime figures it out." It's an interesting characterization, but my disagreement with his characterization is not the point here, at least not of this particular blog entry.

I've been waiting for it for a while, ever since C# 4 was announced, and sure enough, here we go: Scott's blog is the victim of the Static-Typing Fundamentalist, the bearded and grizzled veteran of the Static/Dynamic Code Wars, come out to proclaim the sins of dynamic programming, the evils of those who use(d) it, and why C#/C++/Java was so much better than Visual Basic/Ruby/Python/whatever. Be careful of these creatures. They rival Al-Qaeda in their ferocity and zeal, Fox News in their attention to detail and evidence, and George Bush in their pronouncements of gloom and doom for the future if we don't act now and eliminate this evil.

Allow me to quote (liberally) from Rob's comment on Scott's blog, and comment in turn as we go:

It's such a shame that you promote this stuff. You should've seen the horrific devastation that "Variant" caused in the old VB days. Variant single-handedly create job security for so many people since the late 90's, because of the horrible, horrible, horrible things that developers did with that ridiculous, 12-byte data type!

I just love it when people make comments like "horrific devastation". Nothing like a little hyperbole to liven things up! I mean, it didn't cause exceptions, it didn't make code hard to read, it didn't make it tricky for developers to modify and refactor safely, it leveled cities! burned forests! slaughtered kittens! and even worse, it was 12 bytes in size!

Never mind the fact that Visual Basic developers frequently churned out apps twice, three, five times faster than their C++ cousins did. (I know this—I was one of those C++ developers, and routinely mocked the VB guys across the hall for their crappy language and code.... until they built an app in a few days that I tried to build at home in C++ and gave up after two weeks. And all the damn thing did was basic dialogs-and-data kinds of stuff, too.)

This weak-typing with late-binding is just such a bad idea. I know you'll say "But wait, these are powerful tools that skilled developers can leverage!" - and maybe so, but 98% of the people that truly use these sorts of techniques out in the real world, are unskilled developers making a mess of software all across this great land, because the compiler is so forgiving.

Ah, the "All Developers (Except Me) Are Idiots" argument. I love this one—the hubris involved here is just too precious for words. I have no doubt that the author of this post, being (of course) the classically-trained object-oriented developer and therefore too smart/disciplined/experienced/whatever to fall into such a ridiculous temptation as to use dynamic typing, would never use this feature except in the Most Dire of Emergencies, but his fellow programmers, all of them being much less disciplined/smart/trained/whatever than he is, will fall for the temptation and write code that levels cities! burns forests! kills kittens! and worse, uses 12 bytes! (Oh, wait, it's only 3 bytes, because dynamic is just a placeholder for an object reference, and all object references are 3 bytes in the CLR. Or at least they used to be—I admit, I haven't checked in CLR 4.) Those poor souls, they won't have any hope! There they'll be, staring at Visual Studio, wanting desperately to do the Right Thing, and that evil little programmer devil on their shoulder (probably wearing a T-shirt that says, "P3rl is l33t" or something equally blasphemous) will whisper, "You know, if you just make it a dynamic, you can get the compiler to shut up and you can go home early...."

Oh, right—sorry, I forgot. That devil will whisper, "You know, if you write this code in Visual Basic .NET, you can make the entire codebase Option Strict Off and Option Explicit Off, make the compiler shut up and you can go home early...." Hell, they've been whispering that bit of subversion since 2001. And ye Gods! The leveled cities! burned forests! cute little kitten bodies! all over the place! It's fortunate that we C# developers have kept all those Visual Basic developers on the straight-and-narrow path of true salvation static typing.

This is a huge step backwards for C#, in my opinion - and creates the same scenario VB always did - where it is so forgiving, that it allows developers to write horrible code and you won't so much as see a compiler warning!! I've always tauted that C# was better, simply because it gave the developer "tough love", and forced him/her to be better coder and to "make good choices"!

Ah, yes, the C# compiler and its "tough love". The "prefer compile errors over runtime errors" argument, vis-a-vis Scott Meyers' "Effective C++" circa 1994 or so. It's vastly preferable to see errors early, before the big demo in front of the VP/President/potential customer. (Anybody who disagrees with this obviously hasn't had a demo fail in front of a VP/President/potential customer.) How fortunate that the C# compiler catches all these ugly errors at compile-time, like

   1: static void DoSomething()

   2: {

   3:     List<object> intList = new List<object>();

   4:     intList.Add(5);

   5:     string s = (string) intList[0];

   6:     Console.WriteLine(s);

   7: }

... because boy, that would be embarrassing if it didn't. I mean, can you imagine the horror other disciplined/smart/experienced developers would feel if a lenient compiler actually allowed code like this:

   1: class Point

   2: {

   3:     internal int x;

   4:     internal int y;

   5:     public Point(int x, int y)

   6:     {

   7:         x = x;

   8:         y = y;

   9:     }

  10: }

or this:

   1: class Point

   2: {

   3:     internal int x;

   4:     internal int y;

   5:     public Point(int x, int y)

   6:     {

   7:         this.x = x;

   8:         this.y = y;

   9:     }

  10:     public override string ToString()

  11:     {

  12:         return String.Format("({0},{1})", x, y);

  13:     }

  14: }

  15: static void DoSomething()

  16: {

  17:     Point pt = new Point(12, 12);

  18:     pt.GetType()

  19:         .GetField("x", BindingFlags.Instance | 

  20:             BindingFlags.NonPublic)

  21:         .SetValue(pt, 24);

  22:     Console.WriteLine(pt);

  23: }

to compile? Cities! Forests! Kittens! Thank God C# isn't that kind of lustfully promiscuous... I mean, "lenient"... compiler!

(Now if only we could tout blog comment engines with spellcheck....)

Specific to this blog post, if you are doing somewhere where you can't even quantify what the data type that is coming back? Guess waht, you've got yourself a bad design.

Wow. There's just no arguing with that one. I mean, knowing the actual type on which the method is being dispatched is such a huge part of the C# development experience:

   1: static void DoSomething()

   2: {

   3:     List<Point> ptList = new List<Point>();

   4:     ptList.Add(new Point(12, 12));

   5:     object o = ptList[0];

   6:     Console.WriteLine(o.ToString());

   7: }

Gah. Just the thought of not knowing the concrete type on which the method is being dispatched gives me the heebie-jeebies.

Just because the framework allows you use weak-typing and late-binding, doesn't mean you should - nor should you endorse it's use, in my opinion.

Somebody better tell all those users of NHibernate, NUnit, Spring.NET, MEF and all those other Reflection-based tools... including WinForms, ASP.NET, WPF, Workflow and WCF, come to think about it... that they're using frameworks that clearly were designed by idiots. (The gall of those people.)

I'm just saying, it's a shame that popular "nerd celebrities" like you (and I mean zero offense by that!) - endorse all this loosey-goosey typing. I say that becuase I've never seen a single case where weak typing or late binding: A) made a design better or B) where it didn't make the component or application worse, because it was a looser design.

I'm so glad you were here to set Scott and me straight, Rob. Because otherwise, we might actually get something done. God forbid.

Little tidbits of thought for those who are still thinking about this one.

• Ola Bini describes the application of the right language at the right level of the stack as a three-layer pyramid.
• Any C# or Java developer who's not writing unit tests to test their code "because the compiler will catch all those errors" and provide "tough love" needs to be fired. Immediately. I cannot conceive of a situation where unit tests can be passed over in favor of static typing in a professionally-responsible development project. (Oh, don't mis-read that, I can see lots of situations where unit tests aren't necessary. But not on code that's going to reach Production.)
• The argument for the degree of static typing in C# or Java is completely indefensible compared to what statically-typed type-inferenced languages like Haskell, F# or Scala provide. And their syntax frequently looks like "let x = [ 1; 2; 3; 4; ]", which isn't all that far off from what a dynamically-typed language looks like, despite very very different things happening under the compiler's hood. Until you, the Statically-Typed Fundamentalist, have written code in a Haskell/ML-derived language, you have no right arguing the merits of static typing. (In fact, that's probably also true if you've never written code in Ruby, Python, or PowerShell, either.)
• There's lots more arguments the Static-Typing Fundamentalist can throw, by the way. I'm disappointed Rob never mentioned performance, for one—that's a classic line of attack, too. Never mind the fact that most of those guys are still looping down and doing other silly micro-optimizations because that's way C++ taught them to do it....
• Oh, and never ever show the Static Typing Fundamentalist an XML document and using something like XPath to extract data from it. They inevitably fall into XML Schema and the "if we just write the schema flexibly enough" and.... The last time I did that.... I still visit his gravesite, all these years later, and it still hurts, losing him that way.
• Java guys argued against dynamic typing for years, too... until they tried Groovy and JRuby and Clojure. Now.... not so much.

Peace out.

Cruising the Web late last night, I ran across "10 things you can do to advance your career as a developer", summarized below:

1. Build a PC
2. Participate in an online forum and help others
3. Man the help desk
4. Perform field service
5. Perform DBA functions
6. Perform all phases of the project lifecycle
7. Recognize and learn the latest technologies
8. Be an independent contractor
9. Lead a project, supervise, or manage
10. Seek additional education

I agreed with some of them, I disagreed with others, and in general felt like they were a little too high-level to be of real use. For example, "Seek additional education" seems entirely too vague: In what? How much? How often? And "Recognize and learn the latest technologies" is something like offering advice to the Olympic fencing silver medalist and saying, "You should have tried harder".

So, in the great spirit of "Not Invented Here", I present my own list; as usual, I welcome comment and argument. And, also as usual, caveats apply, since not everybody will be in precisely the same place and be looking for the same things. In general, though, whether you're looking to kick-start your career or just "kick it up a notch", I believe this list will help, because these ideas have been of help to me at some point or another in my own career.

10: Build a PC.

Yes, even developers have to know about hardware. More importantly, a developer at a small organization or team will find himself in a position where he has to take on some system administrator roles, and sometimes that means grabbing a screwdriver, getting a little dusty and dirty, and swapping hardware around. Having said this, though, once you've done it once or twice, leave it alone—the hardware game is an ever-shifting and ever-changing game (much like software is, surprise surprise), and it's been my experience that most of us only really have the time to pursue one or the other.

By the way, "PC" there is something of a generic term—build a Linux box, build a Windows box, or "build" a Mac OS box (meaning, buy a Mac Pro and trick it out a little—add more memory, add another hard drive, and so on), they all get you comfortable with snapping parts together, and discovering just how ridiculously simple the whole thing really is.

And for the record, once you've done it, go ahead and go back to buying pre-built systems or laptops—I've never found building a PC to be any cheaper than buying one pre-built. Particularly for PC systems, I prefer to use smaller local vendors where I can customize and trick out the box. If you're a Mac, that's not really an option unless you're into the "Hackintosh" thing, which is quite possibly the logical equivalent to "Build a PC". Having never done it myself, though, I can't say how useful that is as an educational action.

9: Pick a destination

Do you want to run a team of your own? Become an independent contractor? Teach programming classes? Speak at conferences? Move up into higher management and get out of the programming game altogether? Everybody's got a different idea of what they consider to be the "ideal" career, but it's amazing how many people don't really think about what they want their career path to be.

A wise man once said, "The journey of a thousand miles begins with a single step." I disagree: The journey of a thousand miles begins with the damn map. You have to know where you want to go, and a rough idea of how to get there, before you can really start with that single step. Otherwise, you're just wandering, which in itself isn't a bad thing, but isn't going to get you to a destination except by random chance. (Sometimes that's not a bad result, but at least then you're openly admitting that you're leaving your career in the hands of chance. If you're OK with that, skip to the next item. If you're not, read on.)

Lay out explicitly (as in, write it down someplace) what kind of job you're wanting to grow into, and then lay out a couple of scenarios that move you closer towards that goal. Can you grow within the company you're in? (Have others been able to?) Do you need to quit and strike out on your own? Do you want to lead a team of your own? (Are there new projects coming in to the company that you could put yourself forward as a potential tech lead?) And so on.

Once you've identified the destination, now you can start thinking about steps to get there.

If you want to become a speaker, put your name forward to give some presentations at the local technology user group, or volunteer to hold a "brown bag" session at the company. Sign up with Toastmasters to hone your speaking technique. Watch other speakers give technical talks, and see what they do that you don't, and vice versa.

If you want to be a tech lead, start by quietly assisting other members of the team get their work done. Help them debug thorny problems. Answer questions they have. Offer yourself up as a resource for dealing with hard problems.

If you want to slowly move up the management chain, look to get into the project management side of things. Offer to be a point of contact for the users. Learn the business better. Sit down next to one of your users and watch their interaction with the existing software, and try to see the system from their point of view.

And so on.

8: Be a bell curve

Frequently, at conferences, attendees ask me how I got to know so much on so many things. In some ways, I'm reminded of the story of a world-famous concert pianist giving a concert at Carnegie Hall—when a gushing fan said, "I'd give my life to be able to play like that", the pianist responded quietly, "I did". But as much as I'd like to leave you with the impression that I've dedicated my entire life to knowing everything I could about this industry, that would be something of a lie. The truth is, I don't know anywhere near as much as I'd like, and I'm always poking my head into new areas. Thank God for my ADD, that's all I can say on that one.

For the rest of you, though, that's not feasible, and not really practical, particularly since I have an advantage that the "working" programmer doesn't—I have set aside weeks or months in which to do nothing more than study a new technology or language.

Back in the early days of my career, though, when I was holding down the 9-to-5, I was a Windows/C++ programmer. I was working with the Borland C++ compiler and its associated framework, the ObjectWindows Library (OWL), extending and maintaining applications written in it. One contracting client wanted me to work with Microsoft MFC instead of OWL. Another one was storing data into a relational database using ODBC. And so on. Slowly, over time, I built up a "bell curve"-looking collection of skills that sort of "hovered" around the central position of C++/Windows.

Then, one day, a buddy of mine mentioned the team on which he was a project manager was looking for new blood. They were doing web applications, something with which I had zero experience—this was completely outside of my bell curve. HTML, HTTP, Cold Fusion, NetDynamics (an early Java app server), this was way out of my range, though at least NetDynamics was a little similar, since it was basically a server-side application framework, and I had some experience with app frameworks from my C++ days. So, resting on my C++ experience, I started flirting with Java, and so on.

Before long, my "bell curve" had been readjusted to have Java more or less at its center, and I found that experience in C++ still worked out here—what I knew about ODBC turned out to be incredibly useful in understanding JDBC, what I knew about DLLs from Windows turned out to be helpful in understanding Java's dynamic loading model, and of course syntactically Java looked a lot like C++ even though it behaved a little bit differently under the hood. (One article author suggested that Java was closer to Smalltalk than C++, and that prompted me to briefly flirt with Smalltalk before I concluded said author was out of his frakking mind.)

All of this happened over roughly a three-year period, by the way.

The point here is that you won't be able to assimilate the entire industry in a single sitting, so pick something that's relatively close to what you already know, and use your experience as a springboard to learn something that's new, yet possibly-if-not-probably useful to your current job. You don't have to be a deep expert in it, and the further away it is from what you do, the less you really need to know about it (hence the bell curve metaphor), but you're still exposing yourself to new ideas and new concepts and new tools/technologies that still could be applicable to what you do on a daily basis. Over time the "center" of your bell curve may drift away from what you've done to include new things, and that's OK.

7: Learn one new thing every year

In the last tip, I told you to branch out slowly from what you know. In this tip, I'm telling you to go throw a dart at something entirely unfamiliar to you and learn it. Yes, I realize this sounds contradictory. It's because those who stick to only what they know end up missing the radical shifts of direction that the industry hits every half-decade or so until it's mainstream and commonplace and "everybody's doing it".

In their amazing book "The Pragmatic Programmer", Dave Thomas and Andy Hunt suggest that you learn one new programming language every year. I'm going to amend that somewhat—not because there aren't enough languages in the world to keep you on that pace for the rest of your life—far from it, if that's what you want, go learn Ruby, F#, Scala, Groovy, Clojure, Icon, Io, Erlang, Haskell and Smalltalk, then come back to me for the list for 2020—but because languages aren't the only thing that we as developers need to explore. There's a lot of movement going on in areas beyond languages, and you don't want to be the last kid on the block to know they're happening.

Consider this list: object databases (db4o) and/or the "NoSQL" movement (MongoDB). Dependency injection and composable architectures (Spring, MEF). A dynamic language (Ruby, Python, ECMAScript). A functional language (F#, Scala, Haskell). A Lisp (Common Lisp, Clojure, Scheme, Nu). A mobile platform (iPhone, Android). "Space"-based architecture (Gigaspaces, Terracotta). Rich UI platforms (Flash/Flex, Silverlight). Browser enhancements (AJAX, jQuery, HTML 5) and how they're different from the rich UI platforms. And this is without adding any of the "obvious" stuff, like Cloud, to the list.

(I'm not convinced Cloud is something worth learning this year, anyway.)

You get through that list, you're operating outside of your comfort zone, and chances are, your boss' comfort zone, which puts you into the enviable position of being somebody who can advise him around those technologies. DO NOT TAKE THIS TO MEAN YOU MUST KNOW THEM DEEPLY. Just having a passing familiarity with them can be enough. DO NOT TAKE THIS TO MEAN YOU SHOULD PROPOSE USING THEM ON THE NEXT PROJECT. In fact, sometimes the most compelling evidence that you really know where and when they should be used is when you suggest stealing ideas from the thing, rather than trying to force-fit the thing onto the project as a whole.

6: Practice, practice, practice

Speaking of the concert pianist, somebody once asked him how to get to Carnegie Hall. HIs answer: "Practice, my boy, practice."

The same is true here. You're not going to get to be a better developer without practice. Volunteer some time—even if it's just an hour a week—on an open-source project, or start one of your own. Heck, it doesn't even have to be an "open source" project—just create some requirements of your own, solve a problem that a family member is having, or rewrite the project you're on as an interesting side-project. Do the Nike thing and "Just do it". Write some Scala code. Write some F# code. Once you're past "hello world", write the Scala code to use db4o as a persistent storage. Wire it up behind Tapestry. Or write straight servlets in Scala. And so on.

5: Turn off the TV

Speaking of marketing slogans, if you're like most Americans, surveys have shown that you watch about four hours of TV a day, or 28 hours of TV a week. In that same amount of time (28 hours over 1 week), you could read the entire set of poems by Maya Angelou, one F. Scott Fitzgerald novel, all poems by T.S.Eliot, 2 plays by Thornton Wilder, or all 150 Psalms of the Bible. An average reader, reading just one hour a day, can finish an "average-sized" book (let's assume about the size of a novel) in a week, which translates to 52 books a year.

Let's assume a technical book is going to take slightly longer, since it's a bit deeper in concept and requires you to spend some time experimenting and typing in code; let's assume that reading and going through the exercises of an average technical book will require 4 weeks (a month) instead of just one week. That's 12 new tools/languages/frameworks/ideas you'd be learning per year.

All because you stopped watching David Caruso turn to the camera, whip his sunglasses off and say something stupid. (I guess it's not his fault; CSI:Miami is a crap show. The other two are actually not bad, but Miami just makes me retch.)

After all, when's the last time that David Caruso or the rest of that show did anything that was even remotely realistic from a computer perspective? (I always laugh out loud every time they run a database search against some national database on a completely non-indexable criteria—like a partial license plate number—and it comes back in seconds. What the hell database are THEY using? I want it!) Soon as you hear The Who break into that riff, flip off the TV (or set it to mute) and pick up the book on the nightstand and boost your career. (And hopefully sink Caruso's.)

Or, if you just can't give up your weekly dose of Caruso, then put the book in the bathroom. Think about it—how much time do you spend in there a week?

And this gets even better when you get a Kindle or other e-reader that accepts PDFs, or the book you're interested in is natively supported in the e-readers' format. Now you have it with you for lunch, waiting at dinner for your food to arrive, or while you're sitting guard on your 10-year-old so he doesn't sneak out of his room after his bedtime to play more XBox.

4: Have a life

Speaking of XBox, don't slave your life to work. Pursue other things. Scientists have repeatedly discovered that exercise helps keep the mind in shape, so take a couple of hours a week (buh-bye, American Idol) and go get some exercise. Pick up a new sport you've never played before, or just go work out at the gym. (This year I'm doing Hopkido and fencing.) Read some nontechnical books. (I recommend anything by Malcolm Gladwell as a starting point.) Spend time with your family, if you have one—mine spends at least six or seven hours a week playing "family games" like Settlers of Catan, Dominion, To Court The King, Munchkin, and other non-traditional games, usually over lunch or dinner. I also belong to an informal "Game Night club" in Redmond consisting of several Microsoft employees and their families, as well as outsiders. And so on. Heck, go to a local bar and watch the game, and you'll meet some really interesting people. And some boring people, too, but you don't have to talk to them during the next game if you don't want.

This isn't just about maintaining a healthy work-life balance—it's also about having interests that other people can latch on to, qualities that will make you more "human" and more interesting as a person, and make you more attractive and "connectable" and stand out better in their mind when they hear that somebody they know is looking for a software developer. This will also help you connect better with your users, because like it or not, they do not get your puns involving Klingon. (Besides, the geek stereotype is SO 90's, and it's time we let the world know that.)

Besides, you never know when having some depth in other areas—philosophy, music, art, physics, sports, whatever—will help you create an analogy that will explain some thorny computer science concept to a non-technical person and get past a communication roadblock.

3: Practice on a cadaver

Long before they scrub up for their first surgery on a human, medical students practice on dead bodies. It's grisly, it's not something we really want to think about, but when you're the one going under the general anesthesia, would you rather see the surgeon flipping through the "How-To" manual, "just to refresh himself"?

Diagnosing and debugging a software system can be a hugely puzzling trial, largely because there are so many possible "moving parts" that are creating the problem. Compound that with certain bugs that only appear when multiple users are interacting at the same time, and you've got a recipe for disaster when a production bug suddenly threatens to jeopardize the company's online revenue stream. Do you really want to be sitting in the production center, flipping through "How-To"'s and FAQs online while your boss looks on and your CEO is counting every minute by the thousands of dollars?

Take a tip from the med student: long before the thing goes into production, introduce a bug, deploy the code into a virtual machine, then hand it over to a buddy and let him try to track it down. Have him do the same for you. Or if you can't find a buddy to help you, do it to yourself (but try not to cheat or let your knowledge of where the bug is color your reactions). How do you know the bug is there? Once you know it's there, how do you determine what kind of bug it is? Where do you start looking for it? How would you track it down without attaching a debugger or otherwise disrupting the system's operations? (Remember, we can't always just attach an IDE and step through the code on a production server.) How do you patch the running system? And so on.

Remember, you can either learn these things under controlled circumstances, learn them while you're in the "hot seat", so to speak, or not learn them at all and see how long the company keeps you around.

2: Administer the system

Take off your developer hat for a while—a week, a month, a quarter, whatever—and be one of those thankless folks who have to keep the system running. Wear the pager that goes off at 3AM when a server goes down. Stay all night doing one of those "server upgrades" that have to be done in the middle of the night because the system can't be upgraded while users are using it. Answer the phones or chat requests of those hapless users who can't figure out why they can't find the record they just entered into the system, and after a half-hour of thinking it must be a bug, ask them if they remembered to check the "Save this record" checkbox on the UI (which had to be there because the developers were told it had to be there) before submitting the form. Try adding a user. Try removing a user. Try changing the user's password. Learn what a real joy having seven different properties/XML/configuration files scattered all over the system really is.

Once you've done that, particularly on a system that you built and tossed over the fence into production and thought that was the end of it, you'll understand just why it's so important to keep the system administrators in mind when you're building a system for production. And why it's critical to be able to have a system that tells you when it's down, instead of having to go hunting up the answer when a VP tells you it is (usually because he's just gotten an outage message from a customer or client).

1: Cultivate a peer group

Yes, you can join an online forum, ask questions, answer questions, and learn that way, but that's a poor substitute for physical human contact once in a while. Like it or not, various sociological and psychological studies confirm that a "connection" is really still best made when eyeballs meet flesh. (The "disassociative" nature of email is what makes it so easy to be rude or flamboyant or downright violent in email when we would never say such things in person.) Go to conferences, join a user group, even start one of your own if you can't find one. Yes, the online avenues are still open to you—read blogs, join mailing lists or newsgroups—but don't lose sight of human-to-human contact.

While we're at it, don't create a peer group of people that all look to you for answers—as flattering as that feels, and as much as we do learn by providing answers, frequently we rise (or fall) to the level of our peers—have at least one peer group that's overwhelmingly smarter than you, and as scary as it might be, venture to offer an answer or two to that group when a question comes up. You don't have to be right—in fact, it's often vastly more educational to be wrong. Just maintain an attitude that says "I have no ego wrapped up in being right or wrong", and take the entire experience as a learning opportunity.

I'm excited to say that TechEd has accepted my pre-conference proposal, Multiparadigmatic C#, where the abstract reads:

C# has grown from “just” an object-oriented language into a language that is capable of expressing several different paradigms of software development: object-oriented, functional, and dynamic. In this session, developers will learn how to approach programming in C# to use each of these approaches, and when.

If you're interested in seeing C# used in a variety of different ways, come on out.

And if you're not going to TechEd.... why not? It's in New Orleans, folks!

But too impatient to read a whole book on it? Try the 6-panel RefCard that Chance Coble and I put together for DZone. Free download.

Or, for the more patient type, wait for the books that Chance and I (Professional F#) are each writing; they're remarkably complementary, at least from what Chance has told me about his.

Which reminds me.... if you've not already noticed, Pro F# is now up in Amazon. Call me a romantic fool, but I get just a little thrill run down my spine every time a new book of mine shows up on Amazon, and just a slightly bigger one when it shows up on a shelf (which will happen shortly after VS 2010 hits the streets). Nothing like that little surge of energy to give you the boost you need to cross the finish line.

Here we go again—another year, another set of predictions revisited and offered up for the next 12 months. And maybe, if I'm feeling really ambitious, I'll take that shot I thought about last year and try predicting for the decade. Without further ado, I'll go back and revisit, unedited, my predictions for 2009 ("THEN"), and pontificate on those subjects for 2010 before adding any new material/topics. Just for convenience, here's a link back to last years' predictions.

Last year's predictions went something like this (complete with basketball-scoring):

• THEN: "Cloud" will become the next "ESB" or "SOA", in that it will be something that everybody will talk about, but few will understand and even fewer will do anything with. (Considering the widespread disparity in the definition of the term, this seems like a no-brainer.) NOW: Oh, yeah. Straight up. I get two points for this one. Does anyone have a working definition of "cloud" that applies to all of the major vendors' implementations? Ted, 2; Wrongness, 0.
• THEN: Interest in Scala will continue to rise, as will the number of detractors who point out that Scala is too hard to learn. NOW: Two points for this one, too. Not a hard one, mind you, but one of those "pass-and-shoot" jumpers from twelve feet out. James Strachan even tweeted about this earlier today, pointing out this comparison. As more Java developers who think of themselves as smart people try to pick up Scala and fail, the numbers of sour grapes responses like "Scala's too complex, and who needs that functional stuff anyway?" will continue to rise in 2010. Ted, 4; Wrongness, 0.
• THEN: Interest in F# will continue to rise, as will the number of detractors who point out that F# is too hard to learn. (Hey, the two really are cousins, and the fortunes of one will serve as a pretty good indication of the fortunes of the other, and both really seem to be on the same arc right now.) NOW: Interestingly enough, I haven't heard as many F# detractors as Scala detractors, possibly because I think F# hasn't really reached the masses of .NET developers the way that Scala has managed to find its way in front of Java developers. I think that'll change mighty quickly in 2010, though, once VS 2010 hits the streets. Ted, 4; Wrongness 2.
• THEN: Interest in all kinds of functional languages will continue to rise, and more than one person will take a hint from Bob "crazybob" Lee and liken functional programming to AOP, for good and for ill. People who took classes on Haskell in college will find themselves reaching for their old college textbooks again. NOW: Yep, I'm claiming two points on this one, if only because a bunch of Haskell books shipped this year, and they'll be the last to do so for about five years after this. (By the way, does anybody still remember aspects?) But I'm going the opposite way with this one now; yes, there's Haskell, and yes, there's Erlang, and yes, there's a lot of other functional languages out there, but who cares? They're hard to learn, they don't always translate well to other languages, and developers want languages that work on the platform they use on a daily basis, and that means F# and Scala or Clojure, or its simply not an option. Ted 6; Wrongness 2.
• THEN: The iPhone is going to be hailed as "the enterprise development platform of the future", and companies will be rolling out apps to it. Look for Quicken iPhone edition, PowerPoint and/or Keynote iPhone edition, along with connectors to hook the iPhone up to a presentation device, and (I'll bet) a World of Warcraft iPhone client (legit or otherwise). iPhone is the new hotness in the mobile space, and people will flock to it madly. NOW: Two more points, but let's be honest—this was a fast-break layup, no work required on my part. Ted 8; Wrongness 2.
• THEN: Another Oslo CTP will come out, and it will bear only a superficial resemblance to the one that came out in October at PDC. Betting on Oslo right now is a fools' bet, not because of any inherent weakness in the technology, but just because it's way too early in the cycle to be thinking about for anything vaguely resembling production code. NOW: If you've worked at all with Oslo, you might argue with me, but I'm still taking my two points. The two CTPs were pretty different in a number of ways. Ted 10; Wrongness 2.
• THEN: The IronPython and IronRuby teams will find some serious versioning issues as they try to manage the DLR versioning story between themselves and the CLR as a whole. An initial hack will result, which will be codified into a standard practice when .NET 4.0 ships. Then the next release of IPy or IRb will have to try and slip around its restrictions in 2010/2011. By 2012, IPy and IRb will have to be shipping as part of Visual Studio just to put the releases back into lockstep with one another (and the rest of the .NET universe). NOW: Pressure is still building. Let's see what happens by the time VS 2010 ships, and then see what the IPy/IRb teams start to do to adjust to the versioning issues that arise. Ted 8; Wrongness 2.
• THEN: The death of JSR-277 will spark an uprising among the two leading groups hoping to foist it off on the Java community--OSGi and Maven--while the rest of the Java world will breathe a huge sigh of relief and look to see what "modularity" means in Java 7. Some of the alpha geeks in Java will start using--if not building--JDK 7 builds just to get a heads-up on its impact, and be quietly surprised and, I dare say, perhaps even pleased. NOW: Ah, Ted, you really should never underestimate the community's willingness to take a bad idea, strip all the goodness out of it, and then cycle it back into the mix as something completely different yet somehow just as dangerous and crazy. I give you Project Jigsaw. Ted 10; Wrongness 2;
• THEN: The invokedynamic JSR will leapfrog in importance to the top of the list. NOW: The invokedynamic JSR begat interest in other languages on the JVM. The interest in other languages on the JVM begat the need to start thinking about how to support them in the Java libraries. The need to start thinking about supporting those languages begat a "Holy sh*t moment" somewhere inside Sun and led them to (re-)propose closures for JDK 7. And in local sports news, Ted notched up two more points on the scoreboard. Ted 12; Wrongness 2.
• THEN: Another Windows 7 CTP will come out, and it will spawn huge media interest that will eventually be remembered as Microsoft promises, that will eventually be remembered as Microsoft guarantees, that will eventually be remembered as Microsoft FUD and "promising much, delivering little". Microsoft ain't always at fault for the inflated expectations people have--sometimes, yes, perhaps even a lot of times, but not always. NOW: And then, just when the game started to turn into a runaway, airballs started to fly. The Windows7 release shipped, and contrary to what I expected, the general response to it was pretty warm. Yes, there were a few issues that emerged, but overall the media liked it, the masses liked it, and Microsoft seemed to have dodged a bullet. Ted 12; Wrongness 5.
• THEN: Apple will begin to legally threaten the clone market again, except this time somebody's going to get the DOJ involved. (Yes, this is the iPhone/iTunes prediction from last year, carrying over. I still expect this to happen.) NOW: What clones? The only people trying to clone Macs are those who are building Hackintosh machines, and Apple can't sue them so long as they're using licensed copies of Mac OS X (as far as I know). Which has never stopped them from trying, mind you, and I still think Steve has some part of his brain whispering to him at night, calculating all the hardware sales lost to Hackintosh netbooks out there. But in any event, that's another shot missed. Ted 12; Wrongness 7.
• THEN: Alpha-geek developers will start creating their own languages (even if they're obscure or bizarre ones like Shakespeare or Ook#) just to have that listed on their resume as the DSL/custom language buzz continues to build. NOW: I give you Ioke. If I'd extended this to include outdated CPU interpreters, I'd have made that three-pointer from half-court instead of just the top of the key. Ted 14; Wrongness 7.
• THEN: Roy Fielding will officially disown most of the "REST"ful authors and software packages available. Nobody will care--or worse, somebody looking to make a name for themselves will proclaim that Roy "doesn't really understand REST". And they'll be right--Roy doesn't understand what they consider to be REST, and the fact that he created the term will be of no importance anymore. Being "REST"ful will equate to "I did it myself!", complete with expectations of a gold star and a lollipop. NOW: Does anybody in the REST community care what Roy Fielding wrote way back when? I keep seeing "REST"ful systems that seem to have designers who've never heard of Roy, or his thesis. Roy hasn't officially disowned them, but damn if he doesn't seem close to it. Still.... No points. Ted 14; Wrongness 9.
• THEN: The Parrot guys will make at least one more minor point release. Nobody will notice or care, except for a few doggedly stubborn Perl hackers. They will find themselves having nightmares of previous lives carrying around OS/2 books and Amiga paraphernalia. Perl 6 will celebrate it's seventh... or is it eighth?... anniversary of being announced, and nobody will notice. NOW: Does anybody still follow Perl 6 development? Has the spec even been written yet? Google on "Perl 6 release", and you get varying reports: "It'll ship 'when it's ready'", "There are no such dates because this isn't a commericially-backed effort", and "Spring 2010". Swish—nothin' but net. Ted 16; Wrongness 9.
• THEN: The debate around "Scrum Certification" will rise to a fever pitch as short-sighted money-tight companies start looking for reasons to cut costs and either buy into agile at a superficial level and watch it fail, or start looking to cut the agilists from their company in order to replace them with cheaper labor. NOW: Agile has become another adjective meaning "best practices", and as such, has essentially lost its meaning. Just ask Scott Bellware. Ted 18; Wrongness 9.
• THEN: Adobe will continue to make Flex and AIR look more like C# and the CLR even as Microsoft tries to make Silverlight look more like Flash and AIR. Web designers will now get to experience the same fun that back-end web developers have enjoyed for near-on a decade, as shops begin to artificially partition themselves up as either "Flash" shops or "Silverlight" shops. NOW: Not sure how to score this one—I haven't seen the explicit partitioning happen yet, but the two environments definitely still seem to be looking to start tromping on each others' turf, particularly when we look at the rapid releases coming from the Silverlight team. Ted 16; Wrongness 11.
• THEN: Gartner will still come knocking, looking to hire me for outrageous sums of money to do nothing but blog and wax prophetic. NOW: Still no job offers. Damn. Ah, well. Ted 16; Wrongness 13.

A close game. Could've gone either way. *shrug* Ah, well. It was silly to try and score it in basketball metaphor, anyway—that's the last time I watch ESPN before writing this.

For 2010, I predict....

• ... I will offer 3- and 4-day training classes on F# and Scala, among other things. OK, that's not fair—yes, I have the materials, I just need to work out locations and times. Contact me if you're interested in a private class, by the way.
• ... I will publish two books, one on F# and one on Scala. OK, OK, another plug. Or, rather, more of a resolution. One will be the "Professional F#" I'm doing for Wiley/Wrox, the other isn't yet finalized. But it'll either be published through a publisher, or self-published, by JavaOne 2010.
• ... DSLs will either "succeed" this year, or begin the short slide into the dustbin of obscure programming ideas. Domain-specific language advocates have to put up some kind of strawman for developers to learn from and poke at, or the whole concept will just fade away. Martin's book will help, if it ships this year, but even that might not be enough to generate interest if it doesn't have some kind of large-scale applicability in it. Patterns and refactoring and enterprise containers all had a huge advantage in that developers could see pretty easily what the problem was they solved; DSLs haven't made that clear yet.
• ... functional languages will start to see a backlash. I hate to say it, but "getting" the functional mindset is hard, and there's precious few resources that are making it easy for mainstream (read: O-O) developers make that adjustment, far fewer than there was during the procedural-to-object shift. If the functional community doesn't want to become mainstream, then mainstream developers will find ways to take functional's most compelling gateway use-case (parallel/concurrent programming) and find a way to "git 'er done" in the traditional O-O approach, probably through software transactional memory, and functional languages like Haskell and Erlang will be relegated to the "What Might Have Been" of computer science history. Not sure what I mean? Try this: walk into a functional language forum, and ask what a monad is. Nobody yet has been able to produce an answer that doesn't involve math theory, or that does involve a practical domain-object-based example. In fact, nobody has really said why (or if) monads are even still useful. Or catamorphisms. Or any of the other dime-store words that the functional community likes to toss around.
• ... Visual Studio 2010 will ship on time, and be one of the buggiest and/or slowest releases in its history. I hate to make this prediction, because I really don't want to be right, but there's just so much happening in the Visual Studio refactoring effort that it makes me incredibly nervous. Widespread adoption of VS2010 will wait until SP1 at the earliest. In fact....
• ... Visual Studio 2010 SP 1 will ship within three months of the final product. Microsoft knows that people wait until SP 1 to think about upgrading, so they'll just plan for an eager SP 1 release, and hope that managers will be too hung over from the New Year (still) to notice that the necessary shakeout time hasn't happened.
• ... Apple will ship a tablet with multi-touch on it, and it will flop horribly. Not sure why I think this, but I just don't think the multi-touch paradigm that Apple has cooked up for the iPhone will carry over to a tablet/laptop device. That won't stop them from shipping it, and it won't stop Apple fan-boiz from buying it, but that's about where the interest will end.
• ... JDK 7 closures will be debated for a few weeks, then become a fait accompli as the Java community shrugs its collective shoulders. Frankly, I think the Java community has exhausted its interest in debating new language features for Java. Recent college grads and open-source groups with an axe to grind will continue to try and make an issue out of this, but I think the overall Java community just... doesn't... care. They just want to see JDK 7 ship someday.
• ... Scala either "pops" in 2010, or begins to fall apart. By "pops", I mean reaches a critical mass of developers interested in using it, enough to convince somebody to create a company around it, a la G2One.
• ... Oracle is going to make a serious "cloud" play, probably by offering an Oracle-hosted version of Azure or AppEngine. Oracle loves the enterprise space too much, and derives too much money from it, to not at least appear to have some kind of offering here. Now that they own Java, they'll marry it up against OpenSolaris, the Oracle database, and throw the whole thing into a series of server centers all over the continent, and call it "Oracle 12c" (c for Cloud, of course) or something.
• ... Spring development will slow to a crawl and start to take a left turn toward cloud ideas. VMWare bought SpringSource for a reason, and I believe it's entirely centered around VMWare's movement into the cloud space—they want to be more than "just" a virtualization tool. Spring + Groovy makes a compelling development stack, particularly if VMWare does some interesting hooks-n-hacks to make Spring a virtualization environment in its own right somehow. But from a practical perspective, any community-driven development against Spring is all but basically dead. The source may be downloadable later, like the VMWare Player code is, but making contributions back? Fuhgeddabowdit.
• ... the explosion of e-book readers brings the Kindle 2009 edition way down to size. The era of the e-book reader is here, and honestly, while I'm glad I have a Kindle, I'm expecting that I'll be dusting it off a shelf in a few years. Kinda like I do with my iPods from a few years ago.
• ... "social networking" becomes the "Web 2.0" of 2010. In other words, using the term will basically identify you as a tech wannabe and clearly out of touch with the bleeding edge.
• ... Facebook becomes a developer platform requirement. I don't pretend to know anything about Facebook—I'm not even on it, which amazes my family to no end—but clearly Facebook is one of those mechanisms by which people reach each other, and before long, it'll start showing up as a developer requirement for companies looking to hire. If you're looking to build out your resume to make yourself attractive to companies in 2010, mad Facebook skillz might not be a bad investment.
• ... Nintendo releases an open SDK for building games for its next-gen DS-based device. With the spectacular success of games on the iPhone, Nintendo clearly must see that they're missing a huge opportunity every day developers can't write games for the Nintendo DS that are easily downloadable to the device for playing. Nintendo is not stupid—if they don't open up the SDK and promote "casual" games like those on the iPhone and those that can now be downloaded to the Zune or the XBox, they risk being marginalized out of existence.

And for the next decade, I predict....

• ... colleges and unversities will begin issuing e-book reader devices to students. It's a helluvalot cheaper than issuing laptops or netbooks, and besides....
• ... netbooks and e-book readers will merge before the decade is out. Let's be honest—if the e-book reader could do email and browse the web, you have almost the perfect paperback-sized mobile device. As for the credit-card sized mobile device....
• ... mobile phones will all but disappear as they turn into what PDAs tried to be. "The iPhone makes calls? Really? You mean Voice-over-IP, right? No, wait, over cell signal? It can do that? Wow, there's really an app for everything, isn't there?"
• ... wireless formats will skyrocket in importance all around the office and home. Combine the iPhone's Bluetooth (or something similar yet lower-power-consuming) with an equally-capable (Bluetooth or otherwise) projector, and suddenly many executives can leave their netbook or laptop at home for a business presentation. Throw in the Whispersync-aware e-book reader/netbook-thing, and now most executives have absolutely zero reason to carry anything but their e-book/netbook and their phone/PDA. The day somebody figures out an easy way to combine Bluetooth with PayPal on the iPhone or Android phone, we will have more or less made pocket change irrelevant. And believe me, that day will happen before the end of the decade.
• ... either Android or Windows Mobile will gain some serious market share against the iPhone the day they figure out how to support an open and unrestricted AppStore-like app acquisition model. Let's be honest, the attraction of iTunes and AppStore is that I can see an "Oh, cool!" app on a buddy's iPhone, and have it on mine less than 30 seconds later. If Android or WinMo can figure out how to offer that same kind of experience without the draconian AppStore policies to go with it, they'll start making up lost ground on iPhone in a hurry.
• ... Apple becomes the DOJ target of the decade. Microsoft was it in the 2000's, and Apple's stunning rising success is going to put it squarely in the sights of monopolist accusations before long. Coupled with the unfortunate health distractions that Steve Jobs has to deal with, Apple's going to get hammered pretty hard by the end of the decade, but it will have mastered enough market share and mindshare to weather it as Microsoft has.
• ... Google becomes the next Microsoft. It won't be anything the founders do, but Google will do "something evil", and it will be loudly and screechingly pointed out by all of Google's corporate opponents, and the star will have fallen.
• ... Microsoft finds its way again. Microsoft, as a company, has lost its way. This is a company that's not used to losing, and like Bill Belichick's Patriots, they will find ways to adapt and adjust to the changed circumstances of their position to find a way to win again. What that'll be, I have no idea, but historically, the last decade notwithstanding, betting against Microsoft has historically been a bad idea. My gut tells me they'll figure something new to get that mojo back.
• ... a politician will make himself or herself famous by standing up to the TSA. The scene will play out like this: during a Congressional hearing on airline security, after some nut/terrorist tries to blow up another plane through nitroglycerine-soaked underwear, the TSA director will suggest all passengers should fly naked in order to preserve safety, the congressman/woman will stare open-mouthed at this suggestion, proclaim, "Have you no sense of decency, sir?" and immediately get a standing ovation and never have to worry about re-election again. Folks, if we want to prevent any chance of loss of life from a terrorist act on an airplane, we have to prevent passengers from getting on them. Otherwise, just accept that it might happen, do a reasonable job of preventing it from happening, and let private insurance start offering flight insurance against the possibility to reassure the paranoid.

See you all next year.

Why study new and different programming languages? To change your programming mindset. Not sure what I mean by that? Check this out.

Ever done one of these?

   1: public interface IService

   2: {

   3:   DateTime GetDate();

   4:   int CalculateSomethingInteresting(int lhs, int rhs);

   5: }

   6:  

   7: public class OneServiceImpl : IService

   8: {

   9:   public DateTime GetDate()

  10:   { return DateTime.Now; }

  11:   public int CalculateSomethingInteresting(int lhs, int rhs)

  12:   { return lhs + rhs; }

  13: }

  14:  

  15: public class AnotherServiceImpl : IService

  16: {

  17:   public DateTime GetDate()

  18:   { return new DateTime(); }

  19:   public int CalculateSomethingInteresting(int lhs, int rhs)

  20:   { return lhs * rhs; }

  21: }

  22:  

  23: public class ServiceFactory

  24: {

  25:   public static IService GetInstance(string which)

  26:   {

  27:     if (which == "One") return new OneServiceImpl();

  28:     else if (which == "Another") return new AnotherServiceImpl();

  29:     else throw new ArgumentException();

  30:   }

  31: }

  32:  

  33: public class App

  34: {

  35:   public static void Main(string[] args)

  36:   {

  37:     foreach (string s in args)

  38:     {

  39:       IService serv = ServiceFactory.GetInstance(s);

  40:       Console.WriteLine("serv calc = {0}", serv.CalculateSomethingInteresting(3, 3));

  41:     }

  42:   }

  43: }

So has my client this week. In fact, it's fair to say that they're infatuated with them—they've got services all over the place, including at their communication layer, where they use configuration files to decide which of the two service implementations to use, either a "native" .NET implementation or the "real" Web services implementation that they're supposed to be using. (They end up going back to the native implementation because sometimes—which is to say, apparently a lot of times—the Web services implementation is broken in some fundamental way. Go figure.)

The problem is, very bluntly, that the interfaces they're defining (the IService definition above) are ever-so-slightly different from the communications-based proxy interfaces that they use to communicate outside of this process, so some poor schmuck ends up having to write the service implementation (OneServiceImpl) that simply takes the parameters passed in, translates them into a call through the communications-based interface, then takes the response and hands it back. Tedious, mind-numbing coding, particularly painful when there are dozens of interfaces with (in some cases) hundreds of methods per interface. Ouch.

There had to be a better way.

Based on some of the work/research/play I've been doing with both dynamic and functional programming languages, it occurred to me that what they really wanted was some kind of "forwarding" or "delegating" behavior that certain languages have baked in as a feature. In those languages, it's possible to nominate a "delegate" object to which method calls are automatically forwarded if no such method is implemented on this object; in this particular case, what I'd do to replace all of the above is simply create an IService object instance that has either a OneServiceImpl or a AnotherServiceImpl instance (depending on the value in the configuration file) set up as the "delegate" object. That way the method calls remain statically type-checked, but none of this service interface/service implementation/service factory nonsense has to be created just to switch between the two.

(By the way, all of this pain goes away completely in a language that supports deferred checking of signatures until runtime. In other words, if the client had been programming in IronPython or IronRuby or even Visual Basic, we could get away with not having to do any of the above, and just use Reflection to access the appropriate method on whichever of the two service implementations they want to use at the time. Fan would let us do it if we used "->" instead of "." to invoke the method; Cobra would switch between the two automatically; and so on.)

Now, this is C# 2.0 that they're using, and they're pretty entrenched on that point, so I can't simply suggest that they use a new language, but if we take the basic idea and adapt it to C#, we can get pretty much the same behavior without having to force the poor schmuck on the bottom of the totem pole to write all those service implementations by hand.

We start by transforming the IService interface into an IService "interface" (meaning it's not really an interface anymore, but it'll sure look like one to anybody who's not paying attention):

   1: public class IService

   2: {

   3:   public Func0<DateTime> GetDate;

   4:   public Func2<int, int, int> CalculateSomethingInteresting;

   5: }

IService is now a class with fields (not properties, though I suppose if you really wanted them to be properties you could make them such, not that I see much value to doing so), where each field corresponds in name to the method of the interface it wants to replace, and the type is a delegate type parameterized to match the return type and parameter types of that same method of the original interface. Func0 and Func2 are delegate types I had to create, since nothing like them existed until C# 3.0; their definitions are pretty simple:

   1: public delegate R Func0<R>();

   2: public delegate R Func1<R, P1>(P1 p1);

   3: public delegate R Func2<R, P1, P2>(P1 p1, P2 p2);

Now, assuming we have the implementation classes from before, we have two choices; one is to write a by-hand factory that fills out the fields to point to the appropriate method on the implementation class, like so:

   1: if (which == "One")

   2: {

   3:   servInstance.GetDate = delegate() { return DateTime.Now; };

   4:   servInstance.CalculateSomethingInteresting = delegate (int lhs, int rhs) { return lhs + rhs; };

   5: }

   6: else if (which == "Another")

   7: {

   8:   servInstance.GetDate = delegate() { return new DateTime(); };

   9:   servInstance.CalculateSomethingInteresting = delegate (int lhs, int rhs) { return lhs * rhs; };

  10: }

  11: else

  12:   throw new ArgumentException();

But, quite frankly, this defeats the point—the point was to avoid writing all this stuff by hand, not simply repeat it in a different form. So instead, we leverage Reflection, which depends on the basic assumption that the field name in the IService "interface" matches the method name on the implementation class we wish to invoke. Assuming that holds (which it does, in my client's case, anyway), we can reflect on the IService field, find the matching method name in the implementation, then construct a delegate instance around that method and assign the delegate instance to the field. Once complete, we hand back the completed service instance, and the client literally doesn't know that anything's different:

   1: public class ServiceFactory

   2: {

   3:   public static IService GetInstance(string which)

   4:   {

   5:     IService servInstance = new IService();

   6:  

   7:     Type targetType = Assembly.GetExecutingAssembly().GetType(which + "ServiceImpl");

   8:  

   9:     foreach (FieldInfo fi in servInstance.GetType().GetFields())

  10:     {

  11:       MethodInfo targetMethod = targetType.GetMethod(fi.Name);

  12:       //Console.WriteLine("Wiring up {0} against {1} with {2}", fi.Name, targetType, targetMethod);

  13:       Delegate d = Delegate.CreateDelegate(fi.FieldType, null, targetMethod);

  14:       //Console.WriteLine(d);

  15:       fi.SetValue(servInstance, d);

  16:     }

  17:  

  18:     return servInstance;

  19:   }

  20: }

Remember, the client code still looks the same...

   1: public class App

   2: {

   3:   public static void Main(string[] args)

   4:   {

   5:     foreach (string s in args)

   6:     {

   7:       IService serv = ServiceFactory.GetInstance(s);

   8:       Console.WriteLine("serv calc = {0}", serv.CalculateSomethingInteresting(3, 3));

   9:     }

  10:   }

  11: }

... because what the client doesn't know is that he's accessing a field, then invoking the delegate that's being returned from that field dereference.

What this permits, aside from the automated wiring up of the IService "interface", is a greater degree of flexibility—rather than having to choose which implementation to use on an interface-by-interface basis, we can now configure to use different implementations on a method-by-method basis. But considering how many interfaces and implementations my client was looking at having to write by hand, the real win is in the automated ServiceFactory wiring.

By the way, the only reason we can get away with this sleight-of-hand is because delegates are deliberately designed to act like method calls; no explicit .Invoke() call is required, it's implied with the () after the delegate instance's name. If Java7 closures and/or method handles end up with support for that kind of syntax, then we can do the same thing in Java7 (more or less).

Make sense?

Paul asked me to review this, his first book, and my comment to him was that he had a pretty high bar to match; being of the same "series" as Release It!, Mike Nygard's take on building software ready for production (and, in my repeatedly stated opinion, the most important-to-read book of the decade), Debug It! had some pretty impressive shoes to fill. Paul's comment was pretty predictable: "Thanks for keeping the pressure to a minimum."

My copy arrived in the mail while I was at the NFJS show in Denver this past weekend, and with a certain amount of dread and excitement, I opened the envelope and sat down to read for a few minutes. I managed to get halfway through it before deciding I had to post a review before I get too caught up in my next trip and forget.

Short version

Debug It! is a great resource for anyone looking to learn the science of good debugging. It is entirely language- and platform-agnostic, preferring to focus entirely on the process and mindset of debugging, rather than on edge cases or command-line switches in a tool or language. Overall, the writing is clear and straightforward without being preachy or judgmental, and is liberally annotated with real-life case stories from both the authors' and the Pragmatic Programmers' own history, which keeps the tone lighter and yet still proving the point of the text. Highly recommended for the junior developers on the team; senior developers will likely find some good tidbits in here as well.

Long version

Debug It! is an excellently-written and to-the-point description of the process of not only identifying and fixing defects in software, but also of the attitudes required to keep software from failing. Rather than simply tossing off old maxims or warming them over with new terminology ("You should always verify the parameters to your procedure calls" replaced with "You should always verify the parameters entering a method and ensure the fields follow the invariants established in the specification"), Paul ensures that when making a point, his prose is clear, the rationale carefully explained, and the consequences of not following this advice are clearly spelled out. His advice is pragmatic, and takes into account that developers can't always follow the absolute rules we'd like to—he talks about some of his experiences with "bug priorities" and how users pretty quickly figured out to always set the bug's priority at the highest level in order to get developer attention, for example, and some ways to try and address that all-too-human failing of bug-tracking systems.

It needs to be said, right from the beginning, that Debug It! will not teach you how to use the debugging features of your favorite IDE, however. This is because Paul (deliberately, it seems) takes a platform- and language-agnostic approach to the book—there are no examples of how to set breakpoints in gdb, or how to attach the Visual Studio IDE to a running Windows service, for example. This will likely weed out those readers who are looking for "Google-able" answers to their common debugging problems, and that's a shame, because those are probably the very readers that need to read this book. Having said that, however, I like this agnostic approach, because these ideas and thought processes, the ones that are entirely independent of the language or platform, are exactly the kinds of things that senior developers carry over with them from one platform to the next. Still, the junior developer who picks this book up is going to still need a reference manual or the user manual for their IDE or toolchain, and will need to practice some with both books in hand if they want to maximize the effectiveness of what's in here.

One of the things I like most about this book is that it is liberally adorned with real-life discussions of various scenarios the author team has experienced; the reason I say "author team" here is because although the stories (for the most part) remain unattributed, there are obvious references to "Dave" and "Andy", which I assume pretty obviously refer to Dave Thomas and Andy Hunt, the Pragmatic Programmers and the owners of Pragmatic Bookshelf. Some of the stories are humorous, and some of them probably would be humorous if they didn't strike so close to my own bitterly-remembered experiences. All of them do a good job of reinforcing the point, however, thus rendering the prose more effective in communicating the idea without getting to be too preachy or bombastic.

The book obviously intends to target a junior developer audience, because most senior developers have already intuitively (or experientially) figured out many of the processes described in here. But, quite frankly, I think it would be a shame for senior developers to pass on this one; though the temptation will be to simply toss it aside and say, "I already do all this stuff", senior developers should resist that urge and read it through cover to cover. If nothing else, it'll help reinforce certain ideas, bring some of the intuitive process more to light and allow us to analyze what we do right and what we do wrong, and perhaps most importantly, give us a common backdrop against which we can mentor junior developers in the science of debugging.

One of the chapters I like in particular, "Chapter 7: Pragmatic Zero Tolerance", is particularly good reading for those shops that currently suffer from a deficit of management support for writing good software. In it, Paul talks specifically about some of the triage process about bugs ("When to fix bugs"), the mental approach developers should have to fixing bugs ("The debugging mind-set") and how to get started on creating good software out of bad ("How to dig yourself out of a quality hole"). These are techniques that a senior developer can bring to the team and implement at a grass-roots level, in many cases without management even being aware of what's going on. (It's a sad state of affairs that we sometimes have to work behind management's back to write good-quality code, but I know that some developers out there are in exactly that situation, and simply saying, "Quit and find a new job", although pithy and good for a laugh on a panel, doesn't really offer much in the way of help. Paul doesn't take that route here, and that alone makes this book worth reading.)

Another of the chapters that resonates well with me is the first one in Part III ("Debug Fu"), Chapter 8, entitled "Special Cases", in which he tackles a number of "advanced" debugging topics, such as "Patching Existing Releases" and "Hesenbugs" (Concurrency-related bugs). I won't spoil the punchline for you, but suffice it to say that I wish I'd had that chapter on hand to give out to teammates on a few projects I've worked on in the past.

Overall, this book is going to be a huge win, and I think it's a worthy successor to the Release It! reputation. Development managers and team leads should get a copy for the junior developers on their team as a Christmas gift, but only after the senior developers have read through it as well. (Senior devs, don't despair—at 190 pages, you can rip through this in a single night, and I can almost guarantee that you'll learn a few ideas you can put into practice the next morning to boot.)

Those of you who've seen me speak on Java 7 at various conferences have heard me lament (in a small way) the fact that Sun decided last year (Dec 2008) to forgo the idea of including closures in the Java language. Imagine my surprise, then, to check my Twitter feed and discover that, to everyone's surprise, closures are back in as a consideration for the Java7 release.

Several thoughts come to mind:

• "WTF?!?!? This is a community effort?" Originally, when Sun created the Java Community Process, the tradeoff for a committee-based development process was against the open and fair inclusion of ideas from outside of Sun. But with the Java7 release still lacking a JSR (as of a few weeks ago, anyway; I haven't checked today to see if it was opened), and both the Modules facility and language extensions deferred to "Projects" (not JSRs), it seems Sun is now abandoning the JCP in favor of a Sun-dominant process that is certainly solicitous of the community at large, but not constrained or defined by it. And for the life of me, I can't tell if this is a good thing or a bad thing. It's good in that now we don't have to garner a critical mass of community momentum to get something included into the platform or language, but it's bad in that Sun has historically been the bigger drag on innovation there, not the community.
• "Can we please stop calling them closures?" This is a nit, but technically what we're talking about adding here are either lambda expressions or anonymous methods, depending on whose glossary you're using when you're talking. A true closure is one that will compute all referenced variables from the enclosing scope and automatically include them in the generated code, which (so far as I can tell) none of the Java anonymous method or lambda expression proposals currently include. But it's a nit, so I'll say it this once and then drop it.
• "Will Groovy, Scala, Clojure and all other JVM languages please report to the refactoring room?" People look at me quizzically when I say I'd like to see Java have closures in the language, because in general my take on language features in Java is that the Java language is more or less dead, and I could care less what happens to it; I'd vastly prefer to code in Groovy or Scala or Clojure or JRuby before writing something in Java. My rationale for wanting closures in Java, however, is this: by defining a common implementation for closures in Java, all of the above languages can refactor their implementations of anonymous methods/lambda expressions/etc into something that uses Java's closure implementation, and that'll make calling Groovy anonymous methods from Scala much much easier.
• "Why there, now?" Devoxx is apparently turning into JavaOne Winter, because Sun's been making a lot of pretty big announcements at that show, including last year's "no closures, no built-in XML support, ..." announcement about Java7, and now this year's "well, we lied, we're thinking about closures again". Fortunately I think the Devoxx folks have much better skills at keeping their conference relevant to the Java community than JavaOne's organizers did. And I say that despite the fact (or perhaps because of the fact) that I didn't speak there this year.
• "When is this all supposed to ship again?" Originally, my understanding was that JDK7 was slated to ship in the early part of 2010, but now rumor has it slipping to this time next year (2010). That is a huge postponement, and gives Microsoft a bit of an edge, since Visual Studio 2010 and .NET 4.0 are (again, according to rumor) supposed to ship somewhere around the end of 1Q2010. If Sun/Oracle keeps this up, we could very well be seeing a 2-.NET-releases-to-1-Java-release pattern, and that's disturbing in its own right. (Anybody else remember the days when Sun withdrew Java from ECMA, ISO and ANSI standardization consideration because they wanted to "innovate on the platform faster"?)
• "We really have no clue what we're talking about." Aside from rumors and hearsay (including the one that says that Mark Reinhold, who made the announcement, made up the syntax on the flight from the US to Belgium), we really don't have much by way of Sun-blessed official discussions of what this will look like or act like, at least none so far as I've been able to find, so any sort of supposition on whether it will be good or suck like an inverted hurricane is a tad premature. Trust me, I want to see where this goes, too, so I'll be keeping an eye out.

In the meantime, if you want to keep on top of the Java space, maybe it's time to consider a trip to Antwerp this time next year, since, if the new ship date rumors are to be believed,  it looks like Sun (once again) is planning to use Devoxx as the platform from which to make a large announcement, this time the release Java7 itself.

Update: Ola Bini noted that...

Two things:

• They are definitely closures. Calling them anonymous functions are incorrect, since they aren't really functions. Lambda expression is an OK name, but it has connotations that aren't really correct for a language like Java. A closure is defined as an anonymous piece of code that closes over at least one free variable, which in the case of this proposal will definitely happen. In fact, all of these will be closures, since they will be closing over the this at least.
• This is mostly on the level of compiler, syntax and type checking, and will NOT have any real implications for runtime. This means there will be no real sharing of implementation - at most JRuby, Groovy and Scala blocks will implement another interface (but all of them already implement Runnable and Callable so it's a limited win).

which prompted me to respond thusly:

First off, I actually never used the term "anonymous function"; instead, I said "anonymous method", which, as I understand it, is how the underlying implementation of these proposals will work: the syntax "#() return 42" will create an anonymous inner class instance of an interface defined by the library (in its "SimpleClosure" example, the BGGA compiler uses the interface "javax.lang.function.I", which has one method on it, "invoke()"), which, thus, makes this an anonymous method. We can't call them "anonymous functions" because Java has no function type, and probably never will. (And yes, it may seem like we're splitting hairs somewhat to differentiate between functions and methods,but once you've explored ML, Haskell, Scala, or F#, you really begin to see a huge difference in those terms, so it's important to be precise with our terminology, or else the conversation becomes almost entirely meaningless.)

Neal Gafter uses the definition "A closure is a function that captures the bindings of free variables in its lexical context." (http://gafter.blogspot.com/2007/01/definition-of-closures.html) Given that said same post also claims that Java has no function type (and therefore, by his definition, can't really have a closure), I suppose we could split the hairs even further and suggest that Java will never have closures until it has true function types. Personally, I'm happy to say that we can swap in "methods" for "functions" in this particular discussion, but my understanding is that capturing free variables also implies capturing variables referenced in the enclosing lexical context, which the current "closures" proposal (as reported by Alex Miller's closures page) will not do. (Non-final enclosing parameters will not be accessible, only those passed in formally as parameters. Stephen Colebourne reports as much: "[Mark Reinhold] also indicated that access to non-final variables was unlikely.")

Given that the current proposal suggests the new #() syntax will essentially generate an anonymous inner class with a method of the appropriate signature (though I do believe that method handles are targeted for use at some point, based on what I've been hearing through the rumor mill), to me it feels like the "closures" implementation is generating an anonymous method of an anonymous class with a few other restrictions included--hence my commentary above.

(Having said all that, the FCM proposal does provide complete capture of all referenced variables in enclosing scope, but Mark's keynote hasn't officially endorsed either the BGGA proposal or the FCM proposal, and if Sun keeps to their habits, they won't. They'll build something that's an amalgamation of all of them. Right now the current consensus seems to be to adopt the BGGA implementation behind the FCM syntax, which jives with Neal's 0.6a specification proposal.)

On top of that, the comment "all of these will be closures, since they will be closing over the this at least" is not, I don't think, entirely true. The details of the closures proposal aren't clear, but the "outer this" (which I believe is the "this" Ola refers to above) hasn't been explicitly mentioned in any of the closures proposals I've seen, nor have I seen any text suggesting that they will honor it, so I don't know that this is true. Of course, in absence of a specification or real working bits, all we can do is just speculate. However, having said that, playing around a bit with the BGGA prototype compiler (which, admittedly, is still one minor rev back from Neal's revised proposal), I saw no generated "outer this" in the generated code for the generated inner class implementation of the closure. If the comment above is meant to refer to the "this" of the inner class instance, then that would make all methods of an object-oriented language that provided an implicit "this" a closure, and somehow I doubt that's what Ola means, though I could, as always, be wrong.

As for the runtime implementation, as I said earlier I believe the plan is to use method handles (already on the table for JDK 7), which do have some runtime implications (generally good ones, from what I can tell so far), but not beyond what was already on the table for 7.

Phil Haack wrote a thoughtful, insightful and absolutely correct response to my earlier blog post. But he's still missing the point.

The short version: Phil's right when he says, "Agile is less about managing the complexity of an application itself and more about managing the complexity of building an application." Agile is by far the best approach to take when building complex software.

But that's not where I'm going with this.

As a starting point in the discussion, I'd like to call attention to one of Phil's sidebars: I find it curious (and indicative of the larger point) his earlier comment about "I have to wonder, why is that little school district in western Pennsylvania engaging in custom software development in the first place?" At what point does standing a small Access database up qualify as "custom software development"? And I take huge issue with Phil's comment immediately thereafter: "" That's totally untrue, Phil—you are, in fact, creating custom educational curricula, for your children at home. Not for popular usage, not for commercial use, but clearly you're educating your children at home, because you'd be a pretty crappy parent if you didn't. You also practice an informal form of medicine ("Let me kiss the boo-boo"), psychology ("Now, come on, share the truck"), culinary arts ("Would you like mac and cheese tonight?"), acting ("Aaar! I'm the Tickle Monster!") and a vastly larger array of "professional" skills that any of the "professionals" will do vastly better than you.

In other words, you're not a professional actor/chef/shrink/doctor, you're an amateur one, and you want tools that let you practice your amateur "professions" as you wish, without requiring the skills and trappings (and overhead) of a professional in the same arena.

Consider this, Phil: your child decides it's time to have a puppy. (We all know the kids are the ones who make these choices, not us, right?) So, being the conscientious parent that you are, you decide to build a doghouse for the new puppy to use to sleep outdoors (forgetting, as all parents do, that the puppy will actually end up sleeping in the bed with your child, but that's another discussion for another day). So immediately you head on down to Home Depot, grab some lumber, some nails, maybe a hammer and a screwdriver, some paint, and head on home.

Whoa, there, turbo. Aren't you forgetting a few things? For starters, you need to get the concrete for the foundation, rebar to support the concrete in the event of a bad earthquake, drywall, fire extinguishers, sirens for the emergency exit doors... And of course, you'll need a foreman to coordinate all the work, to make sure the foundation is poured before the carpenters show up to put up the trusses, which in turn has to happen before the drywall can go up...

We in this industry have a jealous and irrational attitude towards the amateur software developer. This was even apparent in the Twitter comments that accompanied the conversation around my blog post: "@tedneward treating the disease would mean... have the client have all their ideas correct from the start" (from @kelps). In other words, "bad client! No biscuit!"?

Why is it that we, IT professionals, consider anything that involves doing something other than simply putting content into an application to be "custom software development"? Why can't end-users create tools of their own to solve their own problems at a scale appropriate to their local problem?

Phil offers a few examples of why end-users creating their own tools is a Bad Idea:

I remember one rescue operation for a company drowning in the complexity of a “simple” Access application they used to run their business. It was simple until they started adding new business processes they needed to track. It was simple until they started emailing copies around and were unsure which was the “master copy”. Not to mention all the data integrity issues and difficulty in changing the monolithic procedural application code.

I also remember helping a teachers union who started off with a simple attendance tracker style app (to use an example Ted mentions) and just scaled it up to an atrociously complex Access database with stranded data and manual processes where they printed excel spreadsheets to paper, then manually entered it into another application.

And you know what?

This is not a bad state of affairs.

Oh, of course, we, the IT professionals, will immediately pounce on all the things wrong with their attempts to extend the once-simple application/solution in ways beyond its capabilities, and we will scoff at their solutions, but you know what? That just speaks to our insecurities, not the effort expended. You think Wolfgang Puck isn't going to throw back his head and roar at my lame attempts at culinary experimentation? You think Frank Lloyd Wright wouldn't cringe in horror at my cobbled-together doghouse? And I'll bet Maya Angelou will be so shocked at the ugliness of my poetry that she'll post it somewhere on the "So You Think You're A Poet" website.

Does that mean I need to abandon my efforts to all of these things?

The agilists' community reaction to my post would seem to imply so. "If you aren't a professional, don't even attempt this?" Really? Is that the message we're preaching these days?

End users have just as much a desire and right to be amateur software developers as we do at being amateur cooks, photographers, poets, construction foremen, and musicians. And what do you do when you want to add an addition to your house instead of just building a doghouse? Or when you want to cook for several hundred people instead of just your family?

You hire a professional, and let them do the project professionally.

The above quote was tossed off by Billy Hollis at the patterns&practices Summit this week in Redmond. I passed the quote out to the Twitter masses, along with my +1, and predictably, the comments started coming in shortly thereafter. Rather than limit the thoughts to the 120 or so characters that Twitter limits us to, I thought this subject deserved some greater expansion.

But before I do, let me try (badly) to paraphrase the lightning talk that Billy gave here, which sets context for the discussion:

• Keeping track of all the stuff Microsoft is releasing is hard work: LINQ, EF, Silverlight, ASP.NET MVC, Enterprise Library, Azure, Prism, Sparkle, MEF, WCF, WF, WPF, InfoCard, CardSpace, the list goes on and on, and frankly, nobody (and I mean nobody) can track it all.
• Microsoft released all this stuff because they were chasing the "enterprise" part of the developer/business curve, as opposed to the "long tail" part of the curve that they used to chase down. They did this because they believed that this was good business practice—like banks, "enterprises are where the money is". (If you're not familiar with this curve, imagine a graph with a single curve asymptotically reaching for both axes, where Y is the number of developers on the project, and X is the number of projects. What you get is a curve of a few high-developer-population projects on the left, to a large number of projects with just 1 or 2 developers. This right-hand portion of the curve is known as "the long tail" of the software industry.)
• A lot of software written back in the 90's was written by 1 or 2 guys working for just a few months to slam something out and see if it was useful. What chances do those kinds of projects have today? What tools would you use to build them?
• The problem is the complexity of the tools we have available to us today preclude that kind of software development.
• Agile doesn't solve this problem—the agile movement suggests that we have to create story cards, we have to build unit tests, we have to have a continuous integration server, we have to have standup meetings every day, .... In short, particularly among the agile evangelists (by which we really mean zealots), if you aren't doing a full agile process, you are simply failing. (If this is true, how on earth did all those thousands of applications written in FoxPro or Access ever manage to succeed? –-Me) At one point, an agilist said point-blank, "If you don't do agile, what happens when your project reaches a thousand users?" As Billy put it, "Think about that for a second: This agile guy is threatening us with success."
• Agile is for managing complexity. What we need is to recognize that there is a place for outright simplicity instead.

By the way, let me say this out loud: if you have not heard Billy Hollis speak, you should. Even if you're a Java or Ruby developer, you should listen to what he has to say. He's been developing software for a long time, has seen a lot of these technology-industry trends come and go, and even if you disagree with him, you need to listen to him.

Let me rephrase Billy's talk this way:

Where is this decade's Access?

It may seem like a snarky and trolling question, but think about it for a moment: for a decade or so, I was brought into project after project that was designed to essentially rebuild/rearchitect the Access database created by one of the department's more tech-savvy employees into something that could scale beyond just the department.

(Actually, in about half of them, the goal wasn't even to scale it up, it was just to put it on the web. It was only in the subsequent meetings and discussions that the issues of scale came up, and if my memory is accurate, I was the one who raised those issues, not the customer. I wonder now, looking back at it, if that was pure gold-plating on my part.)

Others, including many people I care about (Rod Paddock, Markus Eggers, Ken Levy, Cathi Gero, for starters) made a healthy living off of building "line of business" applications in FoxPro, which Microsoft has now officially shut down. For those who did Office applications, Visual Basic for Applications has now been officially deprecated in favor of VSTO (Visual Studio Tools for Office), a set of libraries that are available for use by any .NET application language, and of course classic Visual Basic itself has been "brought into the fold" by making it a fully-fledged object-oriented language complete with XML literals and LINQ query capabilities.

Which means, if somebody working for a small school district in western Pennsylvania wants to build a simple application for tracking students' attendance (rather than tracking it on paper anymore), what do they do?

Bruce Tate alluded to this in his Beyond Java, based on the realization that the Java space was no better—to bring a college/university student up to speed on all the necessary technologies required of a "productive" Java developer, he calculated at least five or six weeks of training was required. And that's not a bad estimate, and might even be a bit on the shortened side. You can maybe get away with less if they're joining a team which collectively has these skills distributed across the entire team, but if we're talking about a standalone developer who's going to be building software by himself, it's a pretty impressive list. Here's my back-of-the-envelope calculations:

• Week one: Java language. (Nobody ever comes out of college knowing all the Java language they need.)
• Week two: Java virtual machine: threading/concurrency, ClassLoaders, Serialization, RMI, XML parsing, reference types (weak, soft, phantom).
• Week three: Infrastructure: Ant, JUnit, continuous integration, Spring.
• Week four: Data access: JDBC, Hibernate. (Yes, I think you need a full week on Hibernate to be able to use it effectively.)
• Week five: Web: HTTP, HTML, servlets, filters, servlet context and listeners, JSP, model-view-controller, and probably some Ajax to boot.

I could go on (seriously! no JMS? no REST? no Web services?), but you get the point. And lest the .NET community start feeling complacent, put together a similar list for the standalone .NET developer, and you'll come out to something pretty equivalent. (Just look at the Pluralsight list of courses—name the one course you would give that college kid to bring him up to speed. Stumped? Don't feel bad—I can't, either. And it's not them—pick on any of the training companies.)

Now throw agile into that mix: how does an agile process reduce the complexity load? And the answer, of course, is that it doesn't—it simply tries to muddle through as best it can, by doing all of the things that developers need to be doing: gathering as much feedback from every corner of their world as they can, through tests, customer interaction, and frequent releases. All of which is good. I'm not here to suggest that we should all give up agile and immediately go back to waterfall and Big Design Up Front. Anybody who uses Billy's quote as a sound bite to suggest that is a subversive and a terrorist and should have their arguments refuted with extreme prejudice.

But agile is not going to reduce the technology complexity load, which is the root cause of the problem.

Or, perhaps, let me ask it this way: your 16-year-old wants to build a system to track the cards in his Magic deck. What language do you teach him?

We are in desperate need of simplicity in this industry. Whoever gets that, and gets it right, defines the "Next Big Thing".

Recently I've had the pleasure to make the acquaintance of Walter Bright, one of the heavyweights of compiler construction, and the creator of the D language (among other things), and he's been great in giving me some hand-holding on some compiler-related topics and ideas.

Thus, it seems appropriate to point out that Walter's willing to give lots of other people the same kind of attention and focus, in exchange for your presence in gorgeous Astoria, OR. The Astoria Compiler Construction Seminar is Walter teaching you about the nuts and bolts of building a compiler, from start to finish:

• Introduction to Compilers
• Lexing and Parsing
• Semantic Analysis
• Intermediate Representation
• Interpreters
• Optimization
• Code Generation
• Special Topics (thread-local storage, exception-handling, and so on)
• Building a Compiler for .NET

If you've got any interest whatsoever in building a language, but you're not sure how or where to get started, this seems like a great chance to sit down with one of the "big boys" and find out how to do it. And it doesn't hurt that Walter's an extremely pleasant guy to hang out with, either. (It doesn't hurt that he was the one who created the original Empire game, either. So at least you know you'll have something to play during the breaks.)

Go. Sign up. You'll thank me later.

Recently, an email crossed my Inbox from a friend who was concerned about some questionable practices involving my content (as well as a few others'); apparently, I have been listed as an "author" for SysCon, I have a "domain" with them, and that I've been writing for them since 10 January, 2003, including two articles, "Effective Enterprise Java" and "Java/.NET Interoperability".

Given that both of those "articles" are summaries from presentations I've done at conferences past, I'm a touch skeptical. In fact, it feels like those summaries were scraped from conferences I've done in the past, and I certainly don't remember ever giving Sys-Con (or any other conference) the right to reprint my presentation as an article.

Then it turns out that apparently I'm not the only one suffering this problem. Go. Read that article, then come back. I promise, I'll wait.

(Seriously, go read it.)

Wow. Just... wow. If even half of what Aral's story is true (and I'm inclined to believe at least part of it, given that he's done some pretty meticulous documentation of at least his side of the story), then this is beyond outrageous, and squarely into "completely unethical".

Now, I'll be the first to admit, I've not heard back from Sys-Con about any of this, so if I get any sort of response I'll be sure to update this blog post. But...

Calling anyone a "homosexual son of a bitch", "terrorist" or "fag" is so unbelievably offensive it staggers the mind. Normally, I'd be a bit hesitant to just give either party the benefit of the doubt on that one, given just how ludicrous the accusation sounds, but Aral includes screen shots of the articles, which in of itself lends an air of credibility to the accusation—either Aral is the world's worst Turkish translator, or Sys-Con's translation into Turkish is a bit on the "edgy" side, or Sys-Con really did call him that. Which implies that whichever way this goes, doesn't look good for one of the two parties. But even if we leave that to one side....

Sys-Con is playing with fire by collecting my content and claiming me as an author. Sys-Con never contacted me about becoming a part of their "Ulitzer" website. They never asked me for permission to reprint my articles, though, I'll admit, I can't find where the articles actually exist, nor links to the articles, so maybe they didn't, actually, reprint the article, but just link to them... except I can't find the links to the articles or the presentations, either. They never asked me for an updated bio or photo, and in fact, they pretty clearly grabbed both bio, photo and "summaries" from an old location, because that bio lists me as a DevelopMentor instructor (which I haven't been for two years or so), and as living in Sacramento, CA (which I haven't been for about three years or so). Let me be very clear about this: I do not write for Sys-Con Media. I never have. They have never asked permission to reuse any of the content I have produced. I am appalled at being included in such a fashion.

Note that I'm not opposed to being linked to, mind you—if I put material on my blog, I generally expect (and hope) that people will link to it, and I don't demand permission or even notification when it happens. But to claim that I've written material for an entity does mean I expect to at least be asked if it's OK to use my likeness, name, or material. No such request was ever made of me, so far as I can remember or find (through my own email archives, which stretch back to 2001).

And I can say that I've thought about this issue before, from the other side of the story—back when I was editor at TheServerSide.NET, we began a "blogger's program" that would take interesting blog posts from around the Internet and "collect" them in some fashion for TSS.NET readers. Originally, the thought was to simply reproduce the content directly on our site, and I hated that idea, for the same reasons as I dislike it when somebody does it to me. Regardless of the licensing model the blog entries are published under, to me, a publication or media firm owes the author at least the right of refusal, and a chance to be notified when their material is reused. (In the end, we chose to ask authors if we could reproduce their material in the program, and we never (to my knowledge) had an author refuse.) It doesn't take a real rocket scientist's brain to figure out that asking permission is never a bad thing to do if you want to maintain good will with your sources of material.

This is an open and public request to Sys-Con media: either contact me about using my name, likeness and material on your website, or remove it. (I have emailed their editorial and asked them to acknowledge receipt of my request.)

In the meantime, I will be making every effort to make sure that other content-producers I know are aware of Sys-Con's practices, so they can act as they see fit.

If you are a reader, and find this distasteful as well, then I suggest you follow some of the suggestions mentioned in Aral's blog post:

• Tell everyone you know about what Sys-Con is doing (but don't link to them so as not to give them Google Juice). If tweeting, leave out the http:// bit so that your URL is not automatically made into a link.
• Sys-Con feeds upon the work of authors and speakers to live. If all authors had their content removed from Sys-Con and Ulitzer, they would not have pages to put ads on. So go through their list of authors and notify the ones you know. If they are unaware that they're listed there, they will most likely want themselves removed. Update: I've created a single list of all Sys-Con's Ulitzer authors. More information and the full list are in this post. The original list of authors is at http://www.ulitzer.com/?q=authors. You can ask for your Ulitzer/Sys-Con author page to be removed by emailing editorial@sys-con.com.
• Contact their advertisers and tell them what you think of their association with Sys-Con.
• If you know any speakers speaking at Sys-Con events, make sure they know the kind of company they are associating themselves with. Do the same with anyone you know who is thinking of attending one of their events. Raise awareness about their events at your place of work.
• Make sure Google knows that Sys-Con/Ulitzer is spamming Google with tons of duplicate content. Report them on Google's spam page for posting duplicate content. According to their terms and conditions, Google should stop indexing Sys-Con/Ulitzer. See this comment for a template you can use when reporting them.
• Make sure Google News knows that they are syndicating libelous articles from Sys-Con. Use the Google News Report an Issue form to report the following articles: http://internetvideo.sys-con.com/node/1017038, http://internetvideo.sys-con.com/node/1028923, http://www.sys-con.com/node/1035252, http://air.ulitzer.com/node/1038383, http://openwebdeveloper.sys-con.com/node/1039556, and http://cloudcomputing.sys-con.com/node/1047589

Meanwhile, I'm going to be talking about this to everybody I know at Microsoft, desperately seeking to find out which department engaged the advertising with Sys-Con, and looking to convince them that they don't need this kind of press or association. Ditto for the contacts (far fewer in number) I have with IBM, and any other Sys-Con advertiser I find.

(Disclaimer: In the spirit of full disclosure, Stu is a friend, fellow NFJS speaker, and former co-worker of mine from DevelopMentor.)

I present this review to you in two parts.

Short version: If you want to learn Clojure, and you're familiar with at least one programming language, you'll find this a great resource. If you don't already know a programming language, or if you already know Clojure, or if you're looking for "best practices" to cut-and-paste, you're going to be disappointed.

Long version: Recently, fellow NFJS speaker Stu Halloway decided to take up a new language, and came to Clojure. He found the language interesting enough to write a book on it, something he hasn't done since his Java days, and the result is a nice walk through the language and its environment for experienced Java developers who want to understand Clojure's language, concurrency concepts, and programming model.

Now, let's be 100% honest about this: if you're coming at this book expecting it to be a language reference, you will probably be disappointed (as this guy obviously is). Stu's not like that—he's not going to re-create material that's available elsewhere, or that can be found with an easy Google search. Stu will not waste your time that way—he wants to tell you a story, one that takes you from "I'm a Java guy, but clueless about Lisp, dynamic languages, functional programming, concurrency, or macros" to "Wow. I know kung-fu." in the shortest path possible, but without trying to lobotomize you. He wants—no, expects—the readers of his book to be propping the text open with a cell phone on one side and the dinner plate on the other, craning your neck over to scan the pages and type in the examples into the REPL shell to try them out, see them work, then spend a few minutes experimenting with them before moving on to the next paragraph or page.

(Oh, I suppose you could just cut and paste them from the PDF version of the book, but where's the fun in that?)

The fact is, the concepts behind Clojure make up what's important to learn here, and readers of this book will come away like the panda from the movie, realizing that "There is no Secret Ingredient", that the power of Clojure comes not from its super-secret language sauce or special libraries, but in the way Clojure programmers approach problems and think about programming. And for that reason, if you're a programmer—even if you don't program on the JVM—you really want to take a look at what Stu's talking about (and Rich Hickey is creating).

Just remember, cellphone and dinner plate. Otherwise you'll be missing out on so much.

Well, OK, the title is trolling ever so slightly, but there is an interesting trend at work, and I'm genuinely concerned about its ultimate expression if the trend continues to its logical conclusion. Have a look and tell me if you agree or disagree.

Apparently the Rails community isn't the only one pursuing that ephemeral goal of "edginess"—another blatantly sexist presentation came off without a hitch, this time at a Flash conference, and if anything, it was worse than the Rails/CouchDB presentation. I excerpt a few choice tidbits from an eyewitness here, but be warned—if you're not comfortable with language, skip the next block paragraph.

Yesterday's afternoon keynote is this guy named Hoss Gifford — I believe his major claim to fame is that viral "spank the monkey" thing that went around a few years back.  Highlights of his talk:

• He opens his keynote with one of those "Ignite"-esque presentations — where you have 5-minutes and 20 slides to tell a story — and the first and last are a close-up of a woman's lower half, her legs spread (wearing stilettos, of course) and her shaved vagina visible through some see-thru panties that say "drink me," with Hoss's Photoshopped, upward-looking face placed below it.
• He later demos a drawing tool he has created (admittedly with someone else's code) and invites a woman to come up to try it.  After she sits back down, he points out that in her doodles she's drawn a "cock."
• Then he decides he wants to give a try at using the tool to draw a "cock" (he loves this word) — and draws a face, then a giant dick (he redraws it three times) that ultimately cums all over the face.
• A multitude of references to penises and lots of swearing — and also "If you are easily offended, fuck you!"
• And then, to top it off, a self-made flash movie of an animated woman's face, positioned as if she's having sex with you, who gradually orgasms based on the speed of your mouse movement on the page.

Wow. Just... wow. To call this unprofessional smacks of calling Hitler a "socially awkward individual"... or using a euphemism like "mild medical condition" to refer to death. This is so far "over the line" that it's unbelievable. Even Mr. Aimonetti's "CouchDB" presentation, as bad as it was, at least tried to tie the analogy together in a meaningful, if offensive, way. This is just male posturing at its worst. (I'm shocked Hoss didn't whip off his pants and demand the women in the room bow down in worship to his obviously superior manhood.)

Fortunately, according to the source, the conference organizer seems to be pretty responsive, so kudos to the one adult in the room, but....

What's worse, apparently the presenter and more than a few of his pals are (in the best traditions of assholery) blatantly unrepentant about the whole thing, claiming the moral high ground in much the same way that the Rails idiots did—it's all in good fun, if you don't find it funny you're a prude, and so on:

I checked Twitter (hashtag #flashbelt) to see what the responses were.  Here are some notable remarks:

• Fonx is reading the #flashbelt rants on Hoss offending the ladies w/ a few swear words & a penis drawing - r u really that prudish & sexist?
• nthitz lol @hoss69 "If you are easily offended, fuck you" #flashbelt
• livenootrac Ladies of #flashbelt , I am sorry for the Hoss preso, but in the flash community he gets a pass, kinda like Don Rickles - that's just Hoss.
• CujoJpn @livenootrac And there were many ladies at #flashbelt who were offended by Hoss' Preso some were thick skinned and took it as is.

So, if you didn't like it then
a) you are a prude - and sexist (?)
b) fuck you
c) suck it because Hoss gets a pass here in the boy's club known as "the flash community" and
d) you are a wimpy girl who isn't strong enough / man enough / "thick-skinned" enough  to deal with it.

Even more... wow. Talk about justification and marginalization. Amazing.

Before I figuratively smack this Hoss guy around the blog for a while, let's take a brief moment for reflection—what's going on here? Why all the misogynistic presentations recently? Is this reflective of a general trend in the programming industry? Of society in general? Is the world coming to an end?

A few possibilities present themselves:

• The lack of women in the IT industry means there's nobody around to act as a "gender filter" to keep things on an even keel. In other words, the genders constantly filter themselves based on the company they keep, and because the boys who put these presentations together don't have female input, they simply don't know where to draw the line for mixed company. This theory also presumes that an industry that's made up primarily of women will also lack such a filter and "girls will be girls" as a result. Unfortunately I have no good counterexamples at hand to examine—anybody know of an industry populated primarily by women, and can weigh in with experience there? The closest I get is my brief experience working in a restaurant with an almost-all-woman serving staff, and from that brief experience, yep, the theory holds. Solution? Easy: get more women in IT, and things will re-balance themselves naturally.
• Programmers are principally males who have no redeeming social skills. In other words, the industry gathers up exactly the kind of men who find objectifying women and reveling in late-acquired testosterone overdoses to be gratifying, and this kind of behavior is the result. If true, it leads to the conclusion that programmers are no more evolved than the Navy sailors involved in the Tailhook scandal of a few years ago. So go ahead, smack your wives and girlfriends around a little if they get a little "uppity", it's OK, 'cuz u r a l33t d00d. Personally? I find the idea ludicrous—there is definitely a strong antisocial streak that runs through the IT ecosystem (how many of you met your friends via World of Warcraft again?), but like all stereotypes, there's some elements of truth to it, and a lot of exaggeration. And frankly, anybody who believes in this theory is welcome to come with me to dinner at a No Fluff Just Stuff show and meet the other speakers, and listen in on our "boys club" conversations, including questions like, "Which movie best represents the book it was made after?" and "If given a mandate to create a programming language, what language would your language most resemble?". Oh, and the odd fart joke. We are boys, after all.
• We're hypersensitive to the subject right now. In other words, these kind of presentations have always been going on, and it's just that we notice them now, in the same way that you notice a particular brand of car on the road a lot more when you're thinking about buying that brand and model of car. Frankly, I don't buy this argument—I've been to a lot of presentations over the past decade, and I've never seen any that were anything like this.
• This is the YouTube generation, with access to everything the Internet has to offer, and this is "just how they do things". After all, how much maturity, sexual discretion and adult behavior can we expect of the generation that gave us "Girls Gone Wild" and its ilk? It's just a "generation gap" thing, and we old fogies who didn't grow up with Internet porn just a browser-click away just don't "get it". Hmm.... somehow, I just don't buy it. Sure, there may be some elements of this involved here (I'm really curious to see what all these "Girls Gone Wild" girls are going to say to their own daughters in a decade or so...), but I think that's too easy an answer, and an eminently unhelpful one.
• We have copycatters out there trying to follow the path of people they respect. If you're looking up at this Hoss character and thinking, "I want to be just like him!", you really should see a therapist and develop a sense of self, before you find yourself without friends. Hoss gets a pass because of your misguided fan-boi hero-worship. So does Paris Hilton. You want to be the Paris Hilton of your social circle? Go for it. After all, she's highly respected and loved, right? Take a clue from the next car wreck you drive past—everybody's slowing to look not because they wish they were in the body bag, folks, but because we have a ghoulish fascination with it. In the case of Ms. Hilton, that ghoulish fascination is with those who self-destruct in spectacular fashion. (Me, I'd love to be the fly on the wall at the Hoss residence when he tries to explain this whole thing to his daughter or his date/girlfriend/wife, if he ever finds one.)
• The presenters taking this tack are looking for an easy path to fame. In the grand traditions of Andrew Dice Clay ("Oh!"), the easiest way for a presenter to "stand out" from the rest of the crowd of presenters is to do something outrageous and call it "edgy", and stake out a claim on the edge of the civilization, rather than try to integrate with the rest of the crowd and build something up slowly. Don Box has already claimed "HTTP is dead", I made the analogy between a technology and a military conflict, and Matt Aimonetti claimed a data storage framework "performs like a pr0n star", so what's left but to stake out ground even further out on the fringe and just be misogynistic? Fortunately, history suggests that people with content-free/shock-heavy presentations (or even content-heavy/shock-heavy ones) don't go the distance, so to speak, and that once there's nowhere more shocking left to go, the audience comes back to the content-heavy/shock-light discussions and stays there for a while. Unfortunately, this means we're going to have to suffer through somebody's "Live YouPorn filming" talk first, which I'm not looking forward to.

And now for the smacking around... but you know, I suddenly realize that the volume of comments on the original post leave with nothing to do or say that's not already being said, so to just "pile on" would only serve to let me vent, and I have other outlets for that. But it would be inappropriate to just "walk away", so to speak, so with that in mind....

Hoss, you're an idiot. Like any sprinter, you're going to head up the pack for a bit, but soon enough, your "shtick" is going to flame out and you'll be left behind with all the other "shock jocks" of the 80's who found their material unwelcome after a while. So enjoy the spotlight (such as it is) while you can. In the meantime, I'm off to revise a few presentations, and stick with solid ideas and analogies, and maybe dropping the odd F-bomb when I want to make a point, just for emphasis, because I know something you apparently don't:

Shock makes a point because of the contrast to the rest of the talk, not because of its inherent "edginess".

Meanwhile, by all means, continue to be an idiot. You just make me look better by comparison, for which I thank you.

It's been going around in developer circles now for a few days, this whole controversy about the "Perform like a pr0n star" presentation from the Golden Gate Ruby Conference and the related accusations of misogyny and sexism and overblown accusations and double-standardisms and what-all else, and I've deliberately waited to let opinions in my head settle out before blogging on the whole thing. Sara J Chipps reacts on her blog, and the comments to her comments are also somewhat... interesting... to note.

Without any particular implied importance or order:

• Matt Aimonetti, you are an idiot. You had to know that this was going to generate more than a few strong reactions. I'll admit, it's a funny title, and it definitely generated a ton of buzz around your name, but for the rest of your life, you're going to be "the porno Rails guy", and in about a year or so, it's not going to be funny anymore. You've touched off a firestorm, and you can't very well hide from it, and frankly, I think the short-term boost to your public recognizance is going to be more than outweighed by the long-term judgments that will be levied against you. "Wait, this is the guy who did that talk? Wow. I bet he's a good developer, but can I risk him pulling the same kind of stunt at a meeting with our VP or clients? Nah, I'll go for this other guy...."
• Clearly we have a lot of issues to work out in the programming industry. I'm not going to go into the rights or wrongs of putting those images into his talk. I'm talking about the discussion that followed (one comment here says, "Matt Aimonetti is obviously an antisocial twerp still living in his mothers basement at the age of 35 who has never even been able to muster up the courage to actually talk to a real-life woman, let alone respect one.", and a follow-up comment says, "Great presentation, nevermind the jackasses, keep up the good work!"), and the fact that at no point in the time leading up to this presentation did anybody pull Mr. Aimonetti off to one side and say, "Dude, it was funny when we thought of it, sure, but it's time to stop." If ever we wanted to convince the rest of the world that the programming industry wasn't populated by a bunch of 13-year-olds giggling over the fact that somebody said, "Boobies".... well, maybe next year.
• Ruby community, you have a long way to go if you want to convince people to spend money on you. Maybe you don't mind that corporations think that you guys are clearly unstable and immature. If/when you want to gain some degree of corporate acceptance, and maybe make it out of your parents' basement someday, you're going to have to learn that how you handle yourself in public goes a long way towards establishing peoples' attitudes towards you as professionals, and right now, you all collectively look like a bunch of 13-year-olds, what between this and DHH's famous "FUCK YOU" presentation of a few years ago. If you're OK with not being taken seriously, then cool, more power to you. But personally, I like the idea of making money at things I like to do and have fun doing, and you're not helping yourselves.
• Why are we such prudes? Whether you agree or not with the rightness of the "porn" metaphor, you have to admit that there is factual basis in the bones of this particular comment: "This is probably the least offensive thing I've seen in 3 weeks." Glance at the billboards in the airport next time you're walking to the gate. Glance at the racks of magazines in the grocery store as you prepare to check out. Glance at the beer commercials on TV during prime-time. In every case, sexy, young, attractive, scantily-clad men and women seek to create an instinctive emotional reaction inside your head to subconsciously create a feel-good link between whatever product is being hawked and your id. Honestly, the photos in the presentation are hardly all that titillating—and a very long ways from the kind of commercials you can see on TV in Europe—so why are we getting up in arms over this?
• Matt Aimonetti, you are an idiot. Notice how nobody's talking about the actual subject of your presentation? A good presenter knows that the message should never outstrip the delivery mechanism, just like a sauce should never overpower the flavor of the dish it accompanies. For all that the content of your presentation might have been spot-on, the lessons that might have been learned from the presentation have drowned in the "He's a pig!" "No he's not!" that has followed. Great job there, mate. Way to get your message across.
• To the commenter on the presentation page who said, "ps [sic] feminism is dead", get a clue. Women still, on average, get paid less than men do for an equivalently-skilled employee in the same job. Maybe it's not $.50 to every $1 as it used to be, but so long as it's even measurable, there's work to be done. This industry in particular has absolutely no reason for gender discrimination in any form, since there's absolutely nothing "physical" about what we do. (Ditto for medicine and law, for that matter.)
• Presentations reach far beyond just the attendees. One commenter on Sara's blog notes, "What an over reaction, there was nothing wrong with that presentation, i wouldn't show it to a board room but as far as showing it to a ruby developers conference then no probs." Frankly, that's a short-sighted attitude, making the presumption that someone of the suit-and-tie set (those supposedly inhabiting the "board room" where this kind of presentation isn't appropriate) wouldn't actually be in the audience at a ruby developers conference. Oh, granted, when in Rome, one has to expect Romans to act like Romans, but that just means that the Ruby community isn't welcome inside the board room, right? (Somehow I doubt this is what the numerous people who are trying to make money off of Ruby really want.) Fact is, that presentation is now captured by the Internet for all time, and it will forever be known as "The Ruby Porno Presentation", and it's an even money bet that somebody in that board room has seen the presentation (and the video, and the play-by-play from the people who had friends who had friends that were there....).
• To the commenters who say, "You asked for it", get a clue. Commenters have suggested that the title should have clued people into what was coming: "I'm totally flabbergasted no one has stated the obvious here: if you see a presentation labeled "CouchDB: Perform like a pr0n star" and you choose to go to it, don't act all surprised when R-rated images are used as props." Sorry, no biscuit. Presenters use analogies and imagery all the time in their titles in order to "sell" their talks. Recently I was part of a talk that was labeled as a "smackdown"—did that mean the audience should have expected to see images of physical violence? If I title my next talk as something that's "hard-core", should you expect to see images of ball gags and snuff film clips? This is what happens when we co-opt terms like "smackdown" and "hard-core"—you can't fall back to the original meanings and then claim ignorance when people misunderstand how you're going to use them. (God only knows what Mr. Aimonetti would have done for a presentation on "Naked Objects". *shudder*)
• Matt Aimonetti, you are an idiot. You could have had your joke and keep it tasteful too. You do, in fact, from time to time in the early part of the presentation: the photo of the "little blue pills" was perfect, offering a hint as to what you meant while keeping the double-entendre alive. Every single "objectionable" photo in that presentation could have been replaced by a more subtle one that kept everybody's mind on the subject and still got the point across. The fact that you resorted to the heavy-handed imagery only proves that you wanted to beat the audience's head with it.
• Please, let the one-ups-manship stop. Can we please agree that moving and powerful presentations can be done without having to resort to cheap tricks? They almost always come off badly, particularly when you have to keep the gag running for a full hour or so. Anybody remember Marc Fleury's "Joker" retinue at TheServerSide a half-decade ago? Can you tell me what his presentation was about? Now, consider Dave Thomas' "Cargo Cults" talk from NFJS around the same time—what was he covering? If you were there for both talks, chances are you remember Dave's talk far better than you remember the Fleury keynote beyond the fact that he wore Joker face paint the entire time. Good presentations are about using humor to underscore and support the message, and not making humor the central point of the message. Think about that before you start reaching for the bad innuendo.
• Is this really the kind of industry we want? Granted, it may seem like all of this is way overblown if you're a 25-year-old guy recently graduated from college and hacking on your first or second Rails project. "What do these grumpy idiots not understand about 'it's a joke'? My God, is everybody nuts? Are they trying to say that we can't have fun at work or with what we do?" To which all I can say is two things: one, check in with yourself five or ten years from now, when your daughters are learning about body images by staring at pictures of women who are entirely artificial (and yes, guys, those pictures you see are entirely artificial, having been touched up and enhanced in many ways), and two, you're more than welcome to have whatever jokes you like at whomever's expense you like, in private. This wasn't in private. A developer conference is not a private locale. More importantly, though, think about it—when you bring your girlfriend to work, do you want her hearing those same jokes that buddies toss off back and forth? What seems like "harmless fun" now, may have a very different feel to it for you a few years from now.

I'll freely admit, I drop profanity from time to time in my presentations. And to everyone who comes up afterwords (figuratively and literally) saying I shouldn't use such offensive language, I apologize, and point out that I did so in order to underscore the point, knowing that I'm taking that risk, and knowing that I may be required to offer up apologies after the fact for having offended them. (To date, those apologies still number in the single digits.) So perhaps I am no better than Mr. Aimonetti in the final accounting of things.

But all of this loses sight of a core principle. Regardless of the efficacy of his presentation, regardless of your feelings about the subject matter, regardless of your thoughts around the overblown-or-not nature of this discussion, a deeper principle is at stake here, that of professional presentation etiquette: Mr. Aimonetti, you owe an apology to anyone and everyone that was offended by your presentation (for whatever reason). Failure to deliver that, in my mind, equates to a personal and professional FAIL on your part.

When you stand up on stage, and you say something that somebody finds offensive, you owe that person an apology, even if you think their reasoning or rationale is bogus.

It's simple common courtesy.

The Simple-Talk newsletter is a monthly e-zine that the folks over at Red Gate Software (makers of some pretty cool toys, including their ANTS Profiler, and recent inheritors of the Reflector utility legacy) produce, usually to good effect.

But this month carried with it an interesting editorial piece, which I reproduce in its entirety here:

When the market is slack, nothing succeeds better at tightening it up than promoting serial group-panic within the community. As an example of this, a wave of multi-core panic spread across the Internet about 18 months ago. IT organizations, it was said, urgently had to improve application performance by an order of magnitude in order to cope with rising demand. We wouldn't be able to meet that need because we were at the "end of the road" with regard to step changes in processor power and clock speed. Multi-core technology was the only sure route to improving the speed of applications but, unfortunately, our current "serial" programming techniques, and the limited multithreading capabilities of our programming languages and programmers, left us ill-equipped to exploit it. Multi-core mania gripped the industry.

However, the fever was surprisingly short-lived. Intel's "largest open-source effort ever" to provide a standard tool for writing multi-threaded code, caused little more than a ripple of interest. Various books, rushed out while the temperature soared, advocated the urgent need for new "multi-core-friendly" programming models, involving such things as "software pipelines". Interesting as they undoubtedly are, they sit stolidly on bookshelves, unread.

The truth is that it's simply not a big issue for the majority of people. Writing truly "concurrent" applications in languages such as C# is difficult, as you get very little help from the language. It means getting involved with low-level concurrency primitives, such as lock statements and so on.

Many programmers lack the skills to do this, but more pertinently lack the need. Increasingly, programmers work in a web environment. As long as these web applications are deployed to a load-balanced web farm, then page requests can be handled in parallel so all available cores will be used efficiently without the need for the programmer to be concerned with fine-grained parallelism.

Furthermore, the SQL Server engine behind these web applications is intrinsically "parallel", and can handle and use effectively about as many cores as you care to throw at it. SQL itself is a declarative rather than procedural language, so it is fundamentally concurrent.

A minority of programmers, for example games programmers or those who deal with "embarrassingly parallel" desktop applications such as Photoshop, do need to start working with the current tools and 'low-level' coding techniques that will allow them to exploit multi-core technology. Although currently perceived to be more of "academic" interest, concurrent languages such as Erlang, and concurrency techniques such as "software transactional memory", may yet prove to be significant.

For most programmers and for most web applications, however, the multi-core furore is a storm in a teacup; it's just not relevant. The web and database platforms already cope with concurrency requirements. We are already doing it.

My hope is that this newsletter, sent on April 1st, was intended to be a joke. Having said that, I can’t find any verbage in the email that suggests that it is, in which case, I have to treat it as a legitimate editorial.

And frankly, I think it’s all crap.

It's dangerously ostrichian in nature—it encourages developers to simply bury their heads in the sand and ignore the freight train that's coming their way. Permit me, if you will, a few minutes of your time, that I may be allowed to go through and demonstrate the reasons why I say this.

To begin ...

When the market is slack, nothing succeeds better at tightening it up than promoting serial group-panic within the community. As an example of this, a wave of multi-core panic spread across the Internet about 18 months ago. IT organizations, it was said, urgently had to improve application performance by an order of magnitude in order to cope with rising demand. [...] Multi-core mania gripped the industry.

Point of fact: The “panic” cited here didn’t start about 18 months ago, it started with Herb Sutter’s most excellent (and not only highly recommended but highly required) article, “The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software”, appeared in the pages of Dr. Dobb’s Journal in March of 2005. (Herb’s website notes that “a much briefer version under the title “The Concurrency Revolution” appeared in C/C++ User’s Journal” the previous month.) And the panic itself wasn’t rooted in the idea that we weren’t going to be able to cope with rising demand, but that multi-core CPUs, back then a rarity and reserved only for hardware systems in highly-specialized roles, were in fact becoming commonplace in servers, and worse, as they migrated into desktops, they would quickly a fact of life that every developer would need to face. Herb demonstrated this by pointing out that CPU speeds had taken an interesting change of pace in early 2003:

Around the beginning of 2003, [looking at the website Figure 1 graph] you’ll note a disturbing sharp turn in the previous trend toward ever-faster CPU clock speeds. I’ve added lines to show the limit trends in maximum clock speed; instead of continuing on the previous path, as indicated by the thin dotted line, there is a sharp flattening. It has become harder and harder to exploit higher clock speeds due to not just one but several physical issues, notably heat (too much of it and too hard to dissipate), power consumption (too high), and current leakage problems.

Joe Armstrong, creator of Erlang, noted in a presentation at QCon London 2007 that another of those physical limitations was the speed of light—that for the first time, CPU signal couldn't get from one end of the chip to the other in a single clock cycle.

Quick: What’s the clock speed on the CPU(s) in your current workstation? Are you running at 10GHz? On Intel chips, we reached 2GHz a long time ago (August 2001), and according to CPU trends before 2003, now in early 2005 we should have the first 10GHz Pentium-family chips.

Just to (re-)emphasize the point, here, now, in early 2009, we should be seeing the first 20 or 40 GHz processors, and clearly we’re still plodding along in the 2 – 3 GHz range. The "Quake Rule" (when asked about perf problems, tell your boss you'll need eighteen months to get a 2X improvement, then bury yourselves in a closet for 18 months playing Quake until the next gen of Intel hardware comes out) no longer works.

For the near-term future, meaning for the next few years, the performance gains in new chips will be fueled by three main approaches, only one of which is the same as in the past. The near-term future performance growth drivers are:

• hyperthreading
• multicore
• cache

Hyperthreading is about running two or more threads in parallel inside a single CPU. Hyperthreaded CPUs are already available today, and they do allow some instructions to run in parallel. A limiting factor, however, is that although a hyper-threaded CPU has some extra hardware including extra registers, it still has just one cache, one integer math unit, one FPU, and in general just one each of most basic CPU features. Hyperthreading is sometimes cited as offering a 5% to 15% performance boost for reasonably well-written multi-threaded applications, or even as much as 40% under ideal conditions for carefully written multi-threaded applications. That’s good, but it’s hardly double, and it doesn’t help single-threaded applications.

Multicore is about running two or more actual CPUs on one chip. Some chips, including Sparc and PowerPC, have multicore versions available already. The initial Intel and AMD designs, both due in 2005, vary in their level of integration but are functionally similar. AMD’s seems to have some initial performance design advantages, such as better integration of support functions on the same die, whereas Intel’s initial entry basically just glues together two Xeons on a single die. The performance gains should initially be about the same as having a true dual-CPU system (only the system will be cheaper because the motherboard doesn’t have to have two sockets and associated “glue” chippery), which means something less than double the speed even in the ideal case, and just like today it will boost reasonably well-written multi-threaded applications. Not single-threaded ones.

Finally, on-die cache sizes can be expected to continue to grow, at least in the near term. Of these three areas, only this one will broadly benefit most existing applications. The continuing growth in on-die cache sizes is an incredibly important and highly applicable benefit for many applications, simply because space is speed. Accessing main memory is expensive, and you really don’t want to touch RAM if you can help it. On today’s systems, a cache miss that goes out to main memory often costs 10 to 50 times as much getting the information from the cache; this, incidentally, continues to surprise people because we all think of memory as fast, and it is fast compared to disks and networks, but not compared to on-board cache which runs at faster speeds. If an application’s working set fits into cache, we’re golden, and if it doesn’t, we’re not. That is why increased cache sizes will save some existing applications and breathe life into them for a few more years without requiring significant redesign: As existing applications manipulate more and more data, and as they are incrementally updated to include more code for new features, performance-sensitive operations need to continue to fit into cache. As the Depression-era old-timers will be quick to remind you, “Cache is king.”

Herb’s article was a pretty serious wake-up call to programmers who hadn’t noticed the trend themselves. (Being one of those who hadn’t noticed, I remember reading his piece, looking at that graph, glancing at the open ad from Fry’s Electronics sitting on the dining room table next to me, and saying to myself, “Holy sh*t, he’s right!”.) Does that qualify it as a “mania”? Perhaps if you’re trying to pooh-pooh the concern, sure. But if you’re a developer who’s wondering where you’re going to get the processing power to address the ever-expanding list of features your users want, something Herb points out as a basic fact of life in the software development world ...

There’s an interesting phenomenon that’s known as “Andy giveth, and Bill taketh away.” No matter how fast processors get, software consistently finds new ways to eat up the extra speed. Make a CPU ten times as fast, and software will usually find ten times as much to do (or, in some cases, will feel at liberty to do it ten times less efficiently).

...  then eking out the best performance from an application is going to remain at the top of the priority list. Users are classic consumers: they will always want more and more for the same money as before. Ignore this truth of software (actually, of basic microeconomics) at your peril.

To get back to the editorial, we next come to ...

However, the fever was surprisingly short-lived. Intel's "largest open-source effort ever" to provide a standard tool for writing multi-threaded code, caused little more than a ripple of interest. Various books, rushed out while the temperature soared, advocated the urgent need for new "multi-core-friendly" programming models, involving such things as "software pipelines". Interesting as they undoubtedly are, they sit stolidly on bookshelves, unread.

Wow. Talk about your pretty aggressive accusation without any supporting evidence or citation whatsoever.

Intel's not big into the open-source space, so it doesn't take much for an open-source project from them to be their "largest open-source effort ever". (What, they're going to open-source the schematics for the Intel chipline? Who could read them even if they did? Who would offer up a patch? What good would it do?) The fact that Intel made the software available in the first place meant that they knew the hurdle that had yet to be overcome, and wanted to aid developers in overcoming it. They're members of the OpenMP group for the same reason.

Rogue Wave's software pipelines programming model is another case where real benefits have accrued, backed by case studies. (Disclaimer: I know this because I ghost-wrote an article for them on their Software Pipelines implementation.) Let's not knock something that's actually delivered value. Pipelines aren't going to be the solution to every problem, granted, but they're a useful way of structuring a design, one that's curiously similar to what I see in functional programming languages.

But simply defending Intel's generosity or the validity of an alternative programming model doesn't support the idea that concurrency is still a hot topic. No, for that, I need real evidence, something with actual concrete numbers and verifiable fact to it.

Thus, I point to Brian Goetz’s Java Concurrency in Practice, one of those “books, rushed out while the temperature soared”, which also turned out to be the best-selling book at Java One 2007, and the second-best-selling book (behind only Joshua Bloch’s unbelievably good Effective Java (2nd Ed) ) at Java One 2008. Clearly, yes, bestselling concurrency books are just a myth, alongside the magical device that will receive messages from all over the world and play them into your brain (by way of your ears) on demand, or the magical silver bird that can wing its way through the air with no visible means of support as it does so. Myths, clearly, all of them.

To continue...

The truth is that it's simply not a big issue for the majority of people. Writing truly "concurrent" applications in languages such as C# is difficult, as you get very little help from the language. It means getting involved with low-level concurrency primitives, such as lock statements and so on.

Many programmers lack the skills to do this, but more pertinently lack the need. Increasingly, programmers work in a web environment. As long as these web applications are deployed to a load-balanced web farm, then page requests can be handled in parallel so all available cores will be used efficiently without the need for the programmer to be concerned with fine-grained parallelism.

He’s right when he says you get very little help from the language, be it C# or Java or C++. And getting involved with low-level concurrency primitives is clearly not in anybody’s best interests, particularly if you’re not a concurrency guru like Brian. (And let’s be honest, even low-level concurrency gurus like Brian, or Joe Duffy, who wrote Concurrent Programming on Windows, or Mike Woodring, who co-authored Win32 Multithreaded Programming, have better things to do.) But to say that they “pertinently lack the need” is a rather impertinent statement. “As long as these web applications are deployed to a load-balanced web farm", which is very likely to continue to happen, “then page requests can be handled in parallel so all available cores will be used …”

Um... excuse me?

Didn’t you just say that programmers didn’t need to learn concurrency constructs? It would strike me that if their page requests are being handled in parallel that they have to learn how to write code that won’t break when it’s accessed in parallel or lead to data-corruption problems or race conditions when their pages are accessed in parallel. If parallelism is a fundamental part of the Web, don’t you think it’s important for them to learn how to write programs that can behave correctly in parallel?

Look for just a moment at the average web application: if data is stored in a per-user collection, and two simultaneous requests come in from a given user (perhaps because the page has AJAX requests being generated by the user on the page, or perhaps because there’s a frameset that’s generating requests for each sub-frame, or ...), what happens if the code is written to read a value from the session, increment it, and store it back? ASP.NET can save you here, a little, in that it used to establish a per-user lock on the entirety of the page request (I don’t know if it still does this—I really have lost any desire to build web apps ever again), but that essentially puts an artificial throttle on the scalability of your system, and makes the end-users’ experience that much slower. Load-balancer going to spray the request all over the farm? So long as the user session state is stored on every machine in the farm, that’ll work... But of course if you store the user’s state in the SQL instance behind each of those machines on the farm, then you take the performance hit of an extra network round-trip (at which point we’re back to concurrency in the database) ...

... all because the programmer couldn’t figure out how to make “lock” work? This is progress?

The Java Servlet specification specifically backed away from this "lock on every request" approach because of the performance implications. I heard a fair amount of wailing and gnashing during the early ASP.NET days over this. I heard the ASP.NET dev team say they made their decision because the average developer can't figure out concurrency correctly anyway.

And, by the way folks, this editorial completely ignores XML services. I guess "real" applications don't write services much, either.

The next part is even better:

Furthermore, the SQL Server engine behind these web applications is intrinsically "parallel", and can handle and use effectively about as many cores as you care to throw at it. SQL itself is a declarative rather than procedural language, so it is fundamentally concurrent.

True… and false. SQL is fundamentally “parallel” (largely because SQL is a non-strict functional language, not just a “declarative” one), but T-SQL isn’t. And how many developers actually know where the line is drawn between SQL and T-SQL? More importantly, though, how many effective applications can be written with a complete ignorance of the underlying locking model? Why do DBAs spend hours tuning the database’s physical constructs, establishing where isolation levels can be turned down, establishing where the scope of a transaction is too large, putting in indexed columns where necessary, and figuring out where page, row, or table locking will be most efficient? Because despite the view that a relational database presents, these queries are being executed in parallel, and if a developer wants to avoid writing an application that requires a new server for each and every new user added to the system, they need to learn how to maximize their use of the database’s parallelism. So even if the language is "fundamentally concurrent" and can thus be relied upon to do the right thing on behalf of the developer, the implementation isn't, and needs to be understood in order to be implemented efficiently.

He finishes:

For most programmers and for most web applications, however, the multi-core furore is a storm in a teacup; it's just not relevant. The web and database platforms already cope with concurrency requirements. We are already doing it.

This is one of those times I wish I had a time machine handy—I'd love to step forward five years, have a look around, then come back and report the findings. I'm tempted to close with the challenge to just let’s come back in five years and see what the programming language landscape and hardware landscape looks like. But that's too easy an "out", and frankly, doesn't do much to really instill confidence, in my opinion.

To ignore the developers building "rich" applications (be they being done in Flex/Flash, Cocoa/iPhone, WinForms, Swing, WPF, or what-have-you) is to also ignore a relatively large segment of the market. Not every application is being built on the web and is backed by a relational database—to simply brush those off and not even consider them as part of the editorial reveals a dangerous bias on the editor's part. And those applications aren't hosted in an "intrinsically 'parallel'" container that developers can just bury their head inside.

Like it or not, folks, the path forward isn't one that you get to choose. Intel, AMD, and other chip manufacturers have already made that clear. They're not going to abandon the multicore approach now, not when doing so would mean trying to wrestle with so many problems (including trying to change the speed of light) that simply aren't there when using a multicore foundation. That isn't up for debate anymore. Multicore has won for the forseeable future. And, as a result, multicore is going to be a fact of the developer's life for the forseeable future. Concurrency is thus also a fact of the developer's life for the forseeable future.

The web and database platforms “cope” with concurrency requirements by either making "one-size-fits-all" decisions that almost always end up being the wrong decision for high-scale systems (but I'm sure your new startup-based idea, like a system that allows people to push "micro-entries" of no more than 140 characters in length to a publicly-trackable feed would never actually take off and start carrying millions and millions of messages every day, right?), or by punting entirely and forcing developers to dig deeper beneath the covers to see the concurrency there. So if you're happy with your applications running no faster than 2GHz for the rest of the forseeable future, then sure, you don't need to worry about learning concurrency-friendly kinds of programming techniques. Bear in mind, by the way, that this essentially locks you in to small-scale, web-plus-database systems for the forseeable future, and clearly nothing with any sort of CPU intensiveness to it whatsoever. Be happy in your niche, and wave to the other COBOL programmers who made the same decision.

This is a leaky abstraction, full stop, end of story. Anyone who tells you otherwise is either trolling for hits, trying to sell you something, or striving to persuade developers that ignorance isn't such a bad place to be.

All you ignorant developers, this is the phrase you will be forced to learn before you start your next job: "Would you like fries with that?"

This crossed my Inbox:

I read your article entitled: The Polyglot Programmer. How about the thought that rather than becoming a polyglot-software engineer; pick a polyglot-language. For example, C# is borrowing techniques from functional and dynamic languages. Let the compiler designer worry about mixing features and software engineers worry about keep up with the mixture. Is this a good approach? [From Phil, at http://greensoftwareengineer.spaces.live.com/]

Phil, it’s an interesting thought you’ve raised—which is the better/easier approach to take, that of incorporating the language features we want into a single language, rather than needing to learn all those different languages (and their own unique syntaxes) in order to take advantage of those features we want?

After all, we’re starting to see this taking place within a certain number of languages already, particularly C#; first, in 3.0, they introduced a number of features in support of LINQ that make C# a useful starting point for working with a functional language. Extension methods, for example, allow us to add a number of different methods to the collection classes that provide some functional capabilities (Select<>, GroupBy<>, and so on), as Matt Podwysocki demonstrates, generics contribute the type-safety that most functional languages embrace, anonymous methods and delegates provide better functions-as-first-class-constructs (including lambdas), and anonymous types make it vastly easier to return and pass tuples. And now, in 4.0, we’re getting the “dynamic” keyword, which will add support for invoking methods and properties dynamically, in the grand tradition of most dynamic languages (like Python and Ruby), and 3.0’s local variable type inference allows us to write “var x = ...”, which feels pretty dynamic (even if it’s not, under the hood).

Unfortunately, I think for the most part, the answer’s going to be, “Yes, it would be nice, if it weren’t for the fact that there are very few languages that won’t collapse underneath their own weight if they did so.”

Consider, for example, the C# language. Already, with the C# 3.0 definition, the language specification weighs in at close to a thousand pages. The additional features in 4.0 could easily push it over a thousand and possibly, with all the places where “dynamic” behavior will need to be factored into the existing specification, could push that well into the 1200 to 1300 page range. What’s the upper limit on a language’s complexity to maintain and enhance, much less for its programmers to comprehend?

(By comparison, the C++ specification, as I can best remember, didn’t weigh in at more than a thousand pages, but given that the current working draft is under password protection, and I can’t find the prior spec as a freely-available download, I can’t see if memory is correct or not.)

Or, consider the various edge cases that came up around the introduction of nullable types in C# 2.0. What started out as a fairly simple suggestion—“let’s let T? represent the idea that this instance of T could be nullable, and at runtime it’ll be a Nullable<T> instance behind the scenes”—turned into a pretty ugly morass of edge cases at the language level that resulted in some serious bug-fixing right up until the final ship date.

Thing is, languages that aren’t written deliberately to allow their own modification and evolution tend to fail over time. C++ was one such example, and I think both Java and C# will stand as successor examples before long.

Right now, in C# 3.0, type inference is limited entirely to local variables because the language isn’t syntactically set up to leave out type names wherever possible—the “var” token is a type placeholder, largely because the parser has to have a type first. (This is the same purpose the “dynamic” keyword seems to be playing for 4.0, though I can’t say so for certain.) In F# and Scala, this syntax is deliberately written Pascal-style, with the name first, optionally followed by a colon and the type, because the parser can see the colon and realize the type is already specified, or see no colon and realize the type should be inferred. That syntax is used consistently throughout the F# and Scala languages, and that means it’s pretty easy, lexically speaking, for the languages to recognize when type inference should kick in.

What’s more, both F# and Scala don’t really support the O-O notion of method overloading, because again, it gets confusing when trying to kick in type inference—something about too many possibilities confusing the type-inferencer. (I’m not entirely positive of this point, by the way, it’s based on some conversations I’ve had with language designers over the last few years. I could be wrong, and would love to see a language that supports both.) Instead, they force developers to be more explicit about parameters being passed—F# won’t even do implicit widening conversions, in fact, such as automatically widening ints to longs.

But both F# and Scala have a very interesting facility to allow definitions of methods/functions using very flexible syntactic rules, such that they look like operators or keywords built into the language; F# defines its pipeline operator ( |> ) in its library definitions, for example. Scala defines numerous “keywords”, like synchronized or transient, as classes in the Scala package extending “StaticAnnotation”—in other words, their syntax and behavior is defined as an annotation, rather than as a built-in part of the language. Ditto for Scala’s XML support.

Lisp, of course, was one of the first (if not the first) language to do this, and it’s my understanding that this has been one of the principal reasons it has survived all these years as a language—because it’s an abstraction built on top of an abstraction built on top of an abstraction, et al, it makes it easier to change those underlying abstractions when the context changes.

This doesn’t mean those “polyactivist” languages like C# are bad things, it just means that there’s a danger that they’ll eventually collapse from too many moving parts all trying to talk to each other at the same time. As an exercise, open the C# 3.0 spec, and start checking off all the sections that will need to be touched by the introduction of the “dynamic” keyword as a new type.

Or, to put it analagously, yes, for a lot of work, a single multifunction tool can be useful, but for a lot of other work, you want tools that are specialized to the task at hand. Let’s not minimize the usefulness of that multifunction tool, but let’s not try to use a Swiss Army knife where a jeweler’s screwdriver is really needed.

This email crossed my Inbox last week while I was on the road:

Due to the current economic situation, TechWeb has made the difficult decision to discontinue the Software Development events, including SD West, SD Best Practices and Architecture & Design World. We are grateful for your support during SD's twenty-four year history and are disappointed to see the events end.

This really bums me out, because the SD shows were some of the best shows I’ve been to, particularly SD West, which always had a great cross-cutting collection of experts from all across the industry’s big technical areas: C++, Java, .NET, security, agile, and more. It was also where I got to meet and interview Bjarne Stroustrup, a personal hero of mine from back in my days as a C++ developer, where I got to hang out each year with Scott Meyers, another personal hero (and now a good friend) as well as editor on Effective Enterprise Java, and Mike Cohn, another good friend as well as a great guy to work for. It was where I first met Gary McGraw, in a rather embarrassing fashion—in the middle of his presentation on security, my cell phone went off with a klaxon alarm ring tone loud enough to be heard throughout the entire room, and as every head turned to look at me, he commented dryly, “That’s the buffer overrun alarm—somewhere in the world, a buffer overrun attack is taking place.”

On a positive note, however, the email goes on to say that “Cloud Connect [will] take over SD West's dates in March 2010 at the Santa Clara Convention Center”, which is good news, since it means (hopefully) that I’ll still get a chance to make my yearly pilgrimage to In-N-Out....

Rest in peace, SD. You will be missed.

Just got this email from Chris Sells:

For twelve 45-minute slots at this year’s DSL DevCon (April 16-17 in Redmond, WA), we had 49 proposals. You have been selected as speakers for the following talks. Please confirm that you’ll be there for both days so that I can put together the schedule and post it on the conference site. This DevCon should rock. Thanks!

Martin Fowler - Keynote

Paul Vick + Gio - Mgrammar Deep Dive

Tom Rodgers - Domain Specific Languages for automated testing of equity order management systems and trading machines

Paul Cowan - DSLs in the Horn Package Manager

Guillaume Laforge - How to implement DSLs with Groovy

Markus Voelter - Eclipse tooling for Model-Driven stuff

Dionysios G. Synodinos - JavaScript DSLs for the Client Side

Ted Neward, Bradford Cross - Functional vs. Dynamic DSLs: The Smackdown

Gilad Bracha - embedding EBNF in a general purpose language

Umit Yalcinalp, Tilman Giese - RUMBA: RUby Managed Business data for Applications

Bob Archer - A DSL for Cool Effects in Adobe Pixel Blender

Chance Coble - Language Oriented Programming in F#

As my 15-year-old son Michael has grown fond of saying... w00t! The list of topics is fascinating, and I'm really looking forward to most, if not all, of them. Chance's talk on LOP in F# should be good, I'm really curious to see Gilad's discussion of EBNF (and wondering if this is Newspeak we'll be seeing), and Guillaume is always fun to watch when he's going on about Groovy. Of course, I'm also excited to be paired up with Brad, who's an insanely smart guy--I have a feeling I'll learn a lot just by standing next to him. (Sort of a speakers' osmosis.)

If you're not planning to be here for this (and the Lang.NET Symposium), either you have life-saving surgery scheduled that can't be pushed back, or you're clearly not interested in DSLs. For your own sake, I hope it's the latter.

Seriously, come for the full week. The Lang.NET Symposium last year was an amazing event, for a number of reasons, not the least of which is that it saw Sun celebrities John Rose, Charlie Nutter and Brian Goetz step on to the Microsoft campus, deliver a great presentation on the JVM, MLVM/invokedynamic, and JRuby, and get good feedback and discussion from Microsoft engineers and other notables. You don't get to see that every day.

Last year I had the opportunity to return to the land of my roots, Poland, and speak at Java Developer Days (JDD). Just today, the organizers from JDD sent me a link with a nice little photo montage from the conference. (I did notice a few photos from the after-party were selectively left out of the montage, however, which is probably a good thing because that was the first time I'd ever met a Polish Mad Dog, and boy did they all go down easy...)

If you're anywhere in the area around Krakow in March, you definitely should swing by for their follow-up conference, 4Developers--it sounds like it's going to be another fun event, and this time it's going to reach out to more than just the Java folks, but also the .NET crowd (and a few others), as well.

(I don't really expect any of the readers of this blog living outside Poland to really pack up and head over to Krakow for a weekend, mind you, but if you're a technology speaker and you're interested in hanging with an extremely good group of people, the people who put these shows on--ProIdea--are top-notch, take great care of the speakers, and overall make the entire experience well worth the trip.)

From Scott Hanselman's blog:

Are you in King County/Seattle/Redmond/Bellevue Washington and surrounding areas? Are you a huge nerd? Perhaps a geek? No? Maybe a dork, dweeb or wonk. Maybe you're in town for an SDR (Software Design Review) visiting BillG. Quite possibly you're just a normal person.

Regardless, why not join us for some Mall Food at the Crossroads Bellevue Mall Food Court on Monday, January 19th around 6:30pm?

...

NOTE: RSVP by leaving a comment here and show up on January 19th at 6:30pm! Feel free to bring friends, kids or family. Bring a Ruby or Java person!

Any of the SeaJUG want to attend? (Anybody know of a Ruby JUG in the Eastside area, by the way?) I'm game....

Chris Sells, an acquaintance (and perhaps friend, when he's not picking on me for my Java leanings) of mine from my DevelopMentor days, has a habit of putting on a "DevCon" whenever a technology seems to have reached a certain maturity level. He did it with XML a few years ago, and ATL before that, both of which were pretty amazing events, filled with the sharpest guys in the subject, gathered into a single room to share ideas and shoot each others' pet theories full of holes.

He's at it again, this time with DSLs; from the announcement on his blog:

Are you interested in presenting a 45-minute talk on some Domain Specific Language (DSL) related topic? It doesn't matter which platform or OS you're targeting. It also doesn't matter whether you're an author, a vendor, a professional speaker or a developer in the trenches (in fact, I tend to be biased toward the latter). We're after interesting and unique applications of DSL technology and if you're doing good work in that area, then I need you to send me a session topic and 2-4 sentence abstract along with a little bit about yourself.

I'll be taking submissions 'til February 9th, 2009, but don't delay. Passion and a burning story to tell count twice as much as anything else.

And don't be shy about spreading this announcement around! I've got good coverage in the .NET and Windows communities, but don't know very many folks in the Java or Unix or hardcore modeling worlds, so if you're in that world, let those guys know! Thanks.

The DSL DevCon itself will be in Redmond, WA on the Microsoft campus April 16-17, 2009, right after the Lang.NET conference. Lang.NET will be focused on general-purpose languages, whereas the DSL DevCon will focus on domain-specific languages. The idea is that if you want to attend one or the other or both, that's totally fine. We'll have 2.5 days of Lang.NET on April 14-16 and then 1.5 days of DSL DevCon content.

Oh, and the cost for both conferences is the same: $0.

We're only accepting 150 attendees to either conference. Every one of the five previous DevCons have sold out, so when we open registration, you'll want to be quick about getting your name on the list.

Submit your DSL-related talk idea!

For those of you who are deep in the Java or Ruby space, I really urge you to take a chance here and come to the event--just because it's being held on the Microsoft campus doesn't mean you're going to be forcibly plugged into the Matrix; the same goes for the Lang.NET event in the earlier part of the week, too. Don't believe me? I have proof: Brian Goetz, John Rose, and Charlie Nutter, Sun employees all, attended last years Lang.NET event, talked about the JVM and JRuby, and not only did they not have to give up their "sun.com" email addresses, but they came away with some new appreciations for the CLR, the ecosystem there, and even a few insights about their own platform in comparison to the JVM. (I won't say this as an absolute fact, but I think a lot of John's work on method handles for Java7 came out of conversations he'd had with some of the CLR guys that week.)

This is a DevCon, not a MarCon or a SaleCon. If you're a dev, you're welcome to come here. Frankly, I'd love to see the Java and Ruby (and LLVM and Parrot and ...) guys storm the castle, so to speak, if for no other reason than so Chris will stop teasing me about being a Java guy.

For a couple of years now, I've been going around the world and giving a talk entitled "Pragmatic Architecture", talking both about what architecture is (and what architects really do), and ending the talk with my own "catalog" of architectural elements and ideas, in an attempt to take some of the mystery and "cloud" nature of architecture out of the discussion. If you've read Effective Enterprise Java, then you've read the first version of that discussion, where Pragmatic Architecture was a second-generation thought process.

Recently, the patterns & practices group at Microsoft went back and refined their Application Architecture Guide, and while there's a lot about it that I wish they'd done differently (less of a Microsoft-centric focus, for one), I think it's a great book for Microsoft-centric architects to pick up and have nearby. In a lot of ways, this is something similar to what I had in mind when I thought about the architectural catalog, though I'll admit that I'd prefer to go one level "deeper" and find more of the "atoms" that make up an architecture.

Nevertheless, I think this is a good PDF to pull down and put somewhere on your reference list.

Notes and caveats: Firstly, this is a book for solution architects; if you're the VP or CTO, don't bother with it, just hand it to somebody further on down the food chain. Secondly, if you're not an architect, this is not the book to pick up to learn how to be one. It's more in the way of a reference guide for existing architects. In fact, my vision is that an architect faced with a new project (that is, a new architecture to create) will think about the problem, sketch out a rough solution in his head, then look at the book to find both potential alternatives (to see if they fit better or worse than the one s/he has in her/his head), and potential consequences (to the one s/he has in her/his head). Thirdly, even if you're a Java or Ruby architect, most of the book is pretty technology-neutral. Just take a black Sharpie to the parts that have the Microsoft trademark around them, and you'll find it a pretty decent reference, too. Fourthly, in the spirit of full disclosure, the p&p guys brought me in for a day of discussion on the Guide, so I can't say that I'm completely unbiased, but I can honestly say that I didn't write any of it, just offered critique (in case that matters to any potential readers).

It's once again that time of year, and in keeping with my tradition, I'll revisit the 2008 predictions to see how close I came before I start waxing prophetic on the coming year. (I'm thinking that maybe the next year--2010's edition--I should actually take a shot at predicting the next decade, but I'm not sure if I'd remember to go back and revisit it in 2020 to see how I did. Anybody want to set a calendar reminder for Dec 31 2019 and remind me, complete with URL? )

Without further preamble, here's what I said for 2008:

• THEN: General: The buzz around building custom languages will only continue to build. More and more tools are emerging to support the creation of custom programming languages, like Microsoft's Phoenix, Scala's parser combinators, the Microsoft DLR, SOOT, Javassist, JParsec/NParsec, and so on. Suddenly, the whole "write your own lexer and parser and AST from scratch" idea seems about as outmoded as the idea of building your own String class. Granted, there are cases where a from-hand scanner/lexer/parser/AST/etc is the Right Thing To Do, but there are times when building your own String class is the Right Thing To Do, too. Between the rich ecosystem of dynamic languages that could be ported to the JVM/CLR, and the interesting strides being made on both platforms (JVM and CLR) to make them more "dynamic-friendly" (such as being able to reify classes or access the call stack directly), the probability that your company will find a need that is best answered by building a custom language are only going to rise. NOW: The buzz has definitely continued to build, but buzz can only take us so far. There's been some scattershot use of custom languages in a few scattershot situations, but it's certainly not "taken the world by storm" in any meaningful way yet.
• THEN: General: The hype surrounding "domain-specific languages" will peak in 2008, and start to generate a backlash. Let's be honest: when somebody looks you straight in the eye and suggests that "scattered, smothered and covered" is a domain-specific language, the term has lost all meaning. A lexicon unique to an industry is not a domain-specific language; it's a lexicon. Period. If you can incorporate said lexicon into your software, thus making it accessible to non-technical professionals, that's a good thing. But simply using the lexicon doesn't make it a domain-specific language. Or, alternatively, if you like, every single API designed for a particular purpose is itself a domain-specific language. This means that Spring configuration files are a DSL. Deployment descriptors are a DSL. The Java language is a DSL (since the domain is that of programmers familiar with the Java language). See how nonsensical this can get? Until somebody comes up with a workable definition of the term "domain" in "domain-specific language", it's a nonsensical term. The idea is a powerful one, mind you--creating something that's more "in tune" with what users understand and can use easily is a technique that's been proven for decades now. Anybody who's ever watched an accountant rip an entirely new set of predictions for the new fiscal outlook based entirely on a few seed numbers and a deeply-nested set of Excel macros knows this already. Whether you call them domain-specific languages or "little languages" or "user-centric languages" or "macro language" is really up to you. NOW: The backlash hasn't begun, but only because the DSL buzz hasn't materialized in much way yet--see previous note. It generally takes a year or two of deployments (and hard-earned experience) before a backlash begins, and we haven't hit that "deployments" stage yet in anything yet resembling "critical mass" yet. But the DSL/custom language buzz continues to grow, and the more the buzz grows, the more the backlash is likey.
• THEN: General: Functional languages will begin to make their presence felt. Between Microsoft's productization plans for F# and the growing community of Scala programmers, not to mention the inherently functional concepts buried inside of LINQ and the concurrency-friendly capabilities of side-effect-free programming, the world is going to find itself working its way into functional thinking either directly or indirectly. And when programmers start to see the inherent capabilities inside of Scala (such as Actors) and/or F# (such as asynchronous workflows), they're going to embrace the strange new world of functional/object hybrid and never look back. NOW: Several books on F# and Scala (and even one or two on Haskell!) were published in 2008, and several more (including one of my own) are on the way. The functional buzz is building, and lots of disparate groups are each evaluating it (functional programming) independently.
• THEN: General: MacOS is going to start posting some serious market share numbers, leading lots of analysts to predict that Microsoft Windows has peaked and is due to collapse sometime within the remainder of the decade. Mac's not only a wonderful OS, but it's some of the best hardware to run Vista on. That will lead not a few customers to buy Mac hardware, wipe the machine, and install Vista, as many of the uber-geeks in the Windows world are already doing. This will in turn lead Gartner (always on the lookout for an established trend they can "predict" on) to suggest that Mac is going to end up with 115% market share by 2012 (.8 probability), then sell you this wisdom for a mere price of $1.5 million (per copy). NOW: Can't speak to the Gartner report--I didn't have $1.5 million handy--but certainly the MacOS is growing in popularity. More on that later.
• THEN: General: Ted will be hired by Gartner... if only to keep him from smacking them around so much. .0001 probability, with probability going up exponentially as my salary offer goes up exponentially. (Hey, I've got kids headed for college in a few years.) NOW: Well, Gartner appears to have lost my email address and phone number, but I'm sure they were planning to make me that offer.
• THEN: General: MacOS is going to start creaking in a few places. The Mac OS is a wonderful OS, but it's got its own creaky parts, and the more users that come to Mac OS, the more that software packages are going to exploit some of those creaky parts, leading to some instability in the Mac OS. It won't be widespread, but for those who are interested in finding it, they're there. Assuming current trends (of customers adopting Mac OS) hold, the Mac OS 10.6 upgrade is going to be a very interesting process, indeed. NOW: Shhh. Don't tell anybody, but I've been seeing it starting to happen. Don't get me wrong, Apple still does a pretty good job with the OS, but the law of numbers has started to create some bad upgrade scenarios for some people.
• THEN: General: Somebody is going to realize that iTunes is the world's biggest monopoly on music, and Apple will be forced to defend itself in the court of law, the court of public opinion, or both. Let's be frank: if this were Microsoft, offering music that can only be played on Microsoft music players, the world would be through the roof. All UI goodness to one side, the iPod represents just as much of a monopoly in the music player business as Internet Explorer did in the operating system business, and if the world doesn't start taking Apple to task over this, then "justice" is a word that only applies when losers in an industry want to drag down the market leader (which I firmly believe to be the case--nobody likes more than to pile on the successful guy). NOW: Nothing this year.
• THEN: General: Somebody is going to realize that the iPhone's "nothing we didn't write will survive the next upgrade process" policy is nothing short of draconian. As my father, who gets it right every once in a while, says, "If I put a third-party stereo in my car, the dealer doesn't get to rip it out and replace it with one of their own (or nothing at all!) the next time I take it in for an oil change". Fact is, if I buy the phone, I own the phone, and I own what's on it. Unfortunately, this takes us squarely into the realm of DRM and IP ownership, and we all know how clear-cut that is... But once the general public starts to understand some of these issues--and I think the iPhone and iTunes may just be the vehicle that will teach them--look out, folks, because the backlash will be huge. As in, "Move over, Mr. Gates, you're about to be joined in infamy by your other buddy Steve...." NOW: Apple released iPhone 2.0, and with it, the iPhone SDK, so at least Apple has opened the dashboard to third-party stereos. But the deployment model (AppStore) is still a bit draconian, and Apple still jealously holds the reins over which apps can be deployed there and which ones can't, so maybe they haven't learned their lesson yet, after all....
• THEN: Java: The OpenJDK in Mercurial will slowly start to see some external contributions. The whole point of Mercurial is to allow for deeper control over which changes you incorporate into your build tree, so once people figure out how to build the JDK and how to hack on it, the local modifications will start to seep across the Internet.... NOW: OpenJDK has started to collect contributions from external (to Sun) sources, but still in relatively small doses, it seems. None of the local modifications I envisioned creeping across the 'Net have begun, that I can see, so maybe it's still waiting to happen. Or maybe the OpenJDK is too complicated to really allow for that kind of customization, and it never will.
• THEN: Java: SpringSource will soon be seen as a vendor like BEA or IBM or Sun. Perhaps with a bit better reputation to begin, but a vendor all the same. NOW: SpringSource's acquisition of G2One (the company behind Groovy just as SpringSource backs Spring) only reinforced this image, but it seems it's still something that some fail to realize or acknowledge due to Spring's open-source (?) nature. (I'm not a Spring expert by any means, but apparently Spring 3 was pulled back inside the SpringSource borders, leading some people to wonder what SpringSource is up to, and whether or not Spring will continue to be open source after all.)
• THEN: .NET: Interest in OpenJDK will bootstrap similar interest in Rotor/SSCLI. After all, they're both VMs, with lots of interesting ideas and information about how the managed platforms work. NOW: Nope, hasn't really happened yet, that I can see. Not even the 2nd edition of the SSCLI book (by Joel Pobar and yours truly, yes that was a plug) seemed to foster the kind of attention or interest that I'd expected, or at least, not on the scale I'd thought might happen.
• THEN: C++/Native: If you've not heard of LLVM before this, you will. It's a compiler and bytecode toolchain aimed at the native platforms, complete with JIT and GC. NOW: Apple sank a lot of investment into LLVM, including hosting an LLVM conference at the corporate headquarters.
• THEN: Java: Somebody will create Yet Another Rails-Killer Web Framework. 'Nuff said. NOW: You know what? I honestly can't say whether this happened or not; I was completely not paying attention.
• THEN: Native: Developers looking for a native programming language will discover D, and be happy. Considering D is from the same mind that was the core behind the Zortech C++ compiler suite, and that D has great native platform integration (building DLLs, calling into DLLs easily, and so on), not to mention automatic memory management (except for those areas where you want manual memory management), it's definitely worth looking into. www.digitalmars.com NOW: D had its own get-together as well, and appears to still be going strong, among the group of developers who still work on native apps (and aren't simply maintaining legacy C/C++ apps).

Now, for the 2009 predictions. The last set was a little verbose, so let me see if I can trim the list down a little and keep it short and sweet:

• General: "Cloud" will become the next "ESB" or "SOA", in that it will be something that everybody will talk about, but few will understand and even fewer will do anything with. (Considering the widespread disparity in the definition of the term, this seems like a no-brainer.)
• Java: Interest in Scala will continue to rise, as will the number of detractors who point out that Scala is too hard to learn.
• .NET: Interest in F# will continue to rise, as will the number of detractors who point out that F# is too hard to learn. (Hey, the two really are cousins, and the fortunes of one will serve as a pretty good indication of the fortunes of the other, and both really seem to be on the same arc right now.)
• General: Interest in all kinds of functional languages will continue to rise, and more than one person will take a hint from Bob "crazybob" Lee and liken functional programming to AOP, for good and for ill. People who took classes on Haskell in college will find themselves reaching for their old college textbooks again.
• General: The iPhone is going to be hailed as "the enterprise development platform of the future", and companies will be rolling out apps to it. Look for Quicken iPhone edition, PowerPoint and/or Keynote iPhone edition, along with connectors to hook the iPhone up to a presentation device, and (I'll bet) a World of Warcraft iPhone client (legit or otherwise). iPhone is the new hotness in the mobile space, and people will flock to it madly.
• .NET: Another Oslo CTP will come out, and it will bear only a superficial resemblance to the one that came out in October at PDC. Betting on Oslo right now is a fools' bet, not because of any inherent weakness in the technology, but just because it's way too early in the cycle to be thinking about for anything vaguely resembling production code.
• .NET: The IronPython and IronRuby teams will find some serious versioning issues as they try to manage the DLR versioning story between themselves and the CLR as a whole. An initial hack will result, which will be codified into a standard practice when .NET 4.0 ships. Then the next release of IPy or IRb will have to try and slip around its restrictions in 2010/2011. By 2012, IPy and IRb will have to be shipping as part of Visual Studio just to put the releases back into lockstep with one another (and the rest of the .NET universe).
• Java: The death of JSR-277 will spark an uprising among the two leading groups hoping to foist it off on the Java community--OSGi and Maven--while the rest of the Java world will breathe a huge sigh of relief and look to see what "modularity" means in Java 7. Some of the alpha geeks in Java will start using--if not building--JDK 7 builds just to get a heads-up on its impact, and be quietly surprised and, I dare say, perhaps even pleased.
• Java: The invokedynamic JSR will leapfrog in importance to the top of the list.
• Windows: Another Windows 7 CTP will come out, and it will spawn huge media interest that will eventually be remembered as Microsoft promises, that will eventually be remembered as Microsoft guarantees, that will eventually be remembered as Microsoft FUD and "promising much, delivering little". Microsoft ain't always at fault for the inflated expectations people have--sometimes, yes, perhaps even a lot of times, but not always.
• Mac OS: Apple will begin to legally threaten the clone market again, except this time somebody's going to get the DOJ involved. (Yes, this is the iPhone/iTunes prediction from last year, carrying over. I still expect this to happen.)
• Languages: Alpha-geek developers will start creating their own languages (even if they're obscure or bizarre ones like Shakespeare or Ook#) just to have that listed on their resume as the DSL/custom language buzz continues to build.
• XML Services: Roy Fielding will officially disown most of the "REST"ful authors and software packages available. Nobody will care--or worse, somebody looking to make a name for themselves will proclaim that Roy "doesn't really understand REST". And they'll be right--Roy doesn't understand what they consider to be REST, and the fact that he created the term will be of no importance anymore. Being "REST"ful will equate to "I did it myself!", complete with expectations of a gold star and a lollipop.
• Parrot: The Parrot guys will make at least one more minor point release. Nobody will notice or care, except for a few doggedly stubborn Perl hackers. They will find themselves having nightmares of previous lives carrying around OS/2 books and Amiga paraphernalia. Perl 6 will celebrate it's seventh... or is it eighth?... anniversary of being announced, and nobody will notice.
• Agile: The debate around "Scrum Certification" will rise to a fever pitch as short-sighted money-tight companies start looking for reasons to cut costs and either buy into agile at a superficial level and watch it fail, or start looking to cut the agilists from their company in order to replace them with cheaper labor.
• Flash: Adobe will continue to make Flex and AIR look more like C# and the CLR even as Microsoft tries to make Silverlight look more like Flash and AIR. Web designers will now get to experience the same fun that back-end web developers have enjoyed for near-on a decade, as shops begin to artificially partition themselves up as either "Flash" shops or "Silverlight" shops.
• Personal: Gartner will still come knocking, looking to hire me for outrageous sums of money to do nothing but blog and wax prophetic.

Well, so much for brief or short. See you all again next year....

It amazes me how insular and inward-facing the software industry is. And how the "agile" movement is reaping the benefits of a very simple characteristic.

For example, consider Jeff Palermo's essay on "The Myth of Self-Organizing Teams". Now, nothing against Jeff, or his post, per se, but it amazes me how our industry believes that they are somehow inventing new concepts, such as, in this case the "self-organizing team". Team dynamics have been a subject of study for decades, and anyone with a background in psychology, business, or sales has probably already been through much of the material on it. The best teams are those that find their own sense of identity, that grow from within, but still accept some leadership from the outside--the classic example here being the championship sports team. Most often, that sense of identity is born of a string of successes, which is why teams without a winning tradition have such a hard time creating the esprit de corps that so often defines the difference between success and failure.

(Editor's note: Here's a free lesson to all of you out there who want to help your team grow its own sense of identity: give them a chance to win a few successes, and they'll start coming together pretty quickly. It's not always that easy, but it works more often than not.)

How many software development managers--much less technical leads or project managers--have actually gone and looked through the management aisle at the local bookstore?

Tom and Mary Poppendieck have been spending years now talking about "lean" software development, which itself (at a casual glance) seems to be a refinement of the concepts Toyota and other Japanese manufacturers were pursuing close to two decades ago. "Total quality management" was a concept introduced in those days, the idea that anyone on the production line was empowered to stop the line if they found something that wasn't right. (My father was one of those "lean" manufacturing advocates back in the 80's, in fact, and has some great stories he can tell to its successes, and failures.)

How many software development managers or project leads give their developers the chance to say, "No, it's not right yet, we can't ship", and back them on it? Wouldn't you, as a developer, feel far more involved in the project if you knew you had that power--and that responsibility?

Or consider the "agile" notion of customer involvement, the classic XP "On-Site Customer" principle. Sales people have known for years, even decades (if not centuries), that if you involve the customer in the process, they are much more likely to feel an ownership stake sooner than if they just take what's on the lot or the shelf. Skilled salespeople have done the "let's walk through what you might buy, if you were buying, of course" trick countless numbers of times, and ended up with a sale where the customer didn't even intend to buy.

How many software development managers or project leads have read a book on basic salesmanship? And yet, isn't that notion of extracting what the customer wants endemic to both software development and basic sales (of anything)?

What is it about the software industry that just collectively refuses to accept that there might be lots of interesting research on topics that aren't technical yet still something that we can use? Why do we feel so compelled to trumpet our own "innovations" to ourselves, when in fact, they've been long-known in dozens of other contexts? When will we wake up and realize that we can learn a lot more if we cross-train in other areas... like, for example, getting your MBA?

Having freshly converted both the Visual Studio 2010 and Oslo SDK VPC images that we received at PDC 2008 last month to VMWare images, I figure it's time to dive into M.

At PDC, the Addison-Wesley folks were giving away copies of "The 'Oslo' Modeling Language" book, which is apparently official canon of the "M" language for Oslo, so I flip to page 1 and start reading:

The "Oslo" Modeling Language (M) is a modern, declarative language for working with data. M lets users write down how they want to structure and query their data using a convenient textual syntax that is convenient to both author and read.

M does not mandate how data is stored or accessed, nor does it mandate a specific implementation technology. Rather, M was designed to allow users to write down what they want from their data without having to specify how those desires are met against a given technology or platform. That stated, M in no way prohibits implementations from providing rich declarative or imperative support for controlling how M constructs are represented and executed in a given environment.

Hmm... I have to admit, all kinds of warning bells and alarm flags are going off in my head, and we're just two sentences into this thing. This sounds like something we've all done before; in fact, though I've not tried it, I have a feeling that if we were to go back through those two paragraphs and replace every instance of "M" with "SQL", we'd find a paragraph that could easily slip into the opening chapter of any introductory SQL or RDBMS book.

The goals of "separation of declaration from intent" have been around for that long, probably longer, and even the fiercest and staunchest defenders of SQL find themselves sometimes wandering through SQL declarations and code that clearly violate Chris Date's politely-worded commands around normal form and separation of declaration from intent and implementation.

I keep reading, though, and a few paragraphs later, find something intriguing.

Another important aspect of data management that M does not address is that of update. M is a functional language that does not have constructs for changing the contents of an extent. (Author's note: an "extent", defined a few paragraphs earlier, is that "an extent provides dynamic storage for values.") How data changes is outside the scope of the language. That said, M anticipates that the contents of an extent can change via external (to M) stimuli. Subsequent versions of M are expected to provide declarative constructs for updating data.

Wow. So the first question becomes, when are those "subsequent versions" expected? Is this simply a state of the PDC Preview bits, or something that's not in scope for v1 of the Oslo SDK?

I flip through the rest of the first chapter, which seems like a decent overview, and what I see there is an interesting type-declaration language; in many ways, it's highly reminiscent of XML Schema Descriptions (XSD) more than SQL declarations, but I suppose that's to be expected, at least for now. I'm sure they're going to cherry-pick a lot of the best data-declarative constructs from XSD, SQL, and any other metadata-based formats/languages, and that the semantics will change as they explore what works well and what doesn't. For now, though, "M" exists essentially as a data-descriptor language, and this is reinforced when I start playing with "m.exe", the "M compiler" (?).

First thing, I simply fire up "m.exe" to see what the options are. And... nothing. Huh? I wait for a bit, then Ctrl-C it, and start hunting through the documentation to see if I'm missing something here. I try a few different tests, like "m /?" or "m -help", and each time, the compiler just seems to wander off into the weeds, requiring a Ctrl-C to kill it.

What the heck? I know that these are PDC pre-alpha CTP "nothing is guaranteed to work" bits, but this seems a bit on the excessive side--I have every faith that Microsoft wouldn't hand these out if you can't even run the compiler! So acting on a hunch, I fire up "m /?" again, and tab away to look at something else. Sure enough, my hunch is rewarded--after a long pause, eventually the help screen comes up. So, apparently, the m.exe tool just takes fricken forever to run, is all.

Currently, the only targets M can compile to is their internal Repository for storing types, and a generic "T-SQL" target for any T-SQL-compliant database (which I presume for now means only SQL Server of various versions, but theoretically, I suppose, Sybase could work too, given those two systems' shared ancestry. And, given a pretty simple sample to work with, m.exe produces a pretty-easily-anticipated result; this:

module Ted
{
  type Person
  {
    Id : Integer32 = AutoNumber();
    Name : Text;
  } where identity Id;
  People : Person*;    
}

turns into this:

set xact_abort on;
go

begin transaction;
go

set ansi_nulls on;
go

create schema [Ted];
go

create table [Ted].[People]
(
  [Id] int not null identity,
  [Name] nvarchar(max) not null,
  constraint [PK_People] primary key clustered ([Id])
);
go

commit transaction;
go

... which, when you look at it, is pretty much what you'd want.

Interestingly enough, there's no reason why people in the Java or Ruby space couldn't use "M" just as easily, so long as the database targeted is one that M understands. (It also wouldn't be a terribly difficult exercise to build an M compiler in Java or Ruby, for that matter. Might be a fun off-time project, in fact.)

One thing that's also pretty clear is that M is very collection-centric, as the first chapter spends probably 50% of its time describing all the various ways that collections in M (written as "{a, b, c}") interact with one another (they can be compared for equality directly, for example, and have some neat projection/filter capabilities that were clearly drawn from the relational algebra and LINQ syntax). Having said that, though, one thing that is obviously missing is the traditional object "reference"-style connection, where A OWNS-A B.

What this seems to imply, then, is that the object/relational-mapping horrors of the past two decades aren't yet over. What's not clear is how M will make it easier (or if it will at all) to access those extents from the languages we traditionally use in the .NET space (C#, VB, C++/CLI, etc), specifically, what the mechanism for conducting a query will be like, and what it's return types will be when it cross the boundary back into C#.

If you're not sure what I mean by that, consider it this way: ADO.NET has a simple mechanism for taking the query--a raw string as a parameter--and executing it, and when it returns, it's handed back to your C# code as a DataSet, or else as an IDataReader for row-based/column-based firehose-style consumption. Much of the criticism of ADO.NET stems around two parts: the untyped nature of the query string, leading to potential typos and errors, and the relative awkwardness for extracting the data from the results, either the DataSet or the IDataReader, at least when compared to languages that have built-in set/tuple constructs.

The one sample that does show any sort of C# -> M kinds of interaction is in the MParserDemo sample, and here, when it queries the database, it does so using traditional ADO.NET API calls, so I'm not sure it's to be taken as a good indicator of the plans around M yet.

If all there was to Oslo was "M", I'd say it was an interesting little side-note at PDC, something that maybe a few folks might find interesting and otherwise not worth studying, but this is not the sum total of the Oslo bits; there is also Mg, the MGrammar language, a language specifically for building DSLs, and that's where my attention (and next blog post) is going next.

Roy Fielding has weighed in on the recent "buzzwordiness" (hey, if Colbert can make up "truthiness", then I can make up "buzzwordiness") of calling everything a "REST API", a tactic that has become more en vogue of late as vendors discover that the general programming population is finding the WSDL-based XML services stack too complex to navigate successfully for all but the simplest of projects. Contrary to what many RESTafarians may be hoping, Roy doesn't gather all these wayward children to his breast and praise their anti-vendor/anti-corporate/anti-proprietary efforts, but instead, blasts them pretty seriously for mangling his term:

I am getting frustrated by the number of people calling any HTTP-based interface a REST API. Today’s example is the SocialSite REST API. That is RPC. It screams RPC. There is so much coupling on display that it should be given an X rating.

Ouch. "So much coupling on display that it should be given an X rating." I have to remember that phrase--that's a keeper. And I'm shocked that Roy even knows what an X rating is; he's such a mellow guy with such an innocent-looking face, I would've bet money he'd never run into one before. (Yes, people, that's a joke.)

What needs to be done to make the REST architectural style clear on the notion that hypertext is a constraint? In other words, if the engine of application state (and hence the API) is not being driven by hypertext, then it cannot be RESTful and cannot be a REST API. Period. Is there some broken manual somewhere that needs to be fixed?

Go Roy!

For those of you who've not read Roy's thesis, and are thinking that this is some kind of betrayal or trick, let's first of all point out that at no point is Roy saying that your nifty HTTP-based API is not useful or simple. He's simply saying that it isn't RESTful. That's a key differentiation. REST has a specific set of goals and constraints it was trying to meet, and as such prescribes a particular kind of architectural style to fit within those constraints. (Yes, REST is essentially an architectural pattern: a solution to a problem within a certain context that yields certain consequences.)

Assuming you haven't tuned me out completely already, allow me to elucidate. In Chapter 5 of Roy's thesis, Roy begins to build up the style that will ultimately be considered REST. I'm not going to quote each and every step here--that's what the hyperlink above is for--but simply call out certain parts. For example, in section 5.1.3, "Stateless", he suggests that this architectural style should be stateless in nature, and explains why; the emphasis/italics are mine:

We next add a constraint to the client-server interaction: communication must be stateless in nature, as in the client-stateless-server (CSS) style of Section 3.4.3 (Figure 5-3), such that each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client.

This constraint induces the properties of visibility, reliability, and scalability. Visibility is improved because a monitoring system does not have to look beyond a single request datum in order to determine the full nature of the request. Reliability is improved because it eases the task of recovering from partial failures [133]. Scalability is improved because not having to store state between requests allows the server component to quickly free resources, and further simplifies implementation because the server doesn't have to manage resource usage across requests.

Like most architectural choices, the stateless constraint reflects a design trade-off. The disadvantage is that it may decrease network performance by increasing the repetitive data (per-interaction overhead) sent in a series of requests, since that data cannot be left on the server in a shared context. In addition, placing the application state on the client-side reduces the server's control over consistent application behavior, since the application becomes dependent on the correct implementation of semantics across multiple client versions.

In the HTTP case, the state is contained entirely in the document itself, the hypertext. This has a couple of implications for those of us building "distributed applications", such as the very real consideration that there's a lot of state we don't necessarily want to be sending back to the client, such as voluminous information (the user's e-commerce shopping cart contents) or sensitive information (the user's credentials or single-signon authentication/authorization token). This is a bitter pill to swallow for the application development world, because much of the applications we develop have some pretty hefty notions of server-based state management that we want or need to preserve, either for legacy support reasons, for legitimate concerns (network bandwidth or security), or just for ease-of-understanding. Fielding isn't apologetic about it, though--look at the third paragraph above. "[T]he stateless constraint reflects a design trade-off."

In other words, if you don't like it, fine, don't follow it, but understand that if you're not leaving all the application state on the client, you're not doing REST.

By the way, note that technically, HTTP is not tied to HTML, since the document sent back and forth could easily be a PDF document, too, particularly since PDF supports hyperlinks to other PDF documents. Nowhere in the thesis do we see the idea that it has to be HTML flying back and forth.

Roy's thesis continues on in the same vein; in section 5.1.4 he describes how "client-cache-stateless-server" provides some additional reliability and performance, but only if the data in the cache is consistent and not stale, which was fine for static documents, but not for dynamic content such as image maps. Extensions were necessary in order to accomodate the new ideas.

In section 5.1.5 ("Uniform Interface") we get to another stinging rebuke of REST as a generalized distributed application scheme; again, the emphasis is mine:

The central feature that distinguishes the REST architectural style from other network-based styles is its emphasis on a uniform interface between components (Figure 5-6). By applying the software engineering principle of generality to the component interface, the overall system architecture is simplified and the visibility of interactions is improved. Implementations are decoupled from the services they provide, which encourages independent evolvability. The trade-off, though, is that a uniform interface degrades efficiency, since information is transferred in a standardized form rather than one which is specific to an application's needs. The REST interface is designed to be efficient for large-grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction.

In order to obtain a uniform interface, multiple architectural constraints are needed to guide the behavior of components. REST is defined by four interface constraints: identification of resources; manipulation of resources through representations; self-descriptive messages; and, hypermedia as the engine of application state. These constraints will be discussed in Section 5.2.

In other words, in order to be doing something that Fielding considers RESTful, you have to be using hypermedia (that is to say, hypertext documents of some form) as the core of your application state. It might seem like this implies that you have to be building a Web application in order to be considered building something RESTful, so therefore all Web apps are RESTful by nature, but pay close attention to the wording: hypermedia must be the core of your application state. The way most Web apps are built today, HTML is clearly not the core of the state, but merely a way to render it. This is the accidental consequence of treating Web applications and desktop client applications as just pale reflections of one another.

The next section, 5.1.6 ("Layered System") again builds on the notion of stateless-server architecture to provide additional flexibility and power:

In order to further improve behavior for Internet-scale requirements, we add layered system constraints (Figure 5-7). As described in Section 3.4.2, the layered system style allows an architecture to be composed of hierarchical layers by constraining component behavior such that each component cannot "see" beyond the immediate layer with which they are interacting. By restricting knowledge of the system to a single layer, we place a bound on the overall system complexity and promote substrate independence. Layers can be used to encapsulate legacy services and to protect new services from legacy clients, simplifying components by moving infrequently used functionality to a shared intermediary. Intermediaries can also be used to improve system scalability by enabling load balancing of services across multiple networks and processors.

The primary disadvantage of layered systems is that they add overhead and latency to the processing of data, reducing user-perceived performance [32]. For a network-based system that supports cache constraints, this can be offset by the benefits of shared caching at intermediaries. Placing shared caches at the boundaries of an organizational domain can result in significant performance benefits [136]. Such layers also allow security policies to be enforced on data crossing the organizational boundary, as is required by firewalls [79].

The combination of layered system and uniform interface constraints induces architectural properties similar to those of the uniform pipe-and-filter style (Section 3.2.2). Although REST interaction is two-way, the large-grain data flows of hypermedia interaction can each be processed like a data-flow network, with filter components selectively applied to the data stream in order to transform the content as it passes [26]. Within REST, intermediary components can actively transform the content of messages because the messages are self-descriptive and their semantics are visible to intermediaries.

The potential of layered systems (itself not something that people building RESTful approaches seem to think much about) is only realized if the entirety of the state being transferred is self-descriptive and visible to the intermediaries--in other words, intermediaries can only be helpful and/or non-performance-inhibitive if they have free reign to make decisions based on the state they see being transferred. If something isn't present in the state being transferred, usually because there is server-side state being maintained, then they have to be concerned about silently changing the semantics of what is happening in the interaction, and intermediaries--and layers as a whole--become a liability. (Which is probably why so few systems seem to do it.)

And if the notion of visible, transported state is not yet made clear in his dissertation, Fielding dissects the discussion even further in section 5.2.1, "Data Elements". It's too long to reprint here in its entirety, and frankly, reading the whole thing is necessary to see the point of hypermedia and its place in the whole system. (The same could be said of the entire chapter, in fact.) But it's pretty clear, once you read the dissertation, that hypermedia/hypertext is a core, critical piece to the whole REST construction. Clients are expected, in a RESTful system, to have no preconceived notions of structure or relationship between resources, and discover all of that through the state of the hypertext documents that are sent back to them. In the HTML case, that discovery occurs inside the human brain; in the SOA/services case, that discovery is much harder to define and describe. RDF and Semantic Web ideas may be of some help here, but JSON can't, and simple XML can't, unless the client has some preconceived notion of what the XML structure looks like, which violates Fielding's rules:

A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API). From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations. The transitions may be determined (or limited by) the client’s knowledge of media types and resource communication mechanisms, both of which may be improved on-the-fly (e.g., code-on-demand). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]

An interesting "fuzzy gray area" here is whether or not the client's knowledge of a variant or schematic structure of XML could be considered to be a "standardized media type", but I'm willing to bet that Fielding will argue against it on the grounds that your application's XML schema is not "standardized" (unless, of course, it is, through a national/international/industry standardization effort).

But in case you'd missed it, let me summarize the past twenty or so paragraphs: hypermedia is a core requirement to being RESTful. If you ain't slinging all of your application state back and forth in hypertext, you ain't REST. Period. Fielding said it, he defined it, and that settles it.

Before the hate mail comes a-flyin', let me reiterate one vitally important point: if you're not doing REST, it doesn't mean that your API sucks. Fielding may have his definition of what REST is, and the idealist in me wants to remain true to his definitions of it (after all, if we can't agree on a common set of definitions, a common lexicon, then we can't really make much progress as an industry), but...

... the pragmatist in me keeps saying, "so what"?

Look, at the end of the day, if your system wants to misuse HTTP, abuse HTML, and carnally violate the principles of loose coupling and resource representation that underlie REST, who cares? Do you get special bonus points from the Apache Foundation if you use HTTP in the way Fielding intended? Will Microsoft and Oracle and Sun and IBM offer you discounts on your next software purchases if you create a REST-faithful system? Will the partisan politics in Washington, or the tribal conflicts in the Middle East, or even the widely-misnamed "REST-vs-SOAP" debates come to an end if you only figure out a way to make hypermedia the core engine of your application state?

Yeah, I didn't think so, either.

Point is, REST is just an architectural style. It is nothing more than another entry alongside such things as client-server, n-tier, distributed objects, service-oriented, and embedded systems. REST is just a tool for thinking about how to build an application, and it's high time we kick it off the pedastal on which we've placed it and let it come back down to earth with the rest of us mortals. HTTP is useful, but not sufficient, so solve our problems. REST is as well.

And at the end of the day, when we put one tool from our tool belt "above all others", we end up building some truly horrendous crap.

Recently, a blog reader asked me if I wasn't doing any speaking any more since I'd joined ThoughtWorks, and that's when I realized I'd been bad about updating my speaking calendar on the website. Sorry, all; no, ThoughtWorks didn't pull my conference visa or anything, I've just been bad about keeping it up to date. I'll fix that ASAP, but in the meantime, three events that I'll be at in the coming wintry months include:

Øredev 2008: 19 - 21 November, Malmoe, Sweden

Øredev will be a first for me, and I've ben invited to give a keynote there, along with a few technical sessions. I'm also told that .NET Rocks! will be on hand, and that they want to record a session, on whichever topic happens to cross the curious, crafty and cunning Carl, or the uh... the uh... sorry, Richard, there's just no good "R" adjectives I can use here. I mean, "rough" and "ready" don't exactly sound flattering in this context, right? Sorry, man.

In any event, I'm looking forward to this event, because it's a curious mix of technologies and ideas (agile, ALT.NET, Java, core .NET, languages, and so on), and because I've never been to Sweden before. One more European country, off my bucket list!

(Yes, I had to cut-and-paste the Ø wherever I needed it. *grin*)

DevTeach 2008: 1 - 5 December, Montreal, Quebec (Canada)

This has been one of my favorite shows since it began, way back in 2003, and a large part of that love has to do with the cast and crew of characters that I see there every year: Julie Lerman, Peter DeBetta, Carl and Richard (again!), Beth Massi, "Yag" Griver, Mario Cardinal and the rest of the Quebecois posse, Ayende, plus some new faces and friends, like Jessica Moss and James Kovacs. (Oh, and for the record, folks, for those of you who are still talking about it, the O/R-M smackdown of a year ago was staged. It was all fake. Ayende and I are really actually friends, we were paid a great deal of money by Carl and Richard to make it sound good, and in fact, we both agree that the only place anybody should really ever store their data is in an XML database.)

If you're near Montreal, and you're a .NET dev, you really owe it to yourself to check this show out.

Update: I just got this email from Jean-Rene, the guy who runs DevTeach:

Every attendees will get Visual Studio 2008 Pro, Expression Web 2 and Tech-Ed DEV set in their bag!

DevTeach believe that all developers need the right tool to be productive. This is what we will give you, free software, when you register to DevTeach or SQLTeach. Yes that right! We’re pleased to announce that we’re giving over a 1000$ of software when you register to DevTeach. You will find in your conference bag a version of Visual Studio 2008 Professional, ExpressionTM Web 2 and the Tech-Ed Conference DVD Set. Is this a good deal or what? DevTeach and SQLTeach are really the training you can’t get any other way.

Not bad. Not bad at all.

DeVoxx 2008: 8 - 12 December, Antwerp, Belgium

DeVoxx, the recently-renamed-formerly-named-JavaPolis conference, has brought me back to team up with Bill Venners to do a University session on Scala, and to record a few more of those Parlays videos that people can't seem to get enough of. Given that this show always seems to draw some of the Java world's best and brightest, I'm definitely looking forward to the chance to point the mike at somebody's grill and give 'em hell! Plus, I love Belgium, and I'm looking forward to getting back there. The fact that it's going to be the middle of winter is only a bonus, as... wait... Belgium, in the middle of winter? Whose bright idea was that?

(And finally, a show that Carl and Richard won't be at!)

Meanwhile, I promise to keep the "Upcoming Events" up to date for 2009. Seriously. I mean it.

PDC 2008 in LA is over now, and like most PDCs, it definitely didn't disappoint on the technical front--Microsoft tossed out a whole slew of new technologies, ideas, releases, and prototypes, all with the eye towards getting bits (in this case, a Western Digital 160 GB USB hard drive) out to the developer community and getting back feedback, either through the usual channels or, more recently, the blogosphere.

These are the things I think I think about this past PDC:

• Windows 7 will be an interesting thing to watch--they handed out DVDs in both 32- and 64-bit versions, and it's somewhat reminiscent of the Longhorn DVDs of the last PDC. If you recall, Longhorn (what eventually became known as Vista) looked surprisingly good--if a bit unstable, something common to any release this early--for a while, then Vista itself pretty much fell flat. I think it will be interesting, as a social experiment, to look at what people say about Windows 7 now, compare it to what was said about Vista back in 2004 (which is I think when the last PDC was), and then compare what people say 1, 2 and 3 years after the PDC release.
• Azure dominated a lot of the focus, commensurate with the growing interest/hype around "the cloud". All of this sounds suspiciously familiar to me, thinking back to the early days of SOAP/WSDL, and the intense pressure for Web services to revolutionize IT as we know it. This didn't happen, largely for technical reasons at first (incompatibilities between toolkits most of all), then because people treated it as CORBA++ or DCOM-with-angle-brackets. Azure and "cloud computing" have a different problem: clear definition of purpose. I think too many people have no idea what "the cloud" really is for this to be something to pay much attention to just yet.
• Conference get-togethers and parties are becoming more and more lavish each year, as the various product teams challenge one another for the coveted title of The "Dude, were you there last night? It was amazing!" Party of PDC. For my money, that party was the party at the J Lounge on Wednesday night, complete with three floors of fun, including a wall-projected image of Rock Band, but--here's the rub--I couldn't tell you which team actually hosted the party. There was a Microsoft Dynamics CRM poster up in the middle of the gaming floor (bunch of XBox 360s, though not networked together, which I found disappointing), so I'm assuming it had something to do with them, but.... I think Microsoft product teams may want to consider saving some budget and instead of hiring six LA Lakers Cheerleaders to sit on a couch and allow drooling geeks to take pictures with them (no touching!), use the money to make the party--and the hosts--stick in my mind more effectively, or at least use it to hand out technical data on whatever it is they're building.
• The vendor floor competition for attention is getting a little cutthroat. DevExpress stole the show this year, importing--no joke--an actor, "Mini-Me", Vern, to essentially echo (badly) anything Mark Miller (dressed, of course, as Austin Powers' arch-nemesis Dr. Evil) tried to say about the most recent version of CodeRush. Granted, Mark's new "do" (and the absurdly large head that was hiding underneath) makes it easy for him to do a good Dr. Evil impression, but other than that, there was really nothing parallel in the situation--despite Mark's insistence on writing code with evil Flying Spaghetti Monsters or what not in it. I think if you're a vendor and you want to make a splash at PDC, you think long and hard about an effective tie-in, like Infragistics' clever "I flew 1500 miles for this T-shirt" they were giving away.
• The language world was a bit abuzz at the barely-concealed C# 4.0 features, mostly centering around the new "dynamic" keyword and the C# REPL loop capabilities, but noticeably absent was any similar kind of talk or buzz around VB 10. Even C++ got more attention than VB did, with a presentation clearly intending to call out a direct reference to Visual C++'s heyday, "Visual C++: Why 10 is the new 6". Conversations I had with a few Microsofties make it pretty clear that VB is now the red-headed stepchild of the .NET language family, and that fact is going to start making itself widely felt through the rest of the ecosystem before long, particularly now that rumors are beginning to circulate that pretty much all the "gifted kids" that were on the VB team have gone to find other places to exercise their intellect and innovation, such as the Oslo team. I think Microsoft is going to find itself in an uncomfortable position soon, of trying to kill VB off without appearing like they are trying to kill VB off, lest they create another "VB revolution" like the one in 2001 when unmanaged VB'ers ("Classic VBers"?) looked at VB.NET and collectively puked.
• Speaking of collective revolution, anybody remember Visual FoxPro? Those guys are still kicking, and they were always a small fraction of the developer community, comparatively against VB, at least. I think Microsoft is in trouble here, of their own making, for not defining distinct and clearly differentiated roles for Visual Basic and C#.
• The DLR is quickly moving into a position of high importance in my mind, and the fact that it now builds on top of expression trees (from C# 3.0/LINQ) and builds its trees in such a way that they look almost identical to what a corresponding C# or VB tree would look like means that the DLR is about a half-step away from becoming the most critical part of the .NET ecosystem, second only to the CLR itself. I think that while certain Microsoft releases, like Oslo, PowerShell, C# or VB, won't adopt the DLR as a core component to their implementation, developers looking to explore the DSL space will find the DLR a very happy place to be, particularly in combination with F# Parser Expression Grammars.
• Speaking of F#, it's pretty clear that it was the developer darling--if not the media darling--of the show. The F# Hands-on-Lab looked to be one of the more popular ones used there, and every time I or my co-author, Amanda Laucher, talked with somebody who didn't already know we were working on F# in a Nutshell, they were asking questions about it and trying to understand its role in the world. I think the "cool kids" of the development community are going to come to check out F#, find that it can do a lot of what the O-O minded C# and VB can do, discover that the functional approach works well in certain scenarios, and start looking to use that on their new projects.
• I think that if the Microsoft languages family were Weasley family from Harry Potter, C++ would be one of the two older brothers (probably Bill or Charlie, the cool older brothers who've gone on to make their name and don't need to impress anybody any more), Visual Basic would be Percy (desperate for validation and respect), C# would be Ron (cleary an up-and-comer in the world, even if he was a little awkward while growing up), and F# would be Ginny (the spunky one who clearly charts her own path despite her initial shyness, her accidental involvement in a Voldemortian scheme and her parents' and big brothers' interference in her life). Oslo, of course, is Professor Snape--we can't be sure if he's a good guy or a bad guy until the last book.
• Continuing that analogy, by the way, I think Java is clearly Hermione: wickedly book smart, but sometimes too clever by half.

Overall, PDC was an amazing show, and there's clearly a lot of stuff to track. I personally plan to take a deep dive into Oslo, and will probably blog about what I find, but in the meantime, remember that all of the PDC bits that we got on the hard drives are available through the various DevCenters (or so I've been told), so have a look. There's a lot more there than just what I mentioned above.

Update: Lisa Feigenbaum emailed me with a correction: there was a session on VB 10 at PDC, and I simply missed it in the schedule. In fact, she was very subtle about it, simply asking me, "Did you make it to the VB talk?" and posted this URL along with it. Lisa, I stand corrected. Having said that, though, I still stand by the other points of that piece: that the buzz I was hearing (which may very well have simply been the social circles I run in, I'll be the first to admit it, but I can only speak to my experience here and am very willing to be told I'm full of poopie on this one) was all C#, no VB, and that it bothers me that notable members of the VB team have departed for other parts of the company. Please, nothing would make me happier than to see VB stand as a full and equal partner in the .NET family of languages, but right now, it really still feels like the red-headed stepchild. Please, prove me wrong.

The full list is here. It's a pretty prestigious group--and I'm totally floored that I'm there next to some pretty big names.

In homage to Ms. Sally Fields, of so many years ago... "You like me, you really like me". Having somebody come up to me at a conference and tell me how much they like my blog is second on my list of "fun things to happen to me at a conference", right behind having somebody come up to me at a conference and tell me how much they like my blog, except for that one entry, where I said something totally ridiculous (and here's why) ....

What I find most fascinating about the list was the means by which it was constructed--the various calculations behind page rank, technorati rating, and so on. Very cool stuff.

Perhaps it's trite to say it, but it's still true: readers are what make writing blogs worthwhile. Thanks to all of you.

As Joel points out, we've made a draft of the SSCLI 2.0 Internals book available for download (via his blog). Rather than tell you all about the book, which Joel summarizes quite well, instead I thought I'd tell you about the process by which the book came to be.

Editor's note: if you have no interest in the process by which a book can get done, skip the rest of this blog entry.

One thing that readers will note that's different about this version of "the Rotor book" is that it's not being done through one of the traditional publishers. This is deliberate. As Joel and I talk about on the .NET Rocks! show we did together, the first Rotor book was on the first version of Rotor, which shipped shortly after the .NET 1.1 bits shipped to customers. That was back in the summer of 2001. Dave, Geoff and I shipped the book, I did a few conference talks on Rotor for the relatively few people who had an interest in what was going on "under the hood" of the CLR, and then we all sort of parted ways. (Dave retired from Microsoft entirely shortly thereafter, in order "to focus on the two things that matter in life: making music and making wine", as he put it.) Mission accomplished, we moved on.

Meanwhile, as we all knew would happen, the world moved on--Whidbey (.NET 2.0) shipped, and with it came a whole slew of CLR enhancements, most notably generics. Unlike how generics happened in the JVM, CLR generics are carried through all the way to the type system, and as a result, a lot of what we said in the first Rotor book was instantly rendered obsolete. Granted, one could always grab the Gyro patch for Rotor and see what generics would have looked like, but even that was pretty much rendered obsolete by the emergence of the SSCLI 2.0 drop, bringing the Rotor code up to date with the Whidbey production CLR release.

Except the book was, to be blunt about it, left behind.

Speaking honestly, the book never broke any sales records. Sure, for a while there it was the #1 best-selling book (in Redmond, WA, to my total shock and surprise) on Amazon, but we never had the kind of best-seller success that that of, say, Programming Ruby or pick-your-favorite-ASP.NET book. In the book publishing world, this was kind of the moral equivalent to watching your neighbors' slide show of their vacation: boring for most people not in the pictures, unless you were really interested in either the place they were visiting or what they did there. Most of our audience were either people working on the CLR itself (hence all the copies sold in Redmond, get it?), people who were researching on the CLR (such as the various Rotor research projects that came over a few years after its release), or people who just had that itch to "get wonky with it" and learn how some of the structures worked. Granted, a lot of what those people in the last category learned turned out to be pretty helpful in the Real World, but it was a payoff that came with a pretty non-trivial learning curve.

Fast-forward a few years, to the end of calendar year 2005.

By this point, .NET 2.0 has been out in production form for a bit, and Mark Lewin, then of Microsoft University Relations (I think that was his job, but to be honest my recollection on that point is kinda fuzzy) approached me: Microsoft was interested in seeing a second edition of the book out, to keep the Rotor community up to date with what was going on in the state of the art in the CLR. Was I interested? Sure, but the rules surrounding a multi-author book and subsequent editions are pretty clear: everybody has to be given right of first refusal. Thus a two-fold task was under way: find a co-author (preferably somebody from the CLR team, since my skills had never really been in navigating the Rotor source code in the first place, and I hadn't really spent a significant amount of time in the code since 2001), and get Geoff and Dave to indicate--in a very proper legal fashion--that they were passing on the second edition.

Ugh. Lawyers. Contracts. Bleah.

John Osborn then broke the bad news: OReilly wasn't interested in doing a second edition. I couldn't really blame them, since the first hadn't broken any kind of sales record, but I was a bit bummed because I thought this was the end of the road.

Mark Lewin to the rescue. Apparently his part of Microsoft really wanted this book out, to the point where they were willing to fund the effort, if I and my co-author were still interested. Sure, that sounded like a workable idea. And once the book was done, maybe we could publish it through MSPress, if that sounded like a good idea to me. Sure, that sounded good. Then Mark dropped the suggestion that maybe I could talk to Joel Pobar, former CLR geek extraordinaire, to see if he was interested. Joel had impressed me back when we'd briefly touched bases during the first book-writing experience, so yeah, sure, that sounded like a good idea. He was on board pretty quickly, and so we had the first step out of the way.

Next, we had to get OReilly to release their copyright on the first book, so we (and possibly MSPress) could work on and publish the second edition. This turned out to be a huge part of the time between then and now, not owing to any one party's deliberate attempt to derail the process, but just because copies of contracts had to be sent to the original three authors (myself, Stutz and Geoff) to sign over our rights with OReilly to a Creative Commons License, then copies had to be sent to everybody else so all the signatures could appear on one document, and so on.

Did I say it already? Ugh. Lawyers. Contracts. Bleah.

Then, we had to get a contract from Microsoft signed, and that meant more contracts flying back and forth across the fax lines, and then later the US (and Australian) postal system, and that was more delays as the same round of signatures had to be exchanged.

Just for the record: Ugh. Lawyers. Contracts. Bleah.

Finally, though, the die was cast, the authors were ready to go, and.... Hey, does anybody have the latest soft copy of the Word docs we used from the first edition? A quick email to John (Osborn) took longer than we thought, as OReilly tried to find the post-QA docs for us to work from. (I had my own copies, of course, but they were pre-QA, and thus not really what we wanted to start from.) More rounds of emails to try and track those down, so we can get started. Oh, and while we're at it, can we get the figures/graphics, too? They're not in the manuscript directly, so.... Oh, wait, does anybody know how to read .EPS files?

Then began the actual writing process, or, to be more precise, the revision process. We decided on a process similar to the way the first book had been written: Joel, being the "subject matter expert", would take a first pass on the text, and sketch in the rough outlines of what needed to be said. I would then take the prose, polish it up (which in many cases didn't require a whole lot of work, Joel being a great writer in his own right) and rearrange sections as necessary to make it flow more easily, as well as flesh out certain sections that didn't require a former position on the CLR team to write. Joel would then have a look at what I wrote, and assuming I didn't get it completely wrong, would sign off on it, and the chapter/section/paragraph/whatever was done.

And now we're in the process of doing that cosmetic cleanup that's part of the overtime period in book-writing, including generating the table of contents and index, since, it turns out, we'd rather publish it ourselves than through MSPress (which they're OK with). So, readers will have a choice: get the free download from Microsoft's website (once we're done, which should be "real soon now") and read it in soft-copy, or buy it off of Amazon in "treeware version", which will put a modest amount of money into Joel's and my collective pocket (once the relatively modest expenses of self-publishing are covered, that is).

This will be my first experience with self-publishing (as it is for Joel, too), so I'm eager to see how the whole things turns out. One thing I will warn the prospective self-publisher, though: do not underestimate the time you will spend doing those things the editorial/QA/copyedit pass normally handles for you, because it's kind of a pain in the *ss to do it yourself. Still, it's worth it, particularly if you're having a hard time selling your book to a publisher who, for reasons of economy of scale, don't want to publish a niche book (like this one).

Anyway, like many of my blog postings, this post has gone on long enough, so I'll sign off here with a "go read the draft", even if you're a Java or other execution engine/virtual machine kind of developer--seeing the nuts and bolts of a complex execution engine in action is a pretty cool exercise.

Oh, and if anybody's interested in doing a similar kind of effort around the OpenJDK (once it ships), let me know, 'cuz I'm a glutton for punishment....

For those of you who were at the Cinncinnati NFJS show, please continue on to the next blog entry in your reader--you've already heard this. For those of you who weren't, then allow me to make the announcement:

Hi. My name's Ted Neward, and I am now a ThoughtWorker.

After four months of discussions, interviews, more discussions and more interviews, I can finally say that ThoughtWorks and I have come to a meeting of the minds, and starting 3 September I will be a Principal Consultant at ThoughtWorks. My role there will be to consult, write, mentor, architect and speak on Java, .NET, XML Services (and maybe even a little Ruby), not to mention help ThoughtWorks' clients achieve IT success in other general ways.

Yep, I'm basically doing the same thing I've been doing for the last five years. Except now I'm doing it with a TW logo attached to my name.

By the way, ThoughtWorkers get to choose their own titles, and I'm curious to know what readers think my title should be. Send me your suggestions, and if one really strikes home, I'll use it and update this entry to reflect the choice. I have a few ideas, but I'm finding that other people can be vastly more creative than I, and I'd love to have a title that rivals Neal's "Meme Wrangler" in coolness.

Oh, and for those of you who were thinking this, "Seat Warmer" has already been taken, from what I understand.

Honestly, this is a connection that's been hovering at the forefront of my mind for several years. I like ThoughtWorks' focus on success, their willingness to explore new ideas (both methodologies and technologies), their commitment to the community, their corporate values, and their overall attitude of "work hard, play hard". There have definitely been people who came away from ThoughtWorks with a negative impression of the company, but they're the minority. Any company that encourages T-shirts and jeans, XBoxes in the office, and wants to promote good corporate values is a winner in my book. In short, ThoughtWorks is, in many ways, the consulting company that I would want to build, if I were going to build a consulting firm. I'm not a wild fan of the travel commitments, mind you, but I am definitely no stranger to travel, we've got some ideas about how I can stay at home a bit more, and frankly I've been champing at the bit to get injected into more agile and team projects, so it feels like a good tradeoff. Plus, I get to think about languages and platforms in a more competitive and hostile way--not that TW is a competitive and hostile place, mind you, but in that my new fellow ThoughtWorkers will not let stupid thoughts stand for long, and will quickly find the holes in my arguments even faster, thus making the arguments as a whole that much stronger... or shooting them down because they really are stupid. (Either outcome works pretty well for me.)

What does this mean to the rest of you? Not much change, really--I'm still logging lots of hours at conferences, I'm still writing (and blogging, when the muse strikes), and I'm still available for consulting/mentoring/speaking; the big difference is that now I come with a thousand-strong developers of proven capability at my back, not to mention two of the more profound and articulate speakers in the industry (in Neal and Martin) as peers. So if you've got some .NET, Java, or Ruby projects you're thinking about, and you want a team to come in and make it happen, you know how to reach me.

Another DZone newsletter crosses my Inbox, and again I feel compelled to comment. Not so much in the uber-aggressive style of my previous attempt, since I find myself more on the fence on this one, but because I think it's a worthwhile debate and worth calling out.

The article in question is "5 Reasons Why You Don't Want A Jack-of-all-Trades Developer", by Rebecca Murphey. In it, she talks about the all-too-common want-ad description that appears on job sites and mailing lists:

I've spent the last couple of weeks trolling Craigslist and have been shocked at the number of ads I've found that seem to be looking for an entire engineering team rolled up into a single person. Descriptions like this aren't at all uncommon:

Candidates must have 5 years experience defining and developing data driven web sites and have solid experience with ASP.NET, HTML, XML, JavaScript, CSS, Flash, SQL, and optimizing graphics for web use. The candidate must also have project management skills and be able to balance multiple, dynamic, and sometimes conflicting priorities. This position is an integral part of executing our web strategy and must have excellent interpersonal and communication skills.

Her disdain for this practice is the focus of the rest of the article:

Now I don't know about you, but if I were building a house, I wouldn't want an architect doing the work of a carpenter, or the foundation guy doing the work of an electrician. But ads like the one above are suggesting that a single person can actually do all of these things, and the simple fact is that these are fundamentally different skills. The foundation guy may build a solid base, but put him in charge of wiring the house and the whole thing could, well, burn down. When it comes to staffing a web project or product, the principle isn't all that different -- nor is the consequence.

I'll admit, when I got to this point in the article, I was fully ready to start the argument right here and now--developers have to have a well-rounded collection of skills, since anecdotal evidence suggests that trying to go the route of programming specialization (along the lines of medical specialization) isn't going to work out, particularly given the shortage of programmers in the industry right now to begin with. But she goes on to make an interesting point:

The thing is, the more you know, the more you find out you don't know. A year ago I'd have told you I could write PHP/MySQL applications, and do the front-end too; now that I've seen what it means to be truly skilled at the back-end side of things, I realize the most accurate thing I can say is that I understand PHP applications and how they relate to my front-end development efforts. To say that I can write them myself is to diminish the good work that truly skilled PHP/MySQL developers are doing, just as I get a little bent when a back-end developer thinks they can do my job.

She really caught me eye (and interest) with that first statement, because it echoes something Bjarne Stroustrup told me almost 15 years ago, in an email reply sent back to me (in response to my rather audacious cold-contact email inquiry about the costs and benefits of writing a book): "The more you know, the more you know you don't know". What I think also caught my eye--and, I admit it, earned respect--was her admission that she maybe isn't as good at something as she thought she was before. This kind of reflective admission is a good thing (and missing far too much from our industry, IMHO), because it leads not only to better job placements for us as well as the companies that want to hire us, but also because the more honest we can be about our own skills, the more we can focus efforts on learning what needs to be learned in order to grow.

She then turns to her list of 5 reasons, phrased more as a list of suggestions to companies seeking to hire programming talent; my comments are in italics:

So to all of those companies who are writing ads seeking one magical person to fill all of their needs, I offer a few caveats before you post your next Craigslist ad:

1. If you're seeking a single person with all of these skills, make sure you have the technical expertise to determine whether a person's skills match their resume. Outsource a tech interview if you need to. Any developer can tell horror stories about inept predecessors, but when a front-end developer like myself can read PHP and think it's appalling, that tells me someone didn't do a very good job of vetting and got stuck with a programmer who couldn't deliver on his stated skills.

(T: I cannot stress this enough--the technical interview process practiced at most companies is a complete sham and travesty, and usually only succeeds in making sure the company doesn't hire a serial killer, would-be terrorist, or financially destitute freeway-underpass resident. I seriously think most companies should outsource the technical interview process entirely.)

2. A single source for all of these skills is a single point of failure on multiple fronts. Think long and hard about what it will mean to your project if the person you hire falls short in some aspect(s), and about the mistakes that will have to be cleaned up when you get around to hiring specialized people. I have spent countless days cleaning up after back-end developers who didn't understand the nuances and power of CSS, or the difference between a div, a paragraph, a list item, and a span. Really.

(T: I'm not as much concerned about the single point of failure argument here, to be honest. Developers will always have "edges" to what they know, and companies will constantly push developers to that edge for various reasons, most of which seem to be financial--"Why pay two people to do what one person can do?" is a really compelling argument to the CFO, particularly when measured against an unquantifiable, namely the quality of the project.)

3. Writing efficient SQL is different from efficiently producing web-optimized graphics. Administering a server is different from troubleshooting cross-browser issues. Trust me. All are integral to the performance and growth of your site, and so you're right to want them all -- just not from the same person. Expecting quality results in every area from the same person goes back to the foundation guy doing the wiring. You're playing with fire.

(T: True, but let's be honest about something here. It's not so much that the company wants to play with fire, or that the company has a manual entitled "Running a Dilbert Company" that says somewhere inside it, "Thou shouldst never hire more than one person to run the IT department", but that the company is dealing with limited budgets and headcount. If you only have room for one head under the budget, you want the maximum for that one head. And please don't tell me how wrong that practice of headcount really is--you're preaching to the choir on that one. The people you want to preach to are the Jack Welches of the world, who apparently aren't listening to us very much.)

4. Asking for a laundry list of skills may end up deterring the candidates who will be best able to fill your actual need. Be precise in your ad: about the position's title and description, about the level of skill you're expecting in the various areas, about what's nice to have and what's imperative. If you're looking to fill more than one position, write more than one ad; if you don't know exactly what you want, try harder to figure it out before you click the publish button.

(T: Asking people to think before publishing? Heresy! Truthfully, I don't think it's a question of not knowing what they want, it's more trying to find what they want. I've seen how some of these same job ads get generated, and it's usually because a programmer on the team has left, and they had some degree of skill in all of those areas. What the company wants, then, is somebody who can step into exactly what that individual was doing before they handed in their resignation, but ads like, "Candidate should look at Joe Smith's resume on Dice.com (http://...) and have exactly that same skill set. Being named Joe Smith a desirable 'plus', since then we won't have to have the sysadmins create a new login handle for you." won't attract much attention. Frankly, what I've found most companies want is to just not lose the programmer in the first place.)

5. If you really do think you want one person to do the task of an entire engineering team, prepare yourself to get someone who is OK at a bunch of things and not particularly good at any of them. Again: the more you know, the more you find out you don't know. I regularly team with a talented back-end developer who knows better than to try to do my job, and I know better than to try to do his. Anyone who represents themselves as being a master of front-to-back web development may very well have no idea just how much they don't know, and could end up imperiling your product or project -- front to back -- as a result.

(T: Or be prepared to pay a lot of money for somebody who is an expert at all of those things, or be prepared to spend a lot of time and money growing somebody into that role. Sometimes the exact right thing to do is have one person do it all, but usually it's cheaper to have a small team work together.)

(On a side note, I find it amusing that she seems to consider PHP a back-end skill, but I don't want to sound harsh doing so--that's just a matter of perspective, I suppose. (I can just imagine the guffaws from the mainframe guys when I talk about EJB, message-queue and Spring systems being "back-end", too.) To me, the whole "web" thing is front-end stuff, whether you're the one generating the HTML from your PHP or servlet/JSP or ASP.NET server-side engine, or you're the one generating the CSS and graphics images that are sent back to the browser by said server-side engine. If a user sees something I did, it's probably because something bad happened and they're looking at a stack trace on the screen.)

The thing I find interesting is that HR hiring practices and job-writing skills haven't gotten any better in the near-to-two-decades I've been in this industry. I can still remember a fresh-faced wet-behind-the-ears Stroustrup-2nd-Edition-toting job candidate named Neward looking at job placement listings and finding much the same kind of laundry list of skills, including those with the impossible number of years of experience. (In 1995, I saw an ad looking for somebody who had "10 years of C++ experience", and wondering, "Gosh, I guess they're looking to hire Stroustrup or Lippmann", since those two are the only people who could possibly have filled that requirement at the time. This was right before reading the ad that was looking for 5 years of Java experience, or the ad below it looking for 15 years of Delphi....)

Given that it doesn't seem likely that HR departments are going to "get a clue" any time soon, it leaves us with an interesting question: if you're a developer, and you're looking at these laundry lists of requirements, how do you respond?

Here's my own list of things for programmers/developers to consider over the next five to ten years:

1. These "laundry list" ads are not going away any time soon. We can rant and rail about the stupidity of HR departments and hiring managers all we want, but the basic fact is, this is the way things are going to work for the forseeable future, it seems. Changing this would require a "sea change" across the industry, and sea change doesn't happen overnight, or even within the span of a few years. So, to me, the right question to ask isn't, "How do I change the industry to make it easier for me to find a job I can do?", but "How do I change what I do when looking for a job to better respond to what the industry is doing?"
2. Exclusively focusing on a single area of technology is the Kiss of Death. If all you know is PHP, then your days are numbered. I mean no disrespect to the PHP developers of the world--in fact, were it not too ambiguous to say it, I would rephrase that as "If all you know is X, your days are numbered." There is no one technical skill that will be as much in demand in ten years as it is now. Technologies age. Industry evolves. Innovations come along that completely change the game and leave our predictions of a few years ago in the dust. Bill Gates (he of the "640K comment") has said, and I think he's spot on with this, "We routinely overestimate where we will be in five years, and vastly underestimate where we will be in ten." If you put all your eggs in the PHP basket, then when PHP gets phased out in favor of (insert new "hotness" here), you're screwed. Unless, of course, you want to wait until you're the last man standing, which seems to have paid off well for the few COBOL developers still alive.... but not so much for the Algol, Simula, or RPG folks....
3. Assuming that you can stop learning is the Kiss of Death. Look, if you want to stop learning at some point and coast on what you know, be prepared to switch industries. This one, for the forseeable future, is one that's predicated on radical innovation and constant change. This means we have to accept that everything is in a constant state of flux--you can either rant and rave against it, or roll with it. This doesn't mean that you don't have to look back, though--anybody who's been in this industry for more than 10 years has seen how we keep reinventing the wheel, particularly now that the relationship between Ruby and Smalltalk has been put up on the big stage, so to speak. Do yourself a favor: learn stuff that's already "done", too, because it turns out there's a lot of lessons we can learn from those who came before us. "Those who cannot remember the past are condemned to repeat it" (George Santanyana). Case in point: if you're trying to get into XML services, spend some time learning CORBA and DCOM, and compare how they do things against WSDL and SOAP. What's similar? What's different? Do some Googling and see if you can find comparison articles between the two, and what XML services were supposed to "fix" from the previous two. You don't have to write a ton of CORBA or DCOM code to see those differences (though writing at least a little CORBA/DCOM code will probably help.)
4. Find a collection of people smarter than you. Chad Fowler calls this "Being the worst player in any band you're in" (My Job Went to India (and All I Got Was This Lousy Book), Pragmatic Press). The more you surround yourself with smart people, the more of these kinds of things (tools, languages, etc) you will pick up merely by osmosis, and find yourself more attractive to those kind of "laundry list" job reqs. If nothing else, it speaks well to you as an employee/consultant if you can say, "I don't know the answer to that question, but I know people who do, and I can get them to help me".
5. Learn to be at least self-sufficient in related, complementary technologies. We see laundry list ads in "clusters". Case in point: if the company is looking for somebody to work on their website, they're going to rattle off a list of five or so things they want he/she to know--HTML, CSS, XML, JavaScript and sometimes Flash (or maybe now Silverlight), in addition to whatever server-side technology they're using (ASP.NET, servlets, PHP, whatever). This is a pretty reasonable request, depending on the depth of each that they want you to know. Here's the thing: the company does not want the guy who says he knows ASP.NET (and nothing but ASP.NET), when asked to make a small HTML or CSS change, to turn to them and say, "I'm sorry, that's not in my job description. I only know ASP.NET. You'll have to get your HTML guy to make that change." You should at least be comfortable with the basic syntax of all of the above (again, with possible exception for Flash, which is the odd man out in that job ad that started this piece), so that you can at least make sure the site isn't going to break when you push your changes live. In the case of the ad above, learn the things that "surround" website development: HTML, CSS, JavaScript, Flash, Java applets, HTTP (!!), TCP/IP, server operating systems, IIS or Apache or Tomcat or some other server engine (including the necessary admin skills to get them installed and up and running), XML (since it's so often used for configuration), and so on. These are all "complementary" skills to being an ASP.NET developer (or a servlet/JSP developer). If you're a C# or Java programmer, learn different programming languages, a la F# (.NET) or Scala (Java), IronRuby (.NET) or JRuby (Java), and so on. If you're a Ruby developer, learn either a JVM language or a CLR language, so you can "plug in" more easily to the large corporate enterprise when that call comes.
6. Learn to "read" the ad at a higher level. It's often possible to "read between the lines" and get an idea of what they're looking for, even before talking to anybody at the company about the job. For example, I read the ad that started this piece, and the internal dialogue that went on went something like this:

   Candidates must have 5 years experience (No entry-level developers wanted, they want somebody who can get stuff done without having their hand held through the process) defining and developing data driven (they want somebody who's comfortable with SQL and databases) web sites (wait for it, the "web cluster" list is coming) and have solid experience with ASP.NET (OK, they're at least marginally a Microsoft shop, that means they probably also want some Windows Server and IIS experience), HTML, XML, JavaScript, CSS (the "web cluster", knew that was coming), Flash (OK, I wonder if this is because they're building rich internet/intranet apps already, or just flirting with the idea?), SQL (knew that was coming), and optimizing graphics for web use (OK, this is another wrinkle--this smells of "we don't want our graphics-heavy website to suck"). The candidate must also have project management skills (in other words, "You're on your own, sucka!"--you're not part of a project team) and be able to balance multiple, dynamic, and sometimes conflicting priorities (in other words, "You're own your own trying to balance between the CTO's demands and the CEO's demands, sucka!", since you're not part of a project team; this also probably means you're not moving into an existing project, but doing more maintenance work on an existing site). This position is an integral part of executing our web strategy (in other words, this project has public visibility and you can't let stupid errors show up on the website and make us all look bad) and must have excellent interpersonal and communication skills (what job doesn't need excellent interpersonal and communication skills?).

   See what I mean? They want an ASP.NET dev. My guess is that they're thinking a lot about Silverlight, since Silverlight's closest competitor is Flash, and so theoretically an ASP.NET-and-Flash dev would know how to use Silverlight well. Thus, I'm guessing that the HTML, CSS, and JavaScript don't need to be "Adept" level, nor even "Master" level, but "Journeyman" is probably necessary, and maybe you could get away with "Apprentice" at those levels, if you're working as part of a team. The SQL part will probably have to be "Journeyman" level, the XML could probably be just "Apprentice", since I'm guessing it's only necessary for the web.config files to control the ASP.NET configuration, and the "optimizing web graphics", push-come-to-shove, could probably be forgiven if you've had some experience at doing some performance tuning of a website.
7. Be insightful. I know, every interview book ever written says you should "ask questions", but what they're really getting at is "Demonstrate that you've thought about this company and this position". Demonstrating insight about the position and the company and technology as a whole is a good way to prove that you're a neck above the other candidates, and will help keep the job once you've got it.
8. Be honest about what you know. Let's be honest--we've all met developers who claimed they were "experts" in a particular tool or technology, and then painfully demonstrated how far from "expert" status they really were. Be honest about yourself: claim your skills on a simple four-point scale. "Apprentice" means "I read a book on it" or "I've looked at it", but "there's no way I could do it on my own without some serious help, and ideally with a Master looking over my shoulder". "Journeyman" means "I'm competent at it, I know the tools/technology"; or, put another way, "I can do 80% of what anybody can ask me to do, and I know how to find the other 20% when those situations arise". "Master" means "I not only claim that I can do what you ask me to do with it, I can optimize systems built with it, I can make it do things others wouldn't attempt, and I can help others learn it better". Masters are routinely paired with Apprentices as mentors or coaches, and should expect to have this as a major part of their responsibilities. (Ideally, anybody claiming "architect" in their title should be a Master at one or two of the core tools/technologies used in their system; or, put another way, architects should be very dubious about architecting with something they can't reasonably claim at least Journeyman status in.) "Adept", shortly put, means you are not only fully capable of pulling off anything a Master can do, but you routinely take the tool/technology way beyond what anybody else thinks possible, or you know the depth of the system so well that you can fix bugs just by thinking about them. With your eyes closed. While drinking a glass of water. Seriously, Adept status is not something to claim lightly--not only had you better know the guys who created the thing personally, but you should have offered up suggestions on how to make it better and had one or more of them accepted.
9. Demonstrate that you have relevant skills beyond what they asked for. Look at the ad in question: they want an ASP.NET dev, so any familiarity with IIS, Windows Server, SQL Server, MSMQ, COM/DCOM/COM+, WCF/Web services, SharePoint, the CLR, IronPython, or IronRuby should be listed prominently on your resume, and brought up at least twice during your interview. These are (again) complementary technologies, and even if the company doesn't have a need for those things right now, it's probably because Joe didn't know any of those, and so they couldn't use them without sending Joe to a training class. If you bring it up during the interview, it can also show some insight on your part: "So, any questions for us?" "Yes, are you guys using Windows Server 2008, or 2003, for your back end?" "Um, we're using 2003, why do you ask?" "Oh, well, when I was working as an ASP.NET dev for my previous company, we moved up to 2008 because it had the Froobinger Enhancement, which let us...., and I was just curious if you guys had a similar need." Or something like that. Again, be entirely honest about what you know--if you helped the server upgrade by just putting the CDs into the drive and punching the power button, then say as much.
10. Demonstrate that you can talk to project stakeholders and users. Communication is huge. The era of the one-developer team is long since over--you have to be able to meet with project champions, users, other developers, and so on. If you can't do that without somebody being offended at your lack of tact and subtlety (or your lack of personal hygiene), then don't expect to get hired too often.
11. Demonstrate that you understand the company, its business model, and what would help it move forward. Developers who actually understand business are surprisingly and unfortunately rare. Be one of the rare ones, and you'll find companies highly reluctant to let you go.

Is this an exhaustive list? Hardly. Is this list guaranteed to keep you employed forever? Nope. But this seems to be working for a lot of the people I run into at conferences and client consulting gigs, so I humbly submit it for your consideration.

But in no way do I consider this conversation completely over, either--feel free to post your own suggestions, or tell me why I'm full of crap on any (or all) of these.

This comment deserves response:

First of all, if you're quoting my post, blocking out my name, and attacking me behind my back by calling me "our intrepid troll", you could have shown the decency of linking back to my original post. Here it is, for those interested in the real discussion:

http://www.agilesoftwaredevelopment.com/blog/jurgenappelo/professionalism-knowledge-first

Well, frankly, I didn't get your post from your blog, I got it from an email 'zine (as indicated by the comment "This crossed my Inbox..."), and I didn't really think that anybody would have any difficulty tracking down where it came from, at least in terms of the email blast that put it into my Inbox. Coupled with the fact that, quite honestly, I don't generally make a practice of using peoples' names without their permission (and my email to the author asking if I could quote the post with his name attached generated no response), so I blocked out the name. Having said that, I'm pleased to offer full credit as to its source.

Now, let's review some of your remarks:

"COBOL is (at least) twenty years old, so therefore any use of COBOL must clearly be as idiotic."

I never talked about COBOL, or any other programming language. I was talking about old practices that are nowadays considered harmful and seriously damaging. (Like practising waterfall project management, instead of agile project management.) I don't see how programming in COBOL could seriously damage a business. Why do you compare COBOL with lobotomies? I don't understand. I couldn't care less about programming languages. I care about management practices.

Frankly, the distinction isn't very clear in your post, and even more frankly, to draw a distinction here is a bit specious. "I didn't mean we should throw away the good stuff that's twenty years old, only the bad stuff!" doesn't seem much like a defense to me. There are cases where waterfall style development is exactly the right thing to do a more agile approach is exactly the wrong thing to do--the difference, as I'm fond of saying, lies entirely in the context of the problem. Analogously, there are cases where keeping an existing COBOL system up and running is the wrong thing to do, and replacing it with a new system is the right thing to do. It all depends on context, and for that reason, any dogmatic suggestion otherwise is flawed.

"How can a developer honestly claim to know "what it can be good for", without some kind of experience to back it?"

I'm talking about gaining knowledge from the experience of others. If I hear 10 experts advising the same best practice, then I still don't have any experience in that best practice. I only have knowledge about it. That's how you can apply your knowledge without your experience.

Leaving aside the notion that there is no such thing as best practices (another favorite rant of mine), what you're suggesting is that you, the individual, don't necessarily have to have experience in the topic but others have to, before we can put faith into it. That's a very different scenario than saying "We don't need no stinkin' experience", and is still vastly more dangerous than saying, "I have used this, it works." I (and lots of IT shops, I've found) will vastly prefer the latter to the former.

"Knowledge, apparently, isn't enough--experience still matters"

Yes, I never said experience doesn't matter. I only said it has no value when you don't have gained the appropriate knowledge (from other sources) on when to apply it, and when not.

You said it when you offered up the title, "Knowledge, not Experience".

"buried under all the ludicrous hyperbole, he has a point"

Thanks for agreeing with me.

You're welcome! Seriously, I think I understand better what you were trying to say, and it's not the horrendously dangerous comments that I thought you were saying, so I will apologize here and now for believing you to be a wet-behind-the-ears/lets-let-technology-solve-all-our-problems/dangerous-to-the-extreme developer that I've run across far too often, particularly in startups. So, please, will you accept my apology?

"developers, like medical professionals, must ensure they are on top of their game and spend some time investing in themselves and their knowledge portfolio"

Exactly.

Exactly.

"this doesn't mean that everything you learn is immediately applicable, or even appropriate, to the situation at hand"

I never said that. You're putting words into my mouth.

My only claim is that you need to KNOW both new and old practices and understand which ones are good and which ones can be seriously damaging. I simply don't trust people who are bragging about their experience. What if a manager tells me he's got 15 years of experience managing developers? If he's a micro-manager I still don't want him. Because micro-management is considered harmful these days. A manager should KNOW that.

Again, this was precisely the read I picked up out of the post, and my apologies for the misinterpretation. But I stand by the idea that this is one of those posts that could be read in a highly dangerous fashion, and used to promote evil, in the form of "Well, he runs a company, so therefore he must know what he's doing, and therefore having any kind of experience isn't really necessary to use something new, so... see, Mr. CEO boss-of-mine? We're safe! Now get out of my way and let me use Software Factories to build our next-generation mission-critical core-of-the-company software system, even though nobody's ever done it before."

To speak to your example for a second, for example: Frankly, there are situations where a micro-manager is a good thing. Young, inexperienced developers, for example, need more hand-holding and mentoring than older, more senior, more experienced developers do (speaking stereotypically, of course). And, quite honestly, the guy with 15 years managing developers is far more likely to know how to manage developers than the guy who's never managed developers before at all. The former is the safer bet; not a guarantee, certainly, but often the safer bet, and that's sometimes the best we can do in this industry.

"And we definitely shouldn't look at anything older than five years ago and equate it to lobotomies."

I never said that either. Why do you claim that I said this? I don't have a problem with old techniques. The daily standup meeting is a 80-year old best practice. It was already used by pilots in the second world war. How could I be against that? It's fine as it is.

Um... because you used the term "lobotomies" first? And because your title pretty clearly implies the statement, perhaps? (And let's lose the term "best practice" entirely, please? There is no such thing--not even the daily standup.)

It's ok you didn't like my article. Sure it's meant to be provocative, and food for thought. The article got twice as many positive votes than negative votes from DZone readers. So I guess I'm adding value. But by taking the discussion away from its original context (both physically and conceptually), and calling me names, you're not adding any value for anyone.

I took it in exactly the context it was given--a DZone email blast. I can't help it if it was taken out of context, because that's how it was handed to me. What's worse, I can see a development team citing this as an "expert opinion" to their management as a justification to try untested approaches or technologies, or as inspiration to a young developer, who reads "knowledge, not experience", and thinks, "Wow, if I know all the cutting-edge latest stuff, I don't need to have those 10 years of experience to get that job as a senior application architect." If your article was aimed more clearly at the development process side of things, then I would wish it had appeared more clearly in the arena of development processes, and made it more clear that your aim was to suggest that managers (who aren't real big on reading blogs anyway, I've sadly found) should be a bit more pragmatic and open-minded about who they hire.

Look, I understand the desire for a provocative title--for me, the author of "The Vietnam of Computer Science", to cast stones at another author for choosing an eye-catching title is so far beyond hypocrisy as to move into sheer wild-eyed audacity. But I have seen, first-hand, how that article has been used to justify the most incredibly asinine technology decisions, and it moves me now to say "Be careful what you wish for" when choosing titles that meant to be provocative and food for thought. Sure, your piece got more positive votes than negative ones. So too, in their day, did articles on client-server, on CORBA, on Six-Sigma, on the necessity for big up-front design....

Let me put it to you this way. Assume your child or your wife is sick, and as you reach the hospital, the admittance nurse offers you a choice of the two doctors on call. Who do you want more: the doctor who just graduated fresh from medical school and knows all the latest in holistic and unconventional medicine, or the doctor with 30 years' experience and a solid track record of healthy patients?

Recently this little gem crossed my Inbox....

Professionalism = Knowledge First, Experience Last
By J----- A-----

Do you trust a doctor with diagnosing your mental problems if the doctor tells you he's got 20 years of experience? Do you still trust that doctor when he picks up his tools, and asks you to prepare for a lobotomy?

Would you still be impressed if the doctor had 20 years of experience in carrying out lobotomies?

I am always skeptic when people tell me they have X years of experience in a certain field or discipline, like "5 years of experience as a .NET developer", "8 years of experience as a project manager" or "12 years of experience as a development manager". It is as if people's professional levels need to be measured in years of practice.

This, of course, is nonsense.

Professionalism is measured by what you are going to do now...

Are you going to use some discredited technique from half a century ago?
•    Are you, as a .NET developer, going to use Response.Write, because you've got 5 years of experience doing exactly that?
•    Are you, as a project manager, going to create Gantt charts, because that's what you've been doing for 8 years?
•    Are you, as a development manager, going to micro-manage your team members, as you did in the 12 years before now?

If so, allow me to illustrate the value of your experience...

(Photo of "Zero" signs)

Here's an example of what it means to be a professional:

There's a concept called Kanban making headlines these days in some parts of the agile community. I honestly and proudly admit that I have no experience at all in applying Kanban. But that's just a minor inconvenience. Because I have attained the knowledge of what it is and what it can be good for. And now there are some planning issues in our organization for which this Kanban-stuff might be the perfect solution. I'm sure we're going to give it a shot, in a controlled setting, with time allocated for a pilot and proper evaluations afterwards. That's the way a professional tries to solve a problem.

Professionals don't match problems with their experiences. They match them with their knowledge.

Sure, experience is useful. But only when you already have the knowledge in place. Experience has no value when there's no knowledge to verify that you are applying the right experience.

Knowledge Comes First, Experience Comes Last

This is my message to anyone who wants to be a professional software developer, a professional project manager, or a professional development manager.

You must gain and apply knowledge first, and experience will help you after that. Professionals need to know about the latest developments and techniques.

They certainly don't bother measuring years of experience.

Are you still practicing lobotomies?

Um....

Wow.

Let's start with the logical fallacy in the first section. Do I trust a doctor with diagnosing my mental problems if he tells me he's got 20 years of experience? Generally, yes, unless I have reasons to doubt this. If the guy picks up a skull-drill and starts looking for a place to start boring into my skull, sure, I'll start to question his judgement.... But what does this have to do with anything? I wouldn't trust the guy if he picked up a chainsaw and started firing it up, either.

Look, I get the analogy: "Doctor has 20 years of experience using outdated skills", har har. Very funny, very memorable, and totally inappropriate metaphor for the situation. To stand here and suggest that developers who aren't using the latest-and-greatest, so-bleeding-edge-even-saying-the-name-cuts-your-skin tools or languages or technologies are somehow practicing lobotomies (which, by the way, are still a recommended practice in certain mental disorder cases, I understand) in order to solve any and all mental-health issues, is a gross mischaracterization--and the worst form of negligence--I've ever heard suggested.

And it comes as no surprise that it's coming from the CIO of a consulting company. (Note to self: here's one more company I don't want anywhere near my clients' IT infrastructure.)

Let's take this analogy to its logical next step, shall we?

COBOL is (at least) twenty years old, so therefore any use of COBOL must clearly be as idiotic as drilling holes in your skull to let the demons out. So any company currently using COBOL has no real option other than to immediately upgrade all of their currently-running COBOL infrastructure (despite the fact that it's tested, works, and cashes most of the US banking industry's checks on a daily basis) with something vastly superior and totally untested (since we don't need experience, just knowlege), like... oh, I dunno.... how about Ruby? Oh, no, wait, that's at least 10 years old. Ditto for Java. And let's not even think about C, Perl, Python....

I know; let's rewrite the entire financial industry's core backbone in Groovy, since it's only, what, 6 years old at this point? I mean, sure, we'll have to do all this over again in just four years, since that's when Groovy will turn 10 and thus obviously begin it's long slide into mediocrity, alongside the "four humors" of medicine and Aristotle's (completely inaccurate) anatomical depictions, but hey, that's progress, right? Forget experience, it has no value compared to the "knowledge" that comes from reading the documentation on a brand-new language, tool, library, or platform....

What I find most appalling is this part right here:

I honestly and proudly admit that I have no experience at all in applying Kanban. But that's just a minor inconvenience. Because I have attained the knowledge of what it is and what it can be good for.

How can a developer honestly claim to know "what it can be good for", without some kind of experience to back it? (Hell, I'll even accept that you have familiarity and experience with something vaguely relating to the thing at hand, if you've got it--after all, experience in Java makes you a pretty damn good C# developer, in my mind, and vice versa.)

And, to make things even more interesting, our intrepid troll, having established the attention-gathering headline, then proceeds to step away from the chasm, by backing away from this "knowledge-not-experience" position in the same paragraph, just one sentence later:

I'm sure we're going to give it a shot, in a controlled setting, with time allocated for a pilot and proper evaluations afterwards.

Ah... In other words, he and his company are going to experiment with this new technique, "in a controlled setting, with time allocated for a pilot and proper evaluations afterwards", in order to gain experience with the technique and see how it works and how it doesn't.

In other words....

.... experience matters.

Knowledge, apparently, isn't enough--experience still matters, and it matters a lot earlier than his "knowledge first, experience last" mantra seems to imply. Otherwise, once you "know" something, why not apply it immediately to your mission-critical core?

At the end of the day, buried under all the ludicrous hyperbole, he has a point: developers, like medical professionals, must ensure they are on top of their game and spend some time investing in themselves and their knowledge portfolio. Jay Zimmerman takes great pains to point this out at every No Fluff Just Stuff show, and he's right: those who spend the time to invest in their own knowledge portfolio, find themselves the last to be fired and the first to be hired. But this doesn't mean that everything you learn is immediately applicable, or even appropriate, to the situation at hand. Just because you learned Groovy last weekend in Austin doesn't mean you have the right--or the responsibility--to immediately start slipping Groovy in to the production servers. Groovy has its share of good things, yes, but it's got its share of negative consequences, too, and you'd better damn well know what they are before you start betting the company's future on it. (No, I will not tell you what those negative consequences are--that's your job, because what if it turns out I'm wrong, or they don't apply to your particular situation?) Every technology, language, library or tool has a positive/negative profile to it, and if you can't point out the pros as well as the cons, then you don't understand the technology and you have no business using it on anything except maybe a prototype that never leaves your local hard disk. Too many projects were built with "flavor-of-the-year" tools and technologies, and a few years later, long after the original "bleeding-edge" developer has gone on to do a new project with a new "bleeding-edge" technology, the IT guys left to admin and upkeep the project are struggling to find ways to keep this thing afloat.

If you're languishing at a company that seems to resist anything and everything new, try this exercise on: go down to the IT guys, and ask them why they resist. Ask them to show you a data flow diagram of how information feeds from one system to another (assuming they even have one). Ask them how many different operating systems they have, how many different languages were used to create the various software programs currently running, what tools they have to know when one of those programs fails, and how many different data formats are currently in use. Then go find the guys currently maintaining and updating and bug-fixing those current programs, and ask to see the code. Figure out how long it would take you to rewrite the whole thing, and keep the company in business while you're at it.

There is a reason "legacy code" exists, and while we shouldn't be afraid to replace it, we shouldn't be cavalier about tossing it out, either.

And we definitely shouldn't look at anything older than five years ago and equate it to lobotomies. COBOL had some good ideas that still echo through the industry today, and Groovy and Scala and Ruby and F# undoubtedly have some buried mines that we will, with benefit of ten years' hindsight, look back at in 2018 and say, "Wow, how dumb were we to think that this was the last language we'd ever have to use!".

That's experience talking.

And the funny thing is, it seems to have served us pretty well. When we don't listen to the guys claiming to know how to use something effectively that they've never actually used before, of course.

Caveat emptor.

If you've peeked at my blog site in the last twenty minutes or so, you've probably noticed some churn in the template in the upper-left corner; by now, it's been finalized, and it reads "JOB REFERRALS".

WTHeck? Has Ted finally sold out? Sort of, not really. At least, I don't think so.

Here's the deal: the company behind those ads, Entice Labs, contacted me to see if I was interested in hosting some job ads on my blog, given that I seem to generate a moderate amount of traffic. I figured it was worthwhile to at least talk to them, and the more I did, the more I liked what I heard--the ads are focused specifically at developers of particular types (I chose a criteria string of "Software Developers", subcategorized by "Java, .NET, .NET (Visual Basic), .NET (C#), C++, Flex, Ruby on Rails, C, SQL, JavaScript, HTML" though I'm not sure whether "HTML" will bring in too many web-designer jobs), and visitors to my blog don't have to click through the ads to get to the content, which was important to me. And, besides, given the current economic climate, if I can help somebody find a new job, I'd like to.

Now for the full disclaimer: I will be getting money back from these job ads, though how much, to be honest with you, I'm not sure. I'm really not doing this for the money, so I make this statement now: I will take 50% of whatever I make through this program and donate it to a charitable organization. The other 50% I will use to offset travel and expenses to user groups and/or CodeCamps and/or for-free conferences put on throughout the country. (Email me if you know of one that you're organizing or attending and would like to see me speak at, and I'll tell you if there's any room in the budget left for it. )

Anyway, I figured if the ads got too obnoxious, I could always remove them; it's an experiment of sorts. Tell me what you think.

The Pragmatic Programmer says, "Learn a new language every year". This is great advice, not just because it puts new tools into your mental toolbox that you can pull out on various occasions to get a job done, but also because it opens your mind to new ideas and new concepts that will filter their way into your code even without explicit language support. For example, suppose you've looked at (J/Iron)Ruby or Groovy, and come to like the "internal iterator" approach as a way of simplifying moving across a collection of objects in a uniform way; for political and cultural reasons, though, you can't write code in anything but Java. You're frustrated, because local anonymous functions (also commonly--and, I think, mistakenly--called closures) are not a first-class concept in Java. Then, you later look at Haskell/ML/Scala/F#, which makes heavy use of what Java programmers would call "static methods" to carry out operations, and realize that this could, in fact, be adapted to Java to give you the "internal iteration" concept over the Java Collections:

   1: package com.tedneward.util;

   2:  

   3: import java.util.*;

   4:  

   5: public interface Acceptor

   6: {

   7:   public void each(Object obj);

   8: }

   9:  

  10: public class Collection

  11: {

  12:   public static void each(List list, Acceptor acc)

  13:   {

  14:     for (Object o : list)

  15:       acc.each(o);

  16:   }

  17: }

Where using it would look like this:

   1: import com.tedneward.util.*;

   2:  

   3: List personList = ...;

   4: Collection.each(new Accpetor() {

   5:   public void each(Object person) {

   6:     System.out.println("Found person " + person + ", isn't that nice?");

   7:   }

   8: });

Is it quite as nice or as clean as using it from a language that has first-class support for anonymous local functions? No, but slowly migrating over to this style has a couple of definitive effects, most notably that you will start grooming the rest of your team (who may be reluctant to pick up these new languages) towards the new ideas that will be present in Groovy, and when they finally do see them (as they will, eventually, unless they hide under rocks on a daily basis), they will realize what's going on here that much more quickly, and start adding their voices to the call to start using (J/Iron)Ruby/Groovy for certain things in the codebase you support.

(By the way, this is so much easier to do in C# 2.0, thanks to generics, static classes and anonymous delegates...

   1: namespace TedNeward.Util

   2: {

   3:   public delegate void EachProc<T>(T obj);

   4:   public static class Collection

   5:   {

   6:     public static void each(ArrayList list, EachProc proc)

   7:     {

   8:       foreach (Object o in list)

   9:         proc(o);

  10:     }

  11:   }

  12: }

  13:  

  14: // ...

  15:  

  16: ArrayList personList = ...;

  17: Collection.each(list, delegate(Object person) {

  18:   System.Console.WriteLine("Found " + person + ", isn't that nice?");

  19: });

... though the collection classes in the .NET FCL are nowhere near as nicely designed as those in the Java Collections library, IMHO. C# programmers take note: spend at least a week studying the Java Collections API.)

This, then, opens the much harder question of, "Which language?" Without trying to infer any sort of order or importance, here's a list of languages to consider, with URLs where applicable; I invite your own suggestions, by the way, as I'm sure there's a lot of languages I don't know about, and quite frankly, would love to. The "current hotness" is to learn the languages marked in bold, so if you want to be daring and different, try one of those that isn't. (I've provided some links, but honestly it's kind of tiring to put all of them in; just remember that Google is your friend, and you should be OK. )

• Visual Basic. Yes, as in Visual Basic--if you haven't played with dynamic languages before, try turning "Option Strict Off", write some code, and see how interacting with the .NET FCL suddenly changes into a duck-typed scenario. If you're really curious, have a look at the generated code in Reflector or ILDasm, and notice how the generated code looks a lot like the generated JVM code from other dynamic languages on an execution environment, a la Groovy.
• Ruby (JRuby, IronRuby):
• Groovy: Some call this "javac 2.0"; I'm not sure it merits that title, or the assumption of the mantle of "King of the JVM" that would seem to go with that title, but the fact is, Groovy's a useful language.
• Scala: A "SCAlable LAnguage" for the JVM (and CLR, though that feature has been left to the community to support), incorporating both object-oriented and functional concepts, plus a few new ideas, into a single package. I'm obviously bullish on Scala, given the talks and articles I've done on it.
• F#: Originally OCaml-on-the-CLR, now F# is starting to take on a personality of its own as Microsoft productizes it. Like Scala and Erlang, F# will be immediately applicable in concurrency scenarios, I think. I'm obviously bullish on F#, given the talks, articles, and book I'm doing on it.
• Erlang: Functional language with a strong emphasis on parallel processing, scalability, and concurrency.
• Perl: People will perhaps be surprised I say this, given my public dislike of Perl's syntax, but I think every programmer should learn Perl, and decide for themselves what's right and what's wrong about Perl. Besides, there's clearly no argument that Perl is one of the power tools in every *nix sysadmin's toolbox.
• Python: Again, given my dislike of Python's significant whitespace, my suggestion to learn it here may surprise some, but Python seems to be stepping into Perl's shoes as the sysadmin language tool of choice, and frankly, lots of people like the significant whitespace, since that's how they format their code anyway.
• C++: The grandaddy of them all, in some ways; if you've never looked at C++ before, you should, particularly what they're doing with templates in the Boost library. As Scott Meyers once put it, "We're a long way from Stack<T>!"
• D: Walter Bright's native-compiling garbage-collected successor to C++/Java/C#.
• Objective-C (part of gcc): Great "other" object-oriented C-based language that never gathered the kind of attention C++ did, yet ended up making its mark on the industry thanks to Steve Jobs' love of the language and its incorporation into the NeXT (and later, Mac OS X) toolchain. Obj-C is a message-passing object language, which has some interesting implications in its own right.
• Common Lisp (Steel Bank Common Lisp): What happens when you create a language that holds as a core principle that the language should hold no clear delineation between "code" and "data"? Or that the syntactic expression of the language should be accessible from within that langauge? You get Lisp, and if you're not sure what I'm talking about, pick up a Lisp or a Scheme implementation and start experimenting.
• Scheme (PLT Scheme, SISC): Scheme is one of the earliest dialects of Lisp, and much of the same syntactic flexibility and power of Lisp is in Scheme, as well. While the syntaxes are usually not directly interchangeable, they're close enough that learning one is usually enough.
• Clojure: Rich Hickey (who also built "dotLisp" for the CLR) has done an amazing job of bringing Lisp to the JVM, including a few new ideas, such as some functional concepts and an implementation of software transactional memory, among other things.
• ECMAScript (E4X, Rhino, ES4): If you've never looking at JavaScript outside of the browser, you're in for a surprise--as Glenn Vanderburg put it during one of his NFJS talks, "There's a real programming language in there!". I'm particularly fond of E4X, which integrates XML as a native primitive type, and the Rhino implementation fully supports it, which makes it attractive to use as an XML services implementation language.
• Haskell (Jaskell): One of the original functional languages. Learning this will give a programmer a leg up on the functional concepts that are creeping into other environments. Jaskell is an implementation of Haskell on the JVM, and they've taken the concept of functional further, creating a build system ("Neptune") on top of Jaskell + Ant, to yield a syntax that's... well... more Haskellian... for building Java projects. (Whether it's better/cleaner than Ant is debatable, but it certainly makes clear the functional nature of build scripts.)
• ML: Another of the original functional languages. Probably don't need to learn this if you learn Haskell, but hey, it can't hurt.
• Heron: Heron is interesting because it looks to take on more of the modeling aspects of programming directly into the language, such as state transitions, which is definitely a novel idea. I'm eagerly looking forward to future drops. (I'm not so interested in the graphical design mode, or the idea of "executable UML", but I think there's a vein of interesting ideas here that could be mined for other languages that aren't quite so lofty in scope.)
• HaXe: A functional language that compiles to three different target platforms: its own (Neko), Flash, and/or Javascript (for use in Web DOMs).
• CAL: A JVM-based statically-typed language from the folks who bring you Crystal Reports.
• E: An interesting tack on distributed systems and security. Not sure if it's production-ready, but it's definitely an eye-opener to look at.
• Prolog: A language built around the idea of logic and logical inference. Would love to see this in play as a "rules engine" in a production system.
• Nemerle: A CLR-based language with functional syntax and semantics, and semantic macros, similar to what we see in Lisp/Scheme.
• Nice: A JVM-based language that permits multi-dispatch methods, sometimes known as multimethods.
• OCaml: An object-functional fusion that was the immediate predecessor of F#. The HaXe and MTASC compilers are both built in OCaml, and frankly, it's in a startlingly small number of lines of code, highlighting how appropriate functional languages are for building compilers and interpreters.
• Smalltalk (Squeak, VisualWorks, Strongtalk): Smalltalk was widely-known as "the O-O language that all the C guys turned to in order to learn how to build object-oriented programs", but very few people at the time understood that Smalltalk was wildly different because of its message-passing and loosely/un-typed semantics. Now we know better (I hope). Have a look.
• TCL (Jacl): Tool Command Language, a procedural scripting language that has some nice embedding capabilities. I'd be curious to try putting a TCL-based language in the hands of end users to see if it was a good DSL base. The Jacl implementation is built on top of the JVM.
• Forth: The original (near as I can tell) stack-based language, in which all execution happens on an execution stack, not unlike what we see in the JVM or CLR. Given how much Lisp has made out of the "atoms and lists" concept, I'm curious if Forth's stack-based approach yields a similar payoff.
• Lua: Dynamically-typed language that lives to be embedded; known for its biggest embedder's popularity: World of Warcraft, along with several other games/game engines. A great demonstration of the power of embedding a language into an engine/environment to allow users to create emergent behavior.
• Fan: Another language that seeks to incorporate both static and dynamic typing, running on top of both the JVM or the CLR.
• Factor: I'm curious about Factor because it's another stack-based language, with a lot of inspiration from some of the other languages on this list.
• Boo: A Python-inspired CLR language that Ayende likes for domain-specific languages.
• Cobra: A Python-inspired language that seeks to encompass both static and dynamic typing into one language. Fascinating stuff.
• Slate: A "prototype-based object-oriented programming language based on Self, CLOS, and Smalltalk-80." Apparently on hold due to loss of interest from the founder, last release was 0.3.5 in August of 2005.
• Joy: Factor's primary inspiration, another stack-based language.
• Raven: A scripting language that "rips off" from Python, Forth, Perl, and the creator's own head.
• Onyx: "Onyx is a powerful stack-based, multi-threaded, interpreted, general purpose programming language similar to PostScript. It can be embedded as an extension language similarly to ficl (Forth), guile (scheme), librep (lisp dialect), s-lang, Lua, and Tcl."
• LOLCode: No, you won't use LOLcode on a project any time soon, but LOLCode has had so many different implementations of it built, it's a great practice tool towards building your own languages, a la DSLs. LOLcode has all the basic components a language would use, so if you can build a parser, AST and execution engine (either interpreter or compiler) for LOLcode, then you've got the basic skills in place to build an external DSL.

There's more, of course, but hopefully there's something in this list to keep you busy for a while. Remember to send me your favorite new-language links, and I'll add them to the list as seems appropriate.

Happy hacking!

As Amanda notes, I'm riding with 46 other folks (and lots of beer) on a bus from Michigan to devLink in Tennessee, as part of sponsoring the show. (I think she got my language preferences just a teensy bit mixed up, though.)

Which brings up a related point, actually: Amanda (of "the great F# T-shirt" fame from TechEd this year) and I are teaming up to do F# In A Nutshell for O'Reilly. The goal is to have a Rough Cut ready (just the language parts) by the time F# goes CTP this summer or fall, so we're on an accelerated schedule. If you don't see much from me via the blog for a while, now you know why. Once that's done, I'm going dark on a Scala book to follow--details to follow when that contract is nailed down.

Meanwhile.... As she suggests, the bus will likely be filled with lots of lively debate. The nice thing about having a technical debate with drunk geeks on a bus traveling down the highway at speed is that it's actually pretty easy to win the debate, if you really want to:

"You are such an idiot! Object-relashunal mappers are just... *burp* so cool! Why can't you see that?"

"Idiot, am I? I demand satisfaction! Step outside, sir!"

"Fine, you--" WHOOSH ... THUMP-THUMP....

"Next?"

I'm looking forward to this.

Editor's note: (Contact Amanda if you're interested in participating on the devLink bus, not the book. Thanks for the interest, but we aren't soliciting co-authors. We think we have this one pretty well covered, but we're always interested in reviewers--for that, you can contact either of us.)

Steve Yegge posted the transcript from a talk on dynamic languages that he gave at Stanford.

Cedric Beust posted a response to Steve's talk, espousing statically-typed languages.

Numerous comments and flamewars erupted, not to mention a Star Wars analogy (which always makes things more fun).

This is my feeble attempt to play galactic peacemaker. Or at least galactic color commentary and play-by-play. I have no doubts about its efficacy, and that it will only fan the flames, for that's how these things work. Still, I feel a certain perverse pleasure in pretending, so....

Enjoy the carnage that results.

First of all, let me be very honest: I like Steve's talk. I think he does a pretty good job of representing the negatives and positives of dynamic languages, though there are obviously places where I'm going to disagree:

• "Because we all know that C++ has some very serious problems, that organizations, you know, put hundreds of staff years into fixing. Portability across compiler upgrades, across platforms, I mean the list goes on and on and on. C++ is like an evolutionary sort of dead-end. But, you know, it's fast, right?" Funny, I doubt Bjarne Stroustrup or Herb Sutter would agree with the "evolutionary dead-end" statement, but they're biased, so let's put that aside for a moment. Have organizations put hundreds of staff years into fixing the problems of C++? Possibly--it would be good to know what Steve considers the "very serious problems" of C++, because that list he does give (compiler/platform/language upgrades and portability across platforms) seems problematic regardless of the langauge or platform you choose--Lord knows we saw that with Java, and Lord knows we see it with ECMAScript in the browser, too. The larger question should be, can, and does, the language evolve? Clearly, based on the work in the Boost libraries and the C++0X standards work, the answer is yes, every bit as much as Java or C#/.NET is, and arguably much more so than what we're seeing in some of the dynamic languages. C++ is getting a standardized memory model, which will make a portable threading package possible, as well as lambda expressions, which is a far cry from the language that I grew up with. That seems evolutionary to me. What's more, Bjarne has said, point-blank, that he prefers taking a slow approach to adopting new features or ideas, so that it can be "done right", and I think that's every bit a fair position to take, regardless of whether I agree with it or not. (I'd probably wish for a faster adoption curve, but that's me.) Oh, and if you're thinking that C++'s problems stem from its memory management approach, you've never written C++ with a garbage collector library.
• "And so you ask them, why not use, like, D? Or Objective-C. And they say, "well, what if there's a garbage collection pause?" " Ah, yes, the "we fear garbage collection" argument. I would hope that Java and C#/.NET have put that particular debate to rest by now, but in the event that said dragon's not yet slain, let's do so now: GC does soak up some cycles, but for the most part, for most applications, the cost is lost in the noise of everything else. As with all things performance related, however, profile.
• "And so, you know, their whole argument is based on these fallacious, you know, sort of almost pseudo-religious... and often it's the case that they're actually based on things that used to be true, but they're not really true anymore, and we're gonna get to some of the interesting ones here." Steve, almost all of these discussions are pseudo-religious in nature. For some reason, programmers like to identify themselves in terms of the language they use, and that just sets up the religious nature of the debate from the get-go.
• "You know how there's Moore's Law, and there are all these conjectures in our industry that involve, you know, how things work. And one of them is that languages get replaced every ten years. ... Because that's what was happening up until like 1995. But the barriers to adoption are really high." I can't tell from the transcript of Steve's talk if this is his opinion, or that this is a conjecture/belief of the industry; in either case, I thoroughly disagree with this sentiment--the barriers to entry to create your own language have never been lower than today, and various elements of research work and available projects just keep making it easier and easier to do, particularly if you target one of the available execution engines. Now, granted, if you want your language to look different from the other languages out there, or if you want to do some seriously cool stuff, yes, there's a fair amount of work you still have to do... but that's always going to be the case. As we find ways to make it easier to build what's "cool" today, the definition of what's "cool" rises in result. (Nowhere is this more clear than in the game industry, for example.) Moore's Law begets Ballmer's Corollary: User expectations double every eighteen months, requiring us to use up all that power trying to meet those expectations with fancier ways of doing things.
• It's a section that's too long to quote directly here, but Steve goes on to talk about how programmers aren't using these alternative languages, and that if you even suggest trying to use D or Scala or [fill in the blank], you're going to get "lynched for trying to use a language that the other engineers don't know. ... And [my intern] is, like, "well I understand the argument" and I'm like "No, no, no! You've never been in a company where there's an engineer with a Computer Science degree and ten years of experience, an architect, who's in your face screaming at you, with spittle flying on you, because you suggested using, you know... D. Or Haskell. Or Lisp, or Erlang, or take your pick." " Steve, with all due respect to your experience, I know plenty of engineers and companies who are using some of these "alternative" languages, and they're having some good success. But frankly, if you work in a company where an architect is "in your face screaming at you, with spittle flying on you", frankly, it's time to move on, because that company is never going to try anything new. Period. I don't care if we're talking about languages, Spring, agile approaches, or trying a new place for lunch today. Companies get into a rut just as much as individuals do, and if the company doesn't challenge that rut every so often, they're going to get bypassed. Period, end of story. That doesn't mean trying every new thing under the sun on your next "mission-critical" project, but for God's sake, Mr. CTO, do you really want to wait until your competition has bypassed you before adopting something new? There's a lot of project work that goes on that has room for some experimentation and experience-gathering before utilizing something on the next big project.
• "I made the famously, horribly, career-shatteringly bad mistake of trying to use Ruby at Google, for this project. ... And I became, very quickly, I mean almost overnight, the Most Hated Person At Google. And, uh, and I'd have arguments with people about it, and they'd be like Nooooooo, WHAT IF... And ultimately, you know, ultimately they actually convinced me that they were right, in the sense that there actually were a few things. There were some taxes that I was imposing on the systems people, where they were gonna have to have some maintenance issues that they wouldn't have [otherwise had]. Those reasons I thought were good ones." Recognizing the cost of deploying a new platform into the IT sphere is a huge deal that programmers frequently try to ignore in their zeal to adopt something new, and as a result, IT departments frequently swing the other way, resisting all change until it becomes inevitable. This is where running on top of one of the existing execution environments (the JVM or the CLR in particular) becomes so powerful--the actual deployment platform doesn't change, and the IT guys remain more or less disconnected from the whole scenario. This is the principal advantage JRuby and IronPython and Jython and IronRuby will have over their native-interpreted counterparts. As for maintenance issues, aside from the "somebody's gonna have to learn this language" tax (which is a real tax but far less costly, I believe, than most people think it to be), I'm not sure what issues would crop up--the IT guys don't usually change your Java or C# or Visual Basic code in production, do they?
• Steve then gets into the discussion about tools around dynamic languages, and I heartily agree with him: the tool vendors have a much deeper toolchest than we (non-tool vendor programmers) give them credit for, and they're proving it left and right as IDEs get better and better for dynamic languages like Groovy and Ruby. In some areas, though, I think we as developers lean too strongly against our tools, expecting them to be able to do the thinking for us, and getting all grumpy when they can't or don't. Granted, I don't want to give up my IntelliJ any time soon, but let's think about this for a second: if I can't program Java today without IntelliJ, then is that my fault, the language's fault, the industry's fault, or some combination thereof? Or is it maybe just a fact of progress? (Would anybody consider building assembly language in Notepad today? Does that make assembly language wrong? Or just the wrong tool for the job?)
• Steve's point about how Java IDE's miss the Reflective case is a good one, and one that every Java programmer should consider. How much of your Java (or C# or C++) code actually isn't capturable directly in the IDE?
• Steve then goes into the ubiquitous Java-generics rant, and I'll have to admit, he's got some good points here--why didn't we (Java, though this applies just as equally to C#) just let the runtime throw the exception when the cast fails, and otherwise just let things go? My guess is that there's probably some good rationale that presumes you already accept the necessity of more verbose syntax in exchange for knowing where the cast might potentially fail, even though there's plenty of other places in the language where exceptions can be thrown without that verbose syntax warning you of that fact, array indexers being a big one. One thing I will point out, however, in what I believe is a refutation of what Steve's suggesting in this discussion: from my research in the area and my memory about the subject from way back when, the javac compiler really doesn't do much in the way of optimizations, and hasn't tried since about JDK 1.1, for the precise reason he points out: the JITter's going to optimize all this stuff anyway, so it's easier to just relax and let the JITter do the heavy lifting.
• The discussion about optimizations is interesting, and while I think he glosses over some issues and hyper-focuses on others, two points stand out, in my mind: performance hits often come from places you don't expect, and that micro-benchmarks generally don't prove much of anything. Sometimes that hit will come from the language, and sometimes that hit will come from something entirely differently. Profile first. Don't let your intuition get in the way, because your intuition sucks. Mine does, too, by the way--there's just too many moving parts to be able to keep it all straight in your head.

Steve then launches into a series of Q&A with the audience, but we'll let the light dim on that stage, and turn our attention over to Cedric's response.

• "... the overall idea is that dynamically typed languages are on the rise and statically typed languages are on their way out." Actually, the transcript I read seemed to imply that Steve thought that dynamically typed languages are cool but that nobody will use them for a variety of reasons, some of which he agreed with. I thoroughly disagree with Steve's conclusion there, by the way, but so be it ...
• "I'm happy to be the Luke Skywalker to his Darth Vader. ... Evil shall not prevail." Yes, let's not let this debate fall into the pseudo-religious category, shall we? Fully religious debates have such a better track record of success, so let's just make it "good vs evil", in order to ensure emotions get all neatly wrapped throughout. Just remember, Cedric, even Satan can quote the Bible... and it was Jesus telling us that, so if you disagree with anything I say below you must be some kind of Al-Qaeda terrorist. Or something.
  • [Editor's note: Oh, shit, he did NOT just call Cedric a terrorist and a Satanist and invoke the name of Christ in all this. Time to roll out the disclaimer... "Ladies and gentlemen, the views and opinions expressed in this blog entry...."]
  • [Author's note: For the humor-challenged in the crowd, no I do not think Cedric is a terrorist. I like Cedric, and hopefully he still likes me, too. Of course, I have also been accused of being the Antichrist, so what that says about Cedric I'm not sure.]
• Cedric on Scala:
  • "Have you taken a look at implicits? Seriously? Just when I thought we were not just done realizing that global variables are bad, but we have actually come up with better ways to leverage the concept with DI frameworks such as Guice, Scala knocks the wind out of us with implicits and all our hardly earned knowledge about side effects is going down the drain again." Umm.... Cedric? One reaction comes to mind here, and it's best expressed as.... WTF?!? Implicits are not global variables or DI, they're more a way of doing conversions, a la autoboxing but more flexible. I agree that casual use of implicits can get you in trouble, but I'd have thought Scala's "there are no operators just methods with funny names" would be the more disconcerting of the two.
  • "As for pattern matching, it makes me feel as if all the careful data abstraction that I have built inside my objects in order to isolate them from the unforgiving world are, again, thrown out of the window because I am now forced to write deconstructors to expose all this state just so my classes can be put in a statement that doesn't even have the courtesy to dress up as something that doesn't smell like a switch/case..." I suppose if you looked at pattern-matching and saw nothing more than a switch/case, then I'd agree with you, but it turns out that pattern-matching is a lot more powerful than just being a switch/case. I think what Cedric's opposing is the fact that pattern-matching can actually bind to variables expressed in the individual match clauses, which might look like deconstructors exposing state... but that's not the way they get used, from what I've seen thus far. But, hey, just because the language offers it, people will use it wrongly, right? So God forbid a language's library should allow me to, say, execute private methods or access private fields....
• Cedric on the difficulty to impose a non-mainstream language in the industry: "Let me turn the table on you and imagine that one of your coworkers comes to you and tells you that he really wants to implement his part of the project in this awesome language called Draco. How would you react? Well, you're a pragmatic kind of guy and even though the idea seems wacky, I'm sure you would start by doing some homework (which would show you that Draco was an awesome language used back in the days on the Amiga). Reading up on Draco, you realize that it's indeed a very cool language that has some features that are a good match for the problem at hand. But even as you realize this, you already know what you need to tell that guy, right? Probably something like "You're out of your mind, go back to Eclipse and get cranking". And suddenly, you've become *that* guy. Just because you showed some common sense." If, I suppose, we equate "common sense" with "thinking the way Cedric does", sure, that makes sense. But you know, if it turned out that I was writing something that targeted the Amiga, and Draco did, in fact, give us a huge boost on the competition, and the drawbacks of using Draco seemed to pale next to the advantages of using it, then... Well, gawrsh, boss, it jus' might make sense to use 'dis har Draco thang, even tho it ain't Java. This is called risk mitigation, and frankly, it's something too few companies go through because they've "standardized" on a language and API set across the company that's hardly applicable to the problem at hand. Don't get me wrong--you don't want the opposite extreme, which is total anarchy in the operations center as people use any and all languages/platforms available to them on a willy-nilly basis, but the funny thing is, this is a continuum, not a binary switch. This is where languages-on-execution-engines (like the JVM or CLR) gets such a great win-win condition: IT can just think in terms of supporting the JVM or CLR, and developers can then think in whatever language they want, so long it compiles/runs on those platforms.
• Cedric on building tools for dynamic languages: "I still strongly disagree with that. It is different *and* harder (and in some cases, impossible). Your point regarding the fact that static refactoring doesn't cover 100% of the cases is well taken, but it's 1) darn close to 100% and 2) getting closer to it much faster than any dynamic tool ever could. By the way, Java refactorings correcting comments, XML and property files are getting pretty common these days, but good luck trying to perform a reliable method renaming in 100 Ruby files." I'm not going to weigh in here, since I don't write tools for either dynamic or static languages, but watching what the IntelliJ IDEA guys are doing with Groovy, and what the NetBeans guys are doing with Ruby, I'm more inclined to believe in what Steve thinks than what Cedric does. As for the "reliable method renaming in 100 Ruby files", I don't know this for a fact, but I'll be willing to be that we're a lot closer to that than Cedric thinks we are. (I'd love to hear comments from somebody neck-deep in the Ruby crowd who's done this and their experience doing so.)
• Cedric on generics: "I no longer bother trying to understand why complex Generic examples are so... well, darn complex. Yes, it's pretty darn hard to follow sometimes, but here are a few points for you to ponder:
  • 90% of the Java programmers (including myself) only ever use Generics for Collections.
  • These same programmers never go as far as nesting two Generic declarations.
  • For API developers and users alike, Generics are a huge progress.
  • Scala still requires you to understand covariance and contravariance (but with different rules. People seem to say that Scala's rules are simpler, I'm not so sure, but not interested in finding out for the aforementioned reasons)."
  Honestly, Cedric, the fact that 90% of the Java programmers are only using generics for collections doesn't sway me in the slightest. 90% of the world's population doesn't use Calculus, either, but that doesn't mean that it's not useful, or that we shouldn't be trying to improve our understanding of it and how to do useful things with it. After looking at what the C++ community has done with templates (the Boost libraries) and what .NET is doing with its generic system (LINQ and F# to cite two examples), I think Java missed a huge opportunity with generics. Type erasure may have made sense in a world where Java was the only practical language on top of the JVM, but in a world that's coming to accept Groovy and JRuby and Scala as potential equals on the JVM, it makes no sense whatsoever. Meanwhile, when thinking about Scala, let's take careful note that a Scala programmer can go a long way with the langauge before having to think about covariance, contravariance, upper and lower type bounds, simpler or not. (For what it's worth, I agree with you, I'm not sure if they're simpler, either.)
• Cedric on dynamic language performance: "What will keep preventing dynamically typed languages from displacing statically typed ones in large scale software is not performance, it's the simple fact that it's impossible to make sense of a giant ball of typeless source files, which causes automatic refactorings to be unreliable, hence hardly applicable, which in turn makes developers scared of refactoring. And it's all downhill from there. Hello bit rot." There's a certain circular logic here--if we presume that IDEs can't make sense of "typeless source files" (I wasn't aware that any source file was statically typed, honestly--this must be something Google teaches), then it follows that refactoring will be impossible or at least unreliable, and thus a giant ball of them will be unmanageable. I disagree with Cedric's premise--that IDEs can't make sense of dynamic language code--so therefore I disagree with the entire logical chain as a result. What I don't disagree with is the implicit presumption that the larger the dynamic language source base, the harder it is to keep straight in your head. In fact, I'll even amend that statement further: the larger the source base (dynamic or otherwise), the harder it is to keep straight in your head. Abstractions are key to the long-term success of any project, so the language I work with had best be able to help me create those abstractions, or I'm in trouble once I cross a certain threshold. That's true regardless of the language: C++, Java, C#, Ruby, or whatever. That's one of the reasons I'm spending time trying to get my head around Lisp and Scheme, because those languages were all about building abstractions upon abstractions upon abstractions, but in libraries, rather than in the language itself, so they could be swapped out and replaced with something else when the abstractions failed or needed evolution.
• Cedric on program unmaintainability: "I hate giving anecdotal evidence to support my points, but that won't stop me from telling a short story that happened to me just two weeks ago: I found myself in this very predicament when trying to improve a Ruby program that 1) I just wrote a few days before and 2) is 200 lines long. I was staring at an object, trying to remember what it does, failing, searching manually in emacs where it was declared, found it as a "Hash", and then realized I still had no idea what the darn thing is. You see my point..." Ain't nothing wrong with anecdotal evidence, Cedric. We all have it, and if we all examine it en masse, some interesting patterns can emerge. Funny thing is, I've had exactly the same experience with C++ code, Java code, and C# code. What does that tell you? It tells me that I probably should have cooked up some better abstractions for those particular snippets, and that's what I ended up doing. As a matter of fact, I just helped a buddy of mine untangle some Ruby code to turn it into C#, and despite the fact that he's never written (or read) a Ruby program in his life, we managed to flip it over to C# in a couple of hours, including the execution of Ruby code blocks (I love anonymous methods) stored in a string-keyed hash within an array. And this was Ruby code that neither of us had ever seen before, much less written it a few days prior.
• Cedric (and Steve) on error messages: "[Steve said] And the weird thing is, I realized early in my career that I would actually rather have a runtime error than a compile error. [Cedric responded] You probably already know this, but you drew the wrong conclusion. You didn't want a runtime error, you wanted a clear error. One that doesn't lie to you, like CFront (and a lot of C++ compilers even today, I hear) used to spit in our faces. And once I have a clear error message, I much prefer to have it as early as possible, thank you very much." Honestly, I agree with Cedric here: I would much prefer errors before execution, as early as possible, so that there's less chance of my users finding the errors I haven't found yet. And I agree that some of the error messages we sometimes get are pretty incomprehensible, particularly from the C++ compiler during template expansion. But how is that different from the ubiquitous Java "ClassCastException: Cannot cast Person to Person" that arises from time to time? Once you know what the message is telling you, it's easy to know how to fix it, but getting to the point of knowing what the error message is telling you requires a good working understanding of Java ClassLoaders. Do we really expect that any tool--static or dynamic, compiler or runtime, is going to be able to produce error messages that somehow precludes the need to have the necessary background to understand it? All errors are relative to the context from which they are born. If you lack that context, the error message, no matter how well-written or phrased, is useless.
• Cedric on "The dynamic nuclear winter": "[Steve said] And everybody else went and chased static. And they've been doing it like crazy. And they've, in my opinion, reached the theoretical bounds of what they can deliver, and it has FAILED. [Cedric responded] Quite honestly, words fail me here." Wow. Just... wow. I can't agree with Steve at all, that static(ically typed languages) has FAILED, or that they've reached the theoretical bounds of what they can deliver, but neither can I say with complete confidence that statically-typed languages are The Way Forward, either. I think, for the time, chasing statically-typed languages was the right thing to do, because for a long time we were in a position where programmer time was cheaper than computer time; now, I believe that this particular metric has flipped, and that it's time we started thinking about what the costs of programmer time really are. (Frankly, I'd love to see a double-blind study on this, but I've no idea how one would carry that out in a scientific manner.)

So.... what's left?

Oh, right: if Steve/Vader is Cedric/Luke's father, then who is Cedric/Luke's sister, and why is she wearing a copper-wire bikini while strangling the Haskell/ML crowd/Jabba the Hutt?

Maybe this whole Star Wars analogy thing was a bad idea.

Look, at the end of the day, the whole static-vs-dynamic thing is a red herring. It doesn't matter. The crucial question is whether or not the language being used does two things, and how well it does them:

1. Provide the ability to express the concept in your head, and
2. Provide the ability to evolve as the concepts in your head evolve

There are certain things that are just impossible to do in C++, for example. I cannot represent the C++ AST inside the program itself. (Before you jump all over me, C++ers of the world, take careful note: I'm not saying that C++ cannot represent an AST, but an AST of itself, at the time it is executing.) This is something dynamic languages--most notably Lisp, but also other languages, including Ruby--do pretty well, because they're building the AST at runtime anyway, in order to execute the code in the first place. Could C++ do this? Perhaps, but the larger question is, would any self-respecting C++ programmer want to? Look at your average Ruby program--80% to 90% (the number may vary, but most of the Rubyists I talk to agree its somewhere in this range) of the program isn't really using the meta-object capabilities of the language, and is just a "simpler/easier/scarier/unchecked" object language. Most of the weird-*ss Rubyisms don't show up in your average Ruby program, but are buried away in some library someplace, and away from the view of the average Ruby programmer.

Keep the simple things simple, and make the hard things possible. That' should be the overriding goal of any language, library, or platform.

Erik Meijer coined this idea first, and I like it a lot: Why can't we operate on a basic principle of "static when we can (or should), dynamic otherwise"? (Reverse that if it makes you feel better: "dynamic when we can, static otherwise", because the difference is really only one of gradation. It's also an interesting point for discussion, just how much of each is necessary/desirable.) Doing this means we get the best of both worlds, and we can stop this Galactic Civil War before anybody's planet gets blown up.

'Cuz that would suck.

Steve Vinoski thinks to deflate my arguments with suppositions and presumptions, which I cannot simply let stand.

(Sorry, Steve-O, but I think you're out in left field on this one. I'm happy to argue it further with you over beer, but if you want the last word, have at it, and we'll compare scores when we run into each other at the next conference.)

Steve first takes aim at my comparison of the Erlang process model to the *nix process model:

First, Ted says:

Erlang’s reliability model–that is, the spawn-a-thousand-processes model–is not unique to Erlang. In fact, it’s been the model for Unix programs and servers, most notably the Apache web server, for decades. When building a robust system under Unix, a master-slave model, in which a master process spawns (and monitors) n number of child processes to do the actual work, offers that same kind of reliability and robustness. If one of these processes fail (due to corrupted memory access, operating system fault, or what-have-you), the process can simply die and be replaced by a new child process.

There’s really no comparison between the UNIX process model (which BTW I hold in very high regard) and Erlang’s approach to achieving high reliability. They are simply not at all the same, and there’s no way you can claim that UNIX “offers that same kind of reliability and robustness” as Erlang can. If it could, wouldn’t virtually every UNIX process be consistently yielding reliability of five nines or better?

What Steve misses here is that just because something can, doesn't mean it does. Processes in *nix are just as vulnerable to bad coding practices as are processes in Windows or Mac OS X, and let's be very clear: the robustness and reliability of a system is entirely held hostage to the skill and care of the worst programmer on the system. There is a large difference between theory and practice, Steve, and whether somebody takes *nix up on that offer depends a great deal on how much they're interested in building robust and reliable software.

This is where a system's architecture becomes so important--architecture leads developers down a particular path, enabling them to fall into what Rico Mariani once described as "the pit of success", or what I like to call "correctness by default". Windows leads developers down a single-process/multi-thread-based model, and UNIX leads developers down a multi-process-based model. Which one seems more robust and reliable by default to you?

(By the way, Erlang's model is apparently neither processes nor threads; according to Wikipedia's entry on Erlang,

Erlang processes are neither operating system processes nor operating system threads, but lightweight processes somewhat similar to Java's original “green threads”. Like operating system processes (and unlike green threads and operating system threads) they have no shared state between them.

Now, I grant you, Wikipedia is about as accurate as graffiti scrawled on the wall, but if that's an incorrect summation, please point me to the documentation that contradicts this and let's fix the Wikipedia entry while we're at it. But in the meantime, assuming this is correct, it means that Erlang's model is similar to the CLR's AppDomain construct, which has been around since .NET 1.0, and Java's proposed "Isolate" feature which has yet to be implemented.)

(Oh, and if the argument here is that Erlang's reliability comes from its lack of shared state between threads, hell, man, that's hardly a difficult architecture to cook up. Most transactional systems get there pretty easily, including EJB, though then programmers then go through dozens of hoops to try and break it.)

Next, Steve makes some interesting fundamental assumptions about "high reliability":

Obviously, achieving high reliability requires at least two computers. On those systems, what part of the UNIX process model allows a process on one system to seamlessly fork child processes on another and monitor them over there? Yes, there are ways to do it, but would anyone claim they are as reliable and robust as Erlang’s approach? I sure wouldn’t. Also, UNIX pipes provide IPC for processes on the same host, but what about communicating with processes on other hosts? Yes, there are many, many ways to achieve that as well — after all, I’ve spent most of my career working on distributed computing systems, so I’m well aware of the myriad choices here — but that’s actually a problem in this case: too many choices, too many trade-offs, and far too many ways to get it wrong. Erlang can achieve high reliability in part because it solves these issues, and a whole bunch of other related issues such as live code upgrade/downgrade, extremely well.

I think you're making some serious assumptions about the definition of "high reliability" here, Steve--building reliable software no more depends on having at least two computers as it does having at least two power sources, two network infrastructures, or two continents. Obviously, the more reliable you want to get with your software, the more redundancy you want to have, but that's a continuum, and one man's "high" reliability is another man's "low" reliability. Often, having just two processes running is redundant enough to get the job done.

As for UNIX's process model making it seamless to fork child processes "over there" and monitor them "over here", I know other languages that have supported precisely that since 1995, SR (Synchronizing Resources) being one of them. (SR later was adapted to the JVM to become JR, and the reason I'm aware of them is because I took a couple of classes from both langauges' creator, Ron Olsson, from UC Davis.)

Frankly, I think Steve's reaching with this argument--there's no reasoning here, just "I sure wouldn't [claim that they are as reliable and robust as Erlang's approach]" as persuasion. You like Erlang's ability to spin off child processes, Steve, and that's fine, but let's not pretend that Erlang's doing anything incredibly novel here--it's just taking the UNIX model and incorporating it directly into the language.

And that's part of the problem--any time we incorporate something directly as part of the language, there's all kinds of versioning and revision issues that come with it. This, to my mind, is one of Scala's (and F#'s and Lisp's and Clojure's and Scheme's and other composite languages') greatest strengths, the ability to create constructs that look like they're part of the language, but in fact come from libraries. Libraries are much much easier to revise and adapt right now, largely because we know how to do it, at least compared against what we know about how to do it with languages.

Next, Steve hollers at me for being apparently inconsistent:

Ted continues:

There is no reason a VM (JVM, CLR, Parrot, etc) could not do this. In fact, here’s the kicker: it would be easier for a VM environment to do this, because VM’s, by their nature, seek to abstract away the details of the underlying platform that muddy up the picture.

In your original posting, Ted, you criticized Erlang for having its own VM, yet here you say that a VM approach can yield the best solution for this problem. Aren’t you contradicting yourself?

I do criticize Erlang for having its own VM (though I think it's not a VM, it's an interpreter, which is a far cry from an actual VM), and yet I do believe the VMs can yield the best solution for the problem. The key here is simple: how much energy has gone into making the VM fast, robust, scalable, manageable, monitorable, and so on? The JVM and the CLR have (literally) thousands of man-months sunk into them to reach high levels in all those areas. Can Erlang claim the same? How do I tune Erlang's GC? How do I find out if Erlang's GC is holding objects longer than it should? How do I discover if an Erlang process is about to die due to a low-memory condition? Can I hold objects inside of a weak reference in Erlang? All of these were things developed in the JVM and CLR in response to real customer problems, and once done, were available to any and all languages that run on top of that platform. In order to keep up, Erlang must sink a significant amount of effort into these same scenarios, and I'm willing to bet that Erlang didn't get feature "X" unless Ericsson ran into the need for it themselves.

By the way, the same argument applies to Ruby, at least until you start talking about JRuby or IronRuby. Ditto for Python up until IronPython or Jython (both of which, I understand, now run faster than the native C Python interpreter).

Steve continues attacking my VM-based arguments:

It would be relatively simple to take an Actors-based Java application, such as that currently being built in Scala, and move it away from a threads-based model and over to a process-based model (with the JVM constuction[sic]/teardown being handled entirely by underlying infrastructure) with little to no impact on the programming model.

Would it really be “relatively simple?” Even if what you describe really were relatively simple, which I strongly doubt, there’s still no guarantee that the result would help applications get anywhere near the levels of reliability they can achieve using Erlang.

Actually, yes, it would, because Scala's already done it. Actors is a library, not part of the language, and as such, is extensible in ways we haven't anticipated yet. As for creating and tearing down JVMs automatically, again, JR has done that already. Combining the two is probably not much more than talking to Prof. Olsson and Prof. Odersky for a while, then writing the code for the host; if I get some time, I'll take a stab at it, if nobody's done it before now. More importantly, the lift web framework is seeing some pretty impressive scalability and performance numbers using Actors, thanks to the inherent nature of an Actors model and the intrinsic perf and scale capabilities of the JVM, though I don't know how much anybody's measured its reliability or robustness yet. (How should we measure it, come to think of it?)

Best part is, the IT department doesn't have to do anything different to their existing Java-based network topology to start taking advantage of this. Can you say the same for Erlang?

Continuing...

As to Steve’s comment that the Erlang interpreter isn’t monitorable, I never said that–I said that Erlang was not monitorable using current IT operations monitoring tools. The JVM and CLR both have gone to great lengths to build infrastructure hooks that make it easy to keep an eye not only on what’s going on at the process level (”Is it up? Is it down?”) but also what’s going on inside the system (”How many requests have we processed in the last hour? How many of those were successful? How many database connections have been created?” and so on). Nothing says that Erlang–or any other system–can’t do that, but it requires the Erlang developer build that infrastructure him-or-herself, which usually means it’s either not going to get done, making life harder for the IT support staff, or else it gets done to a minimalist level, making life harder for the IT support staff.

I know what you meant in your original posting, Ted, and my objection still stands. Are you saying here that all Java and .NET applications are by default network-monitoring-friendly, whereas Erlang applications are not? I seem to recall quite a bit of effort spent by various teams at my previous employer to make sure our distributed computing products, including the Java-based products and .NET-based products, played reasonably well with network monitoring systems, and I sure don’t recall any of it being automatic. Yes, it’s nice that the Java and CLR guys have made their infrastructure monitorable, but that doesn’t relieve developers of the need to put actual effort into tying their applications into the monitoring system in a way that provides useful information that makes sense. There is no magic here, and in my experience, even with all this support, it still doesn’t guarantee that monitoring support will be done to the degree that the IT support staff would like to see.

And do you honestly believe Erlang — conceived, designed, implemented, and maintained by a large well-established telecommunications company for use in highly-reliable telecommunications systems — would offer nothing in the way of tying into network monitoring systems? I guess SNMP, for example, doesn’t count anymore?

(Coincidentally, I recently had to tie some of the Erlang stuff I’m currently working on into a monitoring system which isn’t written in Erlang, and it took me maybe a quarter of a workday to integrate them. I’m absolutely certain it would have taken longer in Java.)

Every JVM and CLR process are, by default, network-monitoring-friendly. Java5 introduced JMX, and the CLR has had PerfMon and WMI hooks in it since Day One. Can't make that any clearer. Dunno what kind of efforts your previous employer was going through, but perhaps those efforts were back in the Java 1.4 and earlier days, when JMX wasn't a part of the platform. Frankly, whether the application you're monitoring hooks into the monitoring infrastructure is not really part of the argument, since Erlang doesn't offer that, either. I'm more concerned with whether the infrastructure is monitoring-friendly.

Considering that most IT departments are happy if you give them any monitoring capabilities, having the infrastructure monitoring-friendly is a big deal.

And Steve, if it takes you more than a quarter of a workday to create an MBean-friendly interface, write an implementation of that interface and register the object under an ObjectName with the MBeanServer, then you're woefully out of practice in Java--you should be able to do that in an hour or so.

More to the point, though, if Erlang ties into SNMP out of the box with no work required by the programmer, please tell me where that's doc'ed and how it works! I won't claim to be the smartest Erlang programmer on the block, so anywhere you can point to facts about Erlang that I'm missing, please point 'em out!

Finally, Steve approaches the coup de grace:

But here’s the part of Ted’s response that I really don’t understand:

So given that an execution engine could easily adopt the model that gives Erlang its reliability, and that using Erlang means a lot more work to get the monitorability and manageability (which is a necessary side-effect requirement of accepting that failure happens), hopefully my reasons for saying that Erlang (or Ruby’s or any other native-implemented language) is a non-starter for me becomes more clear.

Ted, first you state that an execution engine could (emphasis mine) “easily adopt the model that gives Erlang its reliability,” and then you say that it’s “a lot more work” for anyone to write an Erlang application that can be monitored and managed? Aren’t you getting those backwards? It should be obvious that in reality, writing a monitorable Erlang app is not hard at all, whereas building Erlang-level reliability into another VM would be a considerably complicated and time-consuming undertaking.

Yes, the JVM could easily adopt the multi-process model if it chose to. (Said work is being done via the Java Isolates JSR.) The CLR already does (via AppDomains). I mean what I say when I say it. If you have facts that disagree, please cite them. You've already stated that Erlang hooks into SNMP, so please, if you want to get the same degree of monitoring and management that the JVM and CLR have, write the full set of monitoring hooks to keep track of all the same things the JVM and CLR track inside their respective VMs and expose them via SNMP. If you're looking for the full set, look either in the javax.management package JavaDoc and take every interface that ends in "MBean", or walk up to any Windows machine with the .NET framework installed and look at the set of PerfMon counters exposed there.

If you really want to prove your point, let's have a bake-off: on the sound of the gun firing, you start adding the monitoring and management hooks to the Erlang interpreter, and I'll add the spin-a-process-off hooks to the CLR or the JVM, and we'll see who's done first. Then we'll release the whole thing to the world and both camps will have been made better for the experience.

Or maybe you could just port Erlang to the JVM or CLR, and then we'll both be happy.

Recently, a former student asked me,

I was in a .NET web services training class that you gave probably 4 or so years ago on-site at a [company name] office in [city], north of Atlanta.  At that time I asked you for a list of the technical blogs that you read, and I am curious which blogs you are reading now.  I am now with a small company where I have to be a jack of all trades, in the last year I have worked in C++ and Perl backend type projects and web frontend projects with Java, C#, and RoR, so I find your perspective interesting since you also work with various technologies and aren't a zealot for a specific one.

Any way, please either respond by email or in your blog, because I think that others may be interested in the list also.

As one might expect, my blog list is a bit eclectic, but I suppose that's part of the charm of somebody looking to study Java, .NET, C++, Smalltalk, Ruby, Parrot, LLVM, and other languages and environments. So, without further ado, I've pasted in the contents of my OPML file for cut&paste and easy import.

Having said that, though, I would strongly suggest not just blindly importing the whole set of feeds into your nearest RSS reader, but take a moment and go visit each one before you add it. It takes longer, granted, but the time spent is a worthy investment--you don't want to have to declare "blog bankruptcy".

Editor's note: We pause here as readers look at each other and go... "WTF?!?"

"Blog bankruptcy" is a condition similar to "email bankruptcy", when otherwise perfectly high-functioning people give up on trying to catch up to the flood of messages in their email client's Inbox and delete the whole mess (usually with some kind of public apology explaining why and asking those who've emailed them in the past to resend something if it was really important), effectively trying to "start over" with their email in much the same way that Chapter Seven or Chapter Eleven allows companies to "start over" with their creditors, or declaring bankruptcy allows private citizens to do the same with theirs. "Blog bankruptcy" is a similar kind of condition: your RSS reader becomes so full of stuff that you can't keep up, and you can't even remember which blogs were the interesting ones, so you nuke the whole thing and get away from the blog-reading thing for a while.

This happened to me, in fact: a few years ago, when I became the editor-in-chief of TheServerSide.NET, I asked a few folks for their OPML lists, so that I could quickly and easily build a list of blogs that would "tune me in" to the software industry around me, and many of them quite agreeably complied. I took my RSS reader (Newsgator, at the time) and dutifully imported all of them, and ended up with a collection of blogs that was easily into the hundreds of feeds long. And, over time, I found myself reading fewer and fewer blogs, mostly because the whole set was so... intimidating. I mean, I would pick at the list of blogs and their entries in the same way that I picked at vegetables on my plate as a child--half-heartedly, with no real enthusiasm, as if this was something my parents were forcing me to do. That just ruined the experience of blog-reading for me, and eventually (after I left TSS.NET for other pastures), I nuked the whole thing--even going so far as to uninstall my copy of Newsgator--and gave up.

Naturally, I missed it, and slowly over time began to rebuild the list, this time, taking each feed one at a time, carefully weighing what value the feed was to me and selecting only those that I thought had a high signal-to-noise ratio. (This is partly why I don't include much "personal" info in this blog--I found myself routinely stripping away those blogs that had more personal content and less technical content, and I figured if I didn't want to read it, others probably felt the same way.) Over the last year or two, I've rebuilt the list to the point where I probably need to prune a bit and close a few of them back down, but for now, I'm happy with the list I've got.

And speaking of which....

   1: <?xml version="1.0"?>

   2: <opml version="1.0">

   3:  <head>

   4:   <title>OPML exported from Outlook</title>

   5:   <dateCreated>Thu, 15 May 2008 20:55:19 -0700</dateCreated>

   6:   <dateModified>Thu, 15 May 2008 20:55:19 -0700</dateModified>

   7:  </head>

   8:  <body>

   9:   <outline text="If broken it is, fix it you should" type="rss"

  10:   xmlUrl="http://blogs.msdn.com/tess/rss.xml"/>

  11:   <outline text="Artima Developer Buzz" type="rss"

  12:   xmlUrl="http://www.artima.com/news/feeds/news.rss"/>

  13:   <outline text="Artima Weblogs" type="rss"

  14:   xmlUrl="http://www.artima.com/weblogs/feeds/weblogs.rss"/>

  15:   <outline text="Artima Chapters Library" type="rss"

  16:   xmlUrl="http://www.artima.com/chapters/feeds/chapters.rss"/>

  17:   <outline text="Neal Gafter's blog" type="rss"

  18:   xmlUrl="http://gafter.blogspot.com/feeds/posts/default"/>

  19:   <outline text="Room 101" type="rss"

  20:   xmlUrl="http://gbracha.blogspot.com/feeds/posts/default"/>

  21:   <outline text="Kelly O'Hair's Blog" type="rss"

  22:   xmlUrl="http://weblogs.java.net/blog/kellyohair/index.rdf"/>

  23:   <outline text="John Rose @ Sun" type="rss"

  24:   xmlUrl="http://blogs.sun.com/jrose/feed/entries/atom"/>

  25:   <outline text="The Daily WTF" type="rss"

  26:   xmlUrl="http://syndication.thedailywtf.com/TheDailyWtf"/>

  27:   <outline text="Brad Wilson" type="rss"

  28:   xmlUrl="http://feeds.feedburner.com/BradWilson"/>

  29:   <outline text="Mike Stall's .NET Debugging Blog" type="rss"

  30:   xmlUrl="http://blogs.msdn.com/jmstall/rss.xml"/>

  31:   <outline text="Stevey's Blog Rants" type="rss"

  32:   xmlUrl="http://steve-yegge.blogspot.com/atom.xml"/>

  33:   <outline text="Brendan's Roadmap Updates" type="rss"

  34:   xmlUrl="http://weblogs.mozillazine.org/roadmap/index.rdf"/>

  35:   <outline text="pl patterns" type="rss"

  36:   xmlUrl="http://plpatterns.blogspot.com/feeds/posts/default"/>

  37:   <outline text="Joel Pobar's weblog" type="rss"

  38:   xmlUrl="http://feeds.feedburner.com/callvirt"/>

  39:   <outline text="Let&amp;#39;s Kill Dave!" type="rss"

  40:   xmlUrl="http://letskilldave.com/rss.aspx"/>

  41:   <outline text="Why does everything suck?" type="rss"

  42:   xmlUrl="http://whydoeseverythingsuck.com/feeds/posts/default"/>

  43:   <outline text="cdiggins.com" type="rss" xmlUrl="http://cdiggins.com/feed"/>

  44:   <outline text="LukeH's WebLog" type="rss"

  45:   xmlUrl="http://blogs.msdn.com/lukeh/rss.xml"/>

  46:   <outline text="Jomo Fisher -- Sharp Things" type="rss"

  47:   xmlUrl="http://blogs.msdn.com/jomo_fisher/rss.xml"/>

  48:   <outline text="Chance Coble" type="rss"

  49:   xmlUrl="http://leibnizdream.wordpress.com/feed/"/>

  50:   <outline text="Don Syme's WebLog on F# and Other Research Projects" type="rss"

  51:   xmlUrl="http://blogs.msdn.com/dsyme/rss.xml"/>

  52:   <outline text="David Broman's CLR Profiling API Blog" type="rss"

  53:   xmlUrl="http://blogs.msdn.com/davbr/rss.xml"/>

  54:   <outline text="JScript Blog" type="rss"

  55:   xmlUrl="http://blogs.msdn.com/jscript/rss.xml"/>

  56:   <outline text="Yet Another Language Geek" type="rss"

  57:   xmlUrl="http://blogs.msdn.com/wesdyer/rss.xml"/>

  58:   <outline text=".NET Languages Weblog" type="rss"

  59:   xmlUrl="http://www.dotnetlanguages.net/DNL/Rss.aspx"/>

  60:   <outline text="DevHawk" type="rss"

  61:   xmlUrl="http://feeds.feedburner.com/Devhawk"/>

  62:   <outline text="The Cobra Programming Language" type="rss"

  63:   xmlUrl="http://cobralang.blogspot.com/feeds/posts/default"/>

  64:   <outline text="Code Miscellany" type="rss"

  65:   xmlUrl="http://codemiscellany.blogspot.com/feeds/posts/default"/>

  66:   <outline text="Fred, Let it go!" type="rss"

  67:   xmlUrl="http://freddy33.blogspot.com/feeds/posts/default"/>

  68:   <outline text="Codedependent" type="rss"

  69:   xmlUrl="http://graphics-geek.blogspot.com/feeds/posts/default"/>

  70:   <outline text="Presentation Zen" type="rss"

  71:   xmlUrl="http://www.presentationzen.com/presentationzen/index.rdf"/>

  72:   <outline text="The Extreme Presentation(tm) Method" type="rss"

  73:   xmlUrl="http://extremepresentation.typepad.com/blog/index.rdf"/>

  74:   <outline text="ZapThink" type="rss"

  75:   xmlUrl="http://feeds.feedburner.com/zapthink"/>

  76:   <outline text="Chris Smith's completely unique view" type="rss"

  77:   xmlUrl="http://feeds.feedburner.com/ChrisSmithsCompletelyUniqueView"/>

  78:   <outline text="Code Commit" type="rss"

  79:   xmlUrl="http://feeds.codecommit.com/codecommit"/>

  80:   <outline

  81:   text="Comments on Ola Bini: Programming Language Synchronicity: A New Hope: Polyglotism"

  82:   type="rss"

  83:   xmlUrl="http://ola-bini.blogspot.com/feeds/5778383724683099288/comments/default"/>

  84:  </body>

  85: </opml>

Happy reading.....

Not too long ago, Don wrote:

The three most “personal” choices a developer makes are language, tool, and OS.

No.

That may be true for somebody who works for a large commercial or open source vendor, whose team is building something that fits into one of those three categories and wants to see that language/tool/OS succeed.

That is not where most of us live. If you do, certainly, you are welcome to your opinion, but please accept with good grace that your agenda is not the same as my own.

Most of us in the practitioner space are using languages, tools and OSes to solve customer problems, and making the decision to use a particular language, tool or OS a personal one generally gets us into trouble--how many developers do you know that identify themselves so closely with that decision that they include it in their personal metadata?

"Hi, I'm Joe, and I'm a Java programmer."

Or, "Oh, good God, you're running Windows? What are you, some kind of Micro$oft lover or something?"

Or, "Linux? You really are a geek, aren't you? Recompiled your kernel lately (snicker, snicker)?"

Sorry, but all of those make me want to hurl. Of these kinds of statements are technical zealotry and flame wars built. When programmers embed their choice so deeply into their psyche that it becomes the tagline by which they identify themselves, it becomes an "ego" thing instead of a "tool" thing.

What's more, it involves customers and people outside the field in an argument that has nothing to do with them. Think about it for a second; the last time you hired a contractor to add a deck to your house, what's your reaction when they introduce themselves as,

"Hi, I'm Kim, and I'm a Craftsman contractor."

Or, overheard at the job site, "Oh, good God, you're using a Skil? What are you, some kind of nut or something?"

Or, as you look at the tools on their belt, "Nokita? You really are a geek, aren't you? Rebuilt your tools from scratch lately (snicker, snicker)?"

Do you, the customer, really care what kind of tools they use? Or do you care more for the quality of solution they build for you?

It's hard to imagine how the discussion can even come up, it's so ludicrous.

Try this one on, instead:

"Hi, I'm Ted, and I'm a programmer."

I use a variety of languages, tools, and OSes, and my choice of which to use are all geared around a single end goal: not to promote my own social or political agenda, but to make my customer happy.

Sometimes that means using C# on Windows. Sometimes that means using Java on Linux. Sometimes that means Ruby on Mac OS X. Sometimes that means creating a DSL. Sometimes that means using EJB, or Spring, or F#, or Scala, or FXCop, or FindBugs, or log4j, or ... ad infinitum.

Don't get me wrong, I have my opinions, just as contractors (and truck drivers, it turns out) do. And, like most professionals in their field, I'm happy to share those opinions with others in my field, and also with my customers when they ask: I think C# provides a good answer in certain contexts, and that Java provides an equally good answer, but in different contexts. I will be happy to explain my recommendation on which languages, tools and OSes to use, because unlike the contractor, the languages, tools, and OSes I use will be visible to the customer when the software goes into Production, at a variety of levels, and thus, the customer should be involved in that decision. (Sometimes the situation is really one where the customer won't see it, in which case the developer can have full confidence in whatever language/tool/OS they choose... but that's far more often the exception than the rule, and will generally only be true in cases where the developer is providing a complete customer "hands-off" hosting solution.)

I choose to be pro-choice.

Recently, it has become the fad to weigh in on the Groovy vs JRuby debate, usually along the lines of "Which is X?", where X is one of "better", "faster", "more powerful", "more acceptable", "easier", and so on. (Everybody seems to have their own adjective/adverb to slide in there, so I won't even begin to try to list them all.)

Rick Hightower, in a blog post from January, weighs in on this and comes down harshly on both Scala and JRuby. Frankly, I found the whole post to ooze bitterness and maybe a touch of jealousy. Some of the highlights:

• "In short: Scala seems like the next over-hyped language." Rick, they're all over-hyped, including your own nominee for the Presidential race, Groovy. I mean, if we're going to weigh this on the grounds of syntax or familiarity, let's throw {ECMA/Java}script into the ring, since it's:
  1. ... been around a lot longer than Groovy and therefore a lot more familiar and comfortable to the programmers that might use either or both,
  2. ... always going to be around, thanks to its inclusion in HTML browsers, and therefore a good investment in your knowledge portfolio regardless of where you end up using it, client- or server- or wherever-side,
  3. ... has many, if not all, of the same features that Groovy (or JRuby) supports,
  4. ... runs on top of the JVM (several ways, including Rhino, which ships with JDK 6 now, and FESI),
  5. ... and has Steve Yegge's vote of confidence, so you know is has to be good, right?
• "Sun please drop JRuby support. It is a waste of time. Take that money and spend it on Groovy which has a compatible syntax to Java. ... Does Ruby and Rails have good ideas? Yes. Borrow them and move on." This seems like a questionable decision to me--why cherry-pick features from one language and port them over to a different language, for no other reason than to say you did? Why not just use said original language in the first place, assuming it can run on your particular platform? Down this path lies the madness that C# and VB have become, as the C# and VB teams seek to create "feature parity" between the two languages, just so that you as a developer can either have your curly-braces-and-semicolons or not. Stupid. Talk about a waste of time and energy. Ruby's syntax is (mostly) vetted, the test cases written, and the featureset understood. Do something different if you're going to create a new language, don't just take the existing features of a language and put new tokens around it. In the South, that's called putting lipstick on the pig: it may be prettier than it was, but underneath, it's still just a pig. (Note: sometimes the new language is designed specifically to be a subset of the feature set of the source language, and I'm completely supportive of that--sometimes it's necessary to scale back just as much as it is to innovate.)
• "After reading the Scala docs, my thought is: while the language features sound great, the syntax makes me want to hurl. Why do things differently just for the sake of it?" Strangely enough, they didn't. (This is frequently the complaint of those who don't understand something. "The designers couldn't have had a good reason for doing it that way, so it must have been just because they wanted to do it differently".) Scala's syntax is actually quite consistent in many ways, particularly if you came from the functional language world, and the underlying rationale is pretty easy to grok... if you bother spending enough time to find out. Scala drops static, for example, because it turns out that Java developers spend a fair amount of time trying to resolve the "should this be a static method or should it be an instance method on a Singleton object" way too often, for example. See Gilad Bracha's arguments against static if you want to find out more of the rationale here. The "def" syntax for method definition is strikingly similar to Groovy, for the same reason: it makes it clear where a definition is taking place. The name-colon-type syntax is deliberate because then it's much easier to leave off the type signatures and let the compiler do the type inferencing for you (a feature, notably, that Rick says he likes). For what it's worth, Rick, here's a lesson I continue to learn the hard way: Spend some time learning the why of something before you take aim and let fly.
• "Final words: I declare the "Ruby will rule the world" fallacy officially over. Remember: Quit pimple pimping Ruby on JRoller! Scala devotees (both of you). Don't even start!" Well, frankly, the "Ruby will rule the world" meme was that over-hyped thing you mentioned earlier, and before anybody starts the next one, let me nip it in the bud: nothing will take over the world. Nothing has taken over the world: not C++, not C, not Java, not C#, not Visual Basic, nothing. The best a language can hope for is to cross what Simon Peyton-Jones calls the "Threshold of Immortality", and lots of languages have done that, too many to list all of them here, though you could probably do so yourself. Some of those include Java, C++, C#, C, Pascal, FORTRAN, COBOL, Perl, Python, Ruby, SQL, maybe even Smalltalk and Lisp and Scheme and the others we normally don't think about.

And we haven't even bothered to go into some kind of feature shootout or performance shootout between any of these guys.

Don't get me wrong--like me, Rick is entirely entitled to his own opinions and he doesn't owe me (or anybody else) a lick of rationale to defend them. But when he comes out and suggests that Sun should drop JRuby entirely in favor of Groovy instead, I feel compelled to point out that there's some logic missing from the reasoning behind that suggestion. Cynics of this blog will suggest that I'm speaking out of both sides of my mouth: that I get to say Perl sucks, and get away with it because it's just one man's opinion but Rick can't say JRuby sucks in turn. Fact is, I'm not suggesting that Larry Wall and chromatic and the others should drop Perl and go work on something more meaningful--quite the opposite, in fact: so long as there are people who continue to use Perl, they have a responsibility to continue to develop and update that language. And Parrot is quite the interesting VM to explore in its own right. But don't expect me any time soon to be writing a bunch of Perl code except under strongly-worded protest to the United Nations.

At the end of the day, the way I think about these languages loosely falls like this:

• C++. For me, programing started here, so I will always have a special place in my heart for it. Templates were vastly more powerful than most people realized until the STL was released, and even to this day, C++ is usually blamed for the complexities of memory management even when garbage collector libraries (like the Boehm collector) were available and could have reduced that complexity significantly. The Boost libraries just blow my mind, and there's some new stuff coming in C++0X that brings C++ to a degree of parity with Java and C#. I wish I could get back to this for a project in the same way that guys fantasize about running into an old high school girlfriend on a business trip.
• C++/CLI. C++ adapted for the CLR. Interesting idea, but it's hard to see why I'd use this, given its syntactical and semantical similarities to C#. Frankly, C++/CLI seems destined to be forever the "glue" language to write managed wrappers on top of unmanaged C/C++ libraries, and that's hardly a compelling reason to pick this guy up for anything beyond that niche area.
• Java. The language I want to feature-freeze, though I do see a value in adding closures, if only to permit closures to enter the design and implementation of the Java libraries, thus making them widely available across all JVM languages. However, if I really got my way, we'd drop the closures-in-Java debate entirely and throw our weight behind John Rose's proposal for method handles in the JVM, since that would enable the same kind of facilities for libraries and without having to rev the Java language significantly. (Lesson to the Java community from the CLR community: not all features of the virtual machine have to be exposed in one language. Not even C# or VB do this.) The JVM I want to continue to enhance and revise and improve.
• Scala. Functional-object hybrid language for the JVM. Pure goodness. Hey, I'm bullish here, I admit it. Scala's type inference makes for lower ceremony, the static type system provides a degree of confidence in code that dynamic languages don't have without programmer-authored unit tests, and the functional nature offers a new design dimension that we haven't been able to easily express before. I won't say that I'm "thinking in Scala", but I'm thinking a lot about Scala these days, and F# too.
• Groovy. "Ruby meets Java in a bar and has a love child." Groovy's syntax is easy and based on Java, and that's both a good and a bad thing. Good if you're a Java programmer who doesn't want to have to reach very far to get some dynamic goodness; bad if you're trying to avoid some of the stranger or syntactically inconsistent aspects of the Java language, or looking to do some entirely new ways of doing things. Personally, I don't find Groovy all that intellectually stimulating, which is both a blessing and a curse.
• JRuby, IronRuby. Ruby on the JVM. 'nuff said. Ditto for IronRuby on to CLR. All the linguistic power (and flaws) of Ruby, on top of the JVM/CLR, which now means it's a far easier sell to the IT boys who run the datacenter.
• C#. The language is great, so long as it retains its original vision and scope. Memo to the C# team: Please let's not try to make C# into a scripting language. Scripting languages have a purpose, and that purpose is generally different from what general-purpose languages do. C# really doesn't need a REPL--don't fall into the trap of trying to make it into Lisp.
• Visual Basic. The language is great (!), so long as it retains its original vision and scope. Yes, I think the language is a good one--you don't really believe how much of a PITA case-sensitivity is until you start programming without it, and suddenly you realize that it's mostly a holdover from the C days. What right-thinking programmer overloads a symbolic name by case? Programmers have died for less than that. So why does case sensitivity matter? More importantly, VB has always been the dynamic language of choice for millions of programmers, it's time for those of us from the C++ community to just own up, admit that there was a place for VB after all, apologize, and let VB go back to being a powerful dynamic language on top of the CLR. Give it a REPL loop, make it the default choice for building top-of-the-stack code, and let VB guys build UIs that call into middle-tier components built by C# and F# guys. Everybody comes out a winner.
• F#. Functional-object hybrid language for the CLR. Pure goodness. The syntax again will seem quirky and strange to people unused to it, but it makes a lot of sense, and compositional construction using higher-order functions is a vastly underestimated and underused design technique. When functions are values, lots of things become possible, as people working in dynamic languages already know.
• Ruby. "Smalltalk meets Perl in a bar and has a love child." I like parts of the Ruby syntax, but there's too many Perl-isms in there for my taste. The fact that Ruby runs on top of its own interpreter (which is neither monitorable nor manageable using IT-datacenter-established tools) is a significant drawback. RoR may be great for vertical silo apps that don't need to integrate with the rest of the datacenter, but that's a pretty scary place to put yourself.
• Python. Dynamic language (goodness) with some functional concepts (goodness) on its own interpreter (badness) with a radical innovation in syntax called significant whitespace to do away with brackets to denote code blocks. Significant whitespace makes it incredibly awkward to embed Python code anywhere but in .py files, meaning Python's suitability for DSLs is reduced significantly. If I could get Python without significant whitespace, I'd be a lot happier camper.
• Jython/IronPython. Python on the JVM/CLR. 'nuff said.
• Perl. Parrot good. Perl syntax and philosophy not one I care for. Use as a shell scripting tool good. Use as a general-purpose programming language not one I recommend. Perl 6's incredibly delayed departure, very bad, unless you're one of those who wants to see Perl become extinct.
• {ECMA/Java}Script. Can we please finally just accept that ES is much more than just a browser extensibility tool? For most developers, this is their first exposure to a classless prototype-based object-oriented language, and unfortunately, most developers don't ever bother exploring it beyond "How do I make my web page do that floating image thing...?" Gah.
• Rhino/FESI/JScript.NET. {ECMA/Java}Script on the JVM/JVM/CLR. 'nuff said, though I wish the JScript guys would incorporate the E4X bits. JScript on the CLR makes for an interesting case study, and maybe (hopefully) they'll use it as another sanity-check for the DLR.
• PowerShell. Scripting language that finally brings much of the power of bash and tcsh and other shells to the Windows world and unifies a ton of different things together into one space: WMI, .NET, COM, and more. Highly necessary for IT admins who've suffered with batch files for decades. Language syntax isn't too bad, and I could even consider using it in an application/system as an extension language to give to power users so I can turn them loose to create emergent behavior without having to keep coming back to me with their feature requests.
• Lisp. With all apologies to Paul Graham, Lisp's window of opportunity (the "woo" factor, as Jay Zimmerman likes to call it) is essentially gone. We will always be looking back at it for ideas, I think, but it's very hard to imagine doing a project that's even remotely near an IT data center in it, for the same reason that Ruby or Erlang are hard to imagine here: running on top of an execution environment that doesn't have managability and monitorability baked in is a non-starter for me. Despite all that, however, programmers owe it to themselves to learn it, because until somebody points it out, you never realize you're color-blind. There's so many interesting ideas in here that you don't even realize what you're missing until you explore it.
• Scheme. See Lisp.
• Haskell. Love it or leave it, but you have to learn it. Functional languages are becoming big, and Haskell is a major influence on them.
• ML. Ditto to Haskell. If you want to see another functional/object hybrid language based on ML, check out OCaml. Note that OCaml is the direct predecessor to F# and the two are frequently (deliberately) syntax-compatible.
• Erlang. Joe Armstrong's baby was built to solve a specific set of problems at Ericsson, and from it we can learn a phenomenal amount about building massively parallel concurrent programs. The fact that it runs on its own interpreter, bad.

And there's still so many more to learn..... but that's the subject of another blog post, coming soon.

Update: Naturally, people complained about the languages that were left off the list. No slight is intended--there's a lot more that I could have included here, and I will go into each of these in more detail (I hope), but there's only so much time in the day, and shipping (or posting, in this case) is always a feature.

From Wikipedia (itself a source of conceptual folk etymology, but that's another rant):

• A commonly held misunderstanding of the origin of a particular word, a false etymology
• "The popular perversion of the form of words in order to render it apparently significant"; "the process by which a word or phrase, usually one of seemingly opaque formation, is arbitrarily reshaped so as to yield a form which is considered to be more transparent"

What do I mean by "technical folk etymology"?

1. If you're a Java developer, consider the term EJB. No, I mean, seriously think about it for a moment. What images are conjured before your eyes when you do so? Horrendous APIs, hideous deployment requirements, complete untestability, and something that takes forty-five minutes to start?
2. If you're a Ruby developer, consider the phrase static typing. Same sort of reaction?
3. If you're a .NET developer, try on COM or COM+ or even ATL. Mystifying collections of code repeated by rote, cut-and-pasted from that very first project you built by hand from Inside OLE 2, and once you got it to work, served as your basic template for every other COM/MTS/COM+ entity you ever built?
4. Or, if you're any of these, how about Visual Basic?
5. Or maybe Web services, specifically all those WS-* specifications?

As the MVP Summit Product Team dinners wound down here in Redmond tonight, I found myself at a table with (not surprisingly) Neal Ford and Venkat Subramaniam, two of my close friends from the NFJS tour, and who should join us but first Amanda Silver (lately of the Office team, but with her heart still firmly rooted in her Visual Basic dev days), then Don Box (lately of the Oslo team) and Chris Sells (also lately of the Oslo team), and a rousing discussion around the concept of DSLs--domain specific languages--arose, largely because Don wanted to sound out Venkat and Neal on the subject.

Listening to the conversation (Don was mostly interested in Neal and Venkat's opinions, so I just relaxed and listened for the most part), I realized that the discussion was entirely rooted in this concept of context, that ephemeral structure surrounding a concept that gives it shape and color and taste and the other aesthetic qualities that lead us to "like" or "dislike" or "accept" or "reject" certain concepts. Don held the position--either for arguments' sake or because he believes it, I'm not sure which--that domain-specific languages lose context too easily once stored on the file system, in ways that data does not. His test was to suggest "What if a random piece of text drops into my email, how do I know what consumes this text?" The answer, of course, is that you don't, unless you somehow have a context by which to understand a piece of text, in many cases based solely on nothing more than filesystem extension, or MIME type, or the "#!/bin/..." line that precedes many shell scripts, and so on.

Interestingly enough, as I drove home after the dinner, I realized that the conversation echoed an exchange Neal and Venkat and I were having in the car on the way over, about how Microsoft (I think) is making a huge mistake by looking to make C# more dynamic in nature[1]. My position was (and is) that Microsoft needs to differentiate the two key languages they offer--C# and Visual Basic--and an obvious way to do so would be to designate VB as the official "dynamic language" for the CLR, and C# as the official "static language" for the CLR, and encourage developers to use C# to build infrastructure (libraries and business types and so on) and VB to build "top of the stack" kinds of code (WinForms, ASP.NET, and so on).

Neal put me squarely back on my heels with this (paraphrased) comment: Microsoft will never do this, because Visual Basic will never be able to shed the image it has gained, that of being the programming language for idiots[2].

Wow.

Sad thing is, he's right. Go back to the terms I suggested you think about at the top of this blog post. If you're like most Java developers, you heard the term "EJB" and immediately got a note of distaste in your mouth. You know that if you suggest EJB on your next Java project, you will be ridiculed and shamed and made to stand in the corner with the Dunce Cap on, even if it makes complete sense from a technical perspective. Companies are choosing instead to build their own transactional-oriented client/server middleware infrastructure, just to avoid the "shame" of using EJB. Because, as we all know, you just can't test EJB.

Which, by the way, is a fallacy, and always has been. Oh, I know, you meant you can't unit-test EJB, but that's a fallacy too. It's always been testable, to the same degree that any servlet application has been testable, it's just that nobody wanted to take the time to figure out how to test it effectively, particularly not once Rod Johnson had unleashed Spring upon the world and Made Everything Better(TM) (or, at least, XML configurable, which is better... right?).

Static typing suffers the same kind of negative prejudice today. Suggest that C++ has a place in the world, and you will be kicked to the curb by any Right-Thinking Technical Leader. Suggest that C++ has a place on your next project, and you're likely to get sternly reprimanded, possibly even cut loose from the project. Suggest anything that doesn't fit with the Way We Build Software Today, and you're swimming upstream, either with management or with your fellow developers.

All because they fall prey to technical folk etymology. They bend the context around the phrases in question to mean something entirely different than what the words actually mean, and as a result, the words take on an aura of snarling, bitter distaste, or, worse, angelic euphoric enlightenment.

Domain-specific languages are the new phrase of the moment, and its emotional context is being built as we speak. Functional languages will be there sometime next year or the year after. For both, the euphoria is growing, and for each, in some period of n (three, maybe four) years will be crashing just as hard as they were built up, just as Ruby's and Visual Basic's and COM's and EJB's and WS-*'s and other technologies have done before it. It's as predictable as the flow of alcohol at an MVP Summit, or the consumption of either caffeine at an all-night code frenzy.

Other industries have varying relationships with this notion of context: the medical field seems to be almost as susceptible to it as we are, particularly the area of weight management and holistic health (remember the water diet? the South Beach diet? the no-sodium diet? the low-cholesterol diet?), whereas traditional engineering disciplines, such as electrical and construction disciplines, seem far less vulnerable to "the hip new thing of the day". I'm not sure why this is, quite honestly, except that software and medicine share the similar characteristics of a rapid influx of new information on a regular, even daily, basis.

People often call me a contrarian, a technical fuddy-duddy who refuses to embrace anything new or anything "bleeding-edge". In many respects, I welcome and accept that label, but frankly, I bristle at the implicit "you just don't want to learn anything new" accusation, because it's a gross misunderstanding and hideous misinterpretation of what I'm really trying to do: Distance myself from the emotional context surrounding a technology, and examine it through the lens of dispassionate observation.

In short, I actively seek to defeat technical folk etymology, if only in the small area I personally can affect.

Do you?

[1] That particular discussion will have to wait for a different blog post on a different day.

[2] I should point out, before the hate mail comes flooding in, that this isn't Neal's own opinion, nor mine--witness my post on "Mort means productivity". What he--and I--refer to here is the reputation Visual Basic has garnered, not the fact surrounding it. And if you care to argue that point, then you're not paying attention to the relative average salary numbers between C# and Visual Basic developers. The laws of economics do not lie.

From the "Where the hell was I that day?" Department....

The JRuby community is pleased to announce the release of JRuby 1.1!

Homepage: http://www.jruby.org/
Download: http://dist.codehaus.org/jruby/

JRuby 1.1 is the second major release for our project.  The main goal for 1.1
has been improving performance.  We have made great strides in performance
during the last nine months.  There have been more and more reports of
applications exceeding Ruby 1.8.6 performance; we are even beating Ruby 1.9
in some microbenchmarks. ...

(Source: http://docs.codehaus.org/display/JRUBY/2008/04/05/JRuby+1.1+Released)

Congratulations to Thomas and Charlie and the rest of the JRuby team; I'm looking forward to playing around with JRuby, specifically in AOT/compiled mode, and for using it as Ruby was originally intended, as a scripting language to make working with systems (in this case, the JVM) easier, a la JMX client scripts to stop-and-start a web application during development/deployment.

(Editor's note: This post is likely to open a huge can of whoop-*ss on this blog, so unless you want to get caught up in the huge bar fight that's about to break out, you're advised to take your whiskey or beer and head outside for a smoke until the cops come.)

As a fellow Scala writer, I've been following Daniel Spiewak's blog with no small amount of interest, as he discovers little tidbits inside the Scala language (like the Option type). Then I ran across this entry, about benchmarks and comparing the performance of Java, Groovy and Scala:

I’ve seen these results dozens of times (looking back at the post), but they never cease to startle me.  How could Groovy be that much slower than everything else?  Granted it is very much a dynamic language, compared to Java and Scala which are peers in static-land.  But still, this is a ray tracer we’re talking about!  There’s no meta-programming involved to muddle the scene, so a halfway-decent optimizer should be able to at least squeeze that gradient down to maybe 5x as slow, rather than a factor of 830.

That's a huge discrepancy, and like Daniel, I'm not sure where the perf hit comes from, particularly when we consider that JRuby, another language with equally powerful metaobject protocol (MOP) capabilities, is turning in performance times that are equal to those we see with the original Ruby interpreter (according to Daniel's blog entry, though I note that the comparison of JRuby to Java isn't given). And if the disbelievers in the crowd are starting to tune this out based on the fact that "Ah, it must be an edge case, after all, there's always one benchmark that any language will fail compared to another one; maybe Groovy's just not cut out to do ray-tracing. Yeah, that must be it. Besides, how often do I really do ray-tracing when I'm writing code at work?", take heed, for Daniel notes this and starts to cite other evidence that seem to establish a disturbing pattern:

If this were an isolated incident, I would probably just blow it off as bad benchmarking, or perhaps an odd corner case that trips badness in the Groovy runtime.  Then a week later, I read this post by Pete Knego (which shows Groovy's performance as equally disappointing, on the order of 7.6x to 56x worse than equivalent Java code --TKN).

All of this is old news, so the question is: Why am I bringing this up now?  Well, I recently saw a post on Groovy Zone by none-other-than Rick Ross, talking about this very subject.  Rick’s post was in response to two posts (here and here), discussing ways to improve Groovy code performance by obfuscating code.

Uh, oh. I don't know about y'all, but anytime somebody is suggesting improving performance by obfuscating code, I'm nervous--almost by definition, code obfuscation makes code run more slowly, not more quickly, because now the bytecode is pulled out of familiar patterns recognizable by the JITter and therefore more aggressively turned into optimized native code. I'm not saying Rick is wrong, but if his experiments are leading him to understand that obfuscated code is somehow running faster than non-obfuscated code, then something deeply strange is afoot.

(Editor's note: Better hurry and head outside folks, the Groovyists in the corner are starting to grumble amongst themselves, working up the courage to toss that first beer in the piano player's face.)

Daniel's not done here, though, and goes on:

Final result?

This text is being written as I was changing and trying things, I gained 20s from
minor changes of which I lost track. " src="http://www.codecommit.com/blog/wp-includes/images/smilies/icon_smile.gif"> I am currently at 1m30s (down from the
original 4m and comparing with Java’s 4s).

I’m sorry, this is acceptable performance?  This is someone who’s spent time trying to optimize Groovy, and by his own admission, Groovy is 23x slower than the equivalent Java code.  Certainly this is a far cry from the 830x slower in the ray tracer benchmark, but in this case it’s simple string manipulation, rather than a mathematically intensive test.

Coming back to Rick’s entry, he looks at the conclusion and has this to say about it:

Language performance is highly overrated

Much is often made of the theoretical “performance” of a language based on benchmarks and arcane tests. There have even been cases where vendors have built cheats into their products specifically so they would score well on benchmarks. In the end, runtime execution speed is not as important a factor as a lot of people would think it is if they only read about performance comparisons. Other factors such as maintainability, interoperability, developer productivity and tool and library support are all very significant, too.

Wait a minute, that sounds a lot like something else I’ve read recently!  Maybe something like this:

Is picking out the few performance weaknesses the right way to judge the
overall speed of Groovy?

To me the Groovy performance is absolutely sufficient because of the
easy integration with Java. If something’s too slow, I do it in Java.
And Java compared to Python is in most cases much faster.

I appreciate the efforts of the Groovy team to improve the performance,
but if they wouldn’t, this would be no real problem to me. Groovy is the
grooviest language with a development team always having the simplicity
and elegance of the language usage in mind - and that counts to me.  " src="http://www.codecommit.com/blog/wp-includes/images/smilies/icon_smile.gif">

This is almost a mantra for the Groovy proponents: performance is irrelevant.  What’s worse, is that the few times where they’ve been pinned down on a particular performance issue that’s obviously a problem, the response seems to be along the lines of: this test doesn’t really show anything, since micro-benchmarks are useless.

I’m sorry, but that’s a cop-out.  Face up to it, Groovy’s performance is terrible.  Anyone who claims otherwise is simply not looking at the evidence.  Oh, and if you’re going to claim that this is just a function of shoe-horning a dynamic language onto the JVM, check out a direct comparison between JRuby and and Groovy.  Groovy comes out ahead in only four of the tests.

Uh, oh.

(Editor's note: Head for the doors, folks--those guys in the corner wearing the black leather jackets sporting the "Grails Rulez" logos on the back have started to head for the center of the room, and they're looking drunk, mean, and angry.)

Here comes the coup de grace

What really bothers me about the Groovy performance debates is that most “Groovyists” seem to believe that performance is in the eye of the beholder.  The thought is that it’s all just a subjective issue and so should be discounted almost completely from the language selection process.  People who say this have obviously forgotten what it means to try to write a scalable non-trivial application which performs decently under load.  When you start getting hundreds of thousands of hits an hour, you’ll be willing to sell your soul for every last millisecond.

The only answer I can think of is that the Groovy core team just doesn’t value performance.  Why else would they consistently bury their heads in the sand, ignoring the issues even when the evidence is right in front of them?  It’s as if they have repeated their own “performance is irrelevant” mantra so many times that they are actually starting to believe it.  It’s unfortunate, because Groovy really is an interesting effort.  I may not see any value for my needs, but I can understand how a lot of people would.  It fills a nice syntactic niche that other languages (such as Ruby) just miss.  But all of its benefits are for naught if it can’t deliver when it counts.

That did it.

(Editor's note: Shiiiiiiiit! I didn't say nothing, HE did, why're you swinging that beer stein at *WHACK*)

^^^^^^^^^^^^^^^^^^^^^

OK, now that we've gotten that out of our system, let's sit back and examine this issue more carefully, shall we?

The fact is, Groovy is slower than it should be. The Groovy guys can mumble about how performance isn't that important and that developer productivity is what really matters and similar kinds of rationale, but at the end of the day, the basic fact remains that Groovy is, by measurement of several different tests, at least an order of magnitude slower than compiled Java code.

Or is it? Funny thing is, looking at some of these tests, they don't say whether the Groovy code was compiled first, or run through the Groovy interpreter. Theoretically this shouldn't matter, since the Groovy architecture essentially compiles the classes generated once read, it might make a difference in practice.

Although Daniel's post doesn't mention it, I went back and double-checked. Peter Knego's benchmark says that "Groovy code was inside a Groovy script, compiled with groovyc.", which leads me to believe that it was compiled code rather than run through the Groovy shell interpreter, but it would be nice if the actual code and batch/command scripts that ran the benchmarks would be available. (He also notes that each time, the benchmark was "warmed up" by running the bechmark five or six times in a loop, presumably to allow the JIT to work its magic, but most notably, doesn't point out which JVM was used, -client or -server.) Meanwhile, Derek Young's ray-tracing example explicitly uses the Groovy interpreter, but defends that decision in comments: "The only reason I didn’t use groovyc was because the difference was so great, and the compilation overhead at the beginning of the run only takes a couple seconds. I decided it wasn’t worth waiting another two and a half hours to time the compiled output. Running with groovy first compiles the code just like groovyc does, then executes that code. It doesn't interpret the source code or run any differently."

So, apparently, it doesn't make much difference. That's not a good development for the Groovy language.

But let's put the hard numbers out of the way for a moment, and concentrate on the much bigger question: does the core Groovy team just not value performance? And, as a corollary to this, does the performance of a language really matter in a day and age when CPUs are still doubling in size and number of cores? Rick's (and others', including myself) positions on this seem fairly clear, that we long ago passed the threshold where programmer time became more expensive than CPU time, and therefore we should optimize based on programmer productivity, not CPU efficiency, and that's important to recognize: a language should enable the programmer to express the core idea without a great deal of "noise" or additional work, what Stu Halloway has coined as "ceremony", and certainly Groovy takes the Java programmer a step closer towards that place of lower ceremony.

But...

But I can't help it, folks. Ted's First Law of Computer Science states that "Dogma is the Root of All Evil", and holding scripting languages up as the last language you'll ever have to use is dogma, plain and simple. Ted's Second Law of Computer Science states, "Context matters", and in this case, the context includes the performance cost of using a language or tool. Taking a performance hit that weighs in at the orders of magnitude mark is just too big to ignore--the ray-tracing example, at its close-to-four-orders-of-magnitude hit, almost suggests that it would have been just as expensive to offload all those calculations through a distributed RPC call to another machine, rather than calculate it locally in Groovy, and when it becomes faster to go off-CPU to do a calculation than to do it locally, something is wrong.

And Daniel's point is good to hear clearly through the noise: "When you start getting hundreds of thousands of hits an hour, you’ll be willing to sell your soul for every last millisecond." Forget getting hundreds or thousands of hits an hour--the real test will be when the system gets hundreds or thousands of hits per second, that's when developers will be scrambling to find ways to eke out those last bits of performance from the system, even if it means selling their last Mountain Dew (which for some is pretty much synonymous to "soul") to whatever entity can give it to them.

So what exactly is my point with this particular entry, besides stirring the pot up a little? In order:

1. Measure for yourself. As with all things performance and scalability related, abstract benchmarks aren't a good measure of how well it works for you and your system. Build a prototype, measure, and then compare that against your performance and scalability goals. You did establish performance and scalability goals as part of your project's runup, right? (If you didn't, then you probably assume your users don't care about performance, and I suspect you'll be rudely surprised on the veracity of that statement before long...)
2. Benchmarks are tricky things. Programmers could learn something from politicians, and that is the imprecise nature of poll results. Thanks to the nature of statistical analysis and the sample size and source used to produce the poll, polls are always cited with a "plus or minus 3 percent" (or 5 percent, or 10 percent) to indicate the imprecision assumed in the poll. Benchmarks, both across languages and across other products, should be assumed to have similar kinds of imprecision. As people have already noted, benchmarks very quickly get "gamed" in order to produce results that are unfairly biased one way or another if they're not explicitly written and administered to be fair and equal to all sides involved. This isn't to say that benchmarks aren't useful, they're just not useful to a point more precise than rounding to the nearest 10% figure.
3. Groovy IS slower than it could or should be. As much as I like the people involved in the Groovy space, and as much as I like the language itself, I can't help but be very very worried that Groovy's performance numbers aren't anywhere close to where they should be. Yes, productivity will get you a long way in the technology-adoption market, but once people have adopted your language or tool, if your system proves to be unresponsive and performance-challenged, the doors letting people in will get blocked by the people trying to get out, and that's not good for Groovy.
4. Groovy's performance may not be a reason not to use Groovy. Let's be honest, again: productivity matters. This is Mort's principal goal, remember, and there's nothing wrong with it. Groovy fits into that most natural of places, a scripting language gluing together pieces written in a system language. Perl and Python serve the same purpose for native/C/C++ code, PowerShell does the same (I believe) for .NET code, and it's high time we see the value in doing that for Java code.
5. I believe that the core Groovy team holds performance as a value. I know Graeme and Guillaume, and I believe they believe in the value of performance. I believe that Groovy will get faster over time, as they discover new and better ways to compile Groovy code into bytecode. That doesn't mean users of Groovy should walk into this exercise with their eyes shut, mind you, but take the whole of this discussion into context as you figure out where Groovy can be used to make your life, as a developer, more productive and powerful. Certain parts of your system are perf-sensitive, and certain parts aren't. Identify which of those parts are which, and apply Groovy (and other tools) judiciously.
6. These discussions are always good, so long as they're held without rancor. Groovy doesn't suck. It has warts, but so does everything. Hushing them up or pooh-poohing them just leads to arguments--I encourage the Groovy team to take the criticism of Groovy's performance the way I intend it in this blog entry: a challenge to be faced and overcome, and not as an indictment of any and all Groovy code everywhere. Because, and I will say this outright, if Groovy's backers seriously mean Groovy as "a better Java 7", then they have a large gap to fill.

Oh, and if some of you wouldn't mind sticking around to clean up the mess...? Getting beer off the ceiling can be tricky.

Over on Channel 9, the video interview recorded with me during Lang.NET has gone live. Have a look, tell me what you think.

This email crossed my desk yesterday, courtesy of the MVP program:

Microsoft has recently released a public beta of IE8. Standards and security are of top importance in this release. To that end, the IE team is planning on releasing IE8 in full standards mode. Releasing in Full Standards Mode offers many benefits in the long term, but short term, could cause some end-user and developer issues. We would love to understand your thoughts around the impact of this specific issue and invite your suggestions on how we can best communicate it.

If you have thoughts and feedback on IE 8 releasing in full standards mode, please respond to the questions below and send your reply to jasontil@microsoft.com with “[IE8 Community Feedback]” in the subject line by this Friday, April 11^th at Noon, PDT.

1) IE8 releasing in expected to release in “standards mode”.

(a) What do people in your communities space think about this decision?

(b) What do you predict the impact to be on the customer and/or Developer experience?

(c) Do you have a recommendations on how best to share this information?

2) Our current plan is to communicate this heavily with web site owners and developers. We will be contacting top sites directly, distributing developer FAQs, and writing Knowledge Base articles on authoring to these standards.

(a) Do you think that will be effective at improving the customer experience?

(b) Are there other suggestions do you could offer to transition web sites to be standards-based or to improve the experience for users?

3) Is there anything else you or those in your communities wish to tell us about this issue to improve how we react and respond as Internet Explorer advances to release?

If you're a web developer, regardless of what your server-side language of choice is, you probably want to sign up for this, if only to get an early look at IE 8 and how it will, once again, change your life as a web developer. (Ditto for Firefox 3 and Safari, while we're at it.)

Kohsuke Kawagachi has posted a blog entry describing how to watch the assembly code get generated by the JVM during execution, using a non-product (debug or fastdebug) build of Hotspot and the -XX:+PrintOptoAssembly flag, a trick he says he learned while at TheServerSide Java Symposium a few weeks ago in Vegas. He goes on to do some analysis of the generated assembly instructions, offering up some interesting insights into the JVM's inner workings.

There's only one problem with this: the flag doesn't exist.

Looking at the source for the most recent builds of the JVM (b24, plus whatever new changesets have been applied to the Mercurial repositories since then), and in particular at the "globals.hpp" file (jdk7/hotspot/src/share/vm/runtime/globals.hpp), where all the -XX flags are described, no such flag exists. It obviously must have at one point, since he's obviously been able to use it to get an assembly dump (as must whomever taught him how to do it), but it's not there anymore.

OK, OK, I lied. It never was there for the client build (near as I can tell), but it is there if you squint hard enough (jdk7/hotspot/src/share/vm/opto/c2_globals.hpp), but as the pathname to the source file implies, it's only there for the server build, which is why Kohsuke has to specify the "-server" flag on the command line; if you leave that off, you get an error message from the JVM saying the flag is unrecognized, leading you to believe Kohsuke (and whomever taught him this trick) is clearly a few megs shy in their mental heap. So when you try this trick, make sure to use "-server", and make sure to run methods enough to force JIT to take place (or set the JIT threshold  using -XX:CompileThreshold=1) in order to see the assembly actually get generated.

Oh, and make sure to swing the dead chicken--fresh, not frozen--by the neck over your head three times, counterclockwise, while facing the moon and chanting, "Ohwah... Tanidd... Eyah... Tiam...". If you don't, the whole thing won't work. Seriously.

...

Ever feel like that's how we tune the JVM? Me too. The whole thing is this huge black box, and it's nearly impossible to extract any kind of useful information without wandering into the scores of mysterious "-XX" flags, each of which is barely documented, not guaranteed to do anything visibly useful, and barely understood by anybody outside of Sun.

Hey, at least we have those flags in the JVM; the CLR developers have to take whatever the Microsoft guys give them. ("And they'll like it, too! Why, when I was their age, I had to program using nothing but pebbles by the side of the road on my way to school! Uphill! Both ways! In the raging blizzards of Arizona!")

Interestingly enough, this conversation got me into an argument with a friend of mine who works for Sun.

During the conversation, I mentioned that I was annoyed at the difficulty a Java developer has in trying to see how the Java code he/she writes turns into assembly, making it hard to understand what's really happening inside the black box. After all, the CLR makes this pretty trivial--when you set a breakpoint in Visual Studio, if you have the right flags turned on, your C# or VB source is displayed alongside the actual native instructions, making it fairly easy to see that the JITted code. This was always of great help when trying to prove to skeptical C++ developers that the CLR wasn't entirely brain-dead, and did a lot of the optimizations their favorite C++ compiler did, in some cases even better than the C++ compiler might have done. "Why don't we have some kind of double-X-show-me-the-code flag, so I can do the same with the JVM?", I lamented.

His contention was that this lack of a flag is a good thing.

Convinced I was misunderstanding his position, I asked him what he meant by that, and he said, roughly paraphrasing, that there are only about 20 or so people in the world who could look at that assembly dump and not draw incredibly misguided impressions of how the JVM operates internally; more importantly, because so few people could do anything useful with that output, it was to our collective benefit that this information was so hard to obtain.

To quote one of my favorite comedians, "Well excuuuuuuuuuuse ME." I was a bit... taken aback, shall we say.

I understand his point--that sometimes knowledge without the right context around it can lead to misinterpretation and misunderstanding. I'll agree totally with the assertion that the JVM is an incredibly complex piece of software that does some very sophisticated runtime analysis to get Java code to run faster and faster. I'll even grant you that the timing involved in displaying the assembly dump is critical, since Hotspot re-JITs methods that get used repeatedly, something the CLR has talked about ("code pitching") but thus far hasn't executed on.

But this idea that only a certain select group of people are smart enough and understand the JVM well enough to interpret the results correctly? That's dangerous, on several levels.

First, it's potentially an elitist attitude to take, essentially presenting a "We look down on you poor peasants who just don't get it" persona, and if word gets out that this is how Sun views Java developers as a whole, then it's a black mark on Sun's PR image and causes them some major pain and credibility loss. Now, let me brutally frank here: For the record, I don't think this is the case--everybody I've met at Sun thus far is helpful and down-to-earth, and scary-smart. I have a hard time believing that they're secretly thumbing their nose at me. I suppose it's possible, but it's also possible that Bill Gates and Scott McNealy were in cahoots the whole time, too.

Second, and more importantly, there will never be any more than those 20 people we have now, unless Sun works to open the deep dark internals of the JVM to more people. I know I'm not alone in the world in wanting to know how the JVM works at the same level as I know how the CLR works, and now that the OpenJDK is up and running, if Sun wants to see any patches or feature enhancements from the community, then they need to invest in more educational infrastructure to get those of us who are interested in this stuff more up to speed on the topic.

Third, and most important of all, so long as the JVM remains a black box, the "myths, legends and lore" will haunt us forever. Remember when all the Java performance articles went on and on about how method marked "final" were better-performing and so therefore should be how you write your Java code? Now, close to ten years later, we can look back at that and laugh, seeing it for the micro-optimization it is, but if challenged on this idea, we have no proof. There is no way to create demonstrable evidence to prove or disprove this point. Which means, then, that Java developers can argue this back and forth based on nothing more than our mental model of the JVM and what "logically makes sense".

Some will suggest that we can use micro-benchmarks to compare the two options and see how, after a million iterations, the total elapsed time compares. Brian Goetz has spent a lot of time and energy refuting this myth, but to put it in some degree of perspective, a micro-benchmark to prove or disprove the performance benefits of "final" methods is like changing the motor oil in your car and then driving across the country over and over again, measuring how long until the engine explodes. You can do it, but there's going to be so much noise from everything else around the experiment--weather, your habits as a driver, the speeds at which you're driving, and so on--that the results will be essentially meaningless unless there is a huge disparity, capable of shining through the noise.

This is a position born out across history--we've never been able to understand a system until we can observe it from outside the system; examples abound, such as the early medical understanding of Aristotle's theories weighed against the medical experiments performed by the Renaissance thinkers. One story says a skeptic, looking at the body in front of him disproving one of Aristotle's theories, shook his head and said, "I would believe you except that it was Aristotle who said it." When mental models are built on faith, rather than on fact and evidence, progress cannot reasonably occur.

Don't think the analogy holds? How long did we as Java developers hold faith with the idea that object pools were a good idea, and that objects are expensive to create, despite the fact that the Hotspot FAQ has explicitly told us otherwise since JDK 1.3? I still run into Java developers who insist that object pools are a good idea all across the board. I show them the Hotspot FAQ page, and they shake their head and say, "I would believe you except that it was (so-and-so article author) who said it."

Oh, and don't get me started on a near-total opacity of the Parrot and Ruby environments, among others--this isn't a "static vs dynamic" thing, this is something everybody running on a managed platform needs to be able to do.

I'm tired of arguing from a position of faith. I want evidence to either prove or disprove my assertions, and more importantly, I want my mental model of how the JVM operates to improve until it's more reflective of and closer to the reality. I can't do that until the JVM offers up some mechanisms for gathering that evidence, or at least for gathering it more easily and comprehensively. You shouldn't have to be a JVM expert to get some verification that your understanding of how the JVM works is correct or incorrect.

Recently I received a press announcement from Waggener-Edstrom, Microsoft's PR company, about their latest move in the interoperability space; I reproduce it here in its entirety for your perusal:

Hi Ted,

Microsoft is announcing another action to promote greater interoperability, opportunity and choice across the IT industry of developers, partners, customers and competitors. 

Today Microsoft is posting additional documentation of the XAML (eXtensible Application Markup Language) formats for advanced user experiences, enabling third parties to access and implement the XAML formats in their own client, server and tool products.  This documentation is publicly available, for no charge, at http://go.microsoft.com/fwlink/?LinkId=113699 . 

It will assist developers building non-Microsoft clients and servers to read and write XAML to process advanced user experiences – with lots of animation, rich 2D and 3D graphic and video. Specifically, non-Microsoft servers can more easily generate XAML files to be handled, for example, by applications running on Windows client machines.  In addition, non-Microsoft clients can be written more easily to interpret XAML files. This action will assist ISVs in creating design tools and file format converters to read and write XAML to create advanced user experiences.

Microsoft is making this documentation available under the Microsoft Open Specification Promise (OSP), which will allow developers of all types anywhere in the world to access and implement the XAML formats in their own client, server or tool products without having to take a license or pay a fee to Microsoft.

The following quote is attributable to Tom Robertson, general manager, Interoperability and Standards, Microsoft.

“Microsoft’s posting of the expanded set of XAML format documentation to assist third parties to access and implement the XAML formats in their own client, server and tool products will help promote interoperability, opportunity and choice across the IT community.  Use of the Open Specification Promise assures developers that they can use any Microsoft patents needed to implement all or part of the XAML formats for free, anywhere in the world, now and in the future.” 

Please let me know if you have any questions or if I can provide you with any additional information. 

Best,

N--

This marks the most recent in a slew of efforts by the Borg of the Pacific Northwest to "promote greater interoperability, opportunity and choice", and I know it's left a lot of people feeling decidedly skeptical and... well, let's just call it what it is, paranoid, about the company's plans and ulterior motive behind all these efforts. After all, this is the company that tried to co-opt Java, put Stacker out of business, used their monopoly operating system power to crush Novell, used their monopoly office suite power to crush the Mac, bribe an entire country to vote their way on the new office-file specifications, and I don't know what all else.

I know, I know, all my blog-readers who work at Microsoft are going nuts right now, protesting, claiming that this isn't the same company that they work for now, and so on. Fact is, folks, if you work at Microsoft, you work for a company whose name is not well-received in many quarters, and while some of it is undeserved... some of it is. Microsoft has done some pretty stupid things in its history, and if that reputation doesn't sit well with you now, I can't help but wonder if somewhere in that great Corporate Heaven, Stac Electronics isn't just jumping up and down, foaming at the mouth and screaming, "Ha! Serves you right!"

I don't want to use this blog as a chance for everybody who ever got burned by Microsoft (or thought they got burned by Microsoft, which is much more widespread and just as much more likely to be in their own minds) to trot out "reap and sow" cliches. Instead, I want to revisit one of my favorite topics, that of interoperability, and see exactly what this new shift in Microsoft's attitude towards interoperability really means.

Let's take these one at a time. Note that I have no "Deep Throat" at Microsoft feeding me "the Redmondagon Papers"; this is all based on my own conjecture and perspective.

What does releasing the XAML spec really mean?

Honestly, it means that now non-Microsoft platforms can try to create competitors to Aero and Windows Presentation Foundation, and have the same kind of rich client experiences that Windows users can enjoy.

Honestly, I expect this to go pretty much nowhere.

Realistically speaking, if a non-Microsoft app server wanted to generate XAML, it was a simple matter of generating the appropriate XML, tagging it with an appropriate MIME type in the HTTP header, and serving it up over an HTTP request; I've been giving this demo at conferences for three or four years now, pretty much since the first betas of WPF were stable enough to use. This really isn't rocket science.

But more importantly, XAMl has always been misunderstood: it's not a presentation format, it's an object graph format. XAML simply "wires up" a collection of objects into a tree, and it's the underlying object model that provides the functionality or power or presentation or whatever. It's an easier way of writing "Button b = new Button(...);", nothing more, nothing less. Sure, it would be nice to have some kind of equivalent for the Swing space, but doing so would tie the corresponding XML (XSML?) to the Swing APIs, just as WPF XAML is tied to the WPF API.

Does releasing the XAML specs mean that now Linux and Mac OS will get WPF features?

They've had them for years, in the guise of the OpenGL APIs, and nobody knew what to do with them, except maybe for a sliver of folks building games and interesting "effects". Unless somebody really feels the desire to try and create an adapter layer to map the WPF Button over to an OpenGL button, I really don't see much point.

This is one of the most dangerous points in the discussion: attempting to build an adapter to another platform's API is almost always a failed experiment from the day it's begun, and Microsoft's own attempt to port the MFC APIs over to the Mac OS (back in the pre-OS X days, circa 1995) were just a miserable, abject failure. Not because of any lack of intelligence on Microsoft's part, mind you, but because the two operating systems are just too different. Want to see what I mean? Bring a Mac guy and a Windows guy into the same room, and ask them each where God intended the menu bar to live.

Then creep, quietly, out of the room, before you get caught in the blood frenzy.

Why does Microsoft suddenly care about interoperability?

This is the crown jewel of the lot: why should this company, so famous for going it alone on so many issues, suddenly decide that it's important for them to embrace the other kids on the playground and make nice? Is this back to the "embrace" part of the "embrace and extend and extinguish" cycle that they're so famous for?

Partly.

To understand the point I am about to make, let's set some context.

(In other words, gather 'round, children, it's story time.)

Truth is, there was a time back there in the '90s when I think Microsoft really thought they could take over the world. COM was on the ascendancy, and it was a better platform for building software than anything else out there (at the time), particularly in the area of building rich media applications (remember when embedding a sound clip into your email message embedded inside your spreadsheet was all the rage?). The CORBA initiative was going strong, true, but its great claim to fame was to allow two remote processes to talk to one another--the rest of the CORBA "push" was in standards that either never materialized, or else materialized but turned out to be really hard to build, or use, or deploy, or all of the above. IBM's great competitor--SOM--wasn't even in beta on anything other than OS/2 (another great IBM product). Then, when DCOM shipped, it was seen by some as the final nail in the CORBA coffin; Microsoft clearly was going to "win".

Along came Java.

Java literally took the rug out from underneath the COM platform, almost overnight. It provided a platform with most of the same benefits as the COM/DCOM platform, but without having to memorize the QueryInterface rules or knowing what IUnknown was or how IDispatch was required to work or how static_cast<> and dynamic_cast<> and QueryInterface were all related. ("Would you, should you, static_cast? Not if you want your code to last..." Ah, those were heady days.) Suddenly, "mere mortals" could program on this platform, and feel a strong sense of confidence that their code would work, over time, regardless of whether they remembered to set references to null when they were done with them.

At first, Microsoft was "down with it", because in Java they saw a great marriage: the Java language as the "sweet spot" between C++'s expressive power and VB's layers of abstraction, running on top of the JVM as a "sweet spot" intermingled with the COM platform to provide the easiest, most powerful Windows programming environment yet. Visual J++ was clearly the favored child of the litter.

And then the lawyers got involved, and Sun saw their chance to steal a march on Microsoft, and maybe break the feared operating system monopolist, and maybe even get a few more percentage points for Solaris (because, after all, "Write Once, Run Anywhere" meant that you wouldn't have to run sucky operating systems like Windows and instead could trade up to real operating systems like Solaris, right? Hey, where'd that penguin come from, anyway, and why is he eating all our fish?). Sun refused to let Microsoft's marriage of the JVM (technically the MSVM) and COM take place, and Microsoft, rather than seek to fight it out, instead decided to cede the battle, and look for a battleground of their own choosing, instead. Thus was the thing that would become called ".NET" born.

But this "master plan" would take four or so years to develop, and in the meantime...

... in the meantime, EJB and Servlets and later J2EE and "app servers" and Spring and all those wonderful things that came with them, they were eating Microsoft's lunch. Comparing J2EE (even with EJB in the mix!) with the complexities of writing unmanaged COM code on top of COM+ is simply no comparison--again, the power of the managed platform simply proved to be too hard to turn away without compelling reason, and the COM/DCOM/COM+ story simply didn't have that compelling reason. Microsoft watched their "inevitable victory" sail into the sunset without them, just as the Department of Justice came up to them and shackled them with the first of many, many papers about "anti-competitive practices".

In many respects, the positions got reversed--Sun inherited a huge share (an unhealthy dose, in fact) of Microsoft's arrogance, and for a long time there, thought they were suddenly destiny's child, that Java (meaning Sun, of course) would be the one to "win", and thus would Sun's assurance of world dominance thus be assured.

Except it didn't play out that way.

Sun found that by embracing standards over implementations, they spent long hours thrashing out specifications, only to provide instant credibility to other vendors' products while their own languished. Weblogic stole the EJB early adopter window. A number of small vendors provided servlet implementations before Tomcat was born... which, although written by Sun employees, was an open-source project and yielded no financial benefit. JMS... well, JMS was always the redheaded stepchild of the J2EE family, at least until vendors like Sonic and Fiorano rescued it for the common Java programmer. (Those who'd been using IBM MQSeries all the while never really could see why you'd want to program against JMS APIs instead of IBM's own.) In each and every case, Sun found their product to be the third or fourth entry into the race, usually years after the others had started, and as a result....

Meanwhile, back in Redmond....

Microsoft comes to the game with .NET in 2003. (The early betas don't count because many people openly wonder if Microsoft is really serious about this ".NET" thing in the first place. After all, remember Microsoft Bob?) And despite .NET's obvious advantage of being formulated nearly a decade after Java's initial release, thus able to apply hindsight to fix or improve the obvious blemishes in the Java environment, Microsoft finds that they're playing catch-up in the all-too-important enterprise space. Microsoft's tools and products have always been seen as "second-class citizens" to the "big boys" in the enterprise space, particularly at the ends of the "high scale" continuum, and the lack of an obvious "app server" in the .NET arena only serves to underscore and reinforce that opinion among many large firms.

More importantly, Microsoft doesn't ever want to get blindsided by the Java experience again. They want to make sure that they are never in a position where it looks like their tools are vastly out-of-date, underfeatured, underpowered, and underused. They need to remain somewhere near the bleeding edge, but not so close that their customers are the ones doing the bleeding.

(We pause for the inevitable Vista joke.)

To Microsoft, Java is that near-death experience that pulls many adrenaline (and other) junkies back from the brink they so callously teetered on before. They need some kind of forward progress, some kind of advancement in the game, so that their customers and their would-be customers feel like Microsoft is on top of it at all times.

Result: Somewhere in the 2000-2003 timeframe, Microsoft looks around, sees the landscape, and realizes it needs to make itself relevant to a largely J2EE-based universe, and fast.

At first, Microsoft sees a play through the establishment of some standards between the big vendors, around this new "XML" thing, a largely portable data format, and so they throw themselves heart and soul into that space. Doing so will allow them to show existing J2EE-based shops that the power of the .NET platform lies in complementing the existing infrastructure, not replacing it. (Microsoft is smart enough to realize that preaching the software equivalent of hellfire-and-brimstone, known as "rip-and-replace", will not cater well to this congregation.)

(Rubyists could have learned a valuable lesson here, but either weren't paying attention, didn't realize the value of the lesson, or else just chose not to.)

But this play doesn't turn out the way they expect: the WS-* standards become top-heavy, and start to resemble the very thing Microsoft sought to smash fifteen years earlier: CORBA. The number of WS- specifications available through the W3C (and OASIS, and WS-I and whatever other industry consortiums are formed) is exceeded only by the number of Cos- specifications available from the OMG. The complexities therein leave many Java--and .NET--programmers confused, bewildered, and hopelessly lost when trying to get all but the most simple services to work. Thus does the community turn to alternatives--JSON, simple sockets, REST, whatever--to try and find something that works, even if it only addresses a subset of the problems they will eventually face.

Meanwhile...

Open source grows ever more important, and Microsoft-the-company realizes they have to either kill it or join it. It's hard to kill something that has no body (unlike their previous competitors), so joining it is the only viable option. Unlike many other software product companies, however, Microsoft has too large an established software base to just "flip the switch", and has far too deeply entrenched a corporate community to take any kind of radical action without a well-thought plan. (Wall Street, a place few programmers ever bother to consider, much less visit, would not take kindly to Microsoft essentially giving away their core product without something in its place to generate revenue, and regardless of how many programmers would like to imagine a world with a bankrupt Microsoft, this would be bad for business for everybody.)

And thus do we come to the present.

Microsoft needs a play that is Wall Street friendly, programmer friendly, and corporate friendly. They are slowly flirting more and more deeply with open source, yet still firmly committed to turning a profit (something a few of these other open-source-based companies should probably learn to do at some point--just maneuvering to the point of being bought out by a larger fish, like Oracle, is not really a long-term competitive strategy, just so you know).

Microsoft wants--arguably, needs--to keep Office relevant in a world where software isn't always paid for, so they need a play that keeps Office ubiquitous and out in the forefront of developer mindshare. If they can't get you to buy Office, then at least let's get you to use tools that keep the Office file formats ubiquitous. If (and this is a big "if") the Office formats turn out to be technically superior to their competition, then Microsoft succeeds. If not, they find a new play.

In the short, Microsoft needs an interoperability story, and they need a real interoperability play, because their reputation is damaged from the many "embrace, extend, extinguish" plays they've made in the past. The era of a large vendor "winning" is clearly well behind us (if it was ever, in fact, more than just a marketing VP's wet dream), and if Microsoft is going to make sure that they're never in a vulnerable come-from-behind position again, they need to make sure that they can work well with all the other new technologies out there, whether up-and-coming or well-established or even fading-fast. They need to have an interoperability story that developers can believe in, which means some kind of open-source-friendly play, and one that carries serious "street cred" for actually working.

What's the lesson that I, a developer, take away from this?

If you are a Java developer, get past your old prejudices and accept that .NET is a viable platform. The Java developer who refuses to learn how to write C# code on the grounds that "Micro$oft is a company that just puts out crap" or that "M$FT sux" is going to be a Java developer whose value to the business is reduced compared to those with less virulent politics. Thanks to tools like VMWare and Virtual PC, you don't have to give up your Mac or your Linux environment to write .NET code and prove that you can offer value to those projects that need to talk to .NET. Look into more than just the WS-* or REST stacks for communication, as well; explore some of the interoperability options I've been ranting about for four years, a la IKVM, Jace, Hessian, even CORBA.

If you are a Ruby developer, get over yourself and your "we're more agile and more powerful" meme. Ruby is a tool, nothing more, and one whose shine is fast coming off. IT organizations are discovering the myriad problems with the original Ruby runtime, and are unwilling to risk enterprise apps on a runtime that has zero monitoring and zero manageability play. Yes, you can certainly do lots of things yourself to make your Ruby apps more manageable and more monitorable--but that's all time you have to spend building it, or figuring out how to hook it into the existing IT infrastructure, and when all that time gets added up, it's not going to look all that different from a Java or .NET app's timecycle arc. If you don't have an answer to the question, "How will we make this work with the existing infrastructure we've got?", then you have a problem, and no amount of chanting "Obi-Dave Thomas-Kenobi, you and dynamic typing are my only hope" will save you.

If you are a .NET developer, it's high time you accepted that the Java folks are about five years ahead of you on this "managed code" arc, and that they suffered through a lot of hard lessons before arriving at the decisions they came to. Don't be stupid, learn from their mistakes. Why do Java programmers chant "dependency injection" with holy fervor? Why do Java programmers put so much stress on unit testing? What has Microsoft not given you with the latest release of Visual Studio that Java developers think you're an idiot for not demanding in the next release? Yes, C# has some interesting new features in it that Java-the-language doesn't have... but why are the Java guys getting all misty-eyed over Groovy? What do they know that you don't?

If you are a developer outside of these areas, you're swimming in dangerous waters, because while I'm sure you're not having any problems finding a job, chances are your next job is going to require you to talk to one of those three environments. Better have your integration/interoperability story worked out, whether its Phalanger for the PHP developer who needs to talk to .NET (and damn if PHP script driving a WinForms app isn't an interesting idea in of itself... and a useful way to bridge yourself into an entirely new area of employment), or its figuring out how to apply your mad Haskell skillz to F# or Scala, you need to have a good idea of what those languages are (and aren't) and how your knowledge of functional concepts can catapult you to the head of the class the next time a massively-scalable system needs to be built.

If you are a Microsoft employee, don't blow this. Don't make this into another "embrace, extend, extinguish" cycle. Accept that your company made some bone-headed maneuvers in the past, and rather than try to defend them, accept that your reputation outside of the Redmond Reality-Distortion Bubble is not what it looks like from the inside. As hard as this will be to do sometimes, just stop and listen to what others are saying about the company and the paranoia that creeps up every time Microsoft moves into an area of interest. Take the extra moment to hear the concerns, not just the words.

And if you are a Google employee, tatoo this on your forehead: Reputation Matters. The first time anybody at your company does something even remotely "evil", you will be branded as "the next Microsoft" and all of these problems will be yours to share and enjoy, as well.

Apparently, I'm drawing enough of an audience through this blog that various folks have started to send me press releases and notifications and requests for... well, I dunno exactly, but I'm assuming some blogging love of some kind. I'm always a little leery about that particular subject, because it always has this dangerous potential to turn the blog into a less-credible marketing device, but people at conferences have suggested that they really are interested in what I think about various products and tools, so perhaps it's time to amend my stance on this.

With that in mind, if you are a vendor and have a product that you'd like me to take a look at and (possibly) offer up a review here, here's the basic rules:

1. No guarantees. Sending me something will in no way guarantee that I will review your product, for several reasons, two of which being (a) I get really busy sometimes, and (b) I may have no interest whatsoever in your product and I refuse to pretend to do so. (Readers can usually tell when the reviewer isn't all that excited about the subject, I've found.)
2. If you're not going to send me a "real" version (meaning not the time-locked or feature-crippled demo), don't bother. I have no idea when I will get around to a review, and I have no desire to review something that isn't "the real deal". I will in turn promise that the licensed version you send me (if necessary) will not be used for any purpose other than my own research and exploration (signing contract if necessary to give you that "fresh-from-the-lawyer's-office" warm and fuzzy feeling).
3. I say what I think, pro and con. I will not edit my review to suit your marketing purpose, and if you ask me to do so I will simply note in the review that you have asked me to do so. I retain full editorial control over what I say about your product.
4. Having established #1, I will try to be as fair as I can about your product, and point out things that I liked and things that I didn't. (Of course, if I hated it from top to bottom, I may end up with the only positive thing being "It didn't set the atmosphere on fire when I started the app", but hey, that's something positive, right?)
5. Also in the spirit of #1, if you send me mail answering questions or complaints in my review, I will of course amend the review with your comments. You are always welcome to post comments to the blog entry itself, too. Unless you insult my grandmother, then I will have to get all DELETE-key on you.

The reason I'm posting this here is twofold: one, so my faithful audience of four blog readers will know the rules under which I'm looking at these products and (hopefully) realize that I'm not financially vested in any of these products, and two, so the various vendor folks can read this and know what the rules are up front before even asking.

I know it sounds a little cheeky to lay this out. The image I get in my head is that of the kid at Christmas declaring to his grandparents as they walk through the door, presents in hand, "Make sure it's not a scratchy sweater, I hate scratchy sweaters. And G.I. Joe was only popular when my Dad was a kid. And if you give me another lunchbox I will scream until you buy me something cool, like a new GameBoy." Ugh. But I value the trust that people seem to have in me, and so I risk the perception of cheekiness for this tiny window in time in order to (hopefully) establish full disclosure over the reviews that come to pass (which, by the way, will always have the category "review" applied to them, so you know which is an official review and which is just me exploring, like the LLVM and Parrot posts of recent time).

We now return you to the regularly-scheduled blog.

A couple of people have asked me over the last few weeks, so it's probably worth saying out loud:

No, I don't work for a large company, so yes, I'm available for consulting and research projects. If you've got one of those burning questions like, "How would our company/project/department/whatever make use of JRuby-and-Rails, and what would the impact to the rest of the system be", or "Could using F# help us write applications faster", or "How would we best integrate Groovy into our application", or "How does the new Adobe Flex/AIR move help us build richer client apps", or "How do we improve the performance of our Java/.NET app", or other questions along those lines, drop me a line and let's talk. Not only will I cook up a prototype describing the answer, but I'll meet with your management and explain the consequences of the research, both pro and con, for them to evaluate.

Shameless call for consulting complete, now back to the regularly-scheduled programming.

Recently, a number of folks in the Java space have taken to openly ridiculing Microsoft's use of the "Mort" persona, latching on to the idea that Mort is somehow equivalent to "Visual Basic programmer", which is itself somehow equivalent to "stupid idiot programmer who doesn't understand what's going on and just clicks through the wizards". This would be a mischaracterization, one which I think Nikhilik's definition helps to clear up:

Mort, the opportunistic developer, likes to create quick-working solutions for immediate problems and focuses on productivity and learn as needed. Elvis, the pragmatic programmer, likes to create long-lasting solutions addressing the problem domain, and learn while working on the solution. Einstein, the paranoid programmer, likes to create the most efficient solution to a given problem, and typically learn in advance before working on the solution. ....

The description above is only rough summarization of several characteristics collected and documented by our usability folks. During the meeting a program manager on our team applied these personas in the context of server controls rather well (I think), and thought I should share it. Mort would be a developer most comfortable and satisfied if the control could be used as-is and it just worked. Elvis would like to able to customize the control to get the desired behavior through properties and code, or be willing to wire up multiple controls together. Einstein would love to be able to deeply understand the control implementation, and want to be able to extend it to give it different behavior, or go so far as to re-implement it.

Phrased this way, it's fairly easy to recognize that it's possible that these are more roles than actual categorizations for programmers as individuals--sometimes you want to know how to create the most efficient solution, and sometimes you just want the damn thing (whatever it is) to get out of your way and let you move on.

Don Box called this latter approach (which is a tendency on the part of all developers, not just the VB guys) the selective ignorance bit: none of us have the time or energy or intellectual capacity to know how everything works, so for certain topics, a programmer flips the selective ignorance bit and just learns enough to make it work. For myself, a lot of the hardware stuff sees that selective ignorance bit flipped on, as well as a lot of the frou-frou UI stuff like graphics and animation and what-not. Sure, I'm curious, and I'd love to know how it works, but frankly, that's way down the priority list.

If trying to find a quick-working solution to a particular problem means labeling yourself as a Mort, then call me Mort. After all, raise your hand if you didn't watch a team of C++ guys argue for months over the most efficient way to create a reusable framework for an application that they ended up not shipping because they couldn't get the framework done in time to actually start work on the application as a whole....

While tracking some of the links relating to the Groovy/Ruby war, I found this website, which purportedly tracks job trends based on a whole mess of different job sites. So, naturally, I had to plug in to get a graph of C#, C++, Java, Ruby, and VB:

java, c#, c++, ruby, visual basic Job Trends

java jobs - c# jobs - c++ jobs - ruby jobs - visual basic jobs

Interesting. I don't think it proves anything one way or another, mind you, but interesting nonetheless. Having said that, a few things stand out to me after looking at this for all of thirty seconds:

• Wow, what the hell happened in 1Q and 2Q of 2005? Java takes a huge drop in 2005, and all of them take a small drop of some form around the same time in 2006. What is it with summertime? Did the HR supervisor suddenly take a look at the company's job board and mutter, "Damn, I thought we closed all those listings already..."? (Or maybe, "Thank God for cheap college interns..."?)
• C++ jobs still outnumber C# jobs, even in 4Q 2007?
• C++ jobs remain essentially flat from 1Q 2005 to 4Q 2007; apparently, there's a lot more C++ going on than most companies are willing to admit to.... (Can't you picture it? The nervous candidate, sitting at the table, as the interviewer shuffles the paper and says, "So, you're here for a programming job?" The candidate sort of squirms in his chair as he replies, "Well, actually, I was hoping for a... a... C++ job." The interviewer quickly looks around to see who might be listening as he says loudly, "C++? What ever gave you the idea that we do C++ here at BigCorp?" Meanwhile, he surreptitiously scribbles on the back of a business card and slides it across the table to the candidate, then stands up and says loudly, "I'm afraid you've come to the wrong place, sir. You can see yourself out, I take it?" The candidate palms the card, and only once has he left the building does he look at the back, which reads, "8PM, corner of Mission and Vine, password is 'Lippman, Stroustrup, Sutter, and Meyers!' Viva C++!"...)
• VB jobs fall to below C#? So much for those vast hordes of VB programmers that supposedly form the "long tail" of the .NET community....
• Java jobs remain essentially flat from 1Q 2005 to 4Q 2007, despite numerous ups and downs. So much for the idea that Java is somehow going away....
• Ruby's penetration into the job market is much smaller than what I would have guessed.
• I couldn't help myself, I did another query with "cobol" added in, but I'll leave it to you to run your own query to see what that looks like. It's surprising....

Of course, statistics without any sort of understanding of how they were gathered or from what sources are essentially meaningless, but ooooh, it's in color....

Since we're examining various aspects of the canonical O-O language (the three principals being C++, Java and C#/VB.NET), let's take in a review of another recent post, this time on the use of "new" in said languages.

All of us have probably written code like this:

Foo f = new Foo();

And what could be simpler?  As long as the logic in the constructor is simple (or better yet, the constructor is empty), it would seem that the simplest code is the best, so just use the constructor.  Certainly the MSDN documentation is rife with code that uses public constructors.  You can probably find plenty of public constructors used right here on my blog.  Why invest the effort in writing (and using) a factory class that will probably never do anything useful, other than call a public constructor?

In his excellent podcast entitled "Emergent Design: The Evolutionary Nature of Software Development," Scott Bain of Net Objectives nevertheless makes a strong case against the routine use of public constructors.  The problem, notes Scott, is that the use of a public constructor ties the calling code to the implementation of Foo as a concrete class.  But suppose that you later discover that there need to be many subtypes of Foo, and Foo should therefore be an abstract class instead of a concrete class--what then?  You've got a big problem, that's what; a lot of client code that has been making use of Foo's public constructor suddenly becomes invalid.

I just love it when people rediscover advice that they could have had much earlier, had they only been aware of the prior art in the field. I refer the curious C#/VB.NET developer to the book Effective Java, by Joshua Bloch, in which Item 1 states, "Consider providing static factory methods instead of constructors". Quoting from said book, we see:

One advantage of static factory methods is that, unlike constructors, they have names. If the parameters to a constructor do not, in and of themselves, describe the object being returned, a static factory with a well-chosen name can make a class easier to user and the resulting client code easier to read. ...

A second advantage of static factory methods is that, unlike constructors, they are not required to create a new object each time they're invoked. This allows immutable classes (Item 13) to use preconstructed instances or to cache instances as they're constructed and to dispense these instances repeatedly so as to avoid creating unnecessary duplicate values. ...

A third advantage of static factory methods is that, unlike constructors, they can return an object of any subtype of their return type. This gives you great flexibility in choosing the class of the returned object. ...

The main disadvantage of static factory methods is that classes without public or protected constructors cannot be subclassed. The same is true for nonpublic classes returned by public static factories.

A second disadvantage of static factory methods is that they are not readily distinguishable from other static methods. They do not stand out in API documentation the way that constructors do.

C# and VB.NET developers are encouraged to read the book to discover about 30 or so other nuggets of wisdom that are directly applicable to the .NET framework. Note that Josh is in the process, this very month, of revising the book for rerelease as a second edition, taking into account the wide variety of changes that have taken place in the Java language since EJ's initial release.

Meanwhile....

One thing that's been nagging at me is how I think Java and C# missed the boat in respect to the various ways we'd like to construct objects. The presumption was always that allocation and initialization would (a) always take place at the same time, and (b) always take place in the same manner--the underlying system would allocate the memory, the object would be laid out in this newly-minted chunk of heap, and your constructor would then initialize the contents. Neither assumption can be taken to be true, as we've seen over the years; the object may need to come from pre-existing storage (a la the object cache), or the object may need to be a derived type (a la the covariant return Josh mentions in #3 advantage above), or in some cases you want to mint the object from an entirely different part of the process.

C++ actually had an advantage over C# and Java here, in that you could overload operator new() for a class (which then meant you had to overload operator delete(), and oh-by-the-way don't forget to overload array new, that is, operator new[]() and its corresponding twin, array delete, operator delete[](), which was a bit of a pain) to gain better control over both allocation and initialization, to a degree. Initially we always used it to control allocation--the idea being one would create a class-specific allocator, on the grounds that knowing some of the assumptions of the class, such as its size, would allow you to write faster allocation routines for it. But one of the rarely-used features of operator new() was that it could take additional parameters, using a truly obscure syntactic corner of C++:

   1: void* operator new(size_t s, const string& message)

   2: {

   3:     cout << "Operator new sez " << message << endl;

   4:         // allocate s bytes and return; Foo ctor will be invoked automagically

   5: }

   6: Foo* newFoo = new ("Howdy, world!") Foo();

Officially, one such overloaded operator was recognized, the placement new operator, which took a void* as a parameter, indicating the exact location in which your object was to be allocated and thus laid down. This meant that C++ developers could allocate from some other part of the process (including shudder a pointer they'd made up out of thin air) and drop the initialized object right there. While useful in its own right, placement new opened up a whole new world of construction options to the C++ developer that we never really took advantage of, since now you could pass parameters to the construction process without involving the constructor.

That's kind of nifty, in an obscure and slightly terrifying fashion. One thought I'd always had was that it would be cool if a C++ O/R-M overloaded operator new() for database-bound objects to indicate which database connection to use during construction:

   1: DBConnection conn;

   2:  

   3: Person* newFoo = new (conn) Person("Ted", "Neward");

 

Of course, such syntax has the immediate drawback of eliciting a chorus of "WTF?!?" at the next code review, but still....

Meanwhile, other languages choose to view new as one of those nasty static methods Gilad dislikes so much, Ruby and Smalltalk being two of them. That is to say, construction now basically calls into a static method on a class, which has the nice effect of keeping the number of "special" parts of the language to a minimum (since now "new" is just a method, not a keyword), makes it easier to have different-yet-similar names to represent slightly different concepts ("create" vs "new" vs "fetch" vs "allocate", and so on) sitting side by side, and helps eliminate Josh's second disadvantage above. I'm not certain how exactly this could eliminate Josh's first disadvantage (that of inheritance and inaccessible constructors), but it's not entirely unimaginable that the language would have a certain amount of incestuous knowledge here to be able to reach those static method (constructors) in the same way it does currently.

(It actually works better if they aren't static methods at all, but instance methods on class objects, to which the language automatically defers when it sees a "classname.new"; that is, when it sees

Person ann = Person.new("Ann", "Sheriff");

the language automatically changes this to read:

Person ann = Person.class.new("Ann", "Sheriff");

which would be eminently doable in Java, were class objects available for modification/definition somehow. In a language built on top of the JVM or CLR, the class object would be a standalone singleton, a la "object" definitions in Scala.)

Gilad makes the case that static, that staple of C++, C#/VB.NET, and Java, does not belong:

Most imperative languages have some notion of static variable. This is unfortunate, since static variables have many disadvantages. I have argued against static state for quite a few years (at least since the dawn of the millennium), and in Newspeak, I’m finally able to eradicate it entirely.

I think Gilad conflates a few things, but he's also got some good points. To the dissecting table!

To begin:

Static variables are bad for security. See the E literature for extensive discussion on this topic. The key idea is that static state represents an ambient capability to do things to your system, that may be taken advantage of by evildoers.

Eh.... I'm not sure I buy into this. For evildoers to be able to change static state, they have to have some kind of "poke" access inside the innards of your application, and if they have that, then just about anything is vulnerable. Now, granted, I haven't spent a great deal of time on the E literature, so maybe I'm missing the point here, but if an attacker has data-manipulability into my program, then I'm in a whole world of pain, whether he's attacking statics or instances. Having said that, statics have to be stored in a particular well-known location inside the process, so maybe that makes them a touch more vulnerable. Still, this seems a specious argument.

Static variables are bad for distribution. Static state needs to either be replicated and sync’ed across all nodes of a distributed system, or kept on a central node accessible by all others, or some compromise between the former and the latter. This is all difficult/expensive/unreliable.

Now this one I buy into, but the issue isn't the "static"ness of the data, but the fact that it's effectively a Singleton, and Singletons in any distributed system are Evil. I talked a great deal about this in Effective Enterprise Java, so I'll leave that alone, but let me point out that any Singleton is evil, whether it's represented in a static, a Singleton object, a Newspeak module, or a database. The "static"ness here is a red herring.

Static variables are bad for re-entrancy. Code that accesses such state is not re-entrant. It is all too easy to produce such code. Case in point: javac. Originally conceived as a batch compiler, javac had to undergo extensive reconstructive surgery to make it suitable for use in IDEs. A major problem was that one could not create multiple instances of the compiler to be used by different parts of an IDE, because javac had significant static state. In contrast, the code in a Newspeak module definition is always re-entrant, which makes it easy to deploy multiple versions of a module definition side-by-side, for example.

Absolutely, but this is true for instance fields, too--any state that is modified as part of two or more method bodies is vulnerable to a re-entrancy concern, since now the field is visibly modified state to that particular instance. How deeply do you want your code to be re-entrant? Gilad's citation of the javac compiler points out that the compiler was hardly re-entrant at any reasonable level, but the fact is that the compiler *could* have been used in a parallelized fashion using the isolational properties of ClassLoaders. (Its ugly, and Java desperately needs Isolates for that reason.)

Static variables are bad for memory management. This state has to be handed specially by implementations, complicating garbage collection. The woeful tale of class unloading in Java revolves around this problem. Early JVMs lost application’s static state when trying to unload classes. Even though the rules for class unloading were already implicit in the specification, I had to add a section to the JLS to state them explicitly, so overzealous implementors wouldn’t throw away static application state that was not entirely obvious.

This one I can't really comment on, since I'm not in the habit of writing memory-management code. I'll take Gilad's word for it, though I'm curious to know why this is so, in more detail.

Static variables are bad for for startup time. They encourage excess initialization up front. Not to mention the complexities that static initialization engenders: it can deadlock, applications can see uninitialized state, and unless you have a really smart runtime, you find it hard to compile efficiently (because you need to test if things are initialized on every use).

I'm not sure I see how this is different for any startup/initialization code--anything that the user can specify as part of startup will run the risk of deadlocks and viewing uninitialized state. Consider the alternative, however--if the user didn't have the ability to specify startup code, then they would have to either write their own, post-runtime, startup code, or else they have to constantly check the state of their uninitialized objects and initialize them on first use, the very thing that he claims is hard to compile efficiently.

Static variables are bad for for concurrency. Of course, any shared state is bad for concurrency, but static state is one more subtle time bomb that can catch you by surprise.

Absolutely: any shared state is bad for concurrency. However, I think we need to go back to first principles here. Since any shared state is bad for concurrency, and since static data is always shared by definition, it follows that static data is bad for concurrency. Pay particular attention to that chain of reasoning, however: any shared state is bad for concurrency, whether it's held by the process in a special non-instance-aligned location or in an data store that happens to be reachable from multiple paths of control. This means that your average database table is also bad for concurrency, were it not for the transactional protections that surround the table. This isn't an indictment of static variables, per se, but of shared state.

Gilad goes on to describe how Newspeak solves this problem of static:

It may seem like you need static state, somewhere to start things off, but you don’t. You start off by creating an object, and you keep your state in that object and in objects it references. In Newspeak, those objects are modules.

Newspeak isn’t the only language to eliminate static state. E has also done so, out of concern for security. And so has Scala, though its close cohabitation with Java means Scala’s purity is easily violated. The bottom line, though, should be clear. Static state will disappear from modern programming languages, and should be eliminated from modern programming practice.

I wish Newspeak were available for widespread use, because I'd love to explore this concept further; in the CLR, for example, there is the same idea of "modules", in that modules are singleton entities in which methods and data can reside, at a higher level than individual objects themselves. Assemblies, for example, form modules, and this is where "global variables" and "global methods" exist (when supported by the compiling language in question). At the end of the day, though, these are just statics by another name, and face most, if not all, of the same problems Gilad lays out above. Scala "objects" have the same basic property.

I think the larger issue here is that one should be careful where one stores state, period. Every piece of data has a corresponding scope of accessibility, and developers have grown complacent about considering that scope when putting data there: they consider the accessibility at the language level (public, private, what-have-you), and fail to consider the scope beyond that (concurrency, re-entrancy, and so on).

At the end of the day, it's simple: static entities and instance entities are just entities. Nothing more, nothing less. Caveat emptor.

Just recently, I got this bit in an email from the Redmond Developer News ezine:

TWO IF BY SEA

In the course of just over a week starting on Jan. 30, a total of five undersea data cables linking Europe, Africa and the Middle East were damaged or disrupted. The first two cables to be lost link Europe with Egypt and terminate near the Port of Alexandria.

http://reddevnews.com/columns/article.aspx?editorialsid=2502

Early speculation placed the blame on ship anchors that might have dragged across the sea floor during heavy weather. But the subsequent loss of cables in the Persian Gulf and the Mediterranean has produced a chilling numbers game. Someone, it seems, may be trying to sabotage the global network.

It's a conclusion that came up at a recent International Telecommunication Union (ITU) press conference. According to an Associated Press report, ITU head of development Sami al-Murshed isn't ready to "rule out that a deliberate act of sabotage caused the damage to the undersea cables over two weeks ago."

http://tinyurl.com/3bjtdg

You think?

In just seven or eight days, five undersea cables were disrupted.

Five. All of them serving or connecting to the Middle East. And thus far, only one cable cut -- linking Oman and the United Arab Emirates -- has been identified as accidental, caused by a dragging ship anchor.

So what does it mean for developers? A lot, actually. Because it means that the coming wave of service-enabled applications needs to take into account the fact that the cloud is, literally, under attack.

This isn't new. For as long as the Internet has been around, concerns about attacks on the network have centered on threats posed by things like distributed denial of service (DDOS) and other network-borne attacks. Twice -- once in 2002 and again in 2007 -- DDOS attacks have targeted the 13 DNS root servers, threatening to disrupt the Internet.

But assaults on the remote physical infrastructure of the global network are especially concerning. These cables lie hundreds or even thousands of feet beneath the surface. This wasn't a script-kiddie kicking off an ill-advised DOS attack on a server. This was almost certainly a sophisticated, well-planned, well-financed and well-thought-out effort to cut off an entire section of the world from the global Internet.

Clearly, efforts need to be made to ensure that the intercontinental cable infrastructure of the Internet is hardened. Redundant, geographically dispersed links, with plenty of excess bandwidth, are a good start.

But development planners need to do their part, as well. Web-based applications shouldn't be crafted with the expectation of limitless bandwidth. Services and apps must be crafted so that they can fail gracefully, shift to lower-bandwidth media (such as satellite) and provide priority to business-critical operations. In short, your critical cloud-reliant apps must continue to work, when almost nothing else will.

And all this, I might add, as the industry prepares to welcome the second generation of rich Internet application tools and frameworks.

Silverlight 2.0 will debut at MIX08 next month. Adobe is upping the ante with its latest offerings. Developers will enjoy a major step up in their ability to craft enriched, Web-entangled applications and environments.

But as you make your plans and write your code, remember this one thing: The people, organization or government that most likely sliced those four or five cables in the Mediterranean and Persian Gulf -- they can do it again.

There's a couple of things to consider here, aside from the geopolitical ramifications of a concerted attack on the global IT infrastructure (which does more to damage corporations and the economy than it does to disrupt military communications, which to my understanding are mostly satellite-based).

First, this attack on the global infrastructure raises a huge issue with respect to outsourcing--if you lose touch with your development staff for a day, a week, a month (just how long does it take to lay down new trunk cable, anyway?), what sort of chaos is this going to strike with your project schedule? In The World is Flat, Friedman mentions that a couple of fast-food restaurants have outsourced the drive-thru--you drive up to the speaker, and as you place your order, you're talking to somebody half a world way who's punching it into a computer that's flashing the data back to the fast-food join in question for harvesting (it's not like they make the food when you order it, just harvest it from the fields of pre-cooked burgers ripening under infrared lamps in the back) and disbursement as you pull forward the remaining fifty feet to the first window.

The ludicrousness of this arrangement notwithstanding, this means that the local fast-food joint is now dependent on the global IT infrastructure in the same way that your ERP system is. Aside from the obvious "geek attraction" to a setup like this, I find it fascinating that at no point did somebody stand up and yell out, "What happened to minimizing the risks?" Effective project development relies heavily on the ability to touch base with the customer every so often to ensure things are progressing in the way the customer was anticipating. When the development team is one ocean and two continents away in one direction, or one ocean and a whole pile of islands away in the other direction, or even just a few states over, that vital communication link is now at the mercy of every single IT node in between them and you.

We can make huge strides, but at the end of the day, the huge distances involved can only be "fractionalized", never eliminated.

Second, as Desmond points out, this has a huge impact on the design of applications that are assuming a 100% or 99.9% Internet uptime. Yes, I'm looking at you, GMail and Google Calendar and the other so-called "next-generation Internet applications" based on technologies like AJAX. (I categorically refuse to call them "Web 2.0" applications--there is no such thing as "Web 2.0".) As much as we keep looking to the future for an "always-on" networking infrastructure, the more we delude ourselves to the practical realities of life: there is no such thing as "always-on" infrastructure. Networking or otherwise.

I know this personally, since last year here in Redmond, some stronger-than-normal winter storms knocked down a whole slew of power lines and left my house without electricity for a week. To very quickly discover how much of modern Western life depends on "always-on" assumptions, go without power to the house for a week. We were fortunate--parts of Redmond and nearby neighborhoods got power back within 24 hours, so if I needed to recharge the laptop or get online to keep doing business, much less get a hot meal or just find a place where it was warm, it meant a quick trip down to the local strip mall where a restaurant with WiFi (Canyon's, for those of you that visit Redmond) kept me going. For others in Redmond, the power outage meant a brief vacation down at the Redmond Town Center Marriott, where power was available pretty much within an hour or two of its disruption.

The First Fallacy of Enterprise Systems states that "The network is reliable". The network is only as reliable as the infrastructure around it, and not just the infrastructure that your company lays down from your workstation to the proxy or gateway or cable modem. Take a "traceroute" reading from your desktop machine to the server on which your application is running--if it's not physically in the building as you, then you're probably looking at 20 - 30 "hops" before it reaches the server. Every single one of those "hops" is a potential point of failure. Granted, the architecture of TCP/IP suggests that we should be able to route around any localized points of failure, but how many of those points are, in fact, to your world view, completely unroutable? If your gateway machine goes down, how does TCP/IP try to route around that? If your ISP gets hammered by a Denial-of-Service attack, how do clients reach the server?

If we cannot guarantee 100% uptime for electricity, something we've had close to a century to perfect, then how can you assume similar kinds of guarantees for network availability? And before any of you point out that "Hey, most of the time, it just works so why worry about it?", I humbly suggest you walk into your Network Operations Center and ask the helpful IT people to point out the Uninterruptible Power Supplies that fuel the servers there "just in case".

When they in turn ask you to point out the "just in case" infrastructure around the application, what will you say?

Remember, the Fallacies only bite you when you ignore them:

1) The network is reliable

2) Latency is zero

3) Bandwidth is infinite

4) The network is secure

5) Topology doesn't change

6) There is one administrator

7) Transport cost is zero

8) The network is homogeneous

9) The system is monolithic

10) The system is finished

Every project needs, at some point, to have somebody stand up in the room and shout out, "But how do we minimize the risks?" If this is truly a "mission-critical" application, then somebody needs the responsibility of cooking up "What if?" scenarios and answers, even if the answer is to say, "There's not much we can reasonably do in that situation, so we'll just accept that the company shuts its doors in that case".

During the Lang.NET Symposium, a couple of things "clicked" all simultaneously, giving me one of those "Oh, I get it now" moments that just doesn't want to leave you alone.

During the Intentional Software presentation, as the demo wound onwards I (and the rest of the small group gathered there) found myself looking at the same source code, but presented in a variety of new ways, some of which appealed to me as the programmer, others of which appealed to the mathematicians in the room, others of which appealed to the non-programmers in the room. (I heard one of the Microsoft hosts, a non-technical program manager, I think, say, "Wow, even I could understand that spreadsheet view, and that was writing code?")

During the spreadsheet-written-in-IronPython presentation (ResolverOne), we were essentially looking at new ways of writing IronPython code, thus leveraging all the syntactic power of a programming language with a nicer front end.

During the aspect-oriented talk (the one by Stefan Wenig and Fabian Schmeid), we found ourselves looking at a tool that essentially takes compiled assemblies and weaves in additional code based on descriptors from outside that codebase; in essence, just another aspect-oriented tool.

But combine this with my own investigations into Soot, LLVM, Parrot, and Phoenix, alongside the usual discussions around the DLR, CLR, JVM and DaVinci machine, couple that with the presentation Harry gave about parser expression grammars and the research in the functional community into parser combinators, throw in the aspect-oriented and metaprogramming facilities that the Rubyists and other dynamic linguists go on for days about, and what do you end up with?

Folks, the future is in modular toolchains.

This is an oversimplification, and a radical oversimplification at that, but imagine for a moment:

1. A parser takes your source code (let's assume it is Java, just for grins) and builds an AST out of it. Not an AST that's inherently deeply coupled to the Java language, mind you, but a general-purpose one that stands as a union of Java, C#, C++, Perl, Python, Smalltalk, and other languages. (Note that some of the linguistic concepts in some of those languages may not end up in this AST, but instead operate on the AST itself, a la C++'s template facilities.) Said parser is now finished, and can either output a binary (or potentially XML, though it'd probably be hideously verbose) version of this AST to disk for later consumption, or would more than likely be passed directly along to the next beast in the chain.
2. In the simplest scenario, the next beast would be a code generator, which takes the AST and seeks to export some kind of back-end code out of it. Here, since we're working with a general-purpose AST, we can assume that this back-end is flexible and open, a la the Phoenix toolkit (where either native or MSIL can be generated).
3. In a slightly more complicated scenario, verification of the correctness of the AST (against whatever libraries are specified) is checked, usually prior to code-gen, thus making this particular toolchain a statically-checked chain; were verification left out, it would need to happen at runtime, in which case we'd be talking about a dynamically-checked chain.

   Note that I stay away from the term "statically-typed" or "dynamically-typed" for the moment. That would be a measurement of the parser, not the verifier. Verification still occurs in a lot of these dynamically-typed languages, just as it does in statically-typed languages.

   Assuming the verification process succeeds, the AST can be again, written out or passed to the next step in the chain.
4. Another potential step in the process, usually post-parser and pre-verification, would be an "aspect" step, in which a tool takes the AST, consults some external descriptors, and modifies the AST based on what it finds there. (This is how most of your non-AspectJ-like AOP tools work today, except that they have to rebuild the AST from compiled .class files or assemblies first.)
5. Naturally, another step in the process would be an optimize step, but this has to be considered carefully, since some "high-level" optimizations can be done without regard to code-gen backend, and some will need to be done with regard to code-gen backend; for example, register spill is (from what I've heard, can't say I know too much about this) generally only useful if you know how many registers you're targeting. Plus, it's not hard to imagine certain optimizations that are only generally useful on the x86 architecture, versus those that are useful on other CPU platforms. Even operating systems I would imagine would have an impact here. (It turns out that many compiler toolchains go through a dozen or so optimization steps today, so it's not hard to imagine a "code-gen backend" being a series of a half-dozen or so targeted optimization steps before actually generating code.)
6. Bear in mind, too, that these ASTs should have enough information to be directly executable, thus giving us an interpreter back-end instead of a code-generation back-end, a la the DLR instead of the CLR.
7. Also, given the standard AST format, it would be relatively trivial to create a whole series of different "parser"s to get to the AST, along the lines of what the Intentional Software guys have created, thus blowing open the whole concept of "DSL" into areas that heretofore have only been imagined. You still get the complete support of the rest of the toolchain, which is what makes the whole DSL concept viable in the first place, including aspects and verification and your choice of either interpretation or compilation.
8. While we're at it, bear in mind that this AST could/should also be reachable from within the code itself, thus giving languages that want to operate on their own AST at runtime the ability to do so, because the AST is in a standard format and the interpreter could be bundled as part of the generated executable, thus providing a compile-when-you-can-interpret-when-you-must flavor that is currently the reigning meme in language/platform environments like JRuby. (It would also have the happy side effect of making Paul Graham shut up about Lisp, at least for a while. Yes, Paul, code-as-data, it's brilliant, it's wonderful, we get it.)
9. Nothing says this toolchain needs be one-way, by the way: many of the toolkits I mentioned before (LLVM, Phoenix, Soot) can start from compiled binary and work back to AST, thus offering us the opportunity to do surgery of either the exploratory kind (static analysis) or the manipulative kind (aspect-weaving, etc) on compiled code in a relatively clean way. Reflector demonstrates the power of being able to go "back and forth" in this way (even in the relatively limited way Reflector does so), so imagine how powerful it would be to do this from end-to-end throughout the toolchain.

How likely is this utopian vision? I'm not sure, honestly--certainly tools like LLVM and Phoenix seem to imply that there's ways to represent code across languages in a fairly generic form, but clearly there's much more work to be done, starting with this notion of the "uber-AST" that I've been so casually tossing around without definition. Every AST is more or less tied to the language it is supposed to represent, and there's clearly no way to imagine an AST that could represent every language ever invented. Just imagine trying to create an AST that could incorporate Java, COBOL and Brainf*ck, for example. But if we can get to a relatively stable 80/20, where we manage to represent the most-commonly-used 80% of languages within this AST (such as an AST that can incorporate Java, C#, and C++, for starters), then maybe there's enough of a critical mass there to move forward.

Now all I need to do is find somebody who'll fund this little bit of research... anybody got a pile of cash they don't know what to do with?

Update: By the way, in case you want a graphical depiction of what I'm thinking about, the Phoenix page has one (though obviously it's limited to the Phoenix scope of vision, and you may have to be a Microsoft CONNECT member to see it).

Normally, I like to stay out of these kinds of wars, but this post by Stu (whom I deeply respect and consider a friend, though he may not reciprocate by the time I'm done here) just really irked me somewhere sensitive. I'm not entirely sure why, but something about it just... rubbed me the wrong way, I guess is the best way to say it.

Let's dissect, shall we?

Stu begins with the following two candidates:

1. Joe has a problem to solve. The problem is specific, the need is immediate, and the scope is well-contrained.
2. Jane has a problem to solve. The problem is poorly understood, the need is ongoing, and the scope is ambiguous.

For starters, Joe doesn't exist. Or rather, exists only in the theoretical. Of course, neither does Jane really exist, either. Fact is, almost all projects are a combination of Joe and Jane. More importantly, Stu's efforts here to force people into the "either/or" approach to categorization is a subtle (or perhaps not so) ploy to force people into the decision-making path he thinks should be taken.

It's sort of like saying, most people fall into two categories:

1. Joe lives in Ghettopia, where all the men are dumb, the women are ugly, and the children are rejects from the ADHD Clinic.
2. Jane lives in Utopia, where all the men are smart, the woman are good-looking, and the children are well-behaved.

Think about it: you're at work, you have a project, and you happen across Stu's page. Faced with the typical project (too little time, too few resources, too vague in the understanding of requirements and domain comprehension), with whom are you likely to identify? Disturblingly happy Joe, who has a specific problem in a well-constrained scope? Hardly. So from the beginning, you're expected to identify with Jane, which (not surprisingly) leads you into Stu's preferred conclusion.

He goes on:

How should Joe and Jane think differently about software platforms?

   1. Joe's platform needs to be mainstream. It needs to offer immediate productivity, and the toolset should closely match the problem. Also, Joe doesn't want to climb a learning curve.
   2. Jane's needs are quite the opposite. Jane needs flexibility. She needs glue that doesn't set. She needs a way to control technical debt (Joe doesn't care.)

For my part, I am interested in Jane's problems. (And anyway, Joe often discovers he is actually Jane midway through projects.)

Hey, Stu, quick reality check for ya: most developers want all of the above. It's not a binary choice, productivity and toolset vs. flexibility and dynamism. The fact is, the Java language has a degree of flexibility, just not as much as is offered by the Ruby language. For that matter, if you want real flexibility, maybe you oughta look into Lisp, or even Smalltalk, since it (ST) can get at the underlying stack frames from the ST language itself! Now that's flexibility you Ruby guys can only dream of. (Oh, I know, Rubinius will give you that flexibility. Someday. Justin even alludes to how Rubinius is essentially an attempt to recapture that dynamism from Smalltalk. Ironic, then, isn't it, that the guys who wrote the fastest Smalltalk VM on the planet (Strongtalk, which is open-source now, by the way) ended up working at Sun... on the thing that later came to be called Hotspot? You think maybe they have a little familiarity and experience with VMs?)

And that crack about "control technical debt (Joe doesn't care)"?

Bullshit.

Let me repeat that in case you missed it: BULL-SHIT.

Joe and Jane both care about technical debt. Each may be willing to spend their currency on different problems, granted, but both of them care about technical debt. Not caring about technical debt is what got Chandler into trouble, and it had nothing to do with language or tools whatsoever. It's insulting to suggest that either of them don't care about technical debt, particularly the guy that chooses differently than you.

(Shame on you, Stu. You know better. Quit trolling.)

We continue:

So how does this affect platform choice? If you are Joe, you care about specific details about what a toolset can do right now. Most of Graeme's Top 10 reasons are in the "Right here, right now" category. This is true regardless of whether you think he is right. (Sometimes he is, sometimes not.)

I'll grant you, some of Graeme's Top 10 reasons are a bit spurious, and Stu-and-company do a good job of pointing those out. Frankly, anybody who makes a technical selection based on version numbers or whether or not a book exists for it seems to be missing the point, if you ask me. Of far greater concern is the stability of the language/tool, or the wealth of documentation for it. (And yes, this may seem to fly in the face of my arguments against Parrot a few posts ago; actually, it's not. If Parrot were more stable and/or more fully fleshed out, and the version updates just kept going, I'd be happy to say, "Go get this thing and give it a spin". But it doesn't feel stable to me, so I can't.)

But Stu's argument here is spurious: I don't care if you're Joe or if you're Jane, you always care about specific details about what a toolset can do, right now or otherwise. Certain concerns may be concerns that you can put off until later, but those concerns are always a part of the platform selection. Consider a hypothetical for a second: you currently are developing on Windows, and your project will run on Windows servers, with a possibility that it may need to run on non-Windows servers at some point in the future. Do you consider .NET or not? This is exactly the kind of detail that needs to be discussed--how likely is the move to a non-Windows server going to be? If it's <25%, then the CLR and ASP.NET might be a good choice, particularly if your developers are less "plumbing wonk" than "GUI designer", and you rely on being able to move the assemblies to a non-Windows server later via Mono.

Note: I'm not suggesting this a good choice in all scenarios. I'm making the point that the details of the toolset matter in your choice of toolsets, based on what your particular project needs are.

Jane cares just as much about toolset details as Joe does. I can't imagine a scenario where either of them don't care.

To continue:

My advice to Joe: Know exactly what you need, and then pick the platform that comes closest to solving it out of the box. Depending on Joe's needs, either Rails or Grails might be appropriate (or neither!). A particular point in Grails' favor would be an established team of Spring ninjas.

"Know exactly what you need"? Ah, right, because Joe belongs to that .01% of projects that have "specific problems, immediate need, and well-constrained scope". Nothing like conceding a point to the other guys, in preparation for the "killer blow":

If you are Jane, you care more about architecture. I mean this term in two senses:

   1. Architecture: the decisions you cannot unmake easily.
   2. Architecture: the constraints on how you think and work.

If you are Jane, you care about how and why the platform was assembled, because you are likely to have to adapt it quite a bit.

You know, I don't think I've ever been on a project where I didn't care about architecture or in having to "adapt it quite a bit". Of course, back in the days when I was writing C++, this meant either subclassing CWnd or TWindow in interesting ways, or else sometimes even going so far as to reach into the source code and making some tweaks, either at compile-time or through some well-established hackery. (Yes, I wrote a template class called THackOMatic that allowed me to bang away on private fields. Sue me. It worked, I documented the hell out of it, and ripped the hack back out once the bug was fixed.) Point is, both Joe and Jane care about the architecture.

Now, I think what Stu means here is that the architecture of the web framework is more malleable in Rails than it is in Grails, because Rails is written on top of Ruby and Grails is written on top of Groovy, Spring, the JEE container architecture, and Java:

Most of the commenters on my earlier post (and Graeme in his addendum) correctly identified the real architectural difference between Grails and Rails. Rails builds on Ruby, while Grails builds on Groovy and Spring.

Yes! I agree with this so far. (In fact, everybody should, because these are simple statements of fact.) But then Stu takes the cake for the Best Parting Non-Supported Shot Ever:

Rails wins this architecture bakeoff twice:

    * Ruby is a better language than Groovy.
    * Spring does most of its heavy lifting in the stable layer, which is not the right place.

Huh?

Ruby is perhaps a more flexible language than Groovy (and that's an arguable point, folks, and one which I really don't care to get into), but Ruby also runs on a less-flexible and less-scalable and less-supported platform than Groovy. I dunno that this makes Ruby better. It simply makes it different. Try convincing your IT guys to add yet another platform into their already-overwhelmingly complex suite of tools, particularly given the surprisingly sparse amount of monitoring information that Ruby platform offers. Stu may want to argue that Ruby-the-language is more flexible, regardless of what platform it runs on, and if so, then we're arguing languages not platforms, and while he might win much of his "Ruby is a better language than Groovy" argument, he's going to lose the "Ruby is more dynamic than Groovy", because on the JVM they have to be implemented under the same set of restrictions. You can't have it both ways.

(By the way, if you're one of those Ruby/Rails enthusiasts who's going to counterclaim that "Ruby-meaning-MRV is fast enough", I've heard the argument, and I think it's specious and ignorant. "Fast enough" is an argument that rests on your project being able to remain within the expected performance and scalability curve known at the beginning of the project, and remember, Jane's problem is that she doesn't know those sorts of things yet. So either you know, and have some better scope around the problem than Stu gives credit to Jane for having, or else you don't know, and can't assume that the Ruby interpreter will be able to handle the load.)

And WTF is up with the idea that "Spring does most of its heavy lifting in the stable layer, which is not the right place"? I think Stu means to say that Spring is a static layer, not stable layer[1], because hey, stability is kinda important to a few folks. (I'll give Stu the benefit of the doubt here and assume he cares about stability, too. I know his customers do.) Spring has its flaws, mind you, but arguing that it's not up to the heavy lifting seems to be like arguing that Java cannot scale. (Even Microsoft has given up on that argument, by the way.)

The worst part of this is, I've had discussions like this with Stu in the past, and he's much more articulate about it in person than he is in this blog post. Frankly, I think the most interesting space here is the intersection of Graeme's and Stu's positions, which is to say JRuby (and IronRuby or Ruby.NET, but that's for a different platform and out of the scope of this discussion entirely... yet still compelling and relevant, strangely enough). At the end of the day, these arguments about "my web framework is better than your web framework" are really just stupid. (As long as you're not trying to claim that Perl is the best web framework, anyway. Yes, Perl enthusiasts, I'm picking on you.)

My advice to Jane: Rails over Grails.

My advice to Jane: pick a consulting firm that doesn't have preconceived dogma about which web framework... or language, or any other toolset... to use. [2]

And if Jane can't afford a consulting firm, then Jane needs to do the research on her own and make her own decision based on the problem set, the context, and the whole range of tools available to her. (Anybody making a decision based solely on the basis of a blog-post-flame-war deserves what they get, regardless.)

As for Joe? Well, Joe could probably benefit from the goodness inherent in the dynamic languages that are popping up all over the place, too, not to mention the goodness inherent in the type-inferred languages that are starting to poke their heads through the Barrier of Adoption, all the while not ignoring the fact that he could probably benefit from the inherent performance and scalability of the major virtual machine technologies that have been a decade or more in production...

Meaning Joe probably needs to go through the same decision-making criteria Jane does. Thank God both of them, it turned out, work on the same project, as is often the case.

Meanwhile, I'm done with this thread. It's a pointless, stupid argument. Use the right tool for the job. Or, if you prefer, "From each language, according to its abilities, to each project, according to its needs."

Just remember that both shipping and supporting are features, too. Don't neglect the other in favor of the one.

[1] Yes, I saw the hyperlink to Ola's post about languages, and his definitions therein. Ironically, Ola's own comments there state that "Java is really the only choice here", which directly contradicts Stu's choice of MRV (the native Ruby interpreter). More importantly, I think Stu's point is resting on the static nature of the Java layer in Groovy, and while it's certainly more flexible to be able to hack at any layer of the stack, this is only realistically possible in small applications--this isn't my opinion, it's the opinion of Gregor Kiczales, who spent many years in CLOS and determined that CLOS's extremely flexible MOP system (more so than what Ruby currently supports, in fact) led to inherent problems in larger-scale projects. It was this thought that led him to create AspectJ in the first place.

[2] By the way, if there's any temptation in you[3] to post commentary and say, "Dude, you just don't understand Ruby" or "How can you agree with Graeme this way?", just don't. I do understand Ruby, and I like the language. (Much more than I do Rails, anyway.) And I'm not intrinsically agreeing that Grails is better than Rails, because I don't believe that, either. I believe in the basic equation that says the solution you pick is the one that is the right solution to the given problem in the stated context that yields the most desirable consequences.

[3] This includes you, Stu. Or Justin, or Graeme, or anybody working for Relevance, or anybody working for G2One, Inc.

My Mac froze when I tried to hook it up to the projector in the afternoon to do a 15-minute chat on Scala, thus losing the running blog entry in its entirety. Crap. This is my attempt to piece this overview together from memory--accordingly, details may suffer. Check the videos for verification when they come out. Of course, details were never my long suit anyway, so you probably want to do that for all of these posts, come to think of it...

I got to the conference about a half-hour late, owing to some personal errands in the morning; as I got there, Wayne Kelly was talking about his work on the Ruby.NET compiler.

Wayne Kelly: Parsing Ruby is made much harder by the fact that there is no Ruby specification to work from, which means the parser can't easily be generated from a parser generator. He tried, but couldn't get it to work cleanly and finally gave up in favor of getting to the "fun stuff" of code generation. Fortunately, the work he spent on the parser was generalized into the Gardens Point Parser Generator tools, which are also used in other environments and are included (?) as part of the Visual Studio SDK download. Good stuff. Ruby.NET uses a "wrapper" class around the .NET type that contains a hash of all the symbols for that type, which permits them to avoid even constructing (or even knowing!) the actual .NET type behind the scenes, except in certain scenarios where they have to know ahead of time. Interesting trick--probably could be used to great effect in a JSR-223 engine. (I know Rhino makes use of something similar, though I don't think they defer construction of the Java object behind the Rhino object.)

In general, I'm hearing this meme that "Ruby's lack of a specification is making X so much harder". I hate to draw the parallel, but it's highly reminiscent of the state of Perl until Perl 6, when Larry decided it was finally time to write a language specification (and the language has languished ever since), but maybe it's time for Matz or another Ruby digerati to sit down and write a formal specification for the Ruby language. Or even just its grammar.

Luke Hoban: Luke is the PM on the F# team, which is a language that I've recently been spending some quality time with, so I'm looking forward to this talk and how he presents the language. (Note to self: steal slides... I mean leverage slides... for my own future presentations on F#.) Not surprisingly, he makes pretty heavy use of the F# Interactive window in Visual Studio, using a trick I hadn't known before this: swipe some text in the editor, then press Alt-Enter, and it sends it to the Interactive window for execution. Nifty.

Then he starts showing off F#'s fidelity to the underlying CLR, and just for effect creates a DirectX surface and starts graphing functions on it. Then he starts playing with the functions, while the graph is still up, which has the neat effect of changing the function's graph in the DirectX surface without any explicit additional coding. Then he tightens up the mesh of the graph, and adds animation. (Mind you, these are all one-to-four lines of F# at a time he's pasting into the Interactive window.) What gets even more fun is when he pastes in a page and a half more of F# code that introduces balls rolling on the graphed surface. Very nifty. Makes Excel's graphing capabilities just look silly by comparison, in terms of "approachability" by programmers.

I will say, though, that I think that the decision to use significant whitespace in F# the same way Python does is a mistake. We don't have to go back to the semicolon everywhere, but surely there has to be A Better Way than significant whitespace.

Harry Pierson: Harry works in MS IT, so he doesn't play with languages on a regular basis, but he likes to explore, and recently has been exploring Parser Expression Grammars, which purport to be an easier way to write parsers based on an existing grammar. He shows off some code he wrote in F# by hand to do this (a port of the original Haskell code from the PEG paper), then shows the version that Don (Syme) sent back, which made use of active patterns in F#. (Check out Don's Expert F# for details.)

Harry predicated this talk with his experience talking with the creators of Glassbox (a C#-based tool that wanted to do something similar to what the C# mixins guys were doing from yesterday), and when he heard how much pain they were going through taking the Mono C# compiler and hacking it to introduce their extensions, he realized that compilers needed to be more modular. I had an interesting thought on this today, which I'll talk about below.

Magnus ???: Again, this was a lightning talk, a quick-hit lecture on the tool that his company is building, and I can't tell if the name of the tool was Intentional Software, or the name of the company was Intentional Software, or both. It's a derivative of what Charles Simonyi was working on at Microsoft (Intentional Programming), and basically they're creating programming language source trees in various ways while preserving the contents of the tree. So, for example, he takes some sample code (looked like C#, I don't think he said exactly what it was--assume some random C-family language), and presto, the curly braces are now in K&R style instead of where they belong (on new lines). Yawn. Then he presses another button, and suddenly the mathematical expressions are using traditional math "one over x" (with the horizontal line, a la MathML-described output) instead of "one slash x". That got a few peoples' attention. As did the next button-press, which essentially transformed whole equations in code into their mathematical equivalents. Then, he promptly button-presses again, and now the if/else constructs that are part of the equation are displayed inside the equation as "when"/"otherwise" clauses. Another button press, and suddenly we have a Lisp-like expression tree of the same function. Another button press, and we have a circuit diagram of the same function.

Wow. I'm floored. With this, completely non-programmer types can write/edit/test code, with full fidelity back to the original textual source. And, in fact, he promptly demonstrates that, with a table-driven representation of some business rules for a Dutch bank. It's a frickin' spreadsheet we're looking at, yet underneath (as he shows us once or twice), it's live and it's really code.

Combine this with some unit tests, and you have a real user-friendly programming environment, one that makes Rails look amateurish by comparison.

Now, if this stuff actually ships.... but this talk leads me to some deeper insight in conjunction with Harry's comments, which I'll go into below.

Wesner Moise: Wesner presents his product, NStatic, which is a static analysis tool that scans .NET assemblies for violations and bugs, much in the same way that FindBugs does in the Java space. It operates on binary assemblies (I think), rather than on source files the way FxCop does (I think), and it has a very spiffy GUI to help present the results. It also offers a sort of "live" view of your code, but I can't be certain of how it works because despite the fact that he takes the time to fire it up, he doesn't actually walk us through using it. (Wesner, if you read this, this is a HUGE mistake. Your slides should be wrapped around a demo, not the other way around. In fact, I'd suggest strongly ditching the slides altogether and just bring up an assembly and display the results.)

As readers of this blog (column?) will know, I'm a big fan of static analysis tools because I think they have the advantageous properties of being "always on" and, generally, "extensible to include new checks". Compilers fall into the first category, but not the second, in addition to being pretty weak in terms of the checks they do perform--given the exposure we're getting to functional languages and type inferencing, this should change pretty dramatically in the next five years. But in the meantime, I'm curious to start experimenting with the AbsIL toolkit (from MS Research) and F# or a rules engine (either a Prolog variant or something similar) to do some of my own tests against assemblies.

Unfortunately, it's a commercial product, so I don't think source will be available (in case you were wondering).

Chuck ...: Chuck stands up and does a quick-hit lecture on his programming language, CORBA... er, sorry about that, flashback from a bad acid trip. I mean of course, the language Cobra, which according to his blog he was working on at the Lang.NET 2006 Symposium. It's a Python derivative (ugh, more significant whitespace) with some interesting features, including a combination of static and dynamic typing, contracts, a "squeaky-clean" syntax, first-class support for unit tests (directly in the method definition!), and uses source-to-source "compilation", in this case from Cobra to C#, rather than compilation directly to IL.

It's a fascinating little piece of work, and I'm planning on playing with it some.

Miguel de Icaza: Miguel is another one of those who has more energy than any human being should have right to, and he spends the entire talk in fast-forward mode, speaking at a rapid-fire pace. He talks first of all about some experiences with Mono and the toolchain, then gets around to the history of Moonlight ("Can you give us a demo in 3 weeks?") and their (Mono's/Novell's) plans to get Moonlight out the door. They're already an impressive amount of the way there, but they have to make use of a "no-release" codecs library that also (potentially) contains some copywrit stuff, so they're instead going to incorporate Microsoft codecs, which they have rights to thanks to the Microsoft/Novell agreement of n months ago.

The thought of all these Linux devs running Microsoft code in their browser as they work with Moonlight just tickles my demented funny bone to no end.

He then switches tacks, and moves into gaming, because apparently a number of game companies are approaching Novell about using Mono for their gaming scripting engine. (Apparently it is being adopted by SecondLife, but the demo tanks because the SecondLife servers aren't up, apparently. That, or the Microsoft firewall is doing its job.) He jumps into some discussion about UnityScript, a {ECMA/Java}Script-like language for a game engine (called Unity, I think) that Rodrigo (creator of Boo) was able to build a parser for (in Boo) in 20 hours.

He then demonstrates the power of game engines and game editors by giving a short demo of the level editor for the game. He modifies the Robot bad guys to shoot each other instead of the player. If you're a game modder, this is old hat. If you're a business programmer, this is wildly interesting, probably because now you have visions of pasting your boss' face on the robots as you blast them.

Aaron Marten and Carl Brochu: I think his co-presenter's name was Carl something, but memory fails me, sorry. These two are from the Visual Studio Ecosystem team (which I think gets the prize for strangest product team name, ever), and they're here to give an overview of the Visual Studio integration API and tooling, with some sample code around how to plug into VS. This is good, because not an hour or two before, during Chuck's Cobra talk, he was talking about wanting to integrate into VS as a formal "next steps" for his language. Frankly, the whole area of IDE integration Dark Art to most folks (ranking behind custom languages, but still high up there), and the more the VSX team can do to dispel that myth, the more we'll start to see interesting and useful plugins for VS a la what we see in the Eclipse space. (Actually, let's hope the plugins we see for VS work more than a quarter of the time--Eclipse has become the dumping ground for every programmer who had an idea for a plugin, created a space on Sourceforge, wrote twenty lines of code, then got stuck and went away, leaving a nifty idea statement and a plugin that crashes Eclipse when you fire it up, not that I'm bitter or anything.)

The code demo they show off is a RegEx language colorization sample, nothing too terribly useful but still a nice small example of how to do it in VS. As VS starts to put more and more of a managed layer into place inside of VS, this sort of thing should become easier and easier, and thus a lot more approachable to the Average Mortal.

Me: I did a 15-minute presentation on Scala, since the name had come up a few times during the week, and promptly watched in horror as hooking my Mac up to the overhead projector locked the Mac completely. Ugh. Hard reboot. Ugh. Shuffle and dance about the history of Scala while waiting for the Mac to reboot and the VMWare image in which I have Scala installed to reboot. Ugh. I have no prepared slides, so I open up a random Scala example and start talking briefly about the syntax of a language whose list of features alone is so long it would take all fifteen minutes just to read aloud, much less try to explain. Cap it off with a leading question from Don Box ("Is this Sun's attempt to catch up to the C# compiler, given that Java is 'done' like the Patriots or the Dolphins?") that I try to answer as honestly and truthfully as possible, and a second question from Don (again) that forcefully reminds me that I'm out of time despite the "5 min" and "1 min" signs being held up by the guy next to him ("What would you say, in the two minutes you have left to you, is the main reason people should look at Scala?"), and I can safely say that I was thoroughly disgusted with myself at presenting what had to be the crappiest talk a the conference. *sigh*

That's it, no more presentations on technical topics, ever.

OK, not really, but a man can dream....

Don Box and Chris Andersen: I had to leave about ten minutes into their talk, so I still have no idea what Don and he are working on deep inside their incubating little cells in Microsoft. Something to do with "modeling and languages", and something that seeks to bring data to the forefront instead of code. *shrug* Not sure what to make of it, but I'm sure the video will make it more clear.

Meanwhile...

Overall: Here are some thoughts I think I think:

• A blog is not a part of your presentation, and your presentation is not part of your blog. I find it frustrating when speakers say, in their presentation, "Oh, you can find Y on my blog" and don't go into any more detail about it. I don't want to have to go look up your blog after the talk, when the context of the question or situation is swapped out of memory, and I don't want to have to go look it up during your presentation and miss whatever follows in your talk. If you blogged it, you should be able to give me a 30-second summary about the blog entry or what not, enough to tell me whether or not I want the deeper details of what's on your blog. Exception: files that contain examples of a concept you're discussing or sample code or whatnot.
• Don't hook your Mac up to the projector when you have a VMWare session on an external USB disk running. This happened to me at Seattle Code Camp, too, with the same result: Mac lockup. Dunno what the deal is, but from now on, the rule is, connect thy Mac, then fire up thy suspended VMWare VM.
• Language design and implementation is a lot more approachable now than it was even five years ago. Don't assume, for even a second, that the only way to go building a "DSL" or "little language" is by way of Rake and Rails--it's still a fair amount of work to build a non-trivial language, but between parser combinators and toolkits like the DLR and Phoenix, I'd go head-to-head against a Ruby-based DSL development process any day of the week.
• Don't go in front of Don Box at a conference. Dude may like to go long on his own talks, but man, he watches the clock like a hawk when it's time for him to start. (I may sound like I'm angry at Don--I'm not--but I'm not going to resist a chance to poke at him, either. *grin*)
• Modular tool chains are the future. Actually, this is a longish idea, so I will defer that for a future post.
• This conference rocks. It's not the largest conference, you get zero swag, and the room is a touch crowded at times, but man, this little get-together has one of the highest signal-to-noise ratio of any get-together I've been to, and without a doubt, within the realm of languages and language design, this is where the Cool Kids like to hang out.

Bye for now, and thanks for listening....

No snow last night, which means we avoid a repeat of the Redmond-wide shutdown of all facilities due to a half-inch of snow, and thus we avoid once again the scorn of cities all across the US for our wimpiness in the face of fluffy cold white stuff.

Erik Meijer: It's obvious why Erik is doing his talk at 9AM, because the man has far more energy than any human being has a right to have at this hour of the morning. Think of your hyperactive five-year-old nephew. On Christmas morning. And he's getting a G.I. Joe Super Bazooka With Real Bayonet Action(TM). Then you amp him up on caffeine and sugar. And speed.

Start with Erik's natural energy, throw in his excitement about Volta, compound in the fact that he's got the mic cranked up to 11 and I'm sitting in the front row and... well, this talk would wake the dead.

Volta, for those who haven't seen it before, is a MSIL->JavaScript transformation engine, among other things. In essence, he wants to let .NET developers write code in their traditional control-eventhandler model, then transform it automatically into a tier-split model when developers want to deploy it to the Web. (Erik posted a description of it to LtU, as well.) He's said a couple of times now that "Volta stretches the .NET platform to cover the Cloud", and from one perspective this is true--Volta automatically "splits" the code (in a manner I don't quite understand yet) to run Javascript in the browser and some amount of server-side code that remains in .NET.

A couple of thoughts came to mind when I first saw this, and they still haven't gone away:

• How do I control the round trips? If Volta is splitting the code, do I have control over what runs locally (on the server) and what runs remotely (in the browser)? The fact that Volta will help break things out from synchronous calls is nice, but I get much better perf and scale from avoiding the remote call entirely. [Erik answers this later, sort of: use of the RunAtOrigin attribute on a class defines that class to run on the server. He also addresses this again later in the section marked "End-to-End Profiling". Apparently you use a tool called "Rotunda" to profile where the tier split would be most effective.]
• How do I avoid the least-common denominator problem? Any time a library or language has tried to "cover up" the differences between the various UI models, it's left a bad taste in my mouth. Volta doesn't try to hide the markup, per se, but it's not hard to imagine a model where somebody says, "Well, if I write a control that I want to use in both WPF and HTML...."
• Is JavaScript really fast enough to handle the whole .NET library translated into JS? This is a general concern for both GWT and Volta--if I'm putting that much weight on top of the JS engine, will it collapse under several megs of JS code and who-knows-how-much data/objects inside of it?

Still, the idea of transforming MSIL into some other interesting useful form is a cool idea, and one I hope gets more play in other ways, too.

Gilad Bracha: Gilad discusses Newspeak, a Smalltalk- and Self-influenced language that, as John Rose puts it, "is one of the world's smallest languages while still remaining powerful". It bases on message send and receive, a la Smalltalk, but there's some immutability and some other ideas in there as well, on top of a pretty small syntactic core (a la Lisp, I think). Most of the discussion is around Newspeak's influences (Smalltalk, Self, Beta, and a little Scala, plus some Scheme and E), with code examples drawn from a compiler framework. Most notably, Gilad shows how because the language is based on message-sends, it becomes pretty trivial to build a parser combinator that combines both scanning and actions by breaking lexing/scanning into a base class and the actions into a derived class. Elegant.

Unfortunately, no implementation is available, though Gilad strongly suggests that anybody who wants to see it should send him a letter on company letterhead so he can show it to the corporate heads back at the office in order to get it out to the world at large. I'm sufficiently intrigued that I'm going to send him one, personally.

Giles Thomas: Giles talks about Resolver One, his company's spreadsheet product, which is built in IronPython and exposes it as the scripting language within the spreadsheet, a la Excel's formula language and VBA combined. It's an interesting talk from sveeral perspectives:

1. he's got 110,000 lines of code written in IronPython and hasn't found the need to go to C# yet (implying that, yes, dynamic languages can scale(1))
2. he's taking the position that spreadsheets are essentially programs, and therefore should be accessible in a variety of ways outside of the spreadsheet itself--as a back-end to a web service or website, for example
3. he's attended a conference in the UK on spreadsheets. Think about that for a moment: a conference... on spreadsheets. That sounds about as exciting as attending the IRS' Annual Tax Code Conference and Social.
4. he's effectively demonstrating the power of scripting languages exposed inside an application engine, in this case, the scripting language runs throughout the product/application. Frankly I personally think he'd be better off writing the UI core in C# or VB and using the IronPython as the calculation engine, but give credit where credit is due: it runs pretty damn fast, there was no crash ever, and it's fascinating watching him put regular .NET objects (like IronPython generators or lambdas) into the spreadsheet grid and use them from other cells. Nifty.

This is a really elegant design. I'm impressed. JVMers (thanks to JSR 233), CLRers (thanks to DLR), take note: this is the way to build applications/systems with emergent behavior.

Seo Sanghyeon: Seo had a few problems with his entirely gratuitous demo for his context-free talk (although I could've sworn he said "content-free" talk, but it was probably just a combination of his accent and my wax-filled ears). In essence, he wants to produce new backends for the DLR, in order to reuse the existing DLR front- and middle-ends and make lots of money (his words). I can get behind that. In fact, he uses a quote from my yesterday's blog (the "DLR should produce assemblies out the back end" one), which is both flattering and a little scary. ("Wait, that means people are actually reading this thing?!?")

Jim Hugunin had an interesting theme threaded through his talk yesterday that I didn't explicitly mention, and that was a mistake, because it's recurring over and over again this week: "Sharing is good, but homogeneity is bad". I can completely agree with this; sharing implies the free exchange of resources (such as assemblies and type systems, in this case) and ideas (at the very least), but homogeneity--in this case, the idea that there exists somewhere in space and time the One Language God Intended--is something that just constrains our ability to get stuff done. Imagine trying to access data out of a relational database using C++, for example.

Paul Vick: Paul's from the VB team [cue bad one-liner disparaging VB here], and he's talking on "Bringing Scripting (Back) to Visual Basic", something that I can definitely agree with.

Editor's Note: I don't know what Visual Basic did to anger the Gods of Computer Science, but think about it for a second: they were a dynamic language that ran on a bytecode-based platform, used dynamic typing and late name-based binding by default, provided a "scripting glue" to existing applications (Office being the big one), focused primarily on productivity, and followed a component model from almost the first release. Then, after languishing for years as the skinny guy on the beach as the C++ developers kicked sand on their blanket, they get the static-typing and early-binding religion, just in time to be the skinny guy on the beach as the Ruby developers kick sand on their blanket.

Oh, and to add insult to injury, the original code name for Visual Basic before it got a marketing name? Ruby.

Whatever you did, VB, your punishment couldn't have fit the crime. Hopefully your highly-publicized personal hell is almost over.

Paul points out that most VBers come to the language not by purchasing the VB tool chain, but through VBA in Office, and demos VB inside of Excel to prove the point. The cool thing is (and I don't know how he did this), he has a Scripting Window inside of Excel 2007 and demos both VB and IronPython in an interactive mode, flipping from one to the other. A couple of people have done this so far, and I'd love to know if that's a core part of the DLR or something they just built themselves. (Note to self: pick apart DLR code base in my copious spare time.) He does an architectural overview of the VB compilation toolchain, which is nice if you're interested in how to architect a modern IDE environment. The VB guys split things into Core services (what you'd expect from a compiler), Project services (for managing assembly references and such), and IDE services (Intellisense and so on). Note that the Project services implementation is different (and simpler) for the command-line compiler, and obviously the command-line compiler has no IDE services. Their goal for Visual Basic v.Next, is to provide the complete range of Core/compiler, Project and even IDE services for people who want to use VB as a scripting engine, and he demos a simple WinForms app that hosts a single control that exposes the VB editor inside of it. Cool beans.

Serge Baranovsky: (Serge goes first because Karl Prosser has problems hooking his laptop up to the projector.) Serge is a VB MVP and works for a tools company, and he talks about doing some code analysis works. He runs a short demo that has an error in it (he tries to serialize a VB class that has a public event, which as Rocky Lhotka has pointed out prior to now, is a problem). The tool seems somewhat nice, but I wish he'd talked more about the implementation of it rather than the various patterns it spots. (The talk kinda feels like it was intended for a very different audience than this one.) Probably the most interesting thing is that he runs the tool over newTelligence's dasBlog codebase, and finds close to 4000 violations of Microsoft's coding practices. While I won't hold that up as a general indictment of dasBlog, I will say that I like static analysis tools precisely because they can find errors or practice violations in an automated form, without requiring human intervention. Compilers need to tap into this more, but until they do, these kinds of standalone tools can hook into your build process and provide that kind of "always on" effect.

Karl Prosser: Karl's talking about PowerShell, but I'm worried as he gets going that he's talking from a deck that's intended for an entirely difference audience than this one. Hopefully I'm just being paranoid. As the talk progresses, he's right down the middle: he's showing off some interesting aspects of PowerShell-the-language, and has some interesting ideas about scripting languages in general (which obviously includes the PowerShell language) in the console vs. in a GUI, but he also spends too much time talking about the advantages of PowerShell-the-tool (and a little bit about his product, which I don't mind--he's got a kick-ass PowerShell console window). He also talks about some of the advantages of offering a console view instead of a GUI view, which I already agree with, and how to create apps to be scripted, which I also already agree with, so maybe I'm just grumpy at not hearing some more about experiences with PowerShell-the-language and how it could be better or lessons learned for other languages. He talks about the value of the REPL loop, which I think is probably already a given with this crowd (even though it most definitely wouldn't be at just about any other conference on the planet, with possible exception of OOPSLA).

One thing he says that I find worth contemplating more is that "Software is a 2-way conversation, which is why I dislike waterfall so much." I think he's mixing metaphors here--developing software may very well be a 2-way conversation which is why agile methodologies have become so important, and using software may very well also be a 2-way conversation, but that has nothing to do with how the software was built. User interaction with software is one of those areas that developers--agile or otherwise--commonly don't think about much beyond "Does the user like it or not?" (and sometimes not even that much, sadly). What makes this so much worse is that half the time, what the user thinks they want is nowhere close to what they actually want, and the worst part about it is you won't know it until they see the result and then weigh in with the, "Oh, man, that's just not what I thought it would look like."

Which raises the question: how do you handle this? I would tend to say, "I really don't think you'll like this when it's done", but then again I've been known to be high-handed and arrogant at times, so maybe that's not the best tack to take. Thoughts?

Wez Furlong: Wez is talking about PHP, which he should know about, because apparently he's a "Core Developer" (his quotes) of PHP. This promises to be interesting, because PHP is one of those language-slash-web-frameworks that I've spent near-zero time with. (If PHP were accessible outside of the web world, I'd be a lot more interested in it; frankly, I don't know why it couldn't be used outside of the web world, and maybe it already can, but I haven't spent any time studying it to know for sure one way or another.) His question: "Wouldn't it be great if the web devs could transfer their language knowledge to the client side--Silverlight?" Honestly, I'm kind of tired of all these dynamic language discussions being framed in the context of Silverlight, because it seems to pigeonhole the whole dynamic language thing as "just a Silverlight thing". (Note to John Lam: do everything you can to get the DLR out of Silverlight as a ship vehicle, because that only reinforces that notion, IMHO.) Direct quote, and I love it: (slide) "PHP was designed to solve the specific problem of making it easy for Rasmus to make his home page; Not a good example of neat language design." (Wez) "It's a kind of mishmash of HTML, script, code, all thrown together into a stinking pile of a language." He's going over the basics of PHP-the-language, which (since I don't know anything about PHP) is quite interesting. PHP has a "resource" type, which is a "magical handle for an arbitrary C structure type", for external integration stuff.

He's been talking to Jim (Hugunin, I presume) about generics in PHP. Dude... generics... in PHP? In a language with no type information and no first-class support for classes and interfaces? That just seems like such a wrong path to consider....

Interesting--another tidbit I didn't know: PHP uses a JIT-compilation scheme to compile into its own opcode and runs it in the Zend (sp?) engine. Yet another VM hiding in plain sight. I have to admit, I am astounded at how many VMs and execution engines I keep running into in various places.

Another direct quote, which I also love: (slide) "PHP 4: Confirmed as a drunken hack." (Wez) "There's this rumor that one night in a bar, somebody said, Wouldn't it be cool if there were objects in PHP, and the next day there was a patch..." If Wez is any indication of the rest of the PHP community, I could learn to like this language, if only for its self-deprecating sense of humor about itself.

He then mentions Phalanger, a CLR implementation of PHP, and hands the floor over to Thomas for his Phalanger talk. Nice very high-level intro of PHP, and probably entirely worthless if you already knew something about PHP... which I didn't, so I liked it.

Thomas Petricek; Peli de Halleux and Nikolai Tillman; Jeffrey Sax:

(I left the room to get a soda, got roped into doing a quick Channel 9 video about why the next five years will be about languages, then ran into Wez and we talked for a bit about PHP's bytecode engine, then ran into with Jeffrey Snover, PM from the PowerShell team, and we talked for a bit about PSH, hosting PSH, and some other things. Since I don't have a lot of call for numeric computing, I didn't catch most of Jeffrey's talk. I wish I'd caught the Phalanger talk, though. I'll have to collar Thomas in the hallway tomorrow.)

(Just as a final postscript to this talk--John Rose of Sun is sitting next to me during Jeff's talk, and he has more notes on this one talk than any other I've seen. Combined with the cluster of CLR guys that swarmed Jeff as soon as he was done, and I'll go out on a really short limb here and say that this was definitely one of the ones you want to catch when the videos go online "in about a week", according to one of the organizers.)

Stefan Wenig and Fabian Schmied: Oh, this was a fun talk. Very humorous opening, particularly the (real) town's sign they show in the first five or so slides. But their point is good, that enterprise software for various different customers is not easy. They write all their code in C#, so they have to handle this. They cite Jacobsen's "Aspect-Oriented Software Development with Use Cases" as an exemplar of the problem, and go through a few scenarios that don't work to solve it: lots of configuration or scripting, multiple inheritance, inheriting one from another, and so on. (slide) "Inheritance is not enough." (To those of you not here--this is a great slide deck and very well delivered. Even if you don't care about C# or mixins, watch this talk if you give presentations.) Stefan sets up the problem, and Fabian discusses their mixin implementation. (slide) "Mixin programming is the McFlurry programming model." *grin* Mixins in their implementation can be configured "either way": either the mixins can declare what classes they apply to, or the target class can declare which mixins it implements. They create a derived class of your class which implements the mixin interface and mixes in the mixin implementation, then you create the generated derived class via a factory method.

I asked if this was a compile-time, or run-time solution; it's run-time, and they generate code using Reflection.Emit once you call through their static factory (which kicks the process off).

Their mixin implementation is available here.

Thought I'd offer a highly-biased interpretation of the goings-on here at the Lang.NET Symposium. Quite an interesting crowd gathered here; I don't have a full attendee roster, but it includes Erik Meijer, Brian Goetz, Anders Hjelsberg, Jim Hugunin, John Lam, Miguel de Icaza, Charlie Nutter, John Rose, Gilad Braha, Paul Vick, Karl Prosser, Wayne Kelly, Jim Hogg, among a crowd in total of about 40. Great opportunities to do those wonderful hallway chats that seem to be the far more interesting part of conferences.

Jason Zander: Jason basically introduces the Symposium, and the intent of the talk was mostly to welcome everybody (including the > 50% non-Microsoft crowd here) and offer up some interesting history of the CLR and .NET, dating all the way back to a memo/email sent by Chris Brumme in 1998 about garbage collection and the "two heaps", one around COM+ objects, and the other for malloc-allocated data. Fun stuff; hardly intellectually challenging, mind you, but interesting.

Anders Hjelsberg: Anders walks us through the various C# 3.0 features and how they combine to create the subtle power that is LINQ (it's for a lot more than just relational databases, folks), but if you've seen his presentation on C# 3 at TechEd or PDC or any of the other conferences he's been to, you know how that story goes. The most interesting part of his presentation was a statement he made that I think has some interesting ramifications for the industry:

I think that the taxonomies of programming languages are breaking down. I think that languages are fast becoming amalgam. ... I think that in 10 years, there won't be any way to categorize languages as dynamic, static, procedural, object, and so on.

(I'm paraphrasing here--I wasn't typing when he said it, so I may have it wrong in the exact wording.)

I think, first of all, he's absolutely right. Looking at both languages like F# and Scala, for example, we see a definite hybridization of both functional and object languages, and it doesn't take much exploration of C#'s and VB's expression trees facility to realize that they're already a half-step shy of a full (semantic or syntactic) macro system, something that traditionally has been associated with dynamic languages.

Which then brings up a new question: if languages are slowly "bleeding" out of their traditional taxonomies, how will the vast myriad hordes of developers categorize themselves? We can't call ourselves "object-oriented" developers if the taxonomy doesn't exist, and this will have one of two effects: either the urge to distinguish ourselves in such a radical fashion will disappear and we'll all "just get along", or else the distinguishing factor will be the language itself and the zealotry will only get worse. Any takers?

Jim Hugunin: Jim talks about the DLR... and IronPython... by way of a Lego Mindstorms robot and balloon animals. (You kinda had to be there. Or watch the videos--they taped it all, I don't know if they're going to make them publicly available, but if they do, it's highly recommended to watch them.) He uses a combination of Microsoft Robotics Studio, the XNA libraries, his Lego mindstorms robot, and IronPython to create an XBox-controller-driven program to drive the robot in a circle around him. (Seriously, try to get the video.)

(Note to self: go grab the XNA libraries and experiment. The idea of using an Xbox controller to drive Excel or a Web browser just appeals at such a deep level, it's probably a sign of serious dementia.)

Jim talks about the benefits of multiple languages running on one platform, something that a large number of the folks here can definitely agree with. As an aside, he shows the amount of code required to build a C-Python extension in C, and the amount of code required to build an IronPython extension in C#. Two or three orders of magnitude difference, seriously. Plus now the Python code can run on top of a "real" garbage collector, not a reference-counted GC such as the one C-Python uses (which was news to me).

Personally, I continue to dislike Python's use of significant whitespace, but I'm sure glad he came to Microsoft and put it there, because his work begat IronRuby, and that work in turn begat the DLR, which will in turn beget a ton more languages.

Thought: What would be truly interesting would be to create a compiler for the DLR--take a DLR AST, combine it with the Phoenix toolkit, and generate assemblies out of it. They may have something like that already in the DLR, but if it's not there, it should be.

Martin Maly: Martin talks about the DLR in more depth, about the expression trees/AST trees, and the advantages of writing a language on top of the DLR instead of building your own custom platform for it. He shows implementation of the Add operation in ToyScript, the language that ships "with" the DLR (which is found, by the way, in the source for the IronPython and IronRuby languages), and how it manages the reflection (if you will) of operations within the DLR to find the appropriate operation.

Martin is also the one responsible for LOLcode-DLR, and pulls it out in the final five minutes because he just had to give it one final hurrah (or GIMMEH, as you wish). The best part is writing "HAI VISIBLE "Howdy" KTHXBYE" at the DLR console, and just to get even more twisted, he uses the DLR console to define a function in ToyScript, then call it from LOLCODE (using his "COL ... WIT ... AN ..." syntax, which is just too precious for words) directly.

I now have a new goal in life: to create a WCF service in LOLCode that calls into a Windows Workflow instance, also written in LOLcode. I don't know why, but I must do this. And create a UI that's driven by an XBox-360 controller, while I'm at it.

I need a life.

Charlie Nutter/John Rose: Charlie (whom I know from a few No Fluff Just Stuff shows) and John (whom I know from a Scala get-together outside of JavaOne last year) give an overview of some of the elements of the JVM and JRuby, some of the implementational details, and some of the things they want to correct in future versions. John spent much time talking about the "parallel universe" he felt he'd walked into, because he kept saying, "Well, in the JVM we have <something>... which is just like what you [referring to the Microsoft CLR folk who'd gone before him] call <something else>...." It was both refreshing (to see Microsoft and Sun folks talking about implementations without firing nasty white papers back and forth at one another) and disappointing (because there really were more parallels there than I'd thought there'd be, meaning there's less interesting bits for each side to learn from the other) at the same time.

In the end, I'm left with the impression that the JVM really needs something akin to the DLR, because I'm not convinced that just modifying the JVM itself (the recently-named Da Vinci Machine) will be the best road to take--if it's implemented inside the VM, then modifications and enhancements will take orders of magnitude longer to work their way into production use, since there will be so much legacy (Java) code that will have to be regression-tested against those proposed changes. Doing it in a layer-on-top will make it easier and more agile, I believe.

That said, though, I'm glad they (Sun) are (finally) taking the steps necessary to put more dynamic hooks inside the JVM. One thing that John said that really has me on tenterhooks is that Java really does need a lightweight method handle, similar (sort of, kind of, well OK exactly just like) .NET delegates (but we'll never admit it out loud). Once they have that, lots of interesting things become possible, but I have no idea if it would be done in time for Java 7. (It would be nice, but first the Mercurial repositories and other OpenJDK transition work needs to be finished; in the meantime, though, John's been posting patches on his personal website, available as a link off of the Da Vinci Machine/mlvm project page.)

Dan Ingalls: Dan shows us the Lively Kernel project from Sun Labs, which appears to be trying to build the same kind of "naked object" model on top of the Browser/JavaScript world that the Naked Objects framework did on top of the CLR/WinForms and JVM/AWT, both of which trying essentially to recapture the view of objects as Alan Kay originally intended them (entities directly manipulable by the user). For example, there's a "JavaScript CodeBrowser" which looks eerily reminiscent of the Object Browser from Smalltalk environments, except that the code inside of it is all {Java/ECMA}Script. A bit strange to see if you're used to seeing ST code there.

I can't help but wonder, how many people are watching this, thinking, "Great, we're back to where we were 30 years ago?" Granted, there's a fair amount of that going on anyway, given how many concepts that are hot today were invented back in the 50's and 60's, but still, reinventing the Smalltalk environment on top of the browser space just... seems... *sigh*...

It's here if you want to play with it, though when I tried just now it presented me with authentication credentials that I don't have; you may have better luck choosing the 0.8b1 version from here, and the official home page (with explanatory text and a tutorial) for it is here.

Pratap Lakshman: Pratap starts with a brief overview of {Java/ECMA}Script, focusing initially on prototype-based construction. Then he moves into how the DLR should associate various DLR Expression and DLR Rule nodes to the language constructs. Interesting, but a tad on the slow/redundant side, and perhaps a little bit more low-level than I would have liked. That said, though, Charlie spotted what he thought would be a race condition in the definition of types in the code demonstrated, and he and Jim had an interesting discussion around lock-free class definition and modification, which was interesting, if just somewhat slightly off-topic.

Roman Ivantsov: Roman's built the Irony parser, which is a non-code-gen C# parser language reminiscent of the growing collection of parser combinators running around, and he had some thoughts on an ERP language with some interesting linguistic features. I'm going to check out Irony (already pulled it down, in fact), but I'm also very interested to see what comes out of Harry's talk on F# Parsing tomorrow.

Dinner: Pizza. Mmmmm, pizza.

More tomorrow, assuming I don't get stuck here on campus due to the City of Redmond shutting almost completely down due to 2 inches (yes, 2 inches) of snow on the ground from last night. (If you're from Boston, New York, Chicago, Vermont, Montana, North Dakota, or anyplace that gets snow, please don't comment--I already know damn well how ludicrous it is to shut down after just 2 frickin' inches.)

... or if you're a-hankering to kick my *ss over my sacreligious statements about Perl, I'll be at Building 20 on the Microsoft campus in Redmond, at the Language.NET Symposium with a few other guys who know something about language and VM implementation: Jim Hugunin, Gilad Bracha, Wayne Kelly, Charlie Nutter, John Rose, John Lam, Erik Meijer, Anders Hejlsberg....

I wish there were more "other VMs" representation showing up (some of the Parrot or Strongtalk or Squeak folks would offer up some great discussion points), but in the event they don't, it'll still be an interesting discussion. Some of the topics I'm looking forward to:

"Targeting DLR" (Martin Maly)

"Multiple Languages on the Java VM" (John Rose and Charles Nutter)

"Vision of the DLR" (Jim Hugunin)

"Retargeting DLR" (Seo Sanghyeon)

"Ruby" (John Lam)

"Ruby.NET" (Wayne Kelly)

"Integrating Languages into the VSS" (Aaron Marten) [I presume VSS means Visual Studio Shell and not Visual Source Safe...]

"JScript" (Pratap Lakshman) [He can't be looking forward to this, based on what I'm hearing about the debates around ECMAScript 4.0....]

"Volta" (Erik Meijer)

"Parsing Expression Grammars in F#" (Harry Pierson) [I can't be certain, but I think I turned Harry on to F# in the first place, so I'm curious to learn what he's doing with it in Real Life]

And for those of you living within easy driving distance of Redmond, take a trip out to DigiPen this Saturday and Sunday for the Seattle Code Camp. I'll be doing a talk on F# and another one on Scala on Saturday (modulo any scheduling changes). Those of you already coming should check out the xUnit.NET presentation (currently scheduled for 4:45PM on Saturday)--some of James' and Brad's ideas of what a unit-testing framework should really look like are kinda radical, very intriguing, and guaranteed to be thought-provoking. Dunno if there's an xUnit.JVM yet...

... but there should be.

A number of folks commented on the last post about my "ignorant and apparently unsupported swipes against Parrot and Perl". Responses:

1. I took exactly one swipe at Perl, and there was a smiley at the end of it. Apparently, based on the heavily-slanted pro-Perl/anti-Perl-bigotry comments I've received, Perl programmers don't understand smileys. So I will translate: "It means I am smiling as I say this, which is intended as a way of conveying light-heartedness or humor."
2. I didn't take any swipes at Parrot. I said, "Parrot may change that in time, but right now it sits at a 0.5 release and doesn't seem to be making huge inroads into reaching a 1.0 release that will be attractive to anyone outside of the "bleeding-edge" crowd." It is sitting at a 0.5 release (up from a 0.4 release at this time last year), and it doesn't seem to be making huge inroads into reaching a 1.0 release, which I have had several CxO types tell me is the major reason they won't even consider looking at it. That's not a "swipe", that's a practical reality. The same CxO types stay the hell away from Microsoft .NET betas and haven't upgraded to JDK 1.6 yet, either, and they're perfectly justified in doing so: it's called the bleeding edge for a reason.
3. Fact: I don't like Perl. Therefore, on my blog, which is a voice for my opinion and statements, Perl sucks. I don't like a language that has as many side effects and (to my mind) strange symbolic syntax as Perl uses. The side effects I think are a bad programming language design artifact; the strange symbolic syntax is purely an aesthetic preference.
4. Fact: I don't pretend that everybody should agree with me. If you like Perl, cool. I also happen to be Lutheran. If you're Catholic, that's cool, too. Doesn't mean we can't get along, so long as you respect my aesthetic preferences so I can respect yours.
5. I don't have to agree with you to learn from you, and vice versa. In fact, I like it better when people argue, because I learn more that way.
6. I also don't have to like your favorite language, and you don't have to like mine (if I had one).
7. I'm not ignorant, and please don't try to assert your supposed superiority by taking that unsupported swipe at me, either. I've tried Perl. I've tried Python, too, and I find its use of significant whitespace to be awkward and ill-considered, and a major drawback to what otherwise feels like an acceptable language. Simply because I disagree with your love of the language doesn't make me ignorant any more than you are if you dislike Java or C# or C++ or any of the languages I like.
8. Fact: I admit to a deep ignorance of the Perl community. I've never claimed anything of the sort. I also admit to a deep ignorance of the Scientology community, yet that doesn't stop me from passing personal judgment on the Scientologists' beliefs, particularly as expressed by Tom Cruise, or Republicans' beliefs, as expressed by Pat Robertson. And honestly, I don't think I need a deep understanding of the Perl community to judge the language, just as I don't need a deep understanding of Tom Cruise to judge Scientology, or just as you don't need a deep understanding of me to judge my opinions.
9. If by "homework", by the way, you mean "Spend years writing Perl until you come to love it as I do", then yes, I admit, by your definition of "homework", I've not done my homework. If by "homework" you mean "Learn Perl until you become reasonably proficient in it", then yes, I have done my homework. I had to maintain some Perl scripts once upon an eon ago, not to mention the periodic deciphering of the Perl scripts that come with the various Linux/Solaris/Mac OS images I work with, and my dislike and familiarity with the language stemmed from that experience. I have a similar dislike of 65C02 assembler.
10. I've met you, chromatic, though you may not remember it: At the second FOO Camp, you and I and Larry Wall and Brad Merrill and Dave Thomas and Peter Drayton had an impromptu discussion about Parrot, virtual machines, the experiences Microsoft learned while building the Common Type System for the CLR, some of the lessons I'd learned from playing with multiple languages on top of the JVM, and some of the difficulties in trying to support multiple languages on top of a single VM platform. I trust that you don't consider Dave Thomas to be ignorant; he and I had a long conversation after that impromptu round table and we came to the conclusion that Parrot was going to be in for a very rough ride without some kind of common type definitions across the various languages built for it. (He was a little stunned at the idea that there wasn't some kind of common hash type across the languages, if that helps to recall the discussion.) This in no way impugns the effort you're putting into Parrot, by the way, nor should you take this criticism to suggest that you should stop your work. Frankly, I'd love to see how Parrot ends up, since it takes a radically different approach to a virtual execution engine than other environments do, and stark contrast is always a good learning experience. The fact that Parrot has advanced all of a minor build number in the last year seems to me, an outsider who periodically grabs the code, builds it and pokes around, to be indicative of the idea that Parrot is taking a while.
11. Oh, and by the way, chromatic, since I've got your attention, while there, you argued that the Parrot register-based approach was superior to the CLR or JVM approach because "passing things in registers is much faster than passing them on the stack". (I may be misquoting what you said, but this is what Peter, Brad, Dave and I all heard.) I wanted to probe that statement further, but Brad jumped in to explain to you (and the subject got changed fairly quickly, so I don't know if you picked up on it) that the execution stack in the CLR (and the JVM) is an abstraction--both virtual machines make use of registers where and when possible, and can do so fairly easily. Numerous stack-based VMs have done this over the years as a performance enhancement. I assume you know this, so I'm curious to know if I misunderstood the rationale behind a register-based VM.
12. Fact: Perl 6 recently celebrated the fifth anniversary of its announcement. Not its ship date, but the announcement. Fact: Perl 6 has not yet shipped.
13. Opinion: I hate to say this if you're a Perl lover, but based on the above, Perl 6 is quickly vying for the Biggest Vaporware Ever award. The only language that rivals this in terms of incubation length is the official C++ standard, which took close to or more than a decade. And it (rightly) was crucified in the popular press for taking that long, too. (And there was a long time where we--a large group of other C++ programmers I worked with--weren't sure it would ship at all, much less before the language was completely dead, because there was no visible progress taking place: no new features, no new libraries, no new changes, nothing.)
14. Fact: I would love for Parrot to ship, because I would love to be able to start experimenting with building languages that emit PIR. I would love to embed Parrot as an execution engine inside of a larger application, using said language as the glue around the core parts of the application. I would love to do all of this in a paid project. When Parrot reaches a 1.0 release, I'll consider it, just as I had to wait until the CLR and C# reached a 1.0 release when I started playing with them in July of 2001.
15. Fact: The JVM and CLR are not nearly as good for heavily-recursive languages (such as what we see in functional languages like Haskell and ML and F# and Erlang and Scala) because neither one, as of this writing, supports tail-call recursion optimization; the CLR pretends to, via the "tail" opcode that is essentially ignored as of CLR v2.0 (the CLR that ships with .NET 2, 3 and 3.5), but the JVM doesn't even go that far. JIT compilers can do a few things to help optimize here, but realistically both environments need this if they're to become reasonable dynamic language platforms.
16. Fact: Lots of large systems have been built in COBOL, too, and scale even better than systems built in Perl, or C#, or Java, or C++. That doesn't mean I like them, want to program in them, or that the COBOL community should be any less proud of them. Again, just because I don't care for abstract art doesn't undermine the brilliance of an artist like Mark Rothko.
17. And I find the statement, "If you need X, don't look at other languages" to be incredibly short-sighted. Even if I were only paid to write Java, I would look at other languages, because I learn more about programming in general by doing so, thus improving my Java code. I would heartily suggest the same thing for the C# programmer, the C++ programmer, the VB programmer, the Ruby programmer, the Perl programmer, ad infinitum.

At the end of the day, the fact that I don't like Perl doesn't undermine its efficacy amongst those who use it. The fact that Perl scale(1)s and scale(2)s doesn't take away from the fact that I don't like its syntax, semantics, or idioms. The fact that the Perl community can't take a ribbing over the large numbers of incomprehensible Perl scripts out there only reinforces the idea that Perl developers like incomprehensible syntax. (If you want a kind of dirty revenge, ask the Java developers about generics.)

Besides, if you listen to Paul Graham, all these languages are just footnotes on Lisp, anyway, so let's all quit yer bitchin' and start REPLing with lots of intuitively selected (or, if you prefer, irritatingly silly) parentheses.

But, in the interests of making peace with the Perl community....

65C02 assembler sucks way worse than Perl. (And no smiley; that's a statement delivered in straight-faced monotone.)

The recent "failure" of the Chandler PIM project generated the question, "Can Dynamic Languages Scale?" on TheServerSide, and, as is all too typical these days, it turned into a "You suck"/"No you suck" flamefest between a couple of posters to the site.

I now make the perhaps vain attempt to address the question meaningfully.

What do you mean by "scale"?

There's an implicit problem with using the word "scale" here, in that we can think of a language scaling in one of two very orthogonal directions:

1. Size of project, as in lines-of-code (LOC)
2. Capacity handling, as in "it needs to scale to 100,000 requests per second"

Part of the problem I think that appears on the TSS thread is that the posters never really clearly delineate the differences between these two. Assembly language can scale(2), but it can't really scale(1) very well. Most people believe that C scales(2) well, but doesn't scale(1) well. C++ scores better on scale(1), and usually does well on scale(2), but you get into all that icky memory-management stuff. (Unless, of course, you're using the Boehm GC implementation, but that's another topic entirely.)

Scale(1) is a measurement of a language's ability to extend or enhance the complexity budget of a project. For those who've not heard the term "complexity budget", I heard it first from Mike Clark (though I can't find a citation for it via Google--if anybody's got one, holler and I'll slip it in here), he of Pragmatic Project Automation fame, and it's essentially a statement that says "Humans can only deal with a fixed amount of complexity in their heads. Therefore, every project has a fixed complexity budget, and the more you spend on infrastructure and tools, the less you have to spend on the actual business logic." In many ways, this is a reflection of the ability of a language or tool to raise the level of abstraction--when projects began to exceed the abstraction level of assembly, for example, we moved to higher-level languages like C to help hide some of the complexity and let us spend more of the project's complexity budget on the program, and not with figuring out which register needed to have the value of the interrupt to be invoked. This same argument can be seen in the argument against EJB in favor of Spring: too much of the complexity budget was spent in getting the details of the EJB beans correct, and Spring reduced that amount and gave us more room to work with. Now, this argument is at the core of the Ruby/Rails-vs-Java/JEE debate, and implicitly it's obviously there in the middle of the room in the whole discussion over Chandler.

Scale(2) is an equally important measurement, since a project that cannot handle the expected user load during peak usage times will have effectively failed just as surely as if the project had never shipped in the first place. Part of this will be reflected in not just the language used but also the tools and libraries that are part of the overall software footprint, but choice of language can obviously have a major impact here: Erlang is being tossed about as a good choice for high-scale systems because of its intrinsic Actors-based model for concurrent processing, for example.

Both of these get tossed back and forth rather carelessly during this debate, usually along the following lines:

1. Pro-Java (and pro-.NET, though they haven't gotten into this particular debate so much as the Java guys have) adherents argue that a dynamic language cannot scale(1) because of the lack of type-safety commonly found in dynamic languages. Since the compiler is not there to methodically ensure that parameters obey a certain type contract, that objects are not asked to execute methods they couldn't possibly satisfy, and so on. In essence, strongly-typed languages are theorem provers, in that they take the assertion (by the programmer) that this program is type-correct, and validate that. This means less work for the programmer, as an automated tool now runs through a series of tests that the programmer doesn't have to write by hand; as one contributor to the TSS thread put it:

   "With static languages like Java, we get a select subset of code tests, with 100% code coverage, every time we compile. We get those tests for "free". The price we pay for those "free" tests is static typing, which certainly has hidden costs."

   Note that this argument frequently derails into the world of IDE support and refactoring (as its witnessed on the TSS thread), pointing out that Eclipse and IntelliJ provide powerful automated refactoring support that is widely believed to be impossible on dynamic language platforms.
2. Pro-Java adherents also argue that dynamic languages cannot scale(2) as well as Java can, because those languages are built on top of their own runtimes, which are arguably vastly inferior to the engineering effort that goes into the garbage collection facilities found in the JVM Hotspot or CLR implementations.
3. Pro-Ruby (and pro-Python, though again they're not in the frame of this argument quite so much) adherents argue that the dynamic nature of these languages means less work during the creation and maintenance of the codebase, resulting in a far fewer lines-of-code count than one would have with a more verbose language like Java, thus implicitly improving the scale(1) of a dynamic language.

   On the subject of IDE refactoring, scripting language proponents point out that the original refactoring browser was an implementation built for (and into) Smalltalk, one of the world's first dynamic languages.

4. Pro-Ruby adherents also point out that there are plenty of web applications and web sites that scale(2) "well enough" on top of the MRV (Matz's Ruby VM?) interpreter that comes "out of the box" with Ruby, despite the widely-described fact that MRV Ruby Threads are what Java used to call "green threads", where the interpreter manages thread scheduling and management entirely on its own, effectively using one native thread underneath.
5. Both sides tend to get caught up in "you don't know as much as me about this" kinds of arguments as well, essentially relying on the idea that the less you've coded in a language, the less you could possibly know about that language, and the more you've coded in a language, the more knowledgeable you must be. Both positions are fallacies: I know a great deal about D, even though I've barely written a thousand lines of code in it, because D inherits much of its feature set and linguistic expression from both Java and C++. Am I a certified expert in it? Hardly--there are likely dozens of D idioms that I don't yet know, and certainly haven't elevated to the state of intuitive use, and those will come as I write more lines of D code. But that doesn't mean I don't already have a deep understanding of how to design D programs, since it fundamentally remains, as its genealogical roots imply, an object-oriented language. Similar rationale holds for Ruby and Python and ECMAScript, as well as for languages like Haskell, ML, Prolog, Scala, F#, and so on: the more you know about "neighboring" languages on the linguistic geography, the more you know about that language in particular. If two of you are learning Ruby, and you're a Python programmer, you already have a leg up on the guy who's never left C++. Along the other end of this continuum, the programmer who's written half a million lines of C++ code and still never uses the "private" keyword is not an expert C++ programmer, no matter what his checkin metrics claim. (And believe me, I've met way too many of these guys, in more than just the C++ domain.)

A couple of thoughts come to mind on this whole mess.

Just how refactorable are you?

First of all, it's a widely debatable point as to the actual refactorability of dynamic languages. On NFJS speaker panels, Dave Thomas (he of the PickAxe book) would routinely admit that not all of the refactorings currently supported in Eclipse were possible on a dynamic language platform given that type information (such as it is in a language like Ruby) isn't present until runtime. He would also take great pains to point out that simple search-and-replace across files, something any non-trivial editor supports, will do many of the same refactorings as Eclipse or IntelliJ provides, since type is no longer an issue. Having said that, however, it's relatively easy to imagine that the IDE could be actively "running" the code as it is being typed, in much the same way that Eclipse is doing constant compiles, tracking type information throughout the editing process. This is an area I personally expect the various IDE vendors will explore in depth as they look for ways to capture the dynamic language dynamic (if you'll pardon the pun) currently taking place.

Who exactly are you for?

What sometimes gets lost in this discussion is that not all dynamic languages need be for programmers; a tremendous amount of success has been achieved by creating a core engine and surrounding it with a scripting engine that non-programmers use to exercise the engine in meaningful ways. Excel and Word do it, Quake and Unreal (along with other equally impressively-successful games) do it, UNIX shells do it, and various enterprise projects I've worked on have done it, all successfully. A model whereby core components are written in Java/C#/C++ and are manipulated from the UI (or other "top-of-the-stack" code, such as might be found in nightly batch execution) by these less-rigorous languages is a powerful and effective architecture to keep in mind, particularly in combination with the next point....

Where do you run again?

With the release of JRuby, and the work on projects like IronRuby and Ruby.NET, it's entirely reasonable to assume that these dynamic languages can and will now run on top of modern virtual machines like the JVM and the CLR, completely negating arguments 2 and 4. While a dynamic language will usually take some kind of performance and memory hit when running on top of VMs that were designed for statically-typed languages, work on the DLR and the MLVM, as well as enhancements to the underlying platform that will be more beneficial to these dynamic language scenarios, will reduce that. Parrot may change that in time, but right now it sits at a 0.5 release and doesn't seem to be making huge inroads into reaching a 1.0 release that will be attractive to anyone outside of the "bleeding-edge" crowd.

So where does that leave us?

The allure of the dynamic language is strong on numerous levels. Without having to worry about type details, the dynamic language programmer can typically slam out more work-per-line-of-code than his statically-typed compatriot, given that both write the same set of unit tests to verify the code. However, I think this idea that the statically-typed developer must produce the same number of unit tests as his dynamically-minded coworker is a fallacy--a large part of the point of a compiler is to provide those same tests, so why duplicate its work? Plus we have the guarantee that the compiler will always execute these tests, regardless of whether the programmer using it remembers to write those tests or not.

Having said that, by the way, I think today's compilers (C++, Java and C#) are pretty weak in the type expressions they require and verify. Type-inferencing languages, like ML or Haskell and their modern descendents, F# and Scala, clearly don't require the degree of verbosity currently demanded by the traditional O-O compilers. I'm pretty certain this will get fixed over time, a la how C# has introduced implicitly typed variables.

Meanwhile, why the rancor between these two camps? It's eerily reminiscent of the ill-will that flowed back and forth between the C++ and Java communities during Java's early days, leading me to believe that it's more a concern over job market and emplyability than it is a real technical argument. In the end, there will continue to be a ton of Java work for the rest of this decade and well into the next, and JRuby (and Groovy) afford the Java developer lots of opportunities to learn those dynamic languages and still remain relevant to her employer.

It's as Marx said, lo these many years ago: "From each language, according to its abilities, to each project, according to its needs."

Oh, except Perl. Perl just sucks, period.

PostScript

I find it deeply ironic that the news piece TSS cited at the top of the discussion claims that the Chandler project failed due to mismanagement, not its choice of implementation language. It doesn't even mention what language was used to build Chandler, leading me to wonder if anybody even read the piece before choosing up their sides and throwing dirt at one another.

A couple of people have left comments that definitely deserve response, so here we go:

Glenn Vanderberg comments in response to the Larraysaywhut? post, and writes: