For a couple of years now, I've been going around the world and giving a talk entitled "Pragmatic Architecture", talking both about what architecture is (and what architects really do), and ending the talk with my own "catalog" of architectural elements and ideas, in an attempt to take some of the mystery and "cloud" nature of architecture out of the discussion. If you've read Effective Enterprise Java, then you've read the first version of that discussion, where Pragmatic Architecture was a second-generation thought process.

Recently, the patterns & practices group at Microsoft went back and refined their Application Architecture Guide, and while there's a lot about it that I wish they'd done differently (less of a Microsoft-centric focus, for one), I think it's a great book for Microsoft-centric architects to pick up and have nearby. In a lot of ways, this is something similar to what I had in mind when I thought about the architectural catalog, though I'll admit that I'd prefer to go one level "deeper" and find more of the "atoms" that make up an architecture.

Nevertheless, I think this is a good PDF to pull down and put somewhere on your reference list.

Notes and caveats: Firstly, this is a book for solution architects; if you're the VP or CTO, don't bother with it, just hand it to somebody further on down the food chain. Secondly, if you're not an architect, this is not the book to pick up to learn how to be one. It's more in the way of a reference guide for existing architects. In fact, my vision is that an architect faced with a new project (that is, a new architecture to create) will think about the problem, sketch out a rough solution in his head, then look at the book to find both potential alternatives (to see if they fit better or worse than the one s/he has in her/his head), and potential consequences (to the one s/he has in her/his head). Thirdly, even if you're a Java or Ruby architect, most of the book is pretty technology-neutral. Just take a black Sharpie to the parts that have the Microsoft trademark around them, and you'll find it a pretty decent reference, too. Fourthly, in the spirit of full disclosure, the p&p guys brought me in for a day of discussion on the Guide, so I can't say that I'm completely unbiased, but I can honestly say that I didn't write any of it, just offered critique (in case that matters to any potential readers).

People are used to the idea of phishing attacks showing up in their email, but in glowing testament to the creativity of potential attackers, Twitter recently has seen a rash of phishing attacks through Twitter's "direct messaging" feature.

The attack plays out like this: someone on your Twitter followers list sends you a direct message saying, "hey! check out this funny blog about you... " with a hyperlink to a website, "http://jannawalitax.blogspot.com/" . Clicking on the hyperlink takes you to a website that redirects to a webpage containing what looks like the Twitter login page. This is an attempt to get you to fill in your username, and more importantly, your password.

Needless to say, I'd avoid it. If you do get suckered in (hey, I admit it, I did), make sure to change your password immediately after.

What I find fascinating about this attack is that the direct messages come from people that are on my followers list--unless Twitter somehow has a hole in it that allows non-followers to direct-message you, it means that this is a classic security Ponzi scheme: I use the attack to gather the credentials for the people that I'm following directly, then log in and use those credentials to attack their followers, then use those gathered credentials to attack their followers, and so on. Fixing this is also going to be a pain--literally, everybody on Twitter has to change their password, or the scheme can continue with the credentials of those who didn't. (Assuming Twitter doesn't somehow lop the attack off at the knees, for example, by disallowing hyperlinks or something equally draconian.)

We won't even stop to consider what damage might be done if a Twitter-user uses the same password and login name for their Twitter account as they do for other accounts (such as email, banking websites, and so on). If you're one of those folks, you seriously might want to reconsider the strategy of using the same password for all your websites, unless you don't care if they get spoofed.

There's two lessons to be learned here.

One, that as a user of a service--any service--you have to be careful about when and how you're entering your credentials. It's easy to simply get into the habit of offering them up every time you see something that looks familiar, and if supposed "computer experts" (as most of the Twitterverse can be described) can be fooled, then how about the casual user?

Two, and perhaps the more important lesson for those of us who build software, that any time you build a system that enables people to communicate, even when you put a lot of energy into making sure that the system is secure, there's always an angle that attackers will find that will expose a vulnerability, even if it's just a partial one (such as the gathering of credentials here). If you don't need to allow hyperlinks, don't. If you don't need to allow Javascript, don't. Start from the bare minimum that people need to make your system work, and only add new capabilities after they've been scrutinized in a variety of ways. (YAGNI sometimes works to our advantage in more ways than one, it turns out.)

Kudos, by the way, to the Twitter-keepers, who had a message describing the direct-message phishing attack on the Twitter Home page within hours.

Thanks again to the folks at Microsoft who've been gracious enough to award me MVP Architect status again this year, and to the INETA Speakers Bureau, who've decided that I'm to remain an INETA speaker for another twelve months.

What's more impressive is the list of new speakers that INETA has added, including Rachel Appel, Alan Stevens, and Steve Andrews, among others. Congratulations to all three of you, you deserve it. Looking forward to seeing you guys on the road in 2009!

It's once again that time of year, and in keeping with my tradition, I'll revisit the 2008 predictions to see how close I came before I start waxing prophetic on the coming year. (I'm thinking that maybe the next year--2010's edition--I should actually take a shot at predicting the next decade, but I'm not sure if I'd remember to go back and revisit it in 2020 to see how I did. Anybody want to set a calendar reminder for Dec 31 2019 and remind me, complete with URL? )

Without further preamble, here's what I said for 2008:

• THEN: General: The buzz around building custom languages will only continue to build. More and more tools are emerging to support the creation of custom programming languages, like Microsoft's Phoenix, Scala's parser combinators, the Microsoft DLR, SOOT, Javassist, JParsec/NParsec, and so on. Suddenly, the whole "write your own lexer and parser and AST from scratch" idea seems about as outmoded as the idea of building your own String class. Granted, there are cases where a from-hand scanner/lexer/parser/AST/etc is the Right Thing To Do, but there are times when building your own String class is the Right Thing To Do, too. Between the rich ecosystem of dynamic languages that could be ported to the JVM/CLR, and the interesting strides being made on both platforms (JVM and CLR) to make them more "dynamic-friendly" (such as being able to reify classes or access the call stack directly), the probability that your company will find a need that is best answered by building a custom language are only going to rise. NOW: The buzz has definitely continued to build, but buzz can only take us so far. There's been some scattershot use of custom languages in a few scattershot situations, but it's certainly not "taken the world by storm" in any meaningful way yet.
• THEN: General: The hype surrounding "domain-specific languages" will peak in 2008, and start to generate a backlash. Let's be honest: when somebody looks you straight in the eye and suggests that "scattered, smothered and covered" is a domain-specific language, the term has lost all meaning. A lexicon unique to an industry is not a domain-specific language; it's a lexicon. Period. If you can incorporate said lexicon into your software, thus making it accessible to non-technical professionals, that's a good thing. But simply using the lexicon doesn't make it a domain-specific language. Or, alternatively, if you like, every single API designed for a particular purpose is itself a domain-specific language. This means that Spring configuration files are a DSL. Deployment descriptors are a DSL. The Java language is a DSL (since the domain is that of programmers familiar with the Java language). See how nonsensical this can get? Until somebody comes up with a workable definition of the term "domain" in "domain-specific language", it's a nonsensical term. The idea is a powerful one, mind you--creating something that's more "in tune" with what users understand and can use easily is a technique that's been proven for decades now. Anybody who's ever watched an accountant rip an entirely new set of predictions for the new fiscal outlook based entirely on a few seed numbers and a deeply-nested set of Excel macros knows this already. Whether you call them domain-specific languages or "little languages" or "user-centric languages" or "macro language" is really up to you. NOW: The backlash hasn't begun, but only because the DSL buzz hasn't materialized in much way yet--see previous note. It generally takes a year or two of deployments (and hard-earned experience) before a backlash begins, and we haven't hit that "deployments" stage yet in anything yet resembling "critical mass" yet. But the DSL/custom language buzz continues to grow, and the more the buzz grows, the more the backlash is likey.
• THEN: General: Functional languages will begin to make their presence felt. Between Microsoft's productization plans for F# and the growing community of Scala programmers, not to mention the inherently functional concepts buried inside of LINQ and the concurrency-friendly capabilities of side-effect-free programming, the world is going to find itself working its way into functional thinking either directly or indirectly. And when programmers start to see the inherent capabilities inside of Scala (such as Actors) and/or F# (such as asynchronous workflows), they're going to embrace the strange new world of functional/object hybrid and never look back. NOW: Several books on F# and Scala (and even one or two on Haskell!) were published in 2008, and several more (including one of my own) are on the way. The functional buzz is building, and lots of disparate groups are each evaluating it (functional programming) independently.
• THEN: General: MacOS is going to start posting some serious market share numbers, leading lots of analysts to predict that Microsoft Windows has peaked and is due to collapse sometime within the remainder of the decade. Mac's not only a wonderful OS, but it's some of the best hardware to run Vista on. That will lead not a few customers to buy Mac hardware, wipe the machine, and install Vista, as many of the uber-geeks in the Windows world are already doing. This will in turn lead Gartner (always on the lookout for an established trend they can "predict" on) to suggest that Mac is going to end up with 115% market share by 2012 (.8 probability), then sell you this wisdom for a mere price of $1.5 million (per copy). NOW: Can't speak to the Gartner report--I didn't have $1.5 million handy--but certainly the MacOS is growing in popularity. More on that later.
• THEN: General: Ted will be hired by Gartner... if only to keep him from smacking them around so much. .0001 probability, with probability going up exponentially as my salary offer goes up exponentially. (Hey, I've got kids headed for college in a few years.) NOW: Well, Gartner appears to have lost my email address and phone number, but I'm sure they were planning to make me that offer.
• THEN: General: MacOS is going to start creaking in a few places. The Mac OS is a wonderful OS, but it's got its own creaky parts, and the more users that come to Mac OS, the more that software packages are going to exploit some of those creaky parts, leading to some instability in the Mac OS. It won't be widespread, but for those who are interested in finding it, they're there. Assuming current trends (of customers adopting Mac OS) hold, the Mac OS 10.6 upgrade is going to be a very interesting process, indeed. NOW: Shhh. Don't tell anybody, but I've been seeing it starting to happen. Don't get me wrong, Apple still does a pretty good job with the OS, but the law of numbers has started to create some bad upgrade scenarios for some people.
• THEN: General: Somebody is going to realize that iTunes is the world's biggest monopoly on music, and Apple will be forced to defend itself in the court of law, the court of public opinion, or both. Let's be frank: if this were Microsoft, offering music that can only be played on Microsoft music players, the world would be through the roof. All UI goodness to one side, the iPod represents just as much of a monopoly in the music player business as Internet Explorer did in the operating system business, and if the world doesn't start taking Apple to task over this, then "justice" is a word that only applies when losers in an industry want to drag down the market leader (which I firmly believe to be the case--nobody likes more than to pile on the successful guy). NOW: Nothing this year.
• THEN: General: Somebody is going to realize that the iPhone's "nothing we didn't write will survive the next upgrade process" policy is nothing short of draconian. As my father, who gets it right every once in a while, says, "If I put a third-party stereo in my car, the dealer doesn't get to rip it out and replace it with one of their own (or nothing at all!) the next time I take it in for an oil change". Fact is, if I buy the phone, I own the phone, and I own what's on it. Unfortunately, this takes us squarely into the realm of DRM and IP ownership, and we all know how clear-cut that is... But once the general public starts to understand some of these issues--and I think the iPhone and iTunes may just be the vehicle that will teach them--look out, folks, because the backlash will be huge. As in, "Move over, Mr. Gates, you're about to be joined in infamy by your other buddy Steve...." NOW: Apple released iPhone 2.0, and with it, the iPhone SDK, so at least Apple has opened the dashboard to third-party stereos. But the deployment model (AppStore) is still a bit draconian, and Apple still jealously holds the reins over which apps can be deployed there and which ones can't, so maybe they haven't learned their lesson yet, after all....
• THEN: Java: The OpenJDK in Mercurial will slowly start to see some external contributions. The whole point of Mercurial is to allow for deeper control over which changes you incorporate into your build tree, so once people figure out how to build the JDK and how to hack on it, the local modifications will start to seep across the Internet.... NOW: OpenJDK has started to collect contributions from external (to Sun) sources, but still in relatively small doses, it seems. None of the local modifications I envisioned creeping across the 'Net have begun, that I can see, so maybe it's still waiting to happen. Or maybe the OpenJDK is too complicated to really allow for that kind of customization, and it never will.
• THEN: Java: SpringSource will soon be seen as a vendor like BEA or IBM or Sun. Perhaps with a bit better reputation to begin, but a vendor all the same. NOW: SpringSource's acquisition of G2One (the company behind Groovy just as SpringSource backs Spring) only reinforced this image, but it seems it's still something that some fail to realize or acknowledge due to Spring's open-source (?) nature. (I'm not a Spring expert by any means, but apparently Spring 3 was pulled back inside the SpringSource borders, leading some people to wonder what SpringSource is up to, and whether or not Spring will continue to be open source after all.)
• THEN: .NET: Interest in OpenJDK will bootstrap similar interest in Rotor/SSCLI. After all, they're both VMs, with lots of interesting ideas and information about how the managed platforms work. NOW: Nope, hasn't really happened yet, that I can see. Not even the 2nd edition of the SSCLI book (by Joel Pobar and yours truly, yes that was a plug) seemed to foster the kind of attention or interest that I'd expected, or at least, not on the scale I'd thought might happen.
• THEN: C++/Native: If you've not heard of LLVM before this, you will. It's a compiler and bytecode toolchain aimed at the native platforms, complete with JIT and GC. NOW: Apple sank a lot of investment into LLVM, including hosting an LLVM conference at the corporate headquarters.
• THEN: Java: Somebody will create Yet Another Rails-Killer Web Framework. 'Nuff said. NOW: You know what? I honestly can't say whether this happened or not; I was completely not paying attention.
• THEN: Native: Developers looking for a native programming language will discover D, and be happy. Considering D is from the same mind that was the core behind the Zortech C++ compiler suite, and that D has great native platform integration (building DLLs, calling into DLLs easily, and so on), not to mention automatic memory management (except for those areas where you want manual memory management), it's definitely worth looking into. www.digitalmars.com NOW: D had its own get-together as well, and appears to still be going strong, among the group of developers who still work on native apps (and aren't simply maintaining legacy C/C++ apps).

Now, for the 2009 predictions. The last set was a little verbose, so let me see if I can trim the list down a little and keep it short and sweet:

• General: "Cloud" will become the next "ESB" or "SOA", in that it will be something that everybody will talk about, but few will understand and even fewer will do anything with. (Considering the widespread disparity in the definition of the term, this seems like a no-brainer.)
• Java: Interest in Scala will continue to rise, as will the number of detractors who point out that Scala is too hard to learn.
• .NET: Interest in F# will continue to rise, as will the number of detractors who point out that F# is too hard to learn. (Hey, the two really are cousins, and the fortunes of one will serve as a pretty good indication of the fortunes of the other, and both really seem to be on the same arc right now.)
• General: Interest in all kinds of functional languages will continue to rise, and more than one person will take a hint from Bob "crazybob" Lee and liken functional programming to AOP, for good and for ill. People who took classes on Haskell in college will find themselves reaching for their old college textbooks again.
• General: The iPhone is going to be hailed as "the enterprise development platform of the future", and companies will be rolling out apps to it. Look for Quicken iPhone edition, PowerPoint and/or Keynote iPhone edition, along with connectors to hook the iPhone up to a presentation device, and (I'll bet) a World of Warcraft iPhone client (legit or otherwise). iPhone is the new hotness in the mobile space, and people will flock to it madly.
• .NET: Another Oslo CTP will come out, and it will bear only a superficial resemblance to the one that came out in October at PDC. Betting on Oslo right now is a fools' bet, not because of any inherent weakness in the technology, but just because it's way too early in the cycle to be thinking about for anything vaguely resembling production code.
• .NET: The IronPython and IronRuby teams will find some serious versioning issues as they try to manage the DLR versioning story between themselves and the CLR as a whole. An initial hack will result, which will be codified into a standard practice when .NET 4.0 ships. Then the next release of IPy or IRb will have to try and slip around its restrictions in 2010/2011. By 2012, IPy and IRb will have to be shipping as part of Visual Studio just to put the releases back into lockstep with one another (and the rest of the .NET universe).
• Java: The death of JSR-277 will spark an uprising among the two leading groups hoping to foist it off on the Java community--OSGi and Maven--while the rest of the Java world will breathe a huge sigh of relief and look to see what "modularity" means in Java 7. Some of the alpha geeks in Java will start using--if not building--JDK 7 builds just to get a heads-up on its impact, and be quietly surprised and, I dare say, perhaps even pleased.
• Java: The invokedynamic JSR will leapfrog in importance to the top of the list.
• Windows: Another Windows 7 CTP will come out, and it will spawn huge media interest that will eventually be remembered as Microsoft promises, that will eventually be remembered as Microsoft guarantees, that will eventually be remembered as Microsoft FUD and "promising much, delivering little". Microsoft ain't always at fault for the inflated expectations people have--sometimes, yes, perhaps even a lot of times, but not always.
• Mac OS: Apple will begin to legally threaten the clone market again, except this time somebody's going to get the DOJ involved. (Yes, this is the iPhone/iTunes prediction from last year, carrying over. I still expect this to happen.)
• Languages: Alpha-geek developers will start creating their own languages (even if they're obscure or bizarre ones like Shakespeare or Ook#) just to have that listed on their resume as the DSL/custom language buzz continues to build.
• XML Services: Roy Fielding will officially disown most of the "REST"ful authors and software packages available. Nobody will care--or worse, somebody looking to make a name for themselves will proclaim that Roy "doesn't really understand REST". And they'll be right--Roy doesn't understand what they consider to be REST, and the fact that he created the term will be of no importance anymore. Being "REST"ful will equate to "I did it myself!", complete with expectations of a gold star and a lollipop.
• Parrot: The Parrot guys will make at least one more minor point release. Nobody will notice or care, except for a few doggedly stubborn Perl hackers. They will find themselves having nightmares of previous lives carrying around OS/2 books and Amiga paraphernalia. Perl 6 will celebrate it's seventh... or is it eighth?... anniversary of being announced, and nobody will notice.
• Agile: The debate around "Scrum Certification" will rise to a fever pitch as short-sighted money-tight companies start looking for reasons to cut costs and either buy into agile at a superficial level and watch it fail, or start looking to cut the agilists from their company in order to replace them with cheaper labor.
• Flash: Adobe will continue to make Flex and AIR look more like C# and the CLR even as Microsoft tries to make Silverlight look more like Flash and AIR. Web designers will now get to experience the same fun that back-end web developers have enjoyed for near-on a decade, as shops begin to artificially partition themselves up as either "Flash" shops or "Silverlight" shops.
• Personal: Gartner will still come knocking, looking to hire me for outrageous sums of money to do nothing but blog and wax prophetic.

Well, so much for brief or short. See you all again next year....

When I was in college, at the University of California, Davis, I lived in the International Relations building (D Building in the Tercero dorm area, for any other UCD alum out there), and got my first real glimpse of the feminist movement up front. It seemed like it was filled with militant, angry members of the female half of the species, who insisted that their gender was spelled "womyn", so that it wasn't somehow derived from "man" (wo-man, wo-men, get it?), who blamed most of the world's problems on the fact that men were running the show, and that therefore, because of my own gender, I was to share equally in the blame for its ills.

Maybe I was--Lord knows I certainly wasn't an entirely nice guy back then (and some will chirp from the back of the theater, "back then?!?")--but it still left the whole "feminist" thing as something I couldn't really be around, much less support.

My sister, it turned out, had a different experience at University of California, Santa Cruz, and became one.

Needless to say, this made family get-togethers somewhat awkward.

Then, a few years later, she asked my help in purchasing a new computer for herself. Basically, she just wanted me along to help explain any of the technical terms that she wasn't entirely familiar with, and to give her some advice on whether they were important to her needs. Not an unreasonable request, and not something I wouldn't do for anybody else, male or female alike. (I sometimes wish my father would ask my help before buying, but that's another story.)

We went to the store, and I got my first lesson in sexual discrimination.

The entire time we were in the store, despite the fact that it was my sister asking the questions, despite the fact that I only answered questions that she asked of me directly (in other words, I was there to help her, not to help the sales guy sell to her), almost the entire conversation was spent with the sales guy talking to me, even if he was answering her question. His body language was unquestionably that of, "She's clearly not capable of making this decision herself", and addressed everything to me, despite her repeated attempts to catch his eye and have him talk to her, the actual purchaser with the question.

I was a bit taken aback. I don't think the sales guy even noticed. That bothered me more than anything.

Ever since that time, I've been curiously and cautiously trying to figure out why there aren't more women in IT.

Several theories have presented themselves over the years:

1. Women, aside from a statistical minority, are structurally incapable of mastering IT. This is the "math is hard" argument, and I think we can all pretty much agree where this one belongs.
2. Women are encouraged/forced down an educational path that leads them away from IT until much later in life. I've heard this from a couple of women my age, and while I think there may be some validity to it, at least back in the day, I don't know if there still is. I'd love to get some feedback from recent high school or college grads who can weigh in with some anecdotal evidence one way or the other.
3. Women are entering IT, but not in the areas that I hang out in. This is definitely possible, and I think is happening, to some degree. At an Adobe "Flex Camp" last night (I was Chet Haase's roadie for the evening), I noticed a far more even split of gender than I'd ever seen at a Java or .NET user group, and when I mentioned this to one of the other speakers, he nodded and said that women were far more prevalent in the "web design" space, which is clearly not a space I play much in. I've also heard that the system admin space is much more "female-friendly", too.
4. Women get in to IT, then out of it or stay "hidden" in it. I've heard the theory that some women choose to get out of IT because they're not willing to put the same kind of time and energy into it as some men are, or they choose to remain at the software developer level instead of trying to advance the corporate ladder into management or other more visible positions.
5. There are exactly the number of women in IT that want to be there. Hey, let's face it, maybe women just don't like software development, and that's OK, because there's a lot of jobs I don't like, either.

My concern is with theories 1 and 2. There should be no reason whatsoever that a woman cannot succeed every bit as much as a man can. This is one of those (few?) industries where the principal qualifications are entirely intellectual/mental, and that means there's absolutely no reason why one gender should be favored over another. (Nursing and teaching are others, for example.)

So, without further ado, those of you who are interested, check out Dana Coffey's link on the Lego Build event at the MSDN events coming up.

It amazes me how insular and inward-facing the software industry is. And how the "agile" movement is reaping the benefits of a very simple characteristic.

For example, consider Jeff Palermo's essay on "The Myth of Self-Organizing Teams". Now, nothing against Jeff, or his post, per se, but it amazes me how our industry believes that they are somehow inventing new concepts, such as, in this case the "self-organizing team". Team dynamics have been a subject of study for decades, and anyone with a background in psychology, business, or sales has probably already been through much of the material on it. The best teams are those that find their own sense of identity, that grow from within, but still accept some leadership from the outside--the classic example here being the championship sports team. Most often, that sense of identity is born of a string of successes, which is why teams without a winning tradition have such a hard time creating the esprit de corps that so often defines the difference between success and failure.

(Editor's note: Here's a free lesson to all of you out there who want to help your team grow its own sense of identity: give them a chance to win a few successes, and they'll start coming together pretty quickly. It's not always that easy, but it works more often than not.)

How many software development managers--much less technical leads or project managers--have actually gone and looked through the management aisle at the local bookstore?

Tom and Mary Poppendieck have been spending years now talking about "lean" software development, which itself (at a casual glance) seems to be a refinement of the concepts Toyota and other Japanese manufacturers were pursuing close to two decades ago. "Total quality management" was a concept introduced in those days, the idea that anyone on the production line was empowered to stop the line if they found something that wasn't right. (My father was one of those "lean" manufacturing advocates back in the 80's, in fact, and has some great stories he can tell to its successes, and failures.)

How many software development managers or project leads give their developers the chance to say, "No, it's not right yet, we can't ship", and back them on it? Wouldn't you, as a developer, feel far more involved in the project if you knew you had that power--and that responsibility?

Or consider the "agile" notion of customer involvement, the classic XP "On-Site Customer" principle. Sales people have known for years, even decades (if not centuries), that if you involve the customer in the process, they are much more likely to feel an ownership stake sooner than if they just take what's on the lot or the shelf. Skilled salespeople have done the "let's walk through what you might buy, if you were buying, of course" trick countless numbers of times, and ended up with a sale where the customer didn't even intend to buy.

How many software development managers or project leads have read a book on basic salesmanship? And yet, isn't that notion of extracting what the customer wants endemic to both software development and basic sales (of anything)?

What is it about the software industry that just collectively refuses to accept that there might be lots of interesting research on topics that aren't technical yet still something that we can use? Why do we feel so compelled to trumpet our own "innovations" to ourselves, when in fact, they've been long-known in dozens of other contexts? When will we wake up and realize that we can learn a lot more if we cross-train in other areas... like, for example, getting your MBA?

Dustin Campbell, a self-professed "IDE guy", is speaking at the .NET Developer's Association of Redmond this evening, on the future of Visual Basic in Visual Studio 2010, and I feel compelled, based on my earlier "dissing" of VB in my thoughts of PDC post, to give VB a little love here.

First of all, he notes publicly that the VB and C# teams have been brought together under one roof, organizationally, so that the two languages can evolve in parallel to one another. I have my concerns about this. Frankly, I think the Managed Languages team at Microsoft is making a mistake by making these two languages mirror images of one another, no matter what their customers are telling them; it's creating an artificial competition between them, because if you can't differentiate between the two on a technical level, then the only thing left to differentiate them on is an aesthetic level (do you prefer curly braces and semicolons, or keywords?). Unfortunately, the market has already done so, to the tune of "C# developers make more than VB developers do (on average)", leaving little doubt in the minds of VB developers where they'd rather be... and even less doubt in the minds of C# developers where they'd rather the VB developers remain, lest the supply and demand curves shift and move the equilibrium point of C# developer salaries further south.

Besides, think about this for a moment: how much time and energy has Microsoft (and other .NET authors) had to invest in making sure that every SDK and every article ever written has both C# and VB sample code? All because Microsoft refuses to simply draw a line in the sand and say, once and for all, "C# is the best statically-typed object-oriented language for the CLR on the planet, and Visual Basic is the best dynamically-typed object-oriented language for the CLR on the planet", and run with it. Then at least there would be solid technical reasons for using one or the other, and at least we could take this out of the realm of aesthetics.

Or, contrarily, do the logical thing and create one language with two parsers, switching between them based on the file extension. That guarantees that the two evolve in parallel, and releases resources from the languages team to work on other things.

Next, he shows some simple spin-off-a-thread code, with the Thread constructor taking a parameter to a function name, traditional delegate kinds of stuff, then notes the disjoint nature of referencing a method defined elsewhere in the class but only to be used once. Yes, he's setting up for the punchline: VB gets anonymous methods, and "VB's support for lambda (expressions) reaches parity with C#'s" in this next release. I don't know if this was a feature that VB really needed to get, since I don't know that the target audience for VB is really one that cares about such things (and, before the VB community tries to lynch me, let me be honest and say that I'm not sure the target audience for C# does, either), but at least it's nice that such a powerful feature is now present in the VB language. Subject to the concerns of last paragraph, of course.

Look, at the end of the day, I want C# and VB to be full-featured languages each with their own raison d'etre, as the French say, their own "reason to be". Having these two "evolve in parallel" or "evolve in concert" with one another is only bound to keep the C#-vs-VB language wars going for far too long.

Along the way, he's showing off some IDE features, which presumably will be in place for both C# and VB (since the teams are now unified under a single banner), what he's calling "highlights": they'll do the moral equivalent of brace matching/highlighting, for both method names (usage as well as declaration/definition) and blocks of code. There's also "pretty listing", where the IDE will format code appropriately, particularly for the anonymous methods syntax. Nice, but not something I'm personally going to get incredibly excited about--to me, IDE features like this aren't as important as language features, but I realize I'm in something of the minority there, and that's OK.

He demonstrates VB calling PLINQ (Parallel LINQ), pointing out some of the inherent benefits (and drawbacks) to parallelism. This isn't really a VB "feature" per se. <<MORE>>

Now he gets into some more interesting stuff: he begins by saying, "Now let's talk about the Dynamic Language Runtime (DLR)." He shows some VB code hosting the IronPython runtime, simple boilerplate to get the IronPython bits up and running inside this CLR process. (See the DLR Hosting Spec for details, it's pretty straightforward stuff: call IronPython.Hosting.Python.CreateRuntime, then call GetEngine("python") and SetSearchPaths() to tell IPy where to find the Python libs and code.) Where he's going with this is to demonstrate using VB's late-binding capabilities to get hold of a Python file ("random.py", using the DLR UseFile() call), and he dynamically calls the "shuffle" function from that Python file against the array of Ints he set up earlier.

(We get into a discussion as to why the IDE can't give Intellisense on the methods he's calling in the Python code. I won't go into the details, but essentially, no, VS isn't going to be able to do that, at least not for this scenario, any time soon. Maybe if the Python code was used directly from within VS, but not in this hosted sense--that would be a bit much for the IDE to analyze and understand.)

Next he points out some of the "ceremony" remaining in Visual Basic, essentially showing how VB's type inferencing is getting better, such as with array literals, including a background compilation warning where the VB compiler finds that it can't find a common type in the array literal declaration and assumes it to be an array of Object (which is a nice "catch" when the wrong type shows up in the array by accident or typo). He shows off multidimensional array literal and jagged array literal syntax (which requires the internal array literals in the jagged array to be wrapped up in parentheses, a la "{({1,2,3}), ({1, 2, 3, 4, 5})}", which I find a touch awkward and counterintuitive, quite frankly), while he's at it.

(We get into a discussion of finer-granularity color syntax highlighting options, such as colorizing different keywords differently, as well as colorizing different identifiers based on their type. Now that's an interesting idea.)

By the way, one thing that I've always found interesting about VB is its "With" keyword, a la "New Student With {.Id=101, .Name="bart", .Score=53, .Gender="male"}".

He then shows how VB 10 has auto-implemented properties: "Property Gender As String" does exactly what .NET programmers have had to do by hand for so long: create a field, generate simple Get and Set blocks and so on. Another nice feature of this: the autogenerated properties can have defaults, as in, "Public Property Age As Integer = 1". That's kinda nice, and something that VB should have had years ago.

And wahoo! THE UNDERSCORE IS (almost) HISTORY! "Implicit line completion" is a feature of VB 10. This has always plagued me like... well... the plague... when writing VB code. It's not gone completely, there's a few cases where ambiguity would reign without it, but it appears to be gone for 95% of the cases. Because this is such a radical change, they've even gone out and created a website to help the underscores that no longer find themselves necessary: www.unemployedunderscores.com .

He goes into a bit about co- and contravariance in generic types, which VB now supports more readily. (His example is about trying to pass a List(Of Student) into a method taking a List(Of Person), which neither he nor I can remember if it's co- or contra-. Sorry.) The solution is to change the method to take an IEnumerable(Of Person), instead. Not a great solution, but not a bad one, either.

... Corey Vidal, you have outdone every YouTube video I've ever seen, and I was a huge fan of "White and Nerdy".

John Williams, if you don't call this kid, you are missing out on some serious talent. To sing all four of those parts a capella and stitch them together like that, that's crazy.

Having freshly converted both the Visual Studio 2010 and Oslo SDK VPC images that we received at PDC 2008 last month to VMWare images, I figure it's time to dive into M.

At PDC, the Addison-Wesley folks were giving away copies of "The 'Oslo' Modeling Language" book, which is apparently official canon of the "M" language for Oslo, so I flip to page 1 and start reading:

The "Oslo" Modeling Language (M) is a modern, declarative language for working with data. M lets users write down how they want to structure and query their data using a convenient textual syntax that is convenient to both author and read.

M does not mandate how data is stored or accessed, nor does it mandate a specific implementation technology. Rather, M was designed to allow users to write down what they want from their data without having to specify how those desires are met against a given technology or platform. That stated, M in no way prohibits implementations from providing rich declarative or imperative support for controlling how M constructs are represented and executed in a given environment.

Hmm... I have to admit, all kinds of warning bells and alarm flags are going off in my head, and we're just two sentences into this thing. This sounds like something we've all done before; in fact, though I've not tried it, I have a feeling that if we were to go back through those two paragraphs and replace every instance of "M" with "SQL", we'd find a paragraph that could easily slip into the opening chapter of any introductory SQL or RDBMS book.

The goals of "separation of declaration from intent" have been around for that long, probably longer, and even the fiercest and staunchest defenders of SQL find themselves sometimes wandering through SQL declarations and code that clearly violate Chris Date's politely-worded commands around normal form and separation of declaration from intent and implementation.

I keep reading, though, and a few paragraphs later, find something intriguing.

Another important aspect of data management that M does not address is that of update. M is a functional language that does not have constructs for changing the contents of an extent. (Author's note: an "extent", defined a few paragraphs earlier, is that "an extent provides dynamic storage for values.") How data changes is outside the scope of the language. That said, M anticipates that the contents of an extent can change via external (to M) stimuli. Subsequent versions of M are expected to provide declarative constructs for updating data.

Wow. So the first question becomes, when are those "subsequent versions" expected? Is this simply a state of the PDC Preview bits, or something that's not in scope for v1 of the Oslo SDK?

I flip through the rest of the first chapter, which seems like a decent overview, and what I see there is an interesting type-declaration language; in many ways, it's highly reminiscent of XML Schema Descriptions (XSD) more than SQL declarations, but I suppose that's to be expected, at least for now. I'm sure they're going to cherry-pick a lot of the best data-declarative constructs from XSD, SQL, and any other metadata-based formats/languages, and that the semantics will change as they explore what works well and what doesn't. For now, though, "M" exists essentially as a data-descriptor language, and this is reinforced when I start playing with "m.exe", the "M compiler" (?).

First thing, I simply fire up "m.exe" to see what the options are. And... nothing. Huh? I wait for a bit, then Ctrl-C it, and start hunting through the documentation to see if I'm missing something here. I try a few different tests, like "m /?" or "m -help", and each time, the compiler just seems to wander off into the weeds, requiring a Ctrl-C to kill it.

What the heck? I know that these are PDC pre-alpha CTP "nothing is guaranteed to work" bits, but this seems a bit on the excessive side--I have every faith that Microsoft wouldn't hand these out if you can't even run the compiler! So acting on a hunch, I fire up "m /?" again, and tab away to look at something else. Sure enough, my hunch is rewarded--after a long pause, eventually the help screen comes up. So, apparently, the m.exe tool just takes fricken forever to run, is all.

Currently, the only targets M can compile to is their internal Repository for storing types, and a generic "T-SQL" target for any T-SQL-compliant database (which I presume for now means only SQL Server of various versions, but theoretically, I suppose, Sybase could work too, given those two systems' shared ancestry. And, given a pretty simple sample to work with, m.exe produces a pretty-easily-anticipated result; this:

module Ted
{
  type Person
  {
    Id : Integer32 = AutoNumber();
    Name : Text;
  } where identity Id;
  People : Person*;    
}

turns into this:

set xact_abort on;
go

begin transaction;
go

set ansi_nulls on;
go

create schema [Ted];
go

create table [Ted].[People]
(
  [Id] int not null identity,
  [Name] nvarchar(max) not null,
  constraint [PK_People] primary key clustered ([Id])
);
go

commit transaction;
go

... which, when you look at it, is pretty much what you'd want.

Interestingly enough, there's no reason why people in the Java or Ruby space couldn't use "M" just as easily, so long as the database targeted is one that M understands. (It also wouldn't be a terribly difficult exercise to build an M compiler in Java or Ruby, for that matter. Might be a fun off-time project, in fact.)

One thing that's also pretty clear is that M is very collection-centric, as the first chapter spends probably 50% of its time describing all the various ways that collections in M (written as "{a, b, c}") interact with one another (they can be compared for equality directly, for example, and have some neat projection/filter capabilities that were clearly drawn from the relational algebra and LINQ syntax). Having said that, though, one thing that is obviously missing is the traditional object "reference"-style connection, where A OWNS-A B.

What this seems to imply, then, is that the object/relational-mapping horrors of the past two decades aren't yet over. What's not clear is how M will make it easier (or if it will at all) to access those extents from the languages we traditionally use in the .NET space (C#, VB, C++/CLI, etc), specifically, what the mechanism for conducting a query will be like, and what it's return types will be when it cross the boundary back into C#.

If you're not sure what I mean by that, consider it this way: ADO.NET has a simple mechanism for taking the query--a raw string as a parameter--and executing it, and when it returns, it's handed back to your C# code as a DataSet, or else as an IDataReader for row-based/column-based firehose-style consumption. Much of the criticism of ADO.NET stems around two parts: the untyped nature of the query string, leading to potential typos and errors, and the relative awkwardness for extracting the data from the results, either the DataSet or the IDataReader, at least when compared to languages that have built-in set/tuple constructs.

The one sample that does show any sort of C# -> M kinds of interaction is in the MParserDemo sample, and here, when it queries the database, it does so using traditional ADO.NET API calls, so I'm not sure it's to be taken as a good indicator of the plans around M yet.

If all there was to Oslo was "M", I'd say it was an interesting little side-note at PDC, something that maybe a few folks might find interesting and otherwise not worth studying, but this is not the sum total of the Oslo bits; there is also Mg, the MGrammar language, a language specifically for building DSLs, and that's where my attention (and next blog post) is going next.

Roy Fielding has weighed in on the recent "buzzwordiness" (hey, if Colbert can make up "truthiness", then I can make up "buzzwordiness") of calling everything a "REST API", a tactic that has become more en vogue of late as vendors discover that the general programming population is finding the WSDL-based XML services stack too complex to navigate successfully for all but the simplest of projects. Contrary to what many RESTafarians may be hoping, Roy doesn't gather all these wayward children to his breast and praise their anti-vendor/anti-corporate/anti-proprietary efforts, but instead, blasts them pretty seriously for mangling his term:

I am getting frustrated by the number of people calling any HTTP-based interface a REST API. Today’s example is the SocialSite REST API. That is RPC. It screams RPC. There is so much coupling on display that it should be given an X rating.

Ouch. "So much coupling on display that it should be given an X rating." I have to remember that phrase--that's a keeper. And I'm shocked that Roy even knows what an X rating is; he's such a mellow guy with such an innocent-looking face, I would've bet money he'd never run into one before. (Yes, people, that's a joke.)

What needs to be done to make the REST architectural style clear on the notion that hypertext is a constraint? In other words, if the engine of application state (and hence the API) is not being driven by hypertext, then it cannot be RESTful and cannot be a REST API. Period. Is there some broken manual somewhere that needs to be fixed?

Go Roy!

For those of you who've not read Roy's thesis, and are thinking that this is some kind of betrayal or trick, let's first of all point out that at no point is Roy saying that your nifty HTTP-based API is not useful or simple. He's simply saying that it isn't RESTful. That's a key differentiation. REST has a specific set of goals and constraints it was trying to meet, and as such prescribes a particular kind of architectural style to fit within those constraints. (Yes, REST is essentially an architectural pattern: a solution to a problem within a certain context that yields certain consequences.)

Assuming you haven't tuned me out completely already, allow me to elucidate. In Chapter 5 of Roy's thesis, Roy begins to build up the style that will ultimately be considered REST. I'm not going to quote each and every step here--that's what the hyperlink above is for--but simply call out certain parts. For example, in section 5.1.3, "Stateless", he suggests that this architectural style should be stateless in nature, and explains why; the emphasis/italics are mine:

We next add a constraint to the client-server interaction: communication must be stateless in nature, as in the client-stateless-server (CSS) style of Section 3.4.3 (Figure 5-3), such that each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client.

This constraint induces the properties of visibility, reliability, and scalability. Visibility is improved because a monitoring system does not have to look beyond a single request datum in order to determine the full nature of the request. Reliability is improved because it eases the task of recovering from partial failures [133]. Scalability is improved because not having to store state between requests allows the server component to quickly free resources, and further simplifies implementation because the server doesn't have to manage resource usage across requests.

Like most architectural choices, the stateless constraint reflects a design trade-off. The disadvantage is that it may decrease network performance by increasing the repetitive data (per-interaction overhead) sent in a series of requests, since that data cannot be left on the server in a shared context. In addition, placing the application state on the client-side reduces the server's control over consistent application behavior, since the application becomes dependent on the correct implementation of semantics across multiple client versions.

In the HTTP case, the state is contained entirely in the document itself, the hypertext. This has a couple of implications for those of us building "distributed applications", such as the very real consideration that there's a lot of state we don't necessarily want to be sending back to the client, such as voluminous information (the user's e-commerce shopping cart contents) or sensitive information (the user's credentials or single-signon authentication/authorization token). This is a bitter pill to swallow for the application development world, because much of the applications we develop have some pretty hefty notions of server-based state management that we want or need to preserve, either for legacy support reasons, for legitimate concerns (network bandwidth or security), or just for ease-of-understanding. Fielding isn't apologetic about it, though--look at the third paragraph above. "[T]he stateless constraint reflects a design trade-off."

In other words, if you don't like it, fine, don't follow it, but understand that if you're not leaving all the application state on the client, you're not doing REST.

By the way, note that technically, HTTP is not tied to HTML, since the document sent back and forth could easily be a PDF document, too, particularly since PDF supports hyperlinks to other PDF documents. Nowhere in the thesis do we see the idea that it has to be HTML flying back and forth.

Roy's thesis continues on in the same vein; in section 5.1.4 he describes how "client-cache-stateless-server" provides some additional reliability and performance, but only if the data in the cache is consistent and not stale, which was fine for static documents, but not for dynamic content such as image maps. Extensions were necessary in order to accomodate the new ideas.

In section 5.1.5 ("Uniform Interface") we get to another stinging rebuke of REST as a generalized distributed application scheme; again, the emphasis is mine:

The central feature that distinguishes the REST architectural style from other network-based styles is its emphasis on a uniform interface between components (Figure 5-6). By applying the software engineering principle of generality to the component interface, the overall system architecture is simplified and the visibility of interactions is improved. Implementations are decoupled from the services they provide, which encourages independent evolvability. The trade-off, though, is that a uniform interface degrades efficiency, since information is transferred in a standardized form rather than one which is specific to an application's needs. The REST interface is designed to be efficient for large-grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction.

In order to obtain a uniform interface, multiple architectural constraints are needed to guide the behavior of components. REST is defined by four interface constraints: identification of resources; manipulation of resources through representations; self-descriptive messages; and, hypermedia as the engine of application state. These constraints will be discussed in Section 5.2.

In other words, in order to be doing something that Fielding considers RESTful, you have to be using hypermedia (that is to say, hypertext documents of some form) as the core of your application state. It might seem like this implies that you have to be building a Web application in order to be considered building something RESTful, so therefore all Web apps are RESTful by nature, but pay close attention to the wording: hypermedia must be the core of your application state. The way most Web apps are built today, HTML is clearly not the core of the state, but merely a way to render it. This is the accidental consequence of treating Web applications and desktop client applications as just pale reflections of one another.

The next section, 5.1.6 ("Layered System") again builds on the notion of stateless-server architecture to provide additional flexibility and power:

In order to further improve behavior for Internet-scale requirements, we add layered system constraints (Figure 5-7). As described in Section 3.4.2, the layered system style allows an architecture to be composed of hierarchical layers by constraining component behavior such that each component cannot "see" beyond the immediate layer with which they are interacting. By restricting knowledge of the system to a single layer, we place a bound on the overall system complexity and promote substrate independence. Layers can be used to encapsulate legacy services and to protect new services from legacy clients, simplifying components by moving infrequently used functionality to a shared intermediary. Intermediaries can also be used to improve system scalability by enabling load balancing of services across multiple networks and processors.

The primary disadvantage of layered systems is that they add overhead and latency to the processing of data, reducing user-perceived performance [32]. For a network-based system that supports cache constraints, this can be offset by the benefits of shared caching at intermediaries. Placing shared caches at the boundaries of an organizational domain can result in significant performance benefits [136]. Such layers also allow security policies to be enforced on data crossing the organizational boundary, as is required by firewalls [79].

The combination of layered system and uniform interface constraints induces architectural properties similar to those of the uniform pipe-and-filter style (Section 3.2.2). Although REST interaction is two-way, the large-grain data flows of hypermedia interaction can each be processed like a data-flow network, with filter components selectively applied to the data stream in order to transform the content as it passes [26]. Within REST, intermediary components can actively transform the content of messages because the messages are self-descriptive and their semantics are visible to intermediaries.

The potential of layered systems (itself not something that people building RESTful approaches seem to think much about) is only realized if the entirety of the state being transferred is self-descriptive and visible to the intermediaries--in other words, intermediaries can only be helpful and/or non-performance-inhibitive if they have free reign to make decisions based on the state they see being transferred. If something isn't present in the state being transferred, usually because there is server-side state being maintained, then they have to be concerned about silently changing the semantics of what is happening in the interaction, and intermediaries--and layers as a whole--become a liability. (Which is probably why so few systems seem to do it.)

And if the notion of visible, transported state is not yet made clear in his dissertation, Fielding dissects the discussion even further in section 5.2.1, "Data Elements". It's too long to reprint here in its entirety, and frankly, reading the whole thing is necessary to see the point of hypermedia and its place in the whole system. (The same could be said of the entire chapter, in fact.) But it's pretty clear, once you read the dissertation, that hypermedia/hypertext is a core, critical piece to the whole REST construction. Clients are expected, in a RESTful system, to have no preconceived notions of structure or relationship between resources, and discover all of that through the state of the hypertext documents that are sent back to them. In the HTML case, that discovery occurs inside the human brain; in the SOA/services case, that discovery is much harder to define and describe. RDF and Semantic Web ideas may be of some help here, but JSON can't, and simple XML can't, unless the client has some preconceived notion of what the XML structure looks like, which violates Fielding's rules:

A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API). From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations. The transitions may be determined (or limited by) the client’s knowledge of media types and resource communication mechanisms, both of which may be improved on-the-fly (e.g., code-on-demand). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]

An interesting "fuzzy gray area" here is whether or not the client's knowledge of a variant or schematic structure of XML could be considered to be a "standardized media type", but I'm willing to bet that Fielding will argue against it on the grounds that your application's XML schema is not "standardized" (unless, of course, it is, through a national/international/industry standardization effort).

But in case you'd missed it, let me summarize the past twenty or so paragraphs: hypermedia is a core requirement to being RESTful. If you ain't slinging all of your application state back and forth in hypertext, you ain't REST. Period. Fielding said it, he defined it, and that settles it.

Before the hate mail comes a-flyin', let me reiterate one vitally important point: if you're not doing REST, it doesn't mean that your API sucks. Fielding may have his definition of what REST is, and the idealist in me wants to remain true to his definitions of it (after all, if we can't agree on a common set of definitions, a common lexicon, then we can't really make much progress as an industry), but...

... the pragmatist in me keeps saying, "so what"?

Look, at the end of the day, if your system wants to misuse HTTP, abuse HTML, and carnally violate the principles of loose coupling and resource representation that underlie REST, who cares? Do you get special bonus points from the Apache Foundation if you use HTTP in the way Fielding intended? Will Microsoft and Oracle and Sun and IBM offer you discounts on your next software purchases if you create a REST-faithful system? Will the partisan politics in Washington, or the tribal conflicts in the Middle East, or even the widely-misnamed "REST-vs-SOAP" debates come to an end if you only figure out a way to make hypermedia the core engine of your application state?

Yeah, I didn't think so, either.

Point is, REST is just an architectural style. It is nothing more than another entry alongside such things as client-server, n-tier, distributed objects, service-oriented, and embedded systems. REST is just a tool for thinking about how to build an application, and it's high time we kick it off the pedastal on which we've placed it and let it come back down to earth with the rest of us mortals. HTTP is useful, but not sufficient, so solve our problems. REST is as well.

And at the end of the day, when we put one tool from our tool belt "above all others", we end up building some truly horrendous crap.

Recently, a blog reader asked me if I wasn't doing any speaking any more since I'd joined ThoughtWorks, and that's when I realized I'd been bad about updating my speaking calendar on the website. Sorry, all; no, ThoughtWorks didn't pull my conference visa or anything, I've just been bad about keeping it up to date. I'll fix that ASAP, but in the meantime, three events that I'll be at in the coming wintry months include:

Øredev 2008: 19 - 21 November, Malmoe, Sweden

Øredev will be a first for me, and I've ben invited to give a keynote there, along with a few technical sessions. I'm also told that .NET Rocks! will be on hand, and that they want to record a session, on whichever topic happens to cross the curious, crafty and cunning Carl, or the uh... the uh... sorry, Richard, there's just no good "R" adjectives I can use here. I mean, "rough" and "ready" don't exactly sound flattering in this context, right? Sorry, man.

In any event, I'm looking forward to this event, because it's a curious mix of technologies and ideas (agile, ALT.NET, Java, core .NET, languages, and so on), and because I've never been to Sweden before. One more European country, off my bucket list!

(Yes, I had to cut-and-paste the Ø wherever I needed it. *grin*)

DevTeach 2008: 1 - 5 December, Montreal, Quebec (Canada)

This has been one of my favorite shows since it began, way back in 2003, and a large part of that love has to do with the cast and crew of characters that I see there every year: Julie Lerman, Peter DeBetta, Carl and Richard (again!), Beth Massi, "Yag" Griver, Mario Cardinal and the rest of the Quebecois posse, Ayende, plus some new faces and friends, like Jessica Moss and James Kovacs. (Oh, and for the record, folks, for those of you who are still talking about it, the O/R-M smackdown of a year ago was staged. It was all fake. Ayende and I are really actually friends, we were paid a great deal of money by Carl and Richard to make it sound good, and in fact, we both agree that the only place anybody should really ever store their data is in an XML database.)

If you're near Montreal, and you're a .NET dev, you really owe it to yourself to check this show out.

Update: I just got this email from Jean-Rene, the guy who runs DevTeach:

Every attendees will get Visual Studio 2008 Pro, Expression Web 2 and Tech-Ed DEV set in their bag!

DevTeach believe that all developers need the right tool to be productive. This is what we will give you, free software, when you register to DevTeach or SQLTeach. Yes that right! We’re pleased to announce that we’re giving over a 1000$ of software when you register to DevTeach. You will find in your conference bag a version of Visual Studio 2008 Professional, ExpressionTM Web 2 and the Tech-Ed Conference DVD Set. Is this a good deal or what? DevTeach and SQLTeach are really the training you can’t get any other way.

Not bad. Not bad at all.

DeVoxx 2008: 8 - 12 December, Antwerp, Belgium

DeVoxx, the recently-renamed-formerly-named-JavaPolis conference, has brought me back to team up with Bill Venners to do a University session on Scala, and to record a few more of those Parlays videos that people can't seem to get enough of. Given that this show always seems to draw some of the Java world's best and brightest, I'm definitely looking forward to the chance to point the mike at somebody's grill and give 'em hell! Plus, I love Belgium, and I'm looking forward to getting back there. The fact that it's going to be the middle of winter is only a bonus, as... wait... Belgium, in the middle of winter? Whose bright idea was that?

(And finally, a show that Carl and Richard won't be at!)

Meanwhile, I promise to keep the "Upcoming Events" up to date for 2009. Seriously. I mean it.

Having created a Window7 VMWare image (which I then later cloned and installed the Windows7 SDK into, successfully, wahoo!), I turned to the Visual Studio 2010 bits they provided on the hard drive. Not surprisingly, though a bit frustratingly, they didn't give us an install image that I could put into a VMWare image of my own creation, but instead gave us a VPC with everything pre-installed in it.

I know that Microsoft prefers to promote its own products, and that it's probably a bit much to ask them to provide both a VMWare image and a VirtualPC image for these kind of pre-alpha things, but it's a bit of a pain considering that Virtual PC doesn't run anymore on the Mac, that I'm aware of. Please, Microsoft, a lot of .NET devs are carrying around MacBookPro machines these days, and if you're really focused on trying to get bits in the hands of developers, it would be quite the bold move to provide a VMWare image right next to the VPC image. Particularly since over half the drive was unused.

So... I don't want to have to carry around a PC (though I do at the moment) just to run VirtualPC just to be able to explore VS 2010, but fortunately VMWare provides a Converter application that can take a VPC image and flip it over to a VMWare image. Sounds like a plan. I fire up the Converter, point it at the VPC, and after the world's... slowest... wizard... takes... my... settings... and... begins... I discover that it will take upwards of 3 hours to convert. Dear God.

I decided to go to bed at that point.

When I woke up, the image had been converted successfully, but I wasn't quite finished yet. First of all, fire it up to make sure it runs, which it does without a problem, but at 640x480 in black-and-white mode (no, seriously, it's not much more than that). Install the VMWare Tools, reboot, and...

... the mouse cursor disappears. WTF?!?

Turns out this has been a nagging problem with several versions of VMWare over the years, and I vaguely remember running into the problem the last time I tried to create a Windows Server 2003/2008 image, too. Ugh. Hunting around the Web doesn't reveal an easy solution, but a couple of things do show up a few times: disconnect the CD-ROM, change the mouse pointer acceleration, delete the VMWare Mouse driver and let Windows rediscover the standard PS/2 mouse driver, or change the display hardware acceleration.

Not being really interested in debugging the problem (I know, my chance at making everybody's life better is now forever lost), I decided to take a bit of a shotgun approach to the problem. I explicitly deleted the VMWare Mouse driver, fiddled with the display settings (including resizing it to a more respectable 1400x1050), turned display hardware acceleration down, couldn't find mouse hardware acceleration settings, allowed it to reboot, and...

... yay. I have a mouse pointer again.

Now I have a VS2010 image on my Drive-o'-Virtual-Machines, and with it I plan on exploring the VS2010/C# 4.0/C++ 10/VB 10 bits some more. I fire up Visual Studio 2010, intending to poke around C# 4.0's new "dynamic" keyword and see if and how it builds on top of the DLR (as a few people have suggested in comments in prior posts). VS comes up pretty quickly (not bad for a pre-alpha), the new interface seems snappy, and I create the ubiquitous "ConsoleApplicationX" C# app.

Wait a minute...

Something niggled at the back of my head, and I went back to File | New Project, and ... something's missing.

There's no "Visual F#" tab. There's an item in the "Project types:" box on the left for Visual Basic, Visual C#, Visual C++, WiX, Modeling Projects, Database Projects, Other Project Types, and Test Projects, but no Visual F#. (And no, it doesn't show up under "Other Project Types" either, I checked.) Considering that my understanding was that F# was going to ship with VS 2010, I'm a little puzzled as to its absence. Hopefully this is just a temporary oversight.

In the meantime, I'm off to play with "dynamic" a bit more and see what comes out of it. But guys, please, let's see some F# love out of the box? Surely, if you can ship WiX with it, shipping F# can't be hard?

The organizers of TSSJS 2009 have asked me to serve as the Languages track chair, and as long-time readers of this blog will already have guessed, I've accepted quite happily. This means that if you're interested in presenting at TSSJS on a language-on-the-JVM, you now know where to send the bottle of Macallan 18.

Having said that (in jest, of course--bribes have to be at least a Macallan 25 or Macallan Cask Strength to have any real effect), I'm curious to get a sense of what languages--and what depth in each--people are interested in seeing presented there. Groovy, JRuby and Scala are obvious suggestions, but how deep would people be interested in seeing these? Would you prefer to see more languages at a shallower depth, or going really deep on a few?

(Disclaimer: emails sent to me directly or comments on this blog will weigh in on my decision-making process, but don't necessarily count as submitted abstracts; make sure you send them via the "official" channels to ensure they get considered, particularly since some proposals will be "borderline" on several different tracks, and thus could conceivably make it in via a different track than mine.)

Y'all know how to reach me....

Update: The deadline for abstracts is November 19th, so make sure to check out the website when it goes live (Nov 11th), and if you can't figure out how to submit an abstract, send it to me directly....

I realized, I'm sitting here in Canyon's (in Redmond), with two laptops plugged into the wall and the WiFi, playing with PDC bits.

It's a Saturday night, for cryin' out loud.

Please, any Redmondites, Kirklannish, or Bellvuevians, rescue me. Where do the cool people hang out in Eastside?

So the first thing I do when I get back from PDC? After taking my youngest trick-or-treating at the Redmond Town Center, and settling down into the weekend, I pull out the PDC hard drive and have a look around.

Obviously, I'm going to eventually spend a lot of time in the "Developer" subdirectory--lots of yummy PDC goodness in there, like the "Oslo_Dublin_WF_WCF_4" subdirectory in which we'll find a Virtual PC image of the latest CSD bits pre-installed, or the Visual_Studio_2010 subdirectory (another VirtualPC image), but before I start trying to covert those over to VMWare images (so I can run them on my Mac), I figured I'd take a wild shot at playing with Windows 7.

That, of course, means installing it into a VMWare image. So here goes.

First step, create the VMWare virtual machine. Because this is clearly not going to be a stock install, I choose the custom option, and set the operating system to be "Windows Server 2008 (experimental)". Not because I think there's anything really different about that option (except the default options that follow), but because it feels like the right homage to the pre-alpha nature of Windows 7. I set RAM to 512MB, chose to give it a 24GB IDE disk (not SCSI, as the default suggested--Windows sometimes has a tentative relationship with SCSI drives, and this way it's just one less thing to worry about), chose a single network adapter set to NAT, pointed the CD to the smaller of the two ISO images on the drive (which I believe to be the non-checked build version), and fired 'er up, not expecting much.

Kudos to the Windows 7 team.

The CD ISO boots, and I get the install screen, and bloody damn fast, at that. I choose the usual options, choose to do a Custom install (since I'm not really doing an Upgrade), and off it starts to churn. As I write this, it's 74% through the "Expanding files" step of the install, but for the record, Vista never got this far installing into VMWare with its first build. As a matter of fact, if I remember correctly, Vista (then Longhorn) didn't even boot to the first installation screen, and then when it finally did it took about a half-hour or so.

I'll post this now, and update it as I find more information as I go, but if you were curious about installing Windows 7 into VMWare, so far the prognosis looks good. Assuming this all goes well, the next step will be to install the Windows 7 SDK and see what I can build with it. After that, probably either VS 2008 or VS 2010, depending on what ISOs they've given me. (I think VS 2010 is just a VHD, so it'll probably have to be 2008.) But before I do any of that, I'll make a backup, just so that I can avoid having to start over from scratch in the event that there's some kind dependency between the two that I haven't discovered so far.

Update: Well, it got through "Expanding files", and going into "Starting Windows...", and now "Setup is starting services".... So far this really looks good.

Update: Uh, oh, possible snag: "Setup is checking video performance".... Nope! Apparently it's OK with whatever crappy video perf numbers VMWare is going to put up. (No, I didn't enable the experimental DirectX support for VMWare--I've had zero luck with that so far, in any VMWare image.)

Update: Woo-hoo! I'm sitting at the "Windows 7 Ultimate" screen, choosing a username and computername for the VM. This was so frickin flawless, I'm waiting for the shoe to drop. Choosing password, time zone, networking setting (Public), and now we're at the final lap....

Update: Un-FRICKIN-believable. Flawless. Absolutely flawless. I'm in the "System and Security" Control Panel applet, and of course the first thing I select is "User Account Control settings", because I want to see what they did here, and it's brilliant--they set up a 4-point slider to control how much you want UAC to bug you when you or another program changes Windows settings. I select the level that says, "Only notify me when programs try to make changes to my computer", which has as a note to it, "Don't notify me when I make changes to Windows settings. Note: You will still be notified if a program tries to make changes to your computer, including Windows settings", which seems like the right level to work from.

But that's beyond the point right now--the point is, folks, Windows 7 installs into a VMWare image flawlessly, which means it's trivial to start playing with this now. Granted, it still kinda looks like Vista at the moment, which may turn some folks off who didn't like its look and feel, but remember that Longhorn went through a few iterations at the UI level before it shipped as Vista, too, and that this is a pre-alpha release of Win7, so....

I tip my hat to the Windows 7 team, at least so far. This is a great start.

Update: Even better--VMWare Tools (the additions to the guest OS that enable better video, sound, etc) installs and works flawlessly, too. I am impressed. Really, really impressed.

PDC 2008 in LA is over now, and like most PDCs, it definitely didn't disappoint on the technical front--Microsoft tossed out a whole slew of new technologies, ideas, releases, and prototypes, all with the eye towards getting bits (in this case, a Western Digital 160 GB USB hard drive) out to the developer community and getting back feedback, either through the usual channels or, more recently, the blogosphere.

These are the things I think I think about this past PDC:

• Windows 7 will be an interesting thing to watch--they handed out DVDs in both 32- and 64-bit versions, and it's somewhat reminiscent of the Longhorn DVDs of the last PDC. If you recall, Longhorn (what eventually became known as Vista) looked surprisingly good--if a bit unstable, something common to any release this early--for a while, then Vista itself pretty much fell flat. I think it will be interesting, as a social experiment, to look at what people say about Windows 7 now, compare it to what was said about Vista back in 2004 (which is I think when the last PDC was), and then compare what people say 1, 2 and 3 years after the PDC release.
• Azure dominated a lot of the focus, commensurate with the growing interest/hype around "the cloud". All of this sounds suspiciously familiar to me, thinking back to the early days of SOAP/WSDL, and the intense pressure for Web services to revolutionize IT as we know it. This didn't happen, largely for technical reasons at first (incompatibilities between toolkits most of all), then because people treated it as CORBA++ or DCOM-with-angle-brackets. Azure and "cloud computing" have a different problem: clear definition of purpose. I think too many people have no idea what "the cloud" really is for this to be something to pay much attention to just yet.
• Conference get-togethers and parties are becoming more and more lavish each year, as the various product teams challenge one another for the coveted title of The "Dude, were you there last night? It was amazing!" Party of PDC. For my money, that party was the party at the J Lounge on Wednesday night, complete with three floors of fun, including a wall-projected image of Rock Band, but--here's the rub--I couldn't tell you which team actually hosted the party. There was a Microsoft Dynamics CRM poster up in the middle of the gaming floor (bunch of XBox 360s, though not networked together, which I found disappointing), so I'm assuming it had something to do with them, but.... I think Microsoft product teams may want to consider saving some budget and instead of hiring six LA Lakers Cheerleaders to sit on a couch and allow drooling geeks to take pictures with them (no touching!), use the money to make the party--and the hosts--stick in my mind more effectively, or at least use it to hand out technical data on whatever it is they're building.
• The vendor floor competition for attention is getting a little cutthroat. DevExpress stole the show this year, importing--no joke--an actor, "Mini-Me", Vern, to essentially echo (badly) anything Mark Miller (dressed, of course, as Austin Powers' arch-nemesis Dr. Evil) tried to say about the most recent version of CodeRush. Granted, Mark's new "do" (and the absurdly large head that was hiding underneath) makes it easy for him to do a good Dr. Evil impression, but other than that, there was really nothing parallel in the situation--despite Mark's insistence on writing code with evil Flying Spaghetti Monsters or what not in it. I think if you're a vendor and you want to make a splash at PDC, you think long and hard about an effective tie-in, like Infragistics' clever "I flew 1500 miles for this T-shirt" they were giving away.
• The language world was a bit abuzz at the barely-concealed C# 4.0 features, mostly centering around the new "dynamic" keyword and the C# REPL loop capabilities, but noticeably absent was any similar kind of talk or buzz around VB 10. Even C++ got more attention than VB did, with a presentation clearly intending to call out a direct reference to Visual C++'s heyday, "Visual C++: Why 10 is the new 6". Conversations I had with a few Microsofties make it pretty clear that VB is now the red-headed stepchild of the .NET language family, and that fact is going to start making itself widely felt through the rest of the ecosystem before long, particularly now that rumors are beginning to circulate that pretty much all the "gifted kids" that were on the VB team have gone to find other places to exercise their intellect and innovation, such as the Oslo team. I think Microsoft is going to find itself in an uncomfortable position soon, of trying to kill VB off without appearing like they are trying to kill VB off, lest they create another "VB revolution" like the one in 2001 when unmanaged VB'ers ("Classic VBers"?) looked at VB.NET and collectively puked.
• Speaking of collective revolution, anybody remember Visual FoxPro? Those guys are still kicking, and they were always a small fraction of the developer community, comparatively against VB, at least. I think Microsoft is in trouble here, of their own making, for not defining distinct and clearly differentiated roles for Visual Basic and C#.
• The DLR is quickly moving into a position of high importance in my mind, and the fact that it now builds on top of expression trees (from C# 3.0/LINQ) and builds its trees in such a way that they look almost identical to what a corresponding C# or VB tree would look like means that the DLR is about a half-step away from becoming the most critical part of the .NET ecosystem, second only to the CLR itself. I think that while certain Microsoft releases, like Oslo, PowerShell, C# or VB, won't adopt the DLR as a core component to their implementation, developers looking to explore the DSL space will find the DLR a very happy place to be, particularly in combination with F# Parser Expression Grammars.
• Speaking of F#, it's pretty clear that it was the developer darling--if not the media darling--of the show. The F# Hands-on-Lab looked to be one of the more popular ones used there, and every time I or my co-author, Amanda Laucher, talked with somebody who didn't already know we were working on F# in a Nutshell, they were asking questions about it and trying to understand its role in the world. I think the "cool kids" of the development community are going to come to check out F#, find that it can do a lot of what the O-O minded C# and VB can do, discover that the functional approach works well in certain scenarios, and start looking to use that on their new projects.
• I think that if the Microsoft languages family were Weasley family from Harry Potter, C++ would be one of the two older brothers (probably Bill or Charlie, the cool older brothers who've gone on to make their name and don't need to impress anybody any more), Visual Basic would be Percy (desperate for validation and respect), C# would be Ron (cleary an up-and-comer in the world, even if he was a little awkward while growing up), and F# would be Ginny (the spunky one who clearly charts her own path despite her initial shyness, her accidental involvement in a Voldemortian scheme and her parents' and big brothers' interference in her life). Oslo, of course, is Professor Snape--we can't be sure if he's a good guy or a bad guy until the last book.
• Continuing that analogy, by the way, I think Java is clearly Hermione: wickedly book smart, but sometimes too clever by half.

Overall, PDC was an amazing show, and there's clearly a lot of stuff to track. I personally plan to take a deep dive into Oslo, and will probably blog about what I find, but in the meantime, remember that all of the PDC bits that we got on the hard drives are available through the various DevCenters (or so I've been told), so have a look. There's a lot more there than just what I mentioned above.

Update: Lisa Feigenbaum emailed me with a correction: there was a session on VB 10 at PDC, and I simply missed it in the schedule. In fact, she was very subtle about it, simply asking me, "Did you make it to the VB talk?" and posted this URL along with it. Lisa, I stand corrected. Having said that, though, I still stand by the other points of that piece: that the buzz I was hearing (which may very well have simply been the social circles I run in, I'll be the first to admit it, but I can only speak to my experience here and am very willing to be told I'm full of poopie on this one) was all C#, no VB, and that it bothers me that notable members of the VB team have departed for other parts of the company. Please, nothing would make me happier than to see VB stand as a full and equal partner in the .NET family of languages, but right now, it really still feels like the red-headed stepchild. Please, prove me wrong.

One of the more interesting logistical problems faced by the people who run the Microsoft Conference Center is that several events are often running in parallel, and each has their own catering provisions--one might get snacks, another may have lunch boxes, and others have full buffet, and so on. Of course, each group will want to make sure their food isn't swiped by people at other events with less-appealing food, so staff members at the Conference Center (literally) stand guard over the snack tables, looking for badges and directing them to the appropriate table as necessary.

This week is no different; during the VSX DevCon, other events have been running, including some internal Microsoft events. And, not surprisingly, the staff are following their directives, turning people away if they're not wearing the VSX DevCon badge.

Even if that guy is Steve Ballmer.

No joke: I watch as Steve Ballmer--meeting with Kevin Turner and other similarly-pedigreed Microsoft management--comes out of his meeting room and heads over to the VSX DevCon table to grab some cookies, only to be turned away by a MSCC staff member. "I'm sorry, sir, those cookies are not for you."

I wonder if George Bush ever gets pulled aside by the TSA?

The full list is here. It's a pretty prestigious group--and I'm totally floored that I'm there next to some pretty big names.

In homage to Ms. Sally Fields, of so many years ago... "You like me, you really like me". Having somebody come up to me at a conference and tell me how much they like my blog is second on my list of "fun things to happen to me at a conference", right behind having somebody come up to me at a conference and tell me how much they like my blog, except for that one entry, where I said something totally ridiculous (and here's why) ....

What I find most fascinating about the list was the means by which it was constructed--the various calculations behind page rank, technorati rating, and so on. Very cool stuff.

Perhaps it's trite to say it, but it's still true: readers are what make writing blogs worthwhile. Thanks to all of you.

As Joel points out, we've made a draft of the SSCLI 2.0 Internals book available for download (via his blog). Rather than tell you all about the book, which Joel summarizes quite well, instead I thought I'd tell you about the process by which the book came to be.

Editor's note: if you have no interest in the process by which a book can get done, skip the rest of this blog entry.

One thing that readers will note that's different about this version of "the Rotor book" is that it's not being done through one of the traditional publishers. This is deliberate. As Joel and I talk about on the .NET Rocks! show we did together, the first Rotor book was on the first version of Rotor, which shipped shortly after the .NET 1.1 bits shipped to customers. That was back in the summer of 2001. Dave, Geoff and I shipped the book, I did a few conference talks on Rotor for the relatively few people who had an interest in what was going on "under the hood" of the CLR, and then we all sort of parted ways. (Dave retired from Microsoft entirely shortly thereafter, in order "to focus on the two things that matter in life: making music and making wine", as he put it.) Mission accomplished, we moved on.

Meanwhile, as we all knew would happen, the world moved on--Whidbey (.NET 2.0) shipped, and with it came a whole slew of CLR enhancements, most notably generics. Unlike how generics happened in the JVM, CLR generics are carried through all the way to the type system, and as a result, a lot of what we said in the first Rotor book was instantly rendered obsolete. Granted, one could always grab the Gyro patch for Rotor and see what generics would have looked like, but even that was pretty much rendered obsolete by the emergence of the SSCLI 2.0 drop, bringing the Rotor code up to date with the Whidbey production CLR release.

Except the book was, to be blunt about it, left behind.

Speaking honestly, the book never broke any sales records. Sure, for a while there it was the #1 best-selling book (in Redmond, WA, to my total shock and surprise) on Amazon, but we never had the kind of best-seller success that that of, say, Programming Ruby or pick-your-favorite-ASP.NET book. In the book publishing world, this was kind of the moral equivalent to watching your neighbors' slide show of their vacation: boring for most people not in the pictures, unless you were really interested in either the place they were visiting or what they did there. Most of our audience were either people working on the CLR itself (hence all the copies sold in Redmond, get it?), people who were researching on the CLR (such as the various Rotor research projects that came over a few years after its release), or people who just had that itch to "get wonky with it" and learn how some of the structures worked. Granted, a lot of what those people in the last category learned turned out to be pretty helpful in the Real World, but it was a payoff that came with a pretty non-trivial learning curve.

Fast-forward a few years, to the end of calendar year 2005.

By this point, .NET 2.0 has been out in production form for a bit, and Mark Lewin, then of Microsoft University Relations (I think that was his job, but to be honest my recollection on that point is kinda fuzzy) approached me: Microsoft was interested in seeing a second edition of the book out, to keep the Rotor community up to date with what was going on in the state of the art in the CLR. Was I interested? Sure, but the rules surrounding a multi-author book and subsequent editions are pretty clear: everybody has to be given right of first refusal. Thus a two-fold task was under way: find a co-author (preferably somebody from the CLR team, since my skills had never really been in navigating the Rotor source code in the first place, and I hadn't really spent a significant amount of time in the code since 2001), and get Geoff and Dave to indicate--in a very proper legal fashion--that they were passing on the second edition.

Ugh. Lawyers. Contracts. Bleah.

John Osborn then broke the bad news: OReilly wasn't interested in doing a second edition. I couldn't really blame them, since the first hadn't broken any kind of sales record, but I was a bit bummed because I thought this was the end of the road.

Mark Lewin to the rescue. Apparently his part of Microsoft really wanted this book out, to the point where they were willing to fund the effort, if I and my co-author were still interested. Sure, that sounded like a workable idea. And once the book was done, maybe we could publish it through MSPress, if that sounded like a good idea to me. Sure, that sounded good. Then Mark dropped the suggestion that maybe I could talk to Joel Pobar, former CLR geek extraordinaire, to see if he was interested. Joel had impressed me back when we'd briefly touched bases during the first book-writing experience, so yeah, sure, that sounded like a good idea. He was on board pretty quickly, and so we had the first step out of the way.

Next, we had to get OReilly to release their copyright on the first book, so we (and possibly MSPress) could work on and publish the second edition. This turned out to be a huge part of the time between then and now, not owing to any one party's deliberate attempt to derail the process, but just because copies of contracts had to be sent to the original three authors (myself, Stutz and Geoff) to sign over our rights with OReilly to a Creative Commons License, then copies had to be sent to everybody else so all the signatures could appear on one document, and so on.

Did I say it already? Ugh. Lawyers. Contracts. Bleah.

Then, we had to get a contract from Microsoft signed, and that meant more contracts flying back and forth across the fax lines, and then later the US (and Australian) postal system, and that was more delays as the same round of signatures had to be exchanged.

Just for the record: Ugh. Lawyers. Contracts. Bleah.

Finally, though, the die was cast, the authors were ready to go, and.... Hey, does anybody have the latest soft copy of the Word docs we used from the first edition? A quick email to John (Osborn) took longer than we thought, as OReilly tried to find the post-QA docs for us to work from. (I had my own copies, of course, but they were pre-QA, and thus not really what we wanted to start from.) More rounds of emails to try and track those down, so we can get started. Oh, and while we're at it, can we get the figures/graphics, too? They're not in the manuscript directly, so.... Oh, wait, does anybody know how to read .EPS files?

Then began the actual writing process, or, to be more precise, the revision process. We decided on a process similar to the way the first book had been written: Joel, being the "subject matter expert", would take a first pass on the text, and sketch in the rough outlines of what needed to be said. I would then take the prose, polish it up (which in many cases didn't require a whole lot of work, Joel being a great writer in his own right) and rearrange sections as necessary to make it flow more easily, as well as flesh out certain sections that didn't require a former position on the CLR team to write. Joel would then have a look at what I wrote, and assuming I didn't get it completely wrong, would sign off on it, and the chapter/section/paragraph/whatever was done.

And now we're in the process of doing that cosmetic cleanup that's part of the overtime period in book-writing, including generating the table of contents and index, since, it turns out, we'd rather publish it ourselves than through MSPress (which they're OK with). So, readers will have a choice: get the free download from Microsoft's website (once we're done, which should be "real soon now") and read it in soft-copy, or buy it off of Amazon in "treeware version", which will put a modest amount of money into Joel's and my collective pocket (once the relatively modest expenses of self-publishing are covered, that is).

This will be my first experience with self-publishing (as it is for Joel, too), so I'm eager to see how the whole things turns out. One thing I will warn the prospective self-publisher, though: do not underestimate the time you will spend doing those things the editorial/QA/copyedit pass normally handles for you, because it's kind of a pain in the *ss to do it yourself. Still, it's worth it, particularly if you're having a hard time selling your book to a publisher who, for reasons of economy of scale, don't want to publish a niche book (like this one).

Anyway, like many of my blog postings, this post has gone on long enough, so I'll sign off here with a "go read the draft", even if you're a Java or other execution engine/virtual machine kind of developer--seeing the nuts and bolts of a complex execution engine in action is a pretty cool exercise.

Oh, and if anybody's interested in doing a similar kind of effort around the OpenJDK (once it ships), let me know, 'cuz I'm a glutton for punishment....

For those of you who were at the Cinncinnati NFJS show, please continue on to the next blog entry in your reader--you've already heard this. For those of you who weren't, then allow me to make the announcement:

Hi. My name's Ted Neward, and I am now a ThoughtWorker.

After four months of discussions, interviews, more discussions and more interviews, I can finally say that ThoughtWorks and I have come to a meeting of the minds, and starting 3 September I will be a Principal Consultant at ThoughtWorks. My role there will be to consult, write, mentor, architect and speak on Java, .NET, XML Services (and maybe even a little Ruby), not to mention help ThoughtWorks' clients achieve IT success in other general ways.

Yep, I'm basically doing the same thing I've been doing for the last five years. Except now I'm doing it with a TW logo attached to my name.

By the way, ThoughtWorkers get to choose their own titles, and I'm curious to know what readers think my title should be. Send me your suggestions, and if one really strikes home, I'll use it and update this entry to reflect the choice. I have a few ideas, but I'm finding that other people can be vastly more creative than I, and I'd love to have a title that rivals Neal's "Meme Wrangler" in coolness.

Oh, and for those of you who were thinking this, "Seat Warmer" has already been taken, from what I understand.

Honestly, this is a connection that's been hovering at the forefront of my mind for several years. I like ThoughtWorks' focus on success, their willingness to explore new ideas (both methodologies and technologies), their commitment to the community, their corporate values, and their overall attitude of "work hard, play hard". There have definitely been people who came away from ThoughtWorks with a negative impression of the company, but they're the minority. Any company that encourages T-shirts and jeans, XBoxes in the office, and wants to promote good corporate values is a winner in my book. In short, ThoughtWorks is, in many ways, the consulting company that I would want to build, if I were going to build a consulting firm. I'm not a wild fan of the travel commitments, mind you, but I am definitely no stranger to travel, we've got some ideas about how I can stay at home a bit more, and frankly I've been champing at the bit to get injected into more agile and team projects, so it feels like a good tradeoff. Plus, I get to think about languages and platforms in a more competitive and hostile way--not that TW is a competitive and hostile place, mind you, but in that my new fellow ThoughtWorkers will not let stupid thoughts stand for long, and will quickly find the holes in my arguments even faster, thus making the arguments as a whole that much stronger... or shooting them down because they really are stupid. (Either outcome works pretty well for me.)

What does this mean to the rest of you? Not much change, really--I'm still logging lots of hours at conferences, I'm still writing (and blogging, when the muse strikes), and I'm still available for consulting/mentoring/speaking; the big difference is that now I come with a thousand-strong developers of proven capability at my back, not to mention two of the more profound and articulate speakers in the industry (in Neal and Martin) as peers. So if you've got some .NET, Java, or Ruby projects you're thinking about, and you want a team to come in and make it happen, you know how to reach me.

Another DZone newsletter crosses my Inbox, and again I feel compelled to comment. Not so much in the uber-aggressive style of my previous attempt, since I find myself more on the fence on this one, but because I think it's a worthwhile debate and worth calling out.

The article in question is "5 Reasons Why You Don't Want A Jack-of-all-Trades Developer", by Rebecca Murphey. In it, she talks about the all-too-common want-ad description that appears on job sites and mailing lists:

I've spent the last couple of weeks trolling Craigslist and have been shocked at the number of ads I've found that seem to be looking for an entire engineering team rolled up into a single person. Descriptions like this aren't at all uncommon:

Candidates must have 5 years experience defining and developing data driven web sites and have solid experience with ASP.NET, HTML, XML, JavaScript, CSS, Flash, SQL, and optimizing graphics for web use. The candidate must also have project management skills and be able to balance multiple, dynamic, and sometimes conflicting priorities. This position is an integral part of executing our web strategy and must have excellent interpersonal and communication skills.

Her disdain for this practice is the focus of the rest of the article:

Now I don't know about you, but if I were building a house, I wouldn't want an architect doing the work of a carpenter, or the foundation guy doing the work of an electrician. But ads like the one above are suggesting that a single person can actually do all of these things, and the simple fact is that these are fundamentally different skills. The foundation guy may build a solid base, but put him in charge of wiring the house and the whole thing could, well, burn down. When it comes to staffing a web project or product, the principle isn't all that different -- nor is the consequence.

I'll admit, when I got to this point in the article, I was fully ready to start the argument right here and now--developers have to have a well-rounded collection of skills, since anecdotal evidence suggests that trying to go the route of programming specialization (along the lines of medical specialization) isn't going to work out, particularly given the shortage of programmers in the industry right now to begin with. But she goes on to make an interesting point:

The thing is, the more you know, the more you find out you don't know. A year ago I'd have told you I could write PHP/MySQL applications, and do the front-end too; now that I've seen what it means to be truly skilled at the back-end side of things, I realize the most accurate thing I can say is that I understand PHP applications and how they relate to my front-end development efforts. To say that I can write them myself is to diminish the good work that truly skilled PHP/MySQL developers are doing, just as I get a little bent when a back-end developer thinks they can do my job.

She really caught me eye (and interest) with that first statement, because it echoes something Bjarne Stroustrup told me almost 15 years ago, in an email reply sent back to me (in response to my rather audacious cold-contact email inquiry about the costs and benefits of writing a book): "The more you know, the more you know you don't know". What I think also caught my eye--and, I admit it, earned respect--was her admission that she maybe isn't as good at something as she thought she was before. This kind of reflective admission is a good thing (and missing far too much from our industry, IMHO), because it leads not only to better job placements for us as well as the companies that want to hire us, but also because the more honest we can be about our own skills, the more we can focus efforts on learning what needs to be learned in order to grow.

She then turns to her list of 5 reasons, phrased more as a list of suggestions to companies seeking to hire programming talent; my comments are in italics:

So to all of those companies who are writing ads seeking one magical person to fill all of their needs, I offer a few caveats before you post your next Craigslist ad:

1. If you're seeking a single person with all of these skills, make sure you have the technical expertise to determine whether a person's skills match their resume. Outsource a tech interview if you need to. Any developer can tell horror stories about inept predecessors, but when a front-end developer like myself can read PHP and think it's appalling, that tells me someone didn't do a very good job of vetting and got stuck with a programmer who couldn't deliver on his stated skills.

(T: I cannot stress this enough--the technical interview process practiced at most companies is a complete sham and travesty, and usually only succeeds in making sure the company doesn't hire a serial killer, would-be terrorist, or financially destitute freeway-underpass resident. I seriously think most companies should outsource the technical interview process entirely.)

2. A single source for all of these skills is a single point of failure on multiple fronts. Think long and hard about what it will mean to your project if the person you hire falls short in some aspect(s), and about the mistakes that will have to be cleaned up when you get around to hiring specialized people. I have spent countless days cleaning up after back-end developers who didn't understand the nuances and power of CSS, or the difference between a div, a paragraph, a list item, and a span. Really.

(T: I'm not as much concerned about the single point of failure argument here, to be honest. Developers will always have "edges" to what they know, and companies will constantly push developers to that edge for various reasons, most of which seem to be financial--"Why pay two people to do what one person can do?" is a really compelling argument to the CFO, particularly when measured against an unquantifiable, namely the quality of the project.)

3. Writing efficient SQL is different from efficiently producing web-optimized graphics. Administering a server is different from troubleshooting cross-browser issues. Trust me. All are integral to the performance and growth of your site, and so you're right to want them all -- just not from the same person. Expecting quality results in every area from the same person goes back to the foundation guy doing the wiring. You're playing with fire.

(T: True, but let's be honest about something here. It's not so much that the company wants to play with fire, or that the company has a manual entitled "Running a Dilbert Company" that says somewhere inside it, "Thou shouldst never hire more than one person to run the IT department", but that the company is dealing with limited budgets and headcount. If you only have room for one head under the budget, you want the maximum for that one head. And please don't tell me how wrong that practice of headcount really is--you're preaching to the choir on that one. The people you want to preach to are the Jack Welches of the world, who apparently aren't listening to us very much.)

4. Asking for a laundry list of skills may end up deterring the candidates who will be best able to fill your actual need. Be precise in your ad: about the position's title and description, about the level of skill you're expecting in the various areas, about what's nice to have and what's imperative. If you're looking to fill more than one position, write more than one ad; if you don't know exactly what you want, try harder to figure it out before you click the publish button.

(T: Asking people to think before publishing? Heresy! Truthfully, I don't think it's a question of not knowing what they want, it's more trying to find what they want. I've seen how some of these same job ads get generated, and it's usually because a programmer on the team has left, and they had some degree of skill in all of those areas. What the company wants, then, is somebody who can step into exactly what that individual was doing before they handed in their resignation, but ads like, "Candidate should look at Joe Smith's resume on Dice.com (http://...) and have exactly that same skill set. Being named Joe Smith a desirable 'plus', since then we won't have to have the sysadmins create a new login handle for you." won't attract much attention. Frankly, what I've found most companies want is to just not lose the programmer in the first place.)

5. If you really do think you want one person to do the task of an entire engineering team, prepare yourself to get someone who is OK at a bunch of things and not particularly good at any of them. Again: the more you know, the more you find out you don't know. I regularly team with a talented back-end developer who knows better than to try to do my job, and I know better than to try to do his. Anyone who represents themselves as being a master of front-to-back web development may very well have no idea just how much they don't know, and could end up imperiling your product or project -- front to back -- as a result.

(T: Or be prepared to pay a lot of money for somebody who is an expert at all of those things, or be prepared to spend a lot of time and money growing somebody into that role. Sometimes the exact right thing to do is have one person do it all, but usually it's cheaper to have a small team work together.)

(On a side note, I find it amusing that she seems to consider PHP a back-end skill, but I don't want to sound harsh doing so--that's just a matter of perspective, I suppose. (I can just imagine the guffaws from the mainframe guys when I talk about EJB, message-queue and Spring systems being "back-end", too.) To me, the whole "web" thing is front-end stuff, whether you're the one generating the HTML from your PHP or servlet/JSP or ASP.NET server-side engine, or you're the one generating the CSS and graphics images that are sent back to the browser by said server-side engine. If a user sees something I did, it's probably because something bad happened and they're looking at a stack trace on the screen.)

The thing I find interesting is that HR hiring practices and job-writing skills haven't gotten any better in the near-to-two-decades I've been in this industry. I can still remember a fresh-faced wet-behind-the-ears Stroustrup-2nd-Edition-toting job candidate named Neward looking at job placement listings and finding much the same kind of laundry list of skills, including those with the impossible number of years of experience. (In 1995, I saw an ad looking for somebody who had "10 years of C++ experience", and wondering, "Gosh, I guess they're looking to hire Stroustrup or Lippmann", since those two are the only people who could possibly have filled that requirement at the time. This was right before reading the ad that was looking for 5 years of Java experience, or the ad below it looking for 15 years of Delphi....)

Given that it doesn't seem likely that HR departments are going to "get a clue" any time soon, it leaves us with an interesting question: if you're a developer, and you're looking at these laundry lists of requirements, how do you respond?

Here's my own list of things for programmers/developers to consider over the next five to ten years:

1. These "laundry list" ads are not going away any time soon. We can rant and rail about the stupidity of HR departments and hiring managers all we want, but the basic fact is, this is the way things are going to work for the forseeable future, it seems. Changing this would require a "sea change" across the industry, and sea change doesn't happen overnight, or even within the span of a few years. So, to me, the right question to ask isn't, "How do I change the industry to make it easier for me to find a job I can do?", but "How do I change what I do when looking for a job to better respond to what the industry is doing?"
2. Exclusively focusing on a single area of technology is the Kiss of Death. If all you know is PHP, then your days are numbered. I mean no disrespect to the PHP developers of the world--in fact, were it not too ambiguous to say it, I would rephrase that as "If all you know is X, your days are numbered." There is no one technical skill that will be as much in demand in ten years as it is now. Technologies age. Industry evolves. Innovations come along that completely change the game and leave our predictions of a few years ago in the dust. Bill Gates (he of the "640K comment") has said, and I think he's spot on with this, "We routinely overestimate where we will be in five years, and vastly underestimate where we will be in ten." If you put all your eggs in the PHP basket, then when PHP gets phased out in favor of (insert new "hotness" here), you're screwed. Unless, of course, you want to wait until you're the last man standing, which seems to have paid off well for the few COBOL developers still alive.... but not so much for the Algol, Simula, or RPG folks....
3. Assuming that you can stop learning is the Kiss of Death. Look, if you want to stop learning at some point and coast on what you know, be prepared to switch industries. This one, for the forseeable future, is one that's predicated on radical innovation and constant change. This means we have to accept that everything is in a constant state of flux--you can either rant and rave against it, or roll with it. This doesn't mean that you don't have to look back, though--anybody who's been in this industry for more than 10 years has seen how we keep reinventing the wheel, particularly now that the relationship between Ruby and Smalltalk has been put up on the big stage, so to speak. Do yourself a favor: learn stuff that's already "done", too, because it turns out there's a lot of lessons we can learn from those who came before us. "Those who cannot remember the past are condemned to repeat it" (George Santanyana). Case in point: if you're trying to get into XML services, spend some time learning CORBA and DCOM, and compare how they do things against WSDL and SOAP. What's similar? What's different? Do some Googling and see if you can find comparison articles between the two, and what XML services were supposed to "fix" from the previous two. You don't have to write a ton of CORBA or DCOM code to see those differences (though writing at least a little CORBA/DCOM code will probably help.)
4. Find a collection of people smarter than you. Chad Fowler calls this "Being the worst player in any band you're in" (My Job Went to India (and All I Got Was This Lousy Book), Pragmatic Press). The more you surround yourself with smart people, the more of these kinds of things (tools, languages, etc) you will pick up merely by osmosis, and find yourself more attractive to those kind of "laundry list" job reqs. If nothing else, it speaks well to you as an employee/consultant if you can say, "I don't know the answer to that question, but I know people who do, and I can get them to help me".
5. Learn to be at least self-sufficient in related, complementary technologies. We see laundry list ads in "clusters". Case in point: if the company is looking for somebody to work on their website, they're going to rattle off a list of five or so things they want he/she to know--HTML, CSS, XML, JavaScript and sometimes Flash (or maybe now Silverlight), in addition to whatever server-side technology they're using (ASP.NET, servlets, PHP, whatever). This is a pretty reasonable request, depending on the depth of each that they want you to know. Here's the thing: the company does not want the guy who says he knows ASP.NET (and nothing but ASP.NET), when asked to make a small HTML or CSS change, to turn to them and say, "I'm sorry, that's not in my job description. I only know ASP.NET. You'll have to get your HTML guy to make that change." You should at least be comfortable with the basic syntax of all of the above (again, with possible exception for Flash, which is the odd man out in that job ad that started this piece), so that you can at least make sure the site isn't going to break when you push your changes live. In the case of the ad above, learn the things that "surround" website development: HTML, CSS, JavaScript, Flash, Java applets, HTTP (!!), TCP/IP, server operating systems, IIS or Apache or Tomcat or some other server engine (including the necessary admin skills to get them installed and up and running), XML (since it's so often used for configuration), and so on. These are all "complementary" skills to being an ASP.NET developer (or a servlet/JSP developer). If you're a C# or Java programmer, learn different programming languages, a la F# (.NET) or Scala (Java), IronRuby (.NET) or JRuby (Java), and so on. If you're a Ruby developer, learn either a JVM language or a CLR language, so you can "plug in" more easily to the large corporate enterprise when that call comes.
6. Learn to "read" the ad at a higher level. It's often possible to "read between the lines" and get an idea of what they're looking for, even before talking to anybody at the company about the job. For example, I read the ad that started this piece, and the internal dialogue that went on went something like this:

   Candidates must have 5 years experience (No entry-level developers wanted, they want somebody who can get stuff done without having their hand held through the process) defining and developing data driven (they want somebody who's comfortable with SQL and databases) web sites (wait for it, the "web cluster" list is coming) and have solid experience with ASP.NET (OK, they're at least marginally a Microsoft shop, that means they probably also want some Windows Server and IIS experience), HTML, XML, JavaScript, CSS (the "web cluster", knew that was coming), Flash (OK, I wonder if this is because they're building rich internet/intranet apps already, or just flirting with the idea?), SQL (knew that was coming), and optimizing graphics for web use (OK, this is another wrinkle--this smells of "we don't want our graphics-heavy website to suck"). The candidate must also have project management skills (in other words, "You're on your own, sucka!"--you're not part of a project team) and be able to balance multiple, dynamic, and sometimes conflicting priorities (in other words, "You're own your own trying to balance between the CTO's demands and the CEO's demands, sucka!", since you're not part of a project team; this also probably means you're not moving into an existing project, but doing more maintenance work on an existing site). This position is an integral part of executing our web strategy (in other words, this project has public visibility and you can't let stupid errors show up on the website and make us all look bad) and must have excellent interpersonal and communication skills (what job doesn't need excellent interpersonal and communication skills?).

   See what I mean? They want an ASP.NET dev. My guess is that they're thinking a lot about Silverlight, since Silverlight's closest competitor is Flash, and so theoretically an ASP.NET-and-Flash dev would know how to use Silverlight well. Thus, I'm guessing that the HTML, CSS, and JavaScript don't need to be "Adept" level, nor even "Master" level, but "Journeyman" is probably necessary, and maybe you could get away with "Apprentice" at those levels, if you're working as part of a team. The SQL part will probably have to be "Journeyman" level, the XML could probably be just "Apprentice", since I'm guessing it's only necessary for the web.config files to control the ASP.NET configuration, and the "optimizing web graphics", push-come-to-shove, could probably be forgiven if you've had some experience at doing some performance tuning of a website.
7. Be insightful. I know, every interview book ever written says you should "ask questions", but what they're really getting at is "Demonstrate that you've thought about this company and this position". Demonstrating insight about the position and the company and technology as a whole is a good way to prove that you're a neck above the other candidates, and will help keep the job once you've got it.
8. Be honest about what you know. Let's be honest--we've all met developers who claimed they were "experts" in a particular tool or technology, and then painfully demonstrated how far from "expert" status they really were. Be honest about yourself: claim your skills on a simple four-point scale. "Apprentice" means "I read a book on it" or "I've looked at it", but "there's no way I could do it on my own without some serious help, and ideally with a Master looking over my shoulder". "Journeyman" means "I'm competent at it, I know the tools/technology"; or, put another way, "I can do 80% of what anybody can ask me to do, and I know how to find the other 20% when those situations arise". "Master" means "I not only claim that I can do what you ask me to do with it, I can optimize systems built with it, I can make it do things others wouldn't attempt, and I can help others learn it better". Masters are routinely paired with Apprentices as mentors or coaches, and should expect to have this as a major part of their responsibilities. (Ideally, anybody claiming "architect" in their title should be a Master at one or two of the core tools/technologies used in their system; or, put another way, architects should be very dubious about architecting with something they can't reasonably claim at least Journeyman status in.) "Adept", shortly put, means you are not only fully capable of pulling off anything a Master can do, but you routinely take the tool/technology way beyond what anybody else thinks possible, or you know the depth of the system so well that you can fix bugs just by thinking about them. With your eyes closed. While drinking a glass of water. Seriously, Adept status is not something to claim lightly--not only had you better know the guys who created the thing personally, but you should have offered up suggestions on how to make it better and had one or more of them accepted.
9. Demonstrate that you have relevant skills beyond what they asked for. Look at the ad in question: they want an ASP.NET dev, so any familiarity with IIS, Windows Server, SQL Server, MSMQ, COM/DCOM/COM+, WCF/Web services, SharePoint, the CLR, IronPython, or IronRuby should be listed prominently on your resume, and brought up at least twice during your interview. These are (again) complementary technologies, and even if the company doesn't have a need for those things right now, it's probably because Joe didn't know any of those, and so they couldn't use them without sending Joe to a training class. If you bring it up during the interview, it can also show some insight on your part: "So, any questions for us?" "Yes, are you guys using Windows Server 2008, or 2003, for your back end?" "Um, we're using 2003, why do you ask?" "Oh, well, when I was working as an ASP.NET dev for my previous company, we moved up to 2008 because it had the Froobinger Enhancement, which let us...., and I was just curious if you guys had a similar need." Or something like that. Again, be entirely honest about what you know--if you helped the server upgrade by just putting the CDs into the drive and punching the power button, then say as much.
10. Demonstrate that you can talk to project stakeholders and users. Communication is huge. The era of the one-developer team is long since over--you have to be able to meet with project champions, users, other developers, and so on. If you can't do that without somebody being offended at your lack of tact and subtlety (or your lack of personal hygiene), then don't expect to get hired too often.
11. Demonstrate that you understand the company, its business model, and what would help it move forward. Developers who actually understand business are surprisingly and unfortunately rare. Be one of the rare ones, and you'll find companies highly reluctant to let you go.

Is this an exhaustive list? Hardly. Is this list guaranteed to keep you employed forever? Nope. But this seems to be working for a lot of the people I run into at conferences and client consulting gigs, so I humbly submit it for your consideration.

But in no way do I consider this conversation completely over, either--feel free to post your own suggestions, or tell me why I'm full of crap on any (or all) of these.

This crossed my Inbox, and I have to say, I'm stunned at this incredible display of teamwork. Frankly... well, see for yourself.

Speaking of things crossing my Inbox, Shane Paterson sent me this email:

Hi Ted,

How’s things in the USA?

I just wrote the following little blog entry I wanted to share with you, which I thought you may find interesting.

---

I used to work with a Naval Architect a few years back. On day we were discussing where the name "Naval Architect" came from. He explained that "Naval Architecture" is really "Naval Engineering" or "Ship Engineering". The word Engineering came into use AFTER the industrial revolution, however ship design existed hundreds of years before. At that time, people needed a name for ship designers, so they reasoned that Architects designed buildings, therefore ship designers must be a kind of Architect too - hence the name "Naval Architect".

Clearly IT didn't exist before the industrial revolution, and IT Architects don't design buildings, so the question begs to be asked "How did we end up with the word Architecture being used for a role in IT". It seems to me to be a rather vague and grandiose name for a role which is probably better described by the use of the word "Engineer".

Perhaps instead of the names "Solution Architect" and "Enterprise Architect" we should actually be using the names "IT Solution Engineer", "IT Enterprise Engineer"?

I've heard this idea put forward before, and I have to say, I'm not overly fond of it, because I believe that what we do as architects is fundamentally different from what a civil engineer does when he engages in the architect's role.

When a civil architect--be it Frank Lloyd Wright or the folks who designed the I-35 bridge in Minnesota--sits down to draw up the plans for a given project, they do so under a fairly strict set of laws and codes governing the process. Not only must the civic restrictions about safety and appearance be honored and respected, but also the basic physical laws of the universe--weight loads, stress, wind shear and harmonics (which the engineers of the first infamous Tacoma Narrows bridge discovered, to everlasting infamy). Ignoring these has disastrous consequences, and a discipline of mathematical calculation joins in with legal regulation to ensure that those laws are obeyed. Only then can the architect engage in the artistry that Lloyd Wright made so much a part of his craft.

Software architecture, though, is a different matter. Not only do we mostly enjoy complete freedom from legal regulation (Sarbannes-Oxley compliance being perhaps the most notable exception, and even then it routinely fails to apply at the small- to medium-sized project levels), we can also ignore most of the laws of physics (the speed of digital signal across a cable or through the air being perhaps our most notable barrier at the moment). "Access data in Tokyo from a web server in Beijing and send the rendered results to a browser in San Francisco? Sure, yeah, no problem, so long as there's a TCP/IP connection, I don't see why not...." There's just so much less by way of physical restrictions in software than there is in civil (or any other kind) of engineering, it seems.

And that sort of hits the central point squarely on the head--there's a lot we don't know about building software yet. We keep concocting these tortured metaphors and imperfect analogies to other practices, industries and disciplines in an attempt to figure out how best to build software, and they keep leading us astray in various ways. When's the last time you heard an accountant say, "Well, what I do is kinda like what the clerk in a retail store does--handle money--so therefore I should take ideas on how to do my job better from retail store clerks"? Sure, maybe the basic premise is true at some levels, but clearly the difference is in the details. And analogies and metaphors have this dangerous habit--they cause us to lose sight of those limitations, in the pursuit of keeping the analogy pure. Remember when everybody was getting purist about objects, such that an automobile repair shop's accounting system had to model "Car" as an object having an "Engine" object and four "Tire" objects and so on, not because these were things that needed to be repaired and tracked somehow, but because cars in real life have an engine and four tires and other things? (Stroustrup even touches on this at one point, talking about an avionics system which ran into design difficulties trying to decide if "Cloud" objects were owned by the "Sky" object, or something along those lines.) All analogies break down somewhere.

Now, to go back to architects and architecture. At the risk of offering up yet another of those tortured metaphors, let me proffer my own architect analogy: an architect is not like a construction architect, but more like the conductor of a band or symphony. Yes, the band could play without him, but at the end of the day, the band plays better with one guy coordinating the whole thing. The larger the band, the more necessary a conductor becomes. Sometimes the conductor is the same thing as the composer (and perhaps that's the most accurate analogous way to view this), in which case it's his "vision" of how the music in his head should come out in real life, and his job is to lead the performers into contributing towards that vision. Each performer has their own skills, freedom to interpret, and so on, but within the larger vision of the work.

Is it a perfect analogy? Heavens, no. It falls apart, just as every other analogy does, if you stress it too hard. But it captures the essence of art and rigor that I think seeing it as "architecture" along the lines of civil engineering just can't. At least, not easily.

Overheard, at the NFJS Phoenix 2008 show:

"We (ThoughtWorkers) are firm believers in aggressively promiscuous pairing." --Neal Ford

This comment deserves response:

First of all, if you're quoting my post, blocking out my name, and attacking me behind my back by calling me "our intrepid troll", you could have shown the decency of linking back to my original post. Here it is, for those interested in the real discussion:

http://www.agilesoftwaredevelopment.com/blog/jurgenappelo/professionalism-knowledge-first

Well, frankly, I didn't get your post from your blog, I got it from an email 'zine (as indicated by the comment "This crossed my Inbox..."), and I didn't really think that anybody would have any difficulty tracking down where it came from, at least in terms of the email blast that put it into my Inbox. Coupled with the fact that, quite honestly, I don't generally make a practice of using peoples' names without their permission (and my email to the author asking if I could quote the post with his name attached generated no response), so I blocked out the name. Having said that, I'm pleased to offer full credit as to its source.

Now, let's review some of your remarks:

"COBOL is (at least) twenty years old, so therefore any use of COBOL must clearly be as idiotic."

I never talked about COBOL, or any other programming language. I was talking about old practices that are nowadays considered harmful and seriously damaging. (Like practising waterfall project management, instead of agile project management.) I don't see how programming in COBOL could seriously damage a business. Why do you compare COBOL with lobotomies? I don't understand. I couldn't care less about programming languages. I care about management practices.

Frankly, the distinction isn't very clear in your post, and even more frankly, to draw a distinction here is a bit specious. "I didn't mean we should throw away the good stuff that's twenty years old, only the bad stuff!" doesn't seem much like a defense to me. There are cases where waterfall style development is exactly the right thing to do a more agile approach is exactly the wrong thing to do--the difference, as I'm fond of saying, lies entirely in the context of the problem. Analogously, there are cases where keeping an existing COBOL system up and running is the wrong thing to do, and replacing it with a new system is the right thing to do. It all depends on context, and for that reason, any dogmatic suggestion otherwise is flawed.

"How can a developer honestly claim to know "what it can be good for", without some kind of experience to back it?"

I'm talking about gaining knowledge from the experience of others. If I hear 10 experts advising the same best practice, then I still don't have any experience in that best practice. I only have knowledge about it. That's how you can apply your knowledge without your experience.

Leaving aside the notion that there is no such thing as best practices (another favorite rant of mine), what you're suggesting is that you, the individual, don't necessarily have to have experience in the topic but others have to, before we can put faith into it. That's a very different scenario than saying "We don't need no stinkin' experience", and is still vastly more dangerous than saying, "I have used this, it works." I (and lots of IT shops, I've found) will vastly prefer the latter to the former.

"Knowledge, apparently, isn't enough--experience still matters"

Yes, I never said experience doesn't matter. I only said it has no value when you don't have gained the appropriate knowledge (from other sources) on when to apply it, and when not.

You said it when you offered up the title, "Knowledge, not Experience".

"buried under all the ludicrous hyperbole, he has a point"

Thanks for agreeing with me.

You're welcome! Seriously, I think I understand better what you were trying to say, and it's not the horrendously dangerous comments that I thought you were saying, so I will apologize here and now for believing you to be a wet-behind-the-ears/lets-let-technology-solve-all-our-problems/dangerous-to-the-extreme developer that I've run across far too often, particularly in startups. So, please, will you accept my apology?

"developers, like medical professionals, must ensure they are on top of their game and spend some time investing in themselves and their knowledge portfolio"

Exactly.

Exactly.

"this doesn't mean that everything you learn is immediately applicable, or even appropriate, to the situation at hand"

I never said that. You're putting words into my mouth.

My only claim is that you need to KNOW both new and old practices and understand which ones are good and which ones can be seriously damaging. I simply don't trust people who are bragging about their experience. What if a manager tells me he's got 15 years of experience managing developers? If he's a micro-manager I still don't want him. Because micro-management is considered harmful these days. A manager should KNOW that.

Again, this was precisely the read I picked up out of the post, and my apologies for the misinterpretation. But I stand by the idea that this is one of those posts that could be read in a highly dangerous fashion, and used to promote evil, in the form of "Well, he runs a company, so therefore he must know what he's doing, and therefore having any kind of experience isn't really necessary to use something new, so... see, Mr. CEO boss-of-mine? We're safe! Now get out of my way and let me use Software Factories to build our next-generation mission-critical core-of-the-company software system, even though nobody's ever done it before."

To speak to your example for a second, for example: Frankly, there are situations where a micro-manager is a good thing. Young, inexperienced developers, for example, need more hand-holding and mentoring than older, more senior, more experienced developers do (speaking stereotypically, of course). And, quite honestly, the guy with 15 years managing developers is far more likely to know how to manage developers than the guy who's never managed developers before at all. The former is the safer bet; not a guarantee, certainly, but often the safer bet, and that's sometimes the best we can do in this industry.

"And we definitely shouldn't look at anything older than five years ago and equate it to lobotomies."

I never said that either. Why do you claim that I said this? I don't have a problem with old techniques. The daily standup meeting is a 80-year old best practice. It was already used by pilots in the second world war. How could I be against that? It's fine as it is.

Um... because you used the term "lobotomies" first? And because your title pretty clearly implies the statement, perhaps? (And let's lose the term "best practice" entirely, please? There is no such thing--not even the daily standup.)

It's ok you didn't like my article. Sure it's meant to be provocative, and food for thought. The article got twice as many positive votes than negative votes from DZone readers. So I guess I'm adding value. But by taking the discussion away from its original context (both physically and conceptually), and calling me names, you're not adding any value for anyone.

I took it in exactly the context it was given--a DZone email blast. I can't help it if it was taken out of context, because that's how it was handed to me. What's worse, I can see a development team citing this as an "expert opinion" to their management as a justification to try untested approaches or technologies, or as inspiration to a young developer, who reads "knowledge, not experience", and thinks, "Wow, if I know all the cutting-edge latest stuff, I don't need to have those 10 years of experience to get that job as a senior application architect." If your article was aimed more clearly at the development process side of things, then I would wish it had appeared more clearly in the arena of development processes, and made it more clear that your aim was to suggest that managers (who aren't real big on reading blogs anyway, I've sadly found) should be a bit more pragmatic and open-minded about who they hire.

Look, I understand the desire for a provocative title--for me, the author of "The Vietnam of Computer Science", to cast stones at another author for choosing an eye-catching title is so far beyond hypocrisy as to move into sheer wild-eyed audacity. But I have seen, first-hand, how that article has been used to justify the most incredibly asinine technology decisions, and it moves me now to say "Be careful what you wish for" when choosing titles that meant to be provocative and food for thought. Sure, your piece got more positive votes than negative ones. So too, in their day, did articles on client-server, on CORBA, on Six-Sigma, on the necessity for big up-front design....

Let me put it to you this way. Assume your child or your wife is sick, and as you reach the hospital, the admittance nurse offers you a choice of the two doctors on call. Who do you want more: the doctor who just graduated fresh from medical school and knows all the latest in holistic and unconventional medicine, or the doctor with 30 years' experience and a solid track record of healthy patients?

Recently this little gem crossed my Inbox....

Professionalism = Knowledge First, Experience Last
By J----- A-----

Do you trust a doctor with diagnosing your mental problems if the doctor tells you he's got 20 years of experience? Do you still trust that doctor when he picks up his tools, and asks you to prepare for a lobotomy?

Would you still be impressed if the doctor had 20 years of experience in carrying out lobotomies?

I am always skeptic when people tell me they have X years of experience in a certain field or discipline, like "5 years of experience as a .NET developer", "8 years of experience as a project manager" or "12 years of experience as a development manager". It is as if people's professional levels need to be measured in years of practice.

This, of course, is nonsense.

Professionalism is measured by what you are going to do now...

Are you going to use some discredited technique from half a century ago?
•    Are you, as a .NET developer, going to use Response.Write, because you've got 5 years of experience doing exactly that?
•    Are you, as a project manager, going to create Gantt charts, because that's what you've been doing for 8 years?
•    Are you, as a development manager, going to micro-manage your team members, as you did in the 12 years before now?

If so, allow me to illustrate the value of your experience...

(Photo of "Zero" signs)

Here's an example of what it means to be a professional:

There's a concept called Kanban making headlines these days in some parts of the agile community. I honestly and proudly admit that I have no experience at all in applying Kanban. But that's just a minor inconvenience. Because I have attained the knowledge of what it is and what it can be good for. And now there are some planning issues in our organization for which this Kanban-stuff might be the perfect solution. I'm sure we're going to give it a shot, in a controlled setting, with time allocated for a pilot and proper evaluations afterwards. That's the way a professional tries to solve a problem.

Professionals don't match problems with their experiences. They match them with their knowledge.

Sure, experience is useful. But only when you already have the knowledge in place. Experience has no value when there's no knowledge to verify that you are applying the right experience.

Knowledge Comes First, Experience Comes Last

This is my message to anyone who wants to be a professional software developer, a professional project manager, or a professional development manager.

You must gain and apply knowledge first, and experience will help you after that. Professionals need to know about the latest developments and techniques.

They certainly don't bother measuring years of experience.

Are you still practicing lobotomies?

Um....

Wow.

Let's start with the logical fallacy in the first section. Do I trust a doctor with diagnosing my mental problems if he tells me he's got 20 years of experience? Generally, yes, unless I have reasons to doubt this. If the guy picks up a skull-drill and starts looking for a place to start boring into my skull, sure, I'll start to question his judgement.... But what does this have to do with anything? I wouldn't trust the guy if he picked up a chainsaw and started firing it up, either.

Look, I get the analogy: "Doctor has 20 years of experience using outdated skills", har har. Very funny, very memorable, and totally inappropriate metaphor for the situation. To stand here and suggest that developers who aren't using the latest-and-greatest, so-bleeding-edge-even-saying-the-name-cuts-your-skin tools or languages or technologies are somehow practicing lobotomies (which, by the way, are still a recommended practice in certain mental disorder cases, I understand) in order to solve any and all mental-health issues, is a gross mischaracterization--and the worst form of negligence--I've ever heard suggested.

And it comes as no surprise that it's coming from the CIO of a consulting company. (Note to self: here's one more company I don't want anywhere near my clients' IT infrastructure.)

Let's take this analogy to its logical next step, shall we?

COBOL is (at least) twenty years old, so therefore any use of COBOL must clearly be as idiotic as drilling holes in your skull to let the demons out. So any company currently using COBOL has no real option other than to immediately upgrade all of their currently-running COBOL infrastructure (despite the fact that it's tested, works, and cashes most of the US banking industry's checks on a daily basis) with something vastly superior and totally untested (since we don't need experience, just knowlege), like... oh, I dunno.... how about Ruby? Oh, no, wait, that's at least 10 years old. Ditto for Java. And let's not even think about C, Perl, Python....

I know; let's rewrite the entire financial industry's core backbone in Groovy, since it's only, what, 6 years old at this point? I mean, sure, we'll have to do all this over again in just four years, since that's when Groovy will turn 10 and thus obviously begin it's long slide into mediocrity, alongside the "four humors" of medicine and Aristotle's (completely inaccurate) anatomical depictions, but hey, that's progress, right? Forget experience, it has no value compared to the "knowledge" that comes from reading the documentation on a brand-new language, tool, library, or platform....

What I find most appalling is this part right here:

I honestly and proudly admit that I have no experience at all in applying Kanban. But that's just a minor inconvenience. Because I have attained the knowledge of what it is and what it can be good for.

How can a developer honestly claim to know "what it can be good for", without some kind of experience to back it? (Hell, I'll even accept that you have familiarity and experience with something vaguely relating to the thing at hand, if you've got it--after all, experience in Java makes you a pretty damn good C# developer, in my mind, and vice versa.)

And, to make things even more interesting, our intrepid troll, having established the attention-gathering headline, then proceeds to step away from the chasm, by backing away from this "knowledge-not-experience" position in the same paragraph, just one sentence later:

I'm sure we're going to give it a shot, in a controlled setting, with time allocated for a pilot and proper evaluations afterwards.

Ah... In other words, he and his company are going to experiment with this new technique, "in a controlled setting, with time allocated for a pilot and proper evaluations afterwards", in order to gain experience with the technique and see how it works and how it doesn't.

In other words....

.... experience matters.

Knowledge, apparently, isn't enough--experience still matters, and it matters a lot earlier than his "knowledge first, experience last" mantra seems to imply. Otherwise, once you "know" something, why not apply it immediately to your mission-critical core?

At the end of the day, buried under all the ludicrous hyperbole, he has a point: developers, like medical professionals, must ensure they are on top of their game and spend some time investing in themselves and their knowledge portfolio. Jay Zimmerman takes great pains to point this out at every No Fluff Just Stuff show, and he's right: those who spend the time to invest in their own knowledge portfolio, find themselves the last to be fired and the first to be hired. But this doesn't mean that everything you learn is immediately applicable, or even appropriate, to the situation at hand. Just because you learned Groovy last weekend in Austin doesn't mean you have the right--or the responsibility--to immediately start slipping Groovy in to the production servers. Groovy has its share of good things, yes, but it's got its share of negative consequences, too, and you'd better damn well know what they are before you start betting the company's future on it. (No, I will not tell you what those negative consequences are--that's your job, because what if it turns out I'm wrong, or they don't apply to your particular situation?) Every technology, language, library or tool has a positive/negative profile to it, and if you can't point out the pros as well as the cons, then you don't understand the technology and you have no business using it on anything except maybe a prototype that never leaves your local hard disk. Too many projects were built with "flavor-of-the-year" tools and technologies, and a few years later, long after the original "bleeding-edge" developer has gone on to do a new project with a new "bleeding-edge" technology, the IT guys left to admin and upkeep the project are struggling to find ways to keep this thing afloat.

If you're languishing at a company that seems to resist anything and everything new, try this exercise on: go down to the IT guys, and ask them why they resist. Ask them to show you a data flow diagram of how information feeds from one system to another (assuming they even have one). Ask them how many different operating systems they have, how many different languages were used to create the various software programs currently running, what tools they have to know when one of those programs fails, and how many different data formats are currently in use. Then go find the guys currently maintaining and updating and bug-fixing those current programs, and ask to see the code. Figure out how long it would take you to rewrite the whole thing, and keep the company in business while you're at it.

There is a reason "legacy code" exists, and while we shouldn't be afraid to replace it, we shouldn't be cavalier about tossing it out, either.

And we definitely shouldn't look at anything older than five years ago and equate it to lobotomies. COBOL had some good ideas that still echo through the industry today, and Groovy and Scala and Ruby and F# undoubtedly have some buried mines that we will, with benefit of ten years' hindsight, look back at in 2018 and say, "Wow, how dumb were we to think that this was the last language we'd ever have to use!".

That's experience talking.

And the funny thing is, it seems to have served us pretty well. When we don't listen to the guys claiming to know how to use something effectively that they've never actually used before, of course.

Caveat emptor.

If you've peeked at my blog site in the last twenty minutes or so, you've probably noticed some churn in the template in the upper-left corner; by now, it's been finalized, and it reads "JOB REFERRALS".

WTHeck? Has Ted finally sold out? Sort of, not really. At least, I don't think so.

Here's the deal: the company behind those ads, Entice Labs, contacted me to see if I was interested in hosting some job ads on my blog, given that I seem to generate a moderate amount of traffic. I figured it was worthwhile to at least talk to them, and the more I did, the more I liked what I heard--the ads are focused specifically at developers of particular types (I chose a criteria string of "Software Developers", subcategorized by "Java, .NET, .NET (Visual Basic), .NET (C#), C++, Flex, Ruby on Rails, C, SQL, JavaScript, HTML" though I'm not sure whether "HTML" will bring in too many web-designer jobs), and visitors to my blog don't have to click through the ads to get to the content, which was important to me. And, besides, given the current economic climate, if I can help somebody find a new job, I'd like to.

Now for the full disclaimer: I will be getting money back from these job ads, though how much, to be honest with you, I'm not sure. I'm really not doing this for the money, so I make this statement now: I will take 50% of whatever I make through this program and donate it to a charitable organization. The other 50% I will use to offset travel and expenses to user groups and/or CodeCamps and/or for-free conferences put on throughout the country. (Email me if you know of one that you're organizing or attending and would like to see me speak at, and I'll tell you if there's any room in the budget left for it. )

Anyway, I figured if the ads got too obnoxious, I could always remove them; it's an experiment of sorts. Tell me what you think.

After the previous post, I just had to look. The implementation of Object.equals is, as was previously noted, just "return this == obj", but the implementation of Object.hashCode is far more complicated.

Taken straight from the latest hg-pulled OpenJDK sources, Object.hashCode is a native method registered from Object.c that calls into a Hotspot-exported function, JVM_IHashCode(), from hotspot\src\share\vm\prims\jvm.cpp:

JVM_ENTRY(jint, JVM_IHashCode(JNIEnv* env, jobject handle))
  JVMWrapper("JVM_IHashCode");
  // as implemented in the classic virtual machine; return 0 if object is NULL
  return handle == NULL ? 0 : ObjectSynchronizer::FastHashCode (THREAD, JNIHandles::resolve_non_null(handle)) ;
JVM_END

which in turn calls ObjectSynchronizer::FastHashCode, defined in hotspot\src\share\vm\runtime\synchronizer.cpp as:

intptr_t ObjectSynchronizer::FastHashCode (Thread * Self, oop obj) {
  if (UseBiasedLocking) {
    // NOTE: many places throughout the JVM do not expect a safepoint
    // to be taken here, in particular most operations on perm gen
    // objects. However, we only ever bias Java instances and all of
    // the call sites of identity_hash that might revoke biases have
    // been checked to make sure they can handle a safepoint. The
    // added check of the bias pattern is to avoid useless calls to
    // thread-local storage.
    if (obj->mark()->has_bias_pattern()) {
      // Box and unbox the raw reference just in case we cause a STW safepoint.
      Handle hobj (Self, obj) ;
      // Relaxing assertion for bug 6320749.
      assert (Universe::verify_in_progress() ||
              !SafepointSynchronize::is_at_safepoint(),
             "biases should not be seen by VM thread here");
      BiasedLocking::revoke_and_rebias(hobj, false, JavaThread::current());
      obj = hobj() ;
      assert(!obj->mark()->has_bias_pattern(), "biases should be revoked by now");
    }
  }

  // hashCode() is a heap mutator ...
  // Relaxing assertion for bug 6320749.
  assert (Universe::verify_in_progress() ||
          !SafepointSynchronize::is_at_safepoint(), "invariant") ;
  assert (Universe::verify_in_progress() ||
          Self->is_Java_thread() , "invariant") ;
  assert (Universe::verify_in_progress() ||
         ((JavaThread *)Self)->thread_state() != _thread_blocked, "invariant") ;

  ObjectMonitor* monitor = NULL;
  markOop temp, test;
  intptr_t hash;
  markOop mark = ReadStableMark (obj);

  // object should remain ineligible for biased locking
  assert (!mark->has_bias_pattern(), "invariant") ;

  if (mark->is_neutral()) {
    hash = mark->hash();              // this is a normal header
    if (hash) {                       // if it has hash, just return it
      return hash;
    }
    hash = get_next_hash(Self, obj);  // allocate a new hash code
    temp = mark->copy_set_hash(hash); // merge the hash code into header
    // use (machine word version) atomic operation to install the hash
    test = (markOop) Atomic::cmpxchg_ptr(temp, obj->mark_addr(), mark);
    if (test == mark) {
      return hash;
    }
    // If atomic operation failed, we must inflate the header
    // into heavy weight monitor. We could add more code here
    // for fast path, but it does not worth the complexity.
  } else if (mark->has_monitor()) {
    monitor = mark->monitor();
    temp = monitor->header();
    assert (temp->is_neutral(), "invariant") ;
    hash = temp->hash();
    if (hash) {
      return hash;
    }
    // Skip to the following code to reduce code size
  } else if (Self->is_lock_owned((address)mark->locker())) {
    temp = mark->displaced_mark_helper(); // this is a lightweight monitor owned
    assert (temp->is_neutral(), "invariant") ;
    hash = temp->hash();              // by current thread, check if the displaced
    if (hash) {                       // header contains hash code
      return hash;
    }
    // WARNING:
    //   The displaced header is strictly immutable.
    // It can NOT be changed in ANY cases. So we have
    // to inflate the header into heavyweight monitor
    // even the current thread owns the lock. The reason
    // is the BasicLock (stack slot) will be asynchronously
    // read by other threads during the inflate() function.
    // Any change to stack may not propagate to other threads
    // correctly.
  }

  // Inflate the monitor to set hash code
  monitor = ObjectSynchronizer::inflate(Self, obj);
  // Load displaced header and check it has hash code
  mark = monitor->header();
  assert (mark->is_neutral(), "invariant") ;
  hash = mark->hash();
  if (hash == 0) {
    hash = get_next_hash(Self, obj);
    temp = mark->copy_set_hash(hash); // merge hash code into header
    assert (temp->is_neutral(), "invariant") ;
    test = (markOop) Atomic::cmpxchg_ptr(temp, monitor, mark);
    if (test != mark) {
      // The only update to the header in the monitor (outside GC)
      // is install the hash code. If someone add new usage of
      // displaced header, please update this code
      hash = test->hash();
      assert (test->is_neutral(), "invariant") ;
      assert (hash != 0, "Trivial unexpected object/monitor header usage.");
    }
  }
  // We finally get the hash
  return hash;
}

Hope this answers all the debates.

Editor's note: Yes, I know it's a long quotation of code completely out of context; my goal here is simply to suggest that the hashCode() implementation is not just a integerification of the object's address in memory, as was suggested in other discussions. For whatever it's worth, the get_next_hash() implementation that's referenced in the FastHashCode() method looks like:

// hashCode() generation :
//
// Possibilities:
// * MD5Digest of {obj,stwRandom}
// * CRC32 of {obj,stwRandom} or any linear-feedback shift register function.
// * A DES- or AES-style SBox[] mechanism
// * One of the Phi-based schemes, such as:
//   2654435761 = 2^32 * Phi (golden ratio)
//   HashCodeValue = ((uintptr_t(obj) >> 3) * 2654435761) ^ GVars.stwRandom ;
// * A variation of Marsaglia's shift-xor RNG scheme.
// * (obj ^ stwRandom) is appealing, but can result
//   in undesirable regularity in the hashCode values of adjacent objects
//   (objects allocated back-to-back, in particular).  This could potentially
//   result in hashtable collisions and reduced hashtable efficiency.
//   There are simple ways to "diffuse" the middle address bits over the
//   generated hashCode values:
//

static inline intptr_t get_next_hash(Thread * Self, oop obj) {
  intptr_t value = 0 ;
  if (hashCode == 0) {
     // This form uses an unguarded global Park-Miller RNG,
     // so it's possible for two threads to race and generate the same RNG.
     // On MP system we'll have lots of RW access to a global, so the
     // mechanism induces lots of coherency traffic.
     value = os::random() ;
  } else
  if (hashCode == 1) {
     // This variation has the property of being stable (idempotent)
     // between STW operations.  This can be useful in some of the 1-0
     // synchronization schemes.
     intptr_t addrBits = intptr_t(obj) >> 3 ;
     value = addrBits ^ (addrBits >> 5) ^ GVars.stwRandom ;
  } else
  if (hashCode == 2) {
     value = 1 ;            // for sensitivity testing
  } else
  if (hashCode == 3) {
     value = ++GVars.hcSequence ;
  } else
  if (hashCode == 4) {
     value = intptr_t(obj) ;
  } else {
     // Marsaglia's xor-shift scheme with thread-specific state
     // This is probably the best overall implementation -- we'll
     // likely make this the default in future releases.
     unsigned t = Self->_hashStateX ;
     t ^= (t << 11) ;
     Self->_hashStateX = Self->_hashStateY ;
     Self->_hashStateY = Self->_hashStateZ ;
     Self->_hashStateZ = Self->_hashStateW ;
     unsigned v = Self->_hashStateW ;
     v = (v ^ (v >> 19)) ^ (t ^ (t >> 8)) ;
     Self->_hashStateW = v ;
     value = v ;
  }

  value &= markOopDesc::hash_mask;
  if (value == 0) value = 0xBAD ;
  assert (value != markOopDesc::no_hash, "invariant") ;
  TEVENT (hashCode: GENERATE) ;
  return value;
}

Thus (hopefully) putting the idea that it might be allocating a hash based on the object's identity completely to rest.

For the record, this is all from the OpenJDK source base--naturally, it's possible that earlier VM implementations did something entirely different.

I'm not sure what it is about our industry that promotes the flame war, but for some reason exchanges like this one, unheard of in any other industry I've ever touched (even tangentially), are far too common, too easy to get into, and entirely too counterproductive.

I'm not going to weigh in on one side or the other here; frankly, I have a hard time following the debate and figuring out who's exactly arguing for what. I can see, however, that the entire debate follows some traditional patterns of the flame war:

1. Citing yourself as the final authority. At no point during the debate does anybody reach for their copy of Effective Java, a widely-accepted source of Java guidance, for a potential resolution to the discussion. Instead, the various players simply say, "Fact A is true" or "Fact A is false", with zero supporting information, citations, or demonstrations either way. (A few people cite the Javadoc, but there is enough ambiguity there to merit further citation.)
2. Refusal to accept the possibility of an alternative viewpoint. At no point, near as I can tell, did any of the participants bother to say, "You know, you could be right, but I remain unconvinced. Can you give me more information to support your point of view?" The entire time, everybody is arguing from "fact", and nobody even considers the possibility that different JVMs can have different implementations, despite the fact that the Javadoc being quoted says as much.
3. Degeneration into personal attacks. I don't care who started it, I don't care who called who the worse name. Fact is, reasonable people can reasonably disagree, and nobody in that transcript seemed overly reasonable to me.
4. Nobody ever really gets around to answering the question because they're too busy arguing their position or point. Poor "doub", the initiator of the question, tries valiantly to circle the conversation back on topic, but the various players are too busy whipping out their instruments of manhood onto the table so everybody can see how much bigger it is than the other guys'. When "doub" points out that writing some sample code "gave me a very loose but still usefull information about my object, and took less time than the conversation about my question ", or in other words, "Hey, guys, I kinda already got my answer, can we move on now?", the conversation continues as if the comment never occurred--the question has turned into a "biggest-geek" argument by this point. "doub" even asks, at 10:12:12, "do i get bad karma points for being the initiator of a conflict?", and the image I get in my head is that of the poor kid, hiding in his bedroom while his parents yell and scream downstairs, feeling awful because the fight started over his backpack lying in the hallway where Mom told him to put it and Dad thought he left it instead of putting it away. ("doub", if you read this, no, you get no bad karma points, at least not in my universe.)

The interesting thing, though, is that this conversation has nothing to do with Scala. "dysinger" twitters:

• "It's no wonder scala sucks at gaining any momentum with that kind of academic egomaniac cliques going on in the community." (http://twitter.com/dysinger/statuses/859189384)
• "@DRMacIver if you often get called an academic programmer, it's probably because you _are_ one. Scala + zero tolerance + ego = academic." (http://twitter.com/dysinger/statuses/859201109)
• "@DRMaclver starts with this "Or, in other words, you're full of shit. Kindly shut up about this now." to my chat. That set me off. Idiot." (http://twitter.com/dysinger/statuses/859204285)
• "ok - I am better now. I was seeing red for a minute there.... R.I.P. Scala You were barely a twinkle in somebody's eye and now your dead." (http://twitter.com/dysinger/statuses/859214117)
• "Felt better when I wrote some scala code to prove to #scala that they have a tard for a channel op -> http://is.gd/Ulx" (http://twitter.com/dysinger/statuses/859457360)

Frankly, "dysinger", it's kinda hard to have much sympathy for somebody when they blame the language or tool for a conversation that's had around it; this would be like blaming Python, the language, for the community around it (which some people do, I understand). I can understand the frustration, on both sides, since everybody was essentially arguing past one another, but why is that Scala's fault, pray tell?

And frankly, I find the dig at the academics to be a tad disingenuous. Yes, academics have a reputation--duly earned in some cases--of being removed from reality and the slings and arrows of a life spent developing software for production environments, but name for me a language in the popular mainstream that doesn't owe a huge debt to the preliminary work laid down by academics before it. In every other industry, academics are revered and honored--it's only in this industry they are used as an example of degradation and insult. Way to bite the hand that makes your life easier, folks....

At the end of the day, these kind of debates do nothing but harm the innocent, "doub", in this case. "dysinger", "DrMacIver", "JamesIry", all of you, right or wrong, didn't exactly cover yourselves in glory, nor did you really convince anybody of anything. Instead, you shouted at each other really loudly, made lots of noise, got angry over nothing in particular, and really failed to achieve much of anything. Regardless of your intentions, now Scala, Java, the JVM and the entire ecosystem have seen their reputation tarnished just a touch more than it was when you started. Great job.

Here's a tip for all of you: Try listening.

Google (or at least some part of it) has now weighed in on the whole XML discussion with the recent release of their "Protocol Buffers" implementation, and, quite naturally, the debates have begun, with all the carefully-weighed logic, respectful discourse, and reasoned analysis that we've come to expect and enjoy from this industry.

Yeah, right.

Anyway, without trying to take sides either way in this debate--yes, the punchline is that I believe in a world where both XML and Protocol Buffers are useful--I thought I'd weigh in on some of the aspects about PBs that are interesting/disturbing, but more importantly, try to frame some of the debate and discussions around these two topics in a vain attempt to wring some coherency and sanity out of what will likely turn into a large shouting match.

For starters, let's take a quick look at how PBs work.

Protocol Buffers 101

The idea behind PBs is pretty straightforward: given a PB definition file, a code-generator tool builds C++, Java or Python accessors/generators that know how to parse and produce files in the Protocol Buffer format. The generated classes follow a pretty standard format, using the traditional POJO/JavaBean style get/set for Java classes, and something similar for both the Python and C++ classes. (The Python implementation is a tad different from the C++ and Java versions, as it makes use of Python metaclasses to generate the class at runtime, rather than at generation-time.) So, for example, given the Google example of:

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
}

... using the corresponding generated C++ class would look something like

Person person;
person.set_name("John Doe");
person.set_id(1234);
person.set_email("jdoe@example.com");
fstream output("myfile", ios::out | ios::binary);
person.SerializeToOstream(&output);

... and the Java implementation would look somewhat similar, except using a Builder to create the Person.

The Protocol Buffer interoperable definition language is relatively complete, with support for a full range of scalar types, enumerations, variable-length collections of these types, nested message types, and so on. Each field in the message is tagged with a unique field number, as you can see above, and the language provides the ability to "reserve" field IDs via the "extension" mechanism, presumably to allow for either side to have flexibility in extending the PB format.

There's certainly more depth to the PB format, including the "service" and "rpc" features of the language that will generate full RPC proxies, but this general overview serves to provide the necessary background to be able to engage in an analysis of PBs.

Protocol Buffers: An Analysis

When looking at Protocol Buffers, it's important to realize that this is an open-source implementation, and that Google has already issued a call to others to modify them and submit patches to the project. Anything that comes across as a criticism or inherent flaw in the implementation is thus correctable, though whether Google would apply those fixes to the version they use internally is an open question--there's a tension in any open-source project sponsored by a company, between "What we need the project for" and "What other people using the project need it for", and it's not always clear how that tension will play out in the long term.

So, without further ado ...

For starters, Protocol Buffers' claim to be language and/or platform-neutral is hardly justifiable, given that they have zero support for the .NET platform out of the box. Now, before the Googlists and Google-fanbois react angrily, let me be the first to admit, that yes, nothing stops anybody from producing said capability and contributing it back to the project. In fact, there's even some verbage to that effect on the Protocol Buffers' FAQ page. But without it coming out of the box, it's not fair to claim language- and platform-neutrality, unless, of course, they are willing to suggest that COM's Structured Storage was also language- and platform-neutral, and that it wasn't Microsoft's fault that nobody went off and built implementations for it under *nix or the Mac. Frankly, any binary format, regardless of how convoluted, could be claimed to be language- and platform-neutral under those conditions, which I think makes the claim spurious to make. The fact that Google doesn't care about the .NET platform doesn't mean that their implementation is incapable of running on it; the fact that Google wrote their code generator to support C++, Java and Python doesn't mean that Protocol Buffers are language- and platform-neutral, either. XML still holds the edge here, by a long shot--until we see implementations of Protocol Buffers for Perl/Parrot, C, D, .NET, Ruby, JavaScript, mainframes and others, PBs will have to take second-place status behind XML in terms of "reach" across the wide array of systems. Does that diminish their usefulness? Hardly. It just depends on how far a developer wants their data format to stretch.

Having gotten that little bit out of the way ...

Remember a time when binary formats were big? Back in the early- to mid-90's, binary formats were all the rage, and criticism of them abounded: they were hard to follow, hard to debug, bloated, inherently inflexible, tightly-coupled, and so on. Consider, for example, this article comparing XML against its predecessor information-exchange technologies:

A great example ... is a data feed of all orders taken electronically via a web site into an internal billing and shipping system. The main advantage of XML here is it's flexibility - developers create their own tags and dictionaries, as they deem necessary. Therefore no matter what type of data is being transferred, the right XML representation of it will accurately describe the data.

Logically, each order can be described as one customer purchasing one or more items using their credit card and potentially entering different billing and shipping addresses. The contents of the file are very easy to read, even for a person who is not familiar with XML. The information within each of the order tags is well structured and organized. This enables developers to use parsing components and easily access any data within the document. Each item in the order is logically a unique entity, and is also represented with a separate tag. All item properties are defined as "child" nodes of the item tag.

XML is the language of choice for two major reasons. First of all, an XML formatted document can be easily processed under any OS, and in any language, as long as XML parsing components are available. On the other hand, XML files are still raw data, which enables merchants to format the data any way they want to. All in all, document structure and wide acceptance of this format has made it possible to enable customers to build more efficient internal order Tracking systems based on XML-formatted order files. Other online merchant sites are making similar functionality available on their Web sites as well.

And yet, now, the criticism goes the other way, complaining that XML is bloated, hard to follow, and so on:

Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update your data structure without breaking deployed programs that are compiled against the "old" format.

How do we reconcile these apparently contradictory positions?

First of all, I don't think anybody in the XML community has ever sought to argue that XML was designed for efficiency; in fact, quite the opposite, as the XML specification itself states in the abstract:

The Extensible Markup Language (XML) is a subset of SGML that is completely described in this document. Its goal is to enable generic SGML to be served, received, and processed on the Web in the way that is now possible with HTML. XML has been designed for ease of implementation and for interoperability with both SGML and HTML. ... This document specifies a syntax created by subsetting an existing, widely used international text processing standard (Standard Generalized Markup Language, ISO 8879:1986(E) as amended and corrected) for use on the World Wide Web.

The Introduction section is even more explicit:

XML documents are made up of storage units called entities, which contain either parsed or unparsed data. Parsed data is made up of characters, some of which form character data, and some of which form markup. Markup encodes a description of the document's storage layout and logical structure. XML provides a mechanism to impose constraints on the storage layout and logical structure.

And if that wasn't clear enough....

The design goals for XML are:

1. XML shall be straightforwardly usable over the Internet.
2. XML shall support a wide variety of applications.
3. XML shall be compatible with SGML.
4. It shall be easy to write programs which process XML documents.
5. The number of optional features in XML is to be kept to the absolute minimum, ideally zero.
6. XML documents should be human-legible and reasonably clear.
7. The XML design should be prepared quickly.
8. The design of XML shall be formal and concise.
9. XML documents shall be easy to create.
10. Terseness in XML markup is of minimal importance.

In essence, then, the goal of XML was never to be small or fast, but still clearly simple. And, despite your personal opinion about the ecosystem that has grown up around XML (SOAP, WS-*, and so on), it's still fairly easy to defend the idea that XML itself is a simple technology, particularly if we make some basic assumptions around things that usually complicate text like character sets and encoding and such.

Note: I am deliberately ignoring the various attempts at a binary Infoset specification, which has been on the TODO list of many XML working groups in the past and has yet to really make any sort of impact in the industry. Theoretically, yes, XML Infoset-compliant documents could be rendered into a binary format that would have a much smaller, more efficient footprint than the textual representation. If and when we ever get there, it would be interesting to see what the results look like. I'm not holding my breath.

Why, then, did XML take on a role as data-transfer format if, on the surface of things, using text here was such a bad idea?

Interoperability:

"With Web services your accounting departments Win 2k servers billing system can connect with your IT suppliers UNIX server." --http://www.w3schools.com/webservices/ws_why.asp

"Conventional application development often means developing for multiple devices, and that form factors of the client devices can be dramatically different. If you base an application on a web browser display size of 800x600, it would never work on a device with a resolution of 4 lines by 20 characters. Conversely, if you took a lowest common denominator approach and sized for the smaller device, the user interface would be lost on an 800x600 device.Using a non-XML approach, this leaves us writing multiple clients speaking to the server, or writing multiple clients speaking to multiple severs ... Limitations of this approach include:

• Tightly coupled to browser
• Multiple code-bases
• Difficult to adapt to new devices
• Major development efforts
• Slow time to market

"In this case, only one application exists. It runs against the back-end database, and produces an XML stream. A "translator" takes this XML stream, and applies an XSLT transformation to it. Every device could either use a generic XSLT, or have a specialized XSLT that would produce the required device-specific output. The transformation occurs on the server, meaning that no special client capabilities are required.This "hub and spoke" architecture yields tremendous flexibility. When a new device appears, anew spoke can be added to accommodate it. The application itself does not need to be changed, only the translator needs to be informed about the existence of the new device, and which XSLT to use for it." --http://www.topxml.com/conference/wrox/2000_vegas/text/brianl_xml.pdf

Certainly the interoperability argument doesn't require a text-based format, it was just always cited that way. In fact, both the CORBA supporters and the developers over at ZeroC will both agree with Google in suggesting that a binary format can and will be an efficient and effective interoperability format. (I'll let the ZeroC folks talk to the pros and cons of their Ice format as compared to IIOP and SOAP.)

Some of the key inherent advantages in XML that are lost in this new binary format, however, center around the XML Infoset itself and the fact that it has a number of ancillary tools around it, which gets us into the next point; consider some of those inherent advantages in XML that are lost in this new binary, structured, tightly-coupled format:

• XPath. The ability to selectively extract nodes from inside a document is incredibly powerful and highly underutilized by most developers.
• XSLT. Although somewhat on the down-and-out in popularity with developers because of its complexity, XSLT stands as a shining example of how a general-purpose transformation tool can be built once the basic structure is well-understood and independent of the actual domain. (SQL is another.)
• Structureless parsing. In order to use Protocol Buffers, each side must have a copy of the .proto file that generated the proxies. (While it's theoretically possible to build an API that picks through the structure element-by-element in the same way an XML API does, I haven't found such an API in the PB code yet, and doing so would mean spending some quality time with the Protocol Buffer binary encoding format. The same was always true of IIOP or the DCOM MEOW format, too.)

In essence, a Protocol Buffer format consumer is tightly-coupled against the .proto file that it was generated against, whereas in XML we can follow Noah Mendelsohn's advice of years ago ("Schema are relative") and parse XML in whatever manner suits the consumer, with or without schema, without or without schema-based-and-bound proxy code. The advantage to the XML approach, of course, is that it provides a degree of flexibility; the advantage of the Protocol Buffer approach is that the code to produce and consume the elements can be much simpler, and therefore, faster.

Note: it's only fair to point out at this point that the Protocol Buffer approach already contains a certain degree of flexibility that earlier binary formats lacked (if I remember correctly), via PB's use of "optional" vs "required" tags for the various fields. Whether this turns out to be sufficient over time is yet to be decided, though Google claims to be using Protocol Buffers throughout the company for its own internal purposes, which is certainly supporting evidence that cannot be discarded casually.

But there's a corollary effect here, as well: because XML documents are intended to be self-descriptive, the Protocol Buffer format can contain just the data, and leave the format and structure to be enforced by the code on either side of the producer/consumer relationship. Whether you consider this a Good Thing or a Bad Thing probably stands as a good indicator of whether you like the Protocol Buffer approach or the XML approach better.

Note, too, by the way, that many of the XML-based binding APIs can now parse objects out of part of an XML document, as opposed to having to read in the whole thing and then convert--JAXB, JiBX and XMLBeans can all pick up parsing an XML document from any node beyond the initial root element, for example--whereas, at least as of this point (as far as I can tell, and I'd love to have somebody at Google tell me I'm wrong here, because I think it's a major flaw), the Protocol Buffers approach assumes it will read in the entire object from the file. I don't see any way, short of putting developer-specified "separation markers" into the stream or some other kind of encoding or convention, of doing a "partial read" of an object model from a PB data file.

To see what I mean by this, consider the AddressBook example. Suppose the AddressBook holds several thousand records, and my processing system only cares about a select few (less than five, perhaps, who all have the last name "Neward"). In a Protocol Buffer scheme, I deserialize the entire AddressBook, then go through the persons item by item, looking for the one I want. In an XML-based scheme, I can pull-mode parse (StAX in Java, or using the pull-mode XML parser in .NET, for example) the nodes, throwing away nodes until I see one where the <lastName> node contains "Neward", and then JAXB-consume the next n number of nodes into a Person object before continuing through the remainder of the AddressBook.

Let's also note that the Protocol Buffer scheme assumes working with a stream-based (which usually means file-based) storage style for when Protocol Buffers are used as a storage mechanism. But frankly, if I want to store objects, I'd rather use a system that understands objects a bit more deeply than PBs do, and gives me some additional support beyond just input and output. This gets us into the long and involved discussion around object databases, which is another one that's likely to devolve into a shouting match, so I'll leave it at that. Suffice it to say that for object storage, I can see using (for example) db4o-storing-to-a-file as a vastly more long-term solution than I can using PBs, at least for now. (Undoubtedly the benchmarks will be along soon to try and convince us one way or another.) One area that is of particular interest along these lines, though, will be the evolutionary capabilities of each--from my (limited) study of PBs thus far, I believe db4o has a more flexible evolutionary scheme, and I have to admit, I don't like the idea of having to run the codegen step before being able to store my objects, but that's a minor nit that's easily solved with tools like Ant and a global rule saying "Nobody touches the .proto file without checking with me first."

Which, by the way, brings up another problem, the same one that plagues CORBA, COM/DCOM, WSDL-based services, and anything that relies on a shared definition file that is used for code-generation purposes, what I often call The Myth of the One True Schema. Assuming a developer creates a working .proto/.idl/.wsdl definition, and two companies agree on it, what happens when one side wants to evolve or change that definition? Who gets to decide the evolutionary progress of that file? Who "owns" that definition, in effect? And this, of course, presumes that we can even get some kind of definition as to what a "Customer" looks like across the various departments of the company in the first place, much less across companies. Granted, the "optional" tag in PBs help with this, but we're still stuck with an inherently unscalable problem as the number of participants in the system grows.

I'd give a great deal of money to see what the 12,000-odd .proto files look like inside Google, and then again at what they look like in five years, particularly if they are handed out to paying customers as APIs against which to compile and link. There's ways to manage this, of course, but they all look remarkably like the ways we managed them back in the COM/DCOM-vs-CORBA days, too.

Long story short, the Protocol Buffer approach looks like a good one, but let's not let the details get lost in the shouting: Protocol Buffers, as with any binary protocol format and/or RPC mechanism (and I'm not going to go there; the weaknesses of RPC are another debate for another day), are great for those situations where performance is critical and both ends of the system are well-known and controlled. If Google wants to open up their services such that third-parties can call into those systems using the Protocol Buffers approach, then more power to them... but let's not lose sight of the fact that it's yet another proprietary API, and that if Microsoft were to do this, the world would be screaming about "vendor lock-in" and "lack of standards compliance". (In fact, I heard exactly these complaints from Java developers during WCF Q&A when they were told that WCF-to-WCF endpoints could "negotiate up" to a faster, binary, protocol between them.)

In the end, if you want an endpoint that is loosely coupled and offers the maximum flexibility, stick with XML, either wrapped in a SOAP envelope or in a RESTful envelope as dictated by the underlying transport (which means HTTP, since REST over anything else has never really been defined clearly by the Restafarians). If you need a binary format, then Protocol Buffers are certainly one answer... but so is ICE, or even CORBA (though this is fast losing its appeal thanks to the slow decline of the players in this space). Don't lose sight of the technical advantages or disadvantages of each of those solutions just because something has the Google name on it.

The Pragmatic Programmer says, "Learn a new language every year". This is great advice, not just because it puts new tools into your mental toolbox that you can pull out on various occasions to get a job done, but also because it opens your mind to new ideas and new concepts that will filter their way into your code even without explicit language support. For example, suppose you've looked at (J/Iron)Ruby or Groovy, and come to like the "internal iterator" approach as a way of simplifying moving across a collection of objects in a uniform way; for political and cultural reasons, though, you can't write code in anything but Java. You're frustrated, because local anonymous functions (also commonly--and, I think, mistakenly--called closures) are not a first-class concept in Java. Then, you later look at Haskell/ML/Scala/F#, which makes heavy use of what Java programmers would call "static methods" to carry out operations, and realize that this could, in fact, be adapted to Java to give you the "internal iteration" concept over the Java Collections:

   1: package com.tedneward.util;

   2:  

   3: import java.util.*;

   4:  

   5: public interface Acceptor

   6: {

   7:   public void each(Object obj);

   8: }

   9:  

  10: public class Collection

  11: {

  12:   public static void each(List list, Acceptor acc)

  13:   {

  14:     for (Object o : list)

  15:       acc.each(o);

  16:   }

  17: }

Where using it would look like this:

   1: import com.tedneward.util.*;

   2:  

   3: List personList = ...;

   4: Collection.each(new Accpetor() {

   5:   public void each(Object person) {

   6:     System.out.println("Found person " + person + ", isn't that nice?");

   7:   }

   8: });

Is it quite as nice or as clean as using it from a language that has first-class support for anonymous local functions? No, but slowly migrating over to this style has a couple of definitive effects, most notably that you will start grooming the rest of your team (who may be reluctant to pick up these new languages) towards the new ideas that will be present in Groovy, and when they finally do see them (as they will, eventually, unless they hide under rocks on a daily basis), they will realize what's going on here that much more quickly, and start adding their voices to the call to start using (J/Iron)Ruby/Groovy for certain things in the codebase you support.

(By the way, this is so much easier to do in C# 2.0, thanks to generics, static classes and anonymous delegates...

   1: namespace TedNeward.Util

   2: {

   3:   public delegate void EachProc<T>(T obj);

   4:   public static class Collection

   5:   {

   6:     public static void each(ArrayList list, EachProc proc)

   7:     {

   8:       foreach (Object o in list)

   9:         proc(o);

  10:     }

  11:   }

  12: }

  13:  

  14: // ...

  15:  

  16: ArrayList personList = ...;

  17: Collection.each(list, delegate(Object person) {

  18:   System.Console.WriteLine("Found " + person + ", isn't that nice?");

  19: });

... though the collection classes in the .NET FCL are nowhere near as nicely designed as those in the Java Collections library, IMHO. C# programmers take note: spend at least a week studying the Java Collections API.)

This, then, opens the much harder question of, "Which language?" Without trying to infer any sort of order or importance, here's a list of languages to consider, with URLs where applicable; I invite your own suggestions, by the way, as I'm sure there's a lot of languages I don't know about, and quite frankly, would love to. The "current hotness" is to learn the languages marked in bold, so if you want to be daring and different, try one of those that isn't. (I've provided some links, but honestly it's kind of tiring to put all of them in; just remember that Google is your friend, and you should be OK. )

• Visual Basic. Yes, as in Visual Basic--if you haven't played with dynamic languages before, try turning "Option Strict Off", write some code, and see how interacting with the .NET FCL suddenly changes into a duck-typed scenario. If you're really curious, have a look at the generated code in Reflector or ILDasm, and notice how the generated code looks a lot like the generated JVM code from other dynamic languages on an execution environment, a la Groovy.
• Ruby (JRuby, IronRuby):
• Groovy: Some call this "javac 2.0"; I'm not sure it merits that title, or the assumption of the mantle of "King of the JVM" that would seem to go with that title, but the fact is, Groovy's a useful language.
• Scala: A "SCAlable LAnguage" for the JVM (and CLR, though that feature has been left to the community to support), incorporating both object-oriented and functional concepts, plus a few new ideas, into a single package. I'm obviously bullish on Scala, given the talks and articles I've done on it.
• F#: Originally OCaml-on-the-CLR, now F# is starting to take on a personality of its own as Microsoft productizes it. Like Scala and Erlang, F# will be immediately applicable in concurrency scenarios, I think. I'm obviously bullish on F#, given the talks, articles, and book I'm doing on it.
• Erlang: Functional language with a strong emphasis on parallel processing, scalability, and concurrency.
• Perl: People will perhaps be surprised I say this, given my public dislike of Perl's syntax, but I think every programmer should learn Perl, and decide for themselves what's right and what's wrong about Perl. Besides, there's clearly no argument that Perl is one of the power tools in every *nix sysadmin's toolbox.
• Python: Again, given my dislike of Python's significant whitespace, my suggestion to learn it here may surprise some, but Python seems to be stepping into Perl's shoes as the sysadmin language tool of choice, and frankly, lots of people like the significant whitespace, since that's how they format their code anyway.
• C++: The grandaddy of them all, in some ways; if you've never looked at C++ before, you should, particularly what they're doing with templates in the Boost library. As Scott Meyers once put it, "We're a long way from Stack<T>!"
• D: Walter Bright's native-compiling garbage-collected successor to C++/Java/C#.
• Objective-C (part of gcc): Great "other" object-oriented C-based language that never gathered the kind of attention C++ did, yet ended up making its mark on the industry thanks to Steve Jobs' love of the language and its incorporation into the NeXT (and later, Mac OS X) toolchain. Obj-C is a message-passing object language, which has some interesting implications in its own right.
• Common Lisp (Steel Bank Common Lisp): What happens when you create a language that holds as a core principle that the language should hold no clear delineation between "code" and "data"? Or that the syntactic expression of the language should be accessible from within that langauge? You get Lisp, and if you're not sure what I'm talking about, pick up a Lisp or a Scheme implementation and start experimenting.
• Scheme (PLT Scheme, SISC): Scheme is one of the earliest dialects of Lisp, and much of the same syntactic flexibility and power of Lisp is in Scheme, as well. While the syntaxes are usually not directly interchangeable, they're close enough that learning one is usually enough.
• Clojure: Rich Hickey (who also built "dotLisp" for the CLR) has done an amazing job of bringing Lisp to the JVM, including a few new ideas, such as some functional concepts and an implementation of software transactional memory, among other things.
• ECMAScript (E4X, Rhino, ES4): If you've never looking at JavaScript outside of the browser, you're in for a surprise--as Glenn Vanderburg put it during one of his NFJS talks, "There's a real programming language in there!". I'm particularly fond of E4X, which integrates XML as a native primitive type, and the Rhino implementation fully supports it, which makes it attractive to use as an XML services implementation language.
• Haskell (Jaskell): One of the original functional languages. Learning this will give a programmer a leg up on the functional concepts that are creeping into other environments. Jaskell is an implementation of Haskell on the JVM, and they've taken the concept of functional further, creating a build system ("Neptune") on top of Jaskell + Ant, to yield a syntax that's... well... more Haskellian... for building Java projects. (Whether it's better/cleaner than Ant is debatable, but it certainly makes clear the functional nature of build scripts.)
• ML: Another of the original functional languages. Probably don't need to learn this if you learn Haskell, but hey, it can't hurt.
• Heron: Heron is interesting because it looks to take on more of the modeling aspects of programming directly into the language, such as state transitions, which is definitely a novel idea. I'm eagerly looking forward to future drops. (I'm not so interested in the graphical design mode, or the idea of "executable UML", but I think there's a vein of interesting ideas here that could be mined for other languages that aren't quite so lofty in scope.)
• HaXe: A functional language that compiles to three different target platforms: its own (Neko), Flash, and/or Javascript (for use in Web DOMs).
• CAL: A JVM-based statically-typed language from the folks who bring you Crystal Reports.
• E: An interesting tack on distributed systems and security. Not sure if it's production-ready, but it's definitely an eye-opener to look at.
• Prolog: A language built around the idea of logic and logical inference. Would love to see this in play as a "rules engine" in a production system.
• Nemerle: A CLR-based language with functional syntax and semantics, and semantic macros, similar to what we see in Lisp/Scheme.
• Nice: A JVM-based language that permits multi-dispatch methods, sometimes known as multimethods.
• OCaml: An object-functional fusion that was the immediate predecessor of F#. The HaXe and MTASC compilers are both built in OCaml, and frankly, it's in a startlingly small number of lines of code, highlighting how appropriate functional languages are for building compilers and interpreters.
• Smalltalk (Squeak, VisualWorks, Strongtalk): Smalltalk was widely-known as "the O-O language that all the C guys turned to in order to learn how to build object-oriented programs", but very few people at the time understood that Smalltalk was wildly different because of its message-passing and loosely/un-typed semantics. Now we know better (I hope). Have a look.
• TCL (Jacl): Tool Command Language, a procedural scripting language that has some nice embedding capabilities. I'd be curious to try putting a TCL-based language in the hands of end users to see if it was a good DSL base. The Jacl implementation is built on top of the JVM.
• Forth: The original (near as I can tell) stack-based language, in which all execution happens on an execution stack, not unlike what we see in the JVM or CLR. Given how much Lisp has made out of the "atoms and lists" concept, I'm curious if Forth's stack-based approach yields a similar payoff.
• Lua: Dynamically-typed language that lives to be embedded; known for its biggest embedder's popularity: World of Warcraft, along with several other games/game engines. A great demonstration of the power of embedding a language into an engine/environment to allow users to create emergent behavior.
• Fan: Another language that seeks to incorporate both static and dynamic typing, running on top of both the JVM or the CLR.
• Factor: I'm curious about Factor because it's another stack-based language, with a lot of inspiration from some of the other languages on this list.
• Boo: A Python-inspired CLR language that Ayende likes for domain-specific languages.
• Cobra: A Python-inspired language that seeks to encompass both static and dynamic typing into one language. Fascinating stuff.
• Slate: A "prototype-based object-oriented programming language based on Self, CLOS, and Smalltalk-80." Apparently on hold due to loss of interest from the founder, last release was 0.3.5 in August of 2005.
• Joy: Factor's primary inspiration, another stack-based language.
• Raven: A scripting language that "rips off" from Python, Forth, Perl, and the creator's own head.
• Onyx: "Onyx is a powerful stack-based, multi-threaded, interpreted, general purpose programming language similar to PostScript. It can be embedded as an extension language similarly to ficl (Forth), guile (scheme), librep (lisp dialect), s-lang, Lua, and Tcl."
• LOLCode: No, you won't use LOLcode on a project any time soon, but LOLCode has had so many different implementations of it built, it's a great practice tool towards building your own languages, a la DSLs. LOLcode has all the basic components a language would use, so if you can build a parser, AST and execution engine (either interpreter or compiler) for LOLcode, then you've got the basic skills in place to build an external DSL.

There's more, of course, but hopefully there's something in this list to keep you busy for a while. Remember to send me your favorite new-language links, and I'll add them to the list as seems appropriate.

Happy hacking!

I recently had the pleasure of meeting Bruce Wilson, a principal with iLink, and we had a pleasant conversation about enterprise applications and trends and such. Last week, in the middle of my trip to Prague and Zurich, he sent me a link to a blog entry he'd written on using Office as a front-end, and it sort of underscored some ideas I've had around Office in general.

The interesting thing is, most of the ideas he talks about here could just as easily be implemented on top of a Java back-end, or a Ruby back-end, as a .NET back-end. Office is a tool that many end-users "get" right away (whether you agree with Microsoft's user interface metaphors or not, or even like the fact that Office is one of the most widely-installed software packages on the planet), and it has a lot of support infrastructure built in. "Mashup" doesn't have to mean YouTube on your website; in fact, I dislike the term "mashup" entirely, since it sounds like something done in the heat of the moment without any planning or thought (which is the antithesis of anything that goes--or should go--into the enterprise). Can we use the term "cardinality" instead? Please?

One of the things that I like about the idea of building a DSL is the idea of users being able to express, in fairly user-friendly terms, the actions they want to take. For example, Daniel Spiewak has a great example of a DSL built in Scala using Scala's parser combinators, and the resulting text, while certainly not English, is a very readable form. But in of itself, it seems it's been a hard sell to the general community, who look at GUIs as a far more intuitive way of doing things. (Note: I disagree with this; I don't think GUIs are more intuitive, I think GUIs are more self-explanatory, once you've learned a few basic principles, like moving the mouse, clicking the button, and recognizing which elements are clickable and which aren't.)

I think I've finally figured out where an English- (or other human spoken language) driven DSL can be far more powerful and intuitive than a GUI.

Voice. Or, specifically the world of telecommunication devices as a user interface device. Not as "putting-a-GUI-on-a-phone"; I think this is a red herring and ultimately unproductive line of research, iPhones notwithstanding. I mean, literally, talking to the computer.

Imagine a field repair agent, coming off of a repair call, calling back to the office to say the repair was done: "Ticket number 451123, status complete, note Mrs Johnson really needs to stop washing her clothes in the dishwasher." Hanging up, he moves on to the next ticket in the list.

Meanwhile, on the other end, voice-analysis software has done the basic job of transforming words into a line of text, which is fed to the DSL for processing.

Or, the field agent texts the message to a company account, which again passes it directly to the DSL for further processing.

I am firmly convinced that this style of user interface--one we use every day--is the way that mobile devices should interact with enterprise systems. Forget trying to do complex GUIs on a device, forget even trying to simplify down the complex GUI into a simple GUI--just use your voice and a well-understood shared protocol (the DSL itself).

It's the ultimate thin client.

Freshly Twittering

Username is tedneward

Come follow my thoughts