Not too long ago, Don wrote:

The three most “personal” choices a developer makes are language, tool, and OS.

No.

That may be true for somebody who works for a large commercial or open source vendor, whose team is building something that fits into one of those three categories and wants to see that language/tool/OS succeed.

That is not where most of us live. If you do, certainly, you are welcome to your opinion, but please accept with good grace that your agenda is not the same as my own.

Most of us in the practitioner space are using languages, tools and OSes to solve customer problems, and making the decision to use a particular language, tool or OS a personal one generally gets us into trouble--how many developers do you know that identify themselves so closely with that decision that they include it in their personal metadata?

"Hi, I'm Joe, and I'm a Java programmer."

Or, "Oh, good God, you're running Windows? What are you, some kind of Micro$oft lover or something?"

Or, "Linux? You really are a geek, aren't you? Recompiled your kernel lately (snicker, snicker)?"

Sorry, but all of those make me want to hurl. Of these kinds of statements are technical zealotry and flame wars built. When programmers embed their choice so deeply into their psyche that it becomes the tagline by which they identify themselves, it becomes an "ego" thing instead of a "tool" thing.

What's more, it involves customers and people outside the field in an argument that has nothing to do with them. Think about it for a second; the last time you hired a contractor to add a deck to your house, what's your reaction when they introduce themselves as,

"Hi, I'm Kim, and I'm a Craftsman contractor."

Or, overheard at the job site, "Oh, good God, you're using a Skil? What are you, some kind of nut or something?"

Or, as you look at the tools on their belt, "Nokita? You really are a geek, aren't you? Rebuilt your tools from scratch lately (snicker, snicker)?"

Do you, the customer, really care what kind of tools they use? Or do you care more for the quality of solution they build for you?

It's hard to imagine how the discussion can even come up, it's so ludicrous.

Try this one on, instead:

"Hi, I'm Ted, and I'm a programmer."

I use a variety of languages, tools, and OSes, and my choice of which to use are all geared around a single end goal: not to promote my own social or political agenda, but to make my customer happy.

Sometimes that means using C# on Windows. Sometimes that means using Java on Linux. Sometimes that means Ruby on Mac OS X. Sometimes that means creating a DSL. Sometimes that means using EJB, or Spring, or F#, or Scala, or FXCop, or FindBugs, or log4j, or ... ad infinitum.

Don't get me wrong, I have my opinions, just as contractors (and truck drivers, it turns out) do. And, like most professionals in their field, I'm happy to share those opinions with others in my field, and also with my customers when they ask: I think C# provides a good answer in certain contexts, and that Java provides an equally good answer, but in different contexts. I will be happy to explain my recommendation on which languages, tools and OSes to use, because unlike the contractor, the languages, tools, and OSes I use will be visible to the customer when the software goes into Production, at a variety of levels, and thus, the customer should be involved in that decision. (Sometimes the situation is really one where the customer won't see it, in which case the developer can have full confidence in whatever language/tool/OS they choose... but that's far more often the exception than the rule, and will generally only be true in cases where the developer is providing a complete customer "hands-off" hosting solution.)

I choose to be pro-choice.

Recently I received a press announcement from Waggener-Edstrom, Microsoft's PR company, about their latest move in the interoperability space; I reproduce it here in its entirety for your perusal:

Hi Ted,

Microsoft is announcing another action to promote greater interoperability, opportunity and choice across the IT industry of developers, partners, customers and competitors. 

Today Microsoft is posting additional documentation of the XAML (eXtensible Application Markup Language) formats for advanced user experiences, enabling third parties to access and implement the XAML formats in their own client, server and tool products.  This documentation is publicly available, for no charge, at http://go.microsoft.com/fwlink/?LinkId=113699 . 

It will assist developers building non-Microsoft clients and servers to read and write XAML to process advanced user experiences – with lots of animation, rich 2D and 3D graphic and video. Specifically, non-Microsoft servers can more easily generate XAML files to be handled, for example, by applications running on Windows client machines.  In addition, non-Microsoft clients can be written more easily to interpret XAML files. This action will assist ISVs in creating design tools and file format converters to read and write XAML to create advanced user experiences.

Microsoft is making this documentation available under the Microsoft Open Specification Promise (OSP), which will allow developers of all types anywhere in the world to access and implement the XAML formats in their own client, server or tool products without having to take a license or pay a fee to Microsoft.

The following quote is attributable to Tom Robertson, general manager, Interoperability and Standards, Microsoft.

“Microsoft’s posting of the expanded set of XAML format documentation to assist third parties to access and implement the XAML formats in their own client, server and tool products will help promote interoperability, opportunity and choice across the IT community.  Use of the Open Specification Promise assures developers that they can use any Microsoft patents needed to implement all or part of the XAML formats for free, anywhere in the world, now and in the future.” 

Please let me know if you have any questions or if I can provide you with any additional information. 

Best,

N--

This marks the most recent in a slew of efforts by the Borg of the Pacific Northwest to "promote greater interoperability, opportunity and choice", and I know it's left a lot of people feeling decidedly skeptical and... well, let's just call it what it is, paranoid, about the company's plans and ulterior motive behind all these efforts. After all, this is the company that tried to co-opt Java, put Stacker out of business, used their monopoly operating system power to crush Novell, used their monopoly office suite power to crush the Mac, bribe an entire country to vote their way on the new office-file specifications, and I don't know what all else.

I know, I know, all my blog-readers who work at Microsoft are going nuts right now, protesting, claiming that this isn't the same company that they work for now, and so on. Fact is, folks, if you work at Microsoft, you work for a company whose name is not well-received in many quarters, and while some of it is undeserved... some of it is. Microsoft has done some pretty stupid things in its history, and if that reputation doesn't sit well with you now, I can't help but wonder if somewhere in that great Corporate Heaven, Stac Electronics isn't just jumping up and down, foaming at the mouth and screaming, "Ha! Serves you right!"

I don't want to use this blog as a chance for everybody who ever got burned by Microsoft (or thought they got burned by Microsoft, which is much more widespread and just as much more likely to be in their own minds) to trot out "reap and sow" cliches. Instead, I want to revisit one of my favorite topics, that of interoperability, and see exactly what this new shift in Microsoft's attitude towards interoperability really means.

Let's take these one at a time. Note that I have no "Deep Throat" at Microsoft feeding me "the Redmondagon Papers"; this is all based on my own conjecture and perspective.

What does releasing the XAML spec really mean?

Honestly, it means that now non-Microsoft platforms can try to create competitors to Aero and Windows Presentation Foundation, and have the same kind of rich client experiences that Windows users can enjoy.

Honestly, I expect this to go pretty much nowhere.

Realistically speaking, if a non-Microsoft app server wanted to generate XAML, it was a simple matter of generating the appropriate XML, tagging it with an appropriate MIME type in the HTTP header, and serving it up over an HTTP request; I've been giving this demo at conferences for three or four years now, pretty much since the first betas of WPF were stable enough to use. This really isn't rocket science.

But more importantly, XAMl has always been misunderstood: it's not a presentation format, it's an object graph format. XAML simply "wires up" a collection of objects into a tree, and it's the underlying object model that provides the functionality or power or presentation or whatever. It's an easier way of writing "Button b = new Button(...);", nothing more, nothing less. Sure, it would be nice to have some kind of equivalent for the Swing space, but doing so would tie the corresponding XML (XSML?) to the Swing APIs, just as WPF XAML is tied to the WPF API.

Does releasing the XAML specs mean that now Linux and Mac OS will get WPF features?

They've had them for years, in the guise of the OpenGL APIs, and nobody knew what to do with them, except maybe for a sliver of folks building games and interesting "effects". Unless somebody really feels the desire to try and create an adapter layer to map the WPF Button over to an OpenGL button, I really don't see much point.

This is one of the most dangerous points in the discussion: attempting to build an adapter to another platform's API is almost always a failed experiment from the day it's begun, and Microsoft's own attempt to port the MFC APIs over to the Mac OS (back in the pre-OS X days, circa 1995) were just a miserable, abject failure. Not because of any lack of intelligence on Microsoft's part, mind you, but because the two operating systems are just too different. Want to see what I mean? Bring a Mac guy and a Windows guy into the same room, and ask them each where God intended the menu bar to live.

Then creep, quietly, out of the room, before you get caught in the blood frenzy.

Why does Microsoft suddenly care about interoperability?

This is the crown jewel of the lot: why should this company, so famous for going it alone on so many issues, suddenly decide that it's important for them to embrace the other kids on the playground and make nice? Is this back to the "embrace" part of the "embrace and extend and extinguish" cycle that they're so famous for?

Partly.

To understand the point I am about to make, let's set some context.

(In other words, gather 'round, children, it's story time.)

Truth is, there was a time back there in the '90s when I think Microsoft really thought they could take over the world. COM was on the ascendancy, and it was a better platform for building software than anything else out there (at the time), particularly in the area of building rich media applications (remember when embedding a sound clip into your email message embedded inside your spreadsheet was all the rage?). The CORBA initiative was going strong, true, but its great claim to fame was to allow two remote processes to talk to one another--the rest of the CORBA "push" was in standards that either never materialized, or else materialized but turned out to be really hard to build, or use, or deploy, or all of the above. IBM's great competitor--SOM--wasn't even in beta on anything other than OS/2 (another great IBM product). Then, when DCOM shipped, it was seen by some as the final nail in the CORBA coffin; Microsoft clearly was going to "win".

Along came Java.

Java literally took the rug out from underneath the COM platform, almost overnight. It provided a platform with most of the same benefits as the COM/DCOM platform, but without having to memorize the QueryInterface rules or knowing what IUnknown was or how IDispatch was required to work or how static_cast<> and dynamic_cast<> and QueryInterface were all related. ("Would you, should you, static_cast? Not if you want your code to last..." Ah, those were heady days.) Suddenly, "mere mortals" could program on this platform, and feel a strong sense of confidence that their code would work, over time, regardless of whether they remembered to set references to null when they were done with them.

At first, Microsoft was "down with it", because in Java they saw a great marriage: the Java language as the "sweet spot" between C++'s expressive power and VB's layers of abstraction, running on top of the JVM as a "sweet spot" intermingled with the COM platform to provide the easiest, most powerful Windows programming environment yet. Visual J++ was clearly the favored child of the litter.

And then the lawyers got involved, and Sun saw their chance to steal a march on Microsoft, and maybe break the feared operating system monopolist, and maybe even get a few more percentage points for Solaris (because, after all, "Write Once, Run Anywhere" meant that you wouldn't have to run sucky operating systems like Windows and instead could trade up to real operating systems like Solaris, right? Hey, where'd that penguin come from, anyway, and why is he eating all our fish?). Sun refused to let Microsoft's marriage of the JVM (technically the MSVM) and COM take place, and Microsoft, rather than seek to fight it out, instead decided to cede the battle, and look for a battleground of their own choosing, instead. Thus was the thing that would become called ".NET" born.

But this "master plan" would take four or so years to develop, and in the meantime...

... in the meantime, EJB and Servlets and later J2EE and "app servers" and Spring and all those wonderful things that came with them, they were eating Microsoft's lunch. Comparing J2EE (even with EJB in the mix!) with the complexities of writing unmanaged COM code on top of COM+ is simply no comparison--again, the power of the managed platform simply proved to be too hard to turn away without compelling reason, and the COM/DCOM/COM+ story simply didn't have that compelling reason. Microsoft watched their "inevitable victory" sail into the sunset without them, just as the Department of Justice came up to them and shackled them with the first of many, many papers about "anti-competitive practices".

In many respects, the positions got reversed--Sun inherited a huge share (an unhealthy dose, in fact) of Microsoft's arrogance, and for a long time there, thought they were suddenly destiny's child, that Java (meaning Sun, of course) would be the one to "win", and thus would Sun's assurance of world dominance thus be assured.

Except it didn't play out that way.

Sun found that by embracing standards over implementations, they spent long hours thrashing out specifications, only to provide instant credibility to other vendors' products while their own languished. Weblogic stole the EJB early adopter window. A number of small vendors provided servlet implementations before Tomcat was born... which, although written by Sun employees, was an open-source project and yielded no financial benefit. JMS... well, JMS was always the redheaded stepchild of the J2EE family, at least until vendors like Sonic and Fiorano rescued it for the common Java programmer. (Those who'd been using IBM MQSeries all the while never really could see why you'd want to program against JMS APIs instead of IBM's own.) In each and every case, Sun found their product to be the third or fourth entry into the race, usually years after the others had started, and as a result....

Meanwhile, back in Redmond....

Microsoft comes to the game with .NET in 2003. (The early betas don't count because many people openly wonder if Microsoft is really serious about this ".NET" thing in the first place. After all, remember Microsoft Bob?) And despite .NET's obvious advantage of being formulated nearly a decade after Java's initial release, thus able to apply hindsight to fix or improve the obvious blemishes in the Java environment, Microsoft finds that they're playing catch-up in the all-too-important enterprise space. Microsoft's tools and products have always been seen as "second-class citizens" to the "big boys" in the enterprise space, particularly at the ends of the "high scale" continuum, and the lack of an obvious "app server" in the .NET arena only serves to underscore and reinforce that opinion among many large firms.

More importantly, Microsoft doesn't ever want to get blindsided by the Java experience again. They want to make sure that they are never in a position where it looks like their tools are vastly out-of-date, underfeatured, underpowered, and underused. They need to remain somewhere near the bleeding edge, but not so close that their customers are the ones doing the bleeding.

(We pause for the inevitable Vista joke.)

To Microsoft, Java is that near-death experience that pulls many adrenaline (and other) junkies back from the brink they so callously teetered on before. They need some kind of forward progress, some kind of advancement in the game, so that their customers and their would-be customers feel like Microsoft is on top of it at all times.

Result: Somewhere in the 2000-2003 timeframe, Microsoft looks around, sees the landscape, and realizes it needs to make itself relevant to a largely J2EE-based universe, and fast.

At first, Microsoft sees a play through the establishment of some standards between the big vendors, around this new "XML" thing, a largely portable data format, and so they throw themselves heart and soul into that space. Doing so will allow them to show existing J2EE-based shops that the power of the .NET platform lies in complementing the existing infrastructure, not replacing it. (Microsoft is smart enough to realize that preaching the software equivalent of hellfire-and-brimstone, known as "rip-and-replace", will not cater well to this congregation.)

(Rubyists could have learned a valuable lesson here, but either weren't paying attention, didn't realize the value of the lesson, or else just chose not to.)

But this play doesn't turn out the way they expect: the WS-* standards become top-heavy, and start to resemble the very thing Microsoft sought to smash fifteen years earlier: CORBA. The number of WS- specifications available through the W3C (and OASIS, and WS-I and whatever other industry consortiums are formed) is exceeded only by the number of Cos- specifications available from the OMG. The complexities therein leave many Java--and .NET--programmers confused, bewildered, and hopelessly lost when trying to get all but the most simple services to work. Thus does the community turn to alternatives--JSON, simple sockets, REST, whatever--to try and find something that works, even if it only addresses a subset of the problems they will eventually face.

Meanwhile...

Open source grows ever more important, and Microsoft-the-company realizes they have to either kill it or join it. It's hard to kill something that has no body (unlike their previous competitors), so joining it is the only viable option. Unlike many other software product companies, however, Microsoft has too large an established software base to just "flip the switch", and has far too deeply entrenched a corporate community to take any kind of radical action without a well-thought plan. (Wall Street, a place few programmers ever bother to consider, much less visit, would not take kindly to Microsoft essentially giving away their core product without something in its place to generate revenue, and regardless of how many programmers would like to imagine a world with a bankrupt Microsoft, this would be bad for business for everybody.)

And thus do we come to the present.

Microsoft needs a play that is Wall Street friendly, programmer friendly, and corporate friendly. They are slowly flirting more and more deeply with open source, yet still firmly committed to turning a profit (something a few of these other open-source-based companies should probably learn to do at some point--just maneuvering to the point of being bought out by a larger fish, like Oracle, is not really a long-term competitive strategy, just so you know).

Microsoft wants--arguably, needs--to keep Office relevant in a world where software isn't always paid for, so they need a play that keeps Office ubiquitous and out in the forefront of developer mindshare. If they can't get you to buy Office, then at least let's get you to use tools that keep the Office file formats ubiquitous. If (and this is a big "if") the Office formats turn out to be technically superior to their competition, then Microsoft succeeds. If not, they find a new play.

In the short, Microsoft needs an interoperability story, and they need a real interoperability play, because their reputation is damaged from the many "embrace, extend, extinguish" plays they've made in the past. The era of a large vendor "winning" is clearly well behind us (if it was ever, in fact, more than just a marketing VP's wet dream), and if Microsoft is going to make sure that they're never in a vulnerable come-from-behind position again, they need to make sure that they can work well with all the other new technologies out there, whether up-and-coming or well-established or even fading-fast. They need to have an interoperability story that developers can believe in, which means some kind of open-source-friendly play, and one that carries serious "street cred" for actually working.

What's the lesson that I, a developer, take away from this?

If you are a Java developer, get past your old prejudices and accept that .NET is a viable platform. The Java developer who refuses to learn how to write C# code on the grounds that "Micro$oft is a company that just puts out crap" or that "M$FT sux" is going to be a Java developer whose value to the business is reduced compared to those with less virulent politics. Thanks to tools like VMWare and Virtual PC, you don't have to give up your Mac or your Linux environment to write .NET code and prove that you can offer value to those projects that need to talk to .NET. Look into more than just the WS-* or REST stacks for communication, as well; explore some of the interoperability options I've been ranting about for four years, a la IKVM, Jace, Hessian, even CORBA.

If you are a Ruby developer, get over yourself and your "we're more agile and more powerful" meme. Ruby is a tool, nothing more, and one whose shine is fast coming off. IT organizations are discovering the myriad problems with the original Ruby runtime, and are unwilling to risk enterprise apps on a runtime that has zero monitoring and zero manageability play. Yes, you can certainly do lots of things yourself to make your Ruby apps more manageable and more monitorable--but that's all time you have to spend building it, or figuring out how to hook it into the existing IT infrastructure, and when all that time gets added up, it's not going to look all that different from a Java or .NET app's timecycle arc. If you don't have an answer to the question, "How will we make this work with the existing infrastructure we've got?", then you have a problem, and no amount of chanting "Obi-Dave Thomas-Kenobi, you and dynamic typing are my only hope" will save you.

If you are a .NET developer, it's high time you accepted that the Java folks are about five years ahead of you on this "managed code" arc, and that they suffered through a lot of hard lessons before arriving at the decisions they came to. Don't be stupid, learn from their mistakes. Why do Java programmers chant "dependency injection" with holy fervor? Why do Java programmers put so much stress on unit testing? What has Microsoft not given you with the latest release of Visual Studio that Java developers think you're an idiot for not demanding in the next release? Yes, C# has some interesting new features in it that Java-the-language doesn't have... but why are the Java guys getting all misty-eyed over Groovy? What do they know that you don't?

If you are a developer outside of these areas, you're swimming in dangerous waters, because while I'm sure you're not having any problems finding a job, chances are your next job is going to require you to talk to one of those three environments. Better have your integration/interoperability story worked out, whether its Phalanger for the PHP developer who needs to talk to .NET (and damn if PHP script driving a WinForms app isn't an interesting idea in of itself... and a useful way to bridge yourself into an entirely new area of employment), or its figuring out how to apply your mad Haskell skillz to F# or Scala, you need to have a good idea of what those languages are (and aren't) and how your knowledge of functional concepts can catapult you to the head of the class the next time a massively-scalable system needs to be built.

If you are a Microsoft employee, don't blow this. Don't make this into another "embrace, extend, extinguish" cycle. Accept that your company made some bone-headed maneuvers in the past, and rather than try to defend them, accept that your reputation outside of the Redmond Reality-Distortion Bubble is not what it looks like from the inside. As hard as this will be to do sometimes, just stop and listen to what others are saying about the company and the paranoia that creeps up every time Microsoft moves into an area of interest. Take the extra moment to hear the concerns, not just the words.

And if you are a Google employee, tatoo this on your forehead: Reputation Matters. The first time anybody at your company does something even remotely "evil", you will be branded as "the next Microsoft" and all of these problems will be yours to share and enjoy, as well.

Apparently, I'm drawing enough of an audience through this blog that various folks have started to send me press releases and notifications and requests for... well, I dunno exactly, but I'm assuming some blogging love of some kind. I'm always a little leery about that particular subject, because it always has this dangerous potential to turn the blog into a less-credible marketing device, but people at conferences have suggested that they really are interested in what I think about various products and tools, so perhaps it's time to amend my stance on this.

With that in mind, if you are a vendor and have a product that you'd like me to take a look at and (possibly) offer up a review here, here's the basic rules:

1. No guarantees. Sending me something will in no way guarantee that I will review your product, for several reasons, two of which being (a) I get really busy sometimes, and (b) I may have no interest whatsoever in your product and I refuse to pretend to do so. (Readers can usually tell when the reviewer isn't all that excited about the subject, I've found.)
2. If you're not going to send me a "real" version (meaning not the time-locked or feature-crippled demo), don't bother. I have no idea when I will get around to a review, and I have no desire to review something that isn't "the real deal". I will in turn promise that the licensed version you send me (if necessary) will not be used for any purpose other than my own research and exploration (signing contract if necessary to give you that "fresh-from-the-lawyer's-office" warm and fuzzy feeling).
3. I say what I think, pro and con. I will not edit my review to suit your marketing purpose, and if you ask me to do so I will simply note in the review that you have asked me to do so. I retain full editorial control over what I say about your product.
4. Having established #1, I will try to be as fair as I can about your product, and point out things that I liked and things that I didn't. (Of course, if I hated it from top to bottom, I may end up with the only positive thing being "It didn't set the atmosphere on fire when I started the app", but hey, that's something positive, right?)
5. Also in the spirit of #1, if you send me mail answering questions or complaints in my review, I will of course amend the review with your comments. You are always welcome to post comments to the blog entry itself, too. Unless you insult my grandmother, then I will have to get all DELETE-key on you.

The reason I'm posting this here is twofold: one, so my faithful audience of four blog readers will know the rules under which I'm looking at these products and (hopefully) realize that I'm not financially vested in any of these products, and two, so the various vendor folks can read this and know what the rules are up front before even asking.

I know it sounds a little cheeky to lay this out. The image I get in my head is that of the kid at Christmas declaring to his grandparents as they walk through the door, presents in hand, "Make sure it's not a scratchy sweater, I hate scratchy sweaters. And G.I. Joe was only popular when my Dad was a kid. And if you give me another lunchbox I will scream until you buy me something cool, like a new GameBoy." Ugh. But I value the trust that people seem to have in me, and so I risk the perception of cheekiness for this tiny window in time in order to (hopefully) establish full disclosure over the reviews that come to pass (which, by the way, will always have the category "review" applied to them, so you know which is an official review and which is just me exploring, like the LLVM and Parrot posts of recent time).

We now return you to the regularly-scheduled blog.

A couple of people have asked me over the last few weeks, so it's probably worth saying out loud:

No, I don't work for a large company, so yes, I'm available for consulting and research projects. If you've got one of those burning questions like, "How would our company/project/department/whatever make use of JRuby-and-Rails, and what would the impact to the rest of the system be", or "Could using F# help us write applications faster", or "How would we best integrate Groovy into our application", or "How does the new Adobe Flex/AIR move help us build richer client apps", or "How do we improve the performance of our Java/.NET app", or other questions along those lines, drop me a line and let's talk. Not only will I cook up a prototype describing the answer, but I'll meet with your management and explain the consequences of the research, both pro and con, for them to evaluate.

Shameless call for consulting complete, now back to the regularly-scheduled programming.

During the Lang.NET Symposium, a couple of things "clicked" all simultaneously, giving me one of those "Oh, I get it now" moments that just doesn't want to leave you alone.

During the Intentional Software presentation, as the demo wound onwards I (and the rest of the small group gathered there) found myself looking at the same source code, but presented in a variety of new ways, some of which appealed to me as the programmer, others of which appealed to the mathematicians in the room, others of which appealed to the non-programmers in the room. (I heard one of the Microsoft hosts, a non-technical program manager, I think, say, "Wow, even I could understand that spreadsheet view, and that was writing code?")

During the spreadsheet-written-in-IronPython presentation (ResolverOne), we were essentially looking at new ways of writing IronPython code, thus leveraging all the syntactic power of a programming language with a nicer front end.

During the aspect-oriented talk (the one by Stefan Wenig and Fabian Schmeid), we found ourselves looking at a tool that essentially takes compiled assemblies and weaves in additional code based on descriptors from outside that codebase; in essence, just another aspect-oriented tool.

But combine this with my own investigations into Soot, LLVM, Parrot, and Phoenix, alongside the usual discussions around the DLR, CLR, JVM and DaVinci machine, couple that with the presentation Harry gave about parser expression grammars and the research in the functional community into parser combinators, throw in the aspect-oriented and metaprogramming facilities that the Rubyists and other dynamic linguists go on for days about, and what do you end up with?

Folks, the future is in modular toolchains.

This is an oversimplification, and a radical oversimplification at that, but imagine for a moment:

1. A parser takes your source code (let's assume it is Java, just for grins) and builds an AST out of it. Not an AST that's inherently deeply coupled to the Java language, mind you, but a general-purpose one that stands as a union of Java, C#, C++, Perl, Python, Smalltalk, and other languages. (Note that some of the linguistic concepts in some of those languages may not end up in this AST, but instead operate on the AST itself, a la C++'s template facilities.) Said parser is now finished, and can either output a binary (or potentially XML, though it'd probably be hideously verbose) version of this AST to disk for later consumption, or would more than likely be passed directly along to the next beast in the chain.
2. In the simplest scenario, the next beast would be a code generator, which takes the AST and seeks to export some kind of back-end code out of it. Here, since we're working with a general-purpose AST, we can assume that this back-end is flexible and open, a la the Phoenix toolkit (where either native or MSIL can be generated).
3. In a slightly more complicated scenario, verification of the correctness of the AST (against whatever libraries are specified) is checked, usually prior to code-gen, thus making this particular toolchain a statically-checked chain; were verification left out, it would need to happen at runtime, in which case we'd be talking about a dynamically-checked chain.

   Note that I stay away from the term "statically-typed" or "dynamically-typed" for the moment. That would be a measurement of the parser, not the verifier. Verification still occurs in a lot of these dynamically-typed languages, just as it does in statically-typed languages.

   Assuming the verification process succeeds, the AST can be again, written out or passed to the next step in the chain.
4. Another potential step in the process, usually post-parser and pre-verification, would be an "aspect" step, in which a tool takes the AST, consults some external descriptors, and modifies the AST based on what it finds there. (This is how most of your non-AspectJ-like AOP tools work today, except that they have to rebuild the AST from compiled .class files or assemblies first.)
5. Naturally, another step in the process would be an optimize step, but this has to be considered carefully, since some "high-level" optimizations can be done without regard to code-gen backend, and some will need to be done with regard to code-gen backend; for example, register spill is (from what I've heard, can't say I know too much about this) generally only useful if you know how many registers you're targeting. Plus, it's not hard to imagine certain optimizations that are only generally useful on the x86 architecture, versus those that are useful on other CPU platforms. Even operating systems I would imagine would have an impact here. (It turns out that many compiler toolchains go through a dozen or so optimization steps today, so it's not hard to imagine a "code-gen backend" being a series of a half-dozen or so targeted optimization steps before actually generating code.)
6. Bear in mind, too, that these ASTs should have enough information to be directly executable, thus giving us an interpreter back-end instead of a code-generation back-end, a la the DLR instead of the CLR.
7. Also, given the standard AST format, it would be relatively trivial to create a whole series of different "parser"s to get to the AST, along the lines of what the Intentional Software guys have created, thus blowing open the whole concept of "DSL" into areas that heretofore have only been imagined. You still get the complete support of the rest of the toolchain, which is what makes the whole DSL concept viable in the first place, including aspects and verification and your choice of either interpretation or compilation.
8. While we're at it, bear in mind that this AST could/should also be reachable from within the code itself, thus giving languages that want to operate on their own AST at runtime the ability to do so, because the AST is in a standard format and the interpreter could be bundled as part of the generated executable, thus providing a compile-when-you-can-interpret-when-you-must flavor that is currently the reigning meme in language/platform environments like JRuby. (It would also have the happy side effect of making Paul Graham shut up about Lisp, at least for a while. Yes, Paul, code-as-data, it's brilliant, it's wonderful, we get it.)
9. Nothing says this toolchain needs be one-way, by the way: many of the toolkits I mentioned before (LLVM, Phoenix, Soot) can start from compiled binary and work back to AST, thus offering us the opportunity to do surgery of either the exploratory kind (static analysis) or the manipulative kind (aspect-weaving, etc) on compiled code in a relatively clean way. Reflector demonstrates the power of being able to go "back and forth" in this way (even in the relatively limited way Reflector does so), so imagine how powerful it would be to do this from end-to-end throughout the toolchain.

How likely is this utopian vision? I'm not sure, honestly--certainly tools like LLVM and Phoenix seem to imply that there's ways to represent code across languages in a fairly generic form, but clearly there's much more work to be done, starting with this notion of the "uber-AST" that I've been so casually tossing around without definition. Every AST is more or less tied to the language it is supposed to represent, and there's clearly no way to imagine an AST that could represent every language ever invented. Just imagine trying to create an AST that could incorporate Java, COBOL and Brainf*ck, for example. But if we can get to a relatively stable 80/20, where we manage to represent the most-commonly-used 80% of languages within this AST (such as an AST that can incorporate Java, C#, and C++, for starters), then maybe there's enough of a critical mass there to move forward.

Now all I need to do is find somebody who'll fund this little bit of research... anybody got a pile of cash they don't know what to do with?

Update: By the way, in case you want a graphical depiction of what I'm thinking about, the Phoenix page has one (though obviously it's limited to the Phoenix scope of vision, and you may have to be a Microsoft CONNECT member to see it).

My eight-year-old son, a few months ago, asked me what it is I do. I tried to explain to him that Daddy works as a consultant, teaching people how to build computer systems that help people do things. He thought about it a moment, then said, "So you build robots and stuff?" No, not exactly, I build software, which controls the computers. "So you program the robots to do things?" No, I build software like what runs Amazon or eBay. "So you build websites?" At which point, wisdom dawned on me, and I said, "Yes, I build websites."

He thought about it a moment, then said, "Then how come your website is so boring?"

With the coming of the new year comes a change in my professional life. Starting on 11 Feb, I will be working as a technical consultant to Cie Studios, an "interactive and entertainment and marketing company", which is about as far away from my traditional consulting client as I can get without leaving the industry completely.

You see, Cie focuses mostly on front-end, high-gloss kinds of graphical UI things. I focus mostly on back-end, deep-in-the-bowels kinds of plumbing things. They use lots of Flash and other animation tools. I haven't figured out how to draw anything more sophisticated than a stick figure (and believe me, my kids laughed at me last time I drew them in stick figures.) They make things like Nitto 1320 Legends, a free online combination of racing and social networking. I make things like HR systems for big corporations. My parents thought the Cie website was cool and attractive; they barely understand what a "high-scale transactional enterprise system" does, much less why anybody would pay for somebody to help them build it.

Talk about your odd couples.

Nevertheless, I've found a nearly-full-time home for a while, and we're all pretty excited about the partnership. The project I'm working on? Can't say much about it now, but suffice it to say, Cie is looking to leverage my love for programming language design & implementation in a new entertainment project.... which, of course, my kids are excited about, because for the first time they'll actually have something they can look at that Dad built. (Actually, I'm kinda excited about that part, too.)

The tradeoff here is obvious: they teach me about Flash and making user interfaces that are more exciting than my usual console application front-end, and I teach them... uh... I teach them... let's see.... well, anyway, they're happy with the arrangement.

Fortunately, they're also happy with my extracurricular activities (such as NFJS and TechEd, among others), which means, beyond the prospect of being incredibly busy this year, that I may end up doing something a little bit more... flashier... this year on the speaking circuit (pun intended).

Meanwhile, look to the blog for more on programming languages (including but not limited to Clojure, Groovy, Ruby, ES4, F# and Scala), virtual machines (particularly the JVM and CLR), and maybe a little bit on programming the MacOS (as I figure it out myself).