In the latest Redmond Developer News, William Zachmann writes "Game programming is fundamental to understanding where software development is headed in the years ahead", which is a position I happen to believe quite strongly myself. And then...

... then he says absolutely nothing at all.

Oh, there's a couple of book recommendations, two paragraphs about how the techniques of game programming mirror the development of the GUI in the 80s and 90s, and since GUIs obviously became important in time, so will game programming. What parts of game programming, you ask? Why, just this list:

Full 3-D modeling, person and vehicle animation, scripting, textures, lighting effects, object physics, particle effects, voice and video creation and streaming, plotting, goal setting and scoring, scenario building, player interaction strategies, lighting effects, heads-up displays (HUDs), object rendering, damage-level maintenance, artificial intelligence (AI) and virtual-reality rendering are just a few of the component technologies that go into game creation and development. Any one of them can be a totally absorbing learning experience all in itself. Mastering game development requires learning about them all -- and more.

Frankly, the whole article was essentially fluff. Zero in the way of logical defense to his argument, and zero in the way of prescriptive advice, aside from "Learn it all, my son, learn it all."

So here's how I think the article should have read:

Only a game? Think Again (the Ted Neward Version)

Developing enterprise software has never been an easy task, and the demands of corporate IT departments in the next decade are only going to get more stringent. Users demand snappier user interfaces, more expressive displays of data and information, higher performance and scalability, much better interaction among the various user- and machine-driven nodes in the network, and more and more "assistance" from the software to get users from "A" to "B" without having to do all the grunt work themselves. (It's a tough job, moving the mouse, clicking it, moving it some more, clicking again and again and again.... And Lord, then you have to type on the keyboard.... It's amazing the average IT knowledge worker doesn't draw hazard pay.)

So where does the enterprise developer find the skills necessary to stand out in the 2010s?

From his free-wheeling high-flying long-haired pizza-snorting DietCoke-mainlining cousin over in the entertainment software industry, of course.

Consider, if you will, the best-selling game World of Warcraft, not from a point of view that describes the domain of the software, but from its non-domain requirements, what some people also refer to as the non-functional requirements:

1. Performance: if the software behaves sluggishly, users will complain and quit using the software, which directly affects their bottom line: they charge access fees on an hourly basis.
2. Security: if users can hack the software to grant themselves higher access or change their data (gaining more gold, items, whatever), users will complain and quit using the software. (This is a huge deal, by the way--an entire economy has sprung up around MMORPGs like WoW, where people will pay real-world money--or other real-world currency--for WoW-world goods and services. If attackers can alter their WoW accounts, that can translate directly into hard real-world cash.
3. Scalability: the more simultaneous users, the more cash in Blizzard's pocket.
4. Concurrency: these users are all interacting in sub-second timeframes with each other and the rest of the system, so accuracy of information exchange is critical.
5. Portability: the more systems the software can run on, the more potential users the software can attract (and, again, the more cash in Blizzard's pocket in return).
6. User Interface: a tremendous amount of information needs to be available to the user at a moment's notice, and a huge variety of options must be quickly and easily selectable/actionable.
7. Extensibility: users will need new and different elements (scenarios/quests, character types, races, worlds, and so on) in order to stay interested in using the software. (This isn't generally a problem with enterprise software, since it's not like you're going to be excited about using the HR system anyway, but extensibility there is still going to look a lot like extensibility here.)
8. Resiliency: In the inevitable event of a crash, data must not be lost, or users will be... miffed... to say the least. Clear distinctions between transient and durable data must be drawn, and must be communicated to the user, so as to manage expectations accordingly. And it goes without saying that if a server (or server farm) goes down, it must come back up or be hot-swapped with another server/farm as quickly as humanly possible.

No doubt hard-core gamers could come up with a variety of other features that would--once the gaming domain is removed from them--be recognizable to the enterprise developer. Naturally, the entertainment industry has other areas that generally a software developer doesn't run into--physics modeling and what-not--but surprisingly a great deal of the modern video game can, and undoubtedly will, make its way into the enterprise software arena. Some thoughts that come to mind:

1. Animation. Apple has certainly been at the forefront of incorporating animation into user interface, but this is just the tip of the iceberg. Particularly for software that will reach out to the general public, first impressions mean a great deal, and a UI that grabs your attention without being overly dramatic will leave users with warm fuzzies and fond memories. This doesn't even begin to consider the more practical applications of animation, such as a travel reservations system providing a map with your trip itinerary graphically plotted, with zoom-in/zoom-out effects as you work with different parts of the trip (zoom out to look at the air routing, zoom in to look at a particular city for the hotel options, and so on).
2. User interface paradigms. The modern video game, particularly those that involve deeper strategic and tactical thought (a la the real-time strategy game like Command & Conquer), make heavy use of the heads-up display, or HUD, to provide a small-real-estate control panel that doesn't distract from the main focus of the user's attention (the map). Microsoft has started to work with this idea some, with the new Office 2007 taking a very different approach to the ubiquitous menubar-across-the-top, going for what they call "ribbon" elements that fly out and fly back, much the same way that the HUD does in C&C. Also something to consider is the map navigation system, where simply moving the mouse to the far edge of the screen starts scrolling in that direction. Consider this before you dismiss the idea: horizontal scrolling is completely verboten in the word processing app, yet we do it all the time (without too much complaint) in the modern RTS. Why? I submit to you that it is because scrolling is so much easier in the RTS than it is in Word/Excel/whatever.
3. Player-to-computer interaction. This is different from UI in that the computer often has to masquerade as a player, and in order to do so in strategy games (a la Civilization IV), the programmers typically limit the interaction to very specific statements. Now consider natural language parsing (an offshoot branch of AI research), which can take English statements, break them down, analyze them and respond according to the content of the statement. How much easier would it be for users to say, "Show me all the unsold merchandise for the Northern California region for the years 2005 to 2006", rather than "SELECT * FROM merchandise WHERE ..." or navigating a complex report form?
4. Speech and sound. Consider the user who is blind, or is missing digits from either or both hands. How useful is a computer then? Now consider the same user who can speak to the machine (a la the natural language point above) or converse with the machine, as blind people do with other people, every day. Not everything has to be presented visually--I eagerly await the interaction of cell phones to Interactive Voice Response systems that are backed by a natural language parser. It's coming, folks.
5. Scripting languages. Most games are built as game engines written by programmers, with scenarios or missions or quests (or whatever) written in some kind of scripting engine and/or scenario editor. This is the epitome of the domain-specific language, and was done to allow the non-technical knowledge worker (the game designer and playtest leads) to be able to adjust the scenarios without requiring complex development steps.
6. Explosions and "ka-boom" sound effects. Well... I suppose you could get one of these when you deleted an employee from the system, but that's just getting a little gratuitous.

The point is, all of these things, and more, could--and I submit, will--radically change how we build business software. And considering that most game development isn't about twiddling assembly instructions but writing in modern high-level languages (native C++ being the most common, with Java and C# bringing up a close-and-rapidly-growing second), complete with high-level abstractions and libraries to handle the ugly details (including lighting effects, object interaction, and more), it's fast becoming reasonable to learn these skills without having to throw away everything the enterprise developer already knows.

As for resources, a trip down to your local computer book store or Amazon will yield a plethora of game-related titles, some of which focus on the details of 3D graphics, others of which focus on game design (the actual modeling of the game domain itself--how many units, hit points, etc). One interesting series to consider picking up is "Game Programming Gems", which are collections of short essays on a huge variety of topics--including the recently-discovered concept of "unit testing" that the entertainment industry has just picked up.

So yes, we have a few things we can contribute to them, as well. *grin*

And besides, it'll finally be nice to explain to your non-technical friends and family what you do for a living. "Well, you see this? I wrote this..." will generate "oohs" and "aahs" rather than "Um... that's just text on a screen, what did that do?"

People visiting my house have commented from time to time on the fact that at my house, there's no WEP key or WPA password to get on the network; in fact, if you were to park your car in my driveway and open up your notebook, you can jump onto the network and start browsing away. For years, I've always shrugged and said, "If I can't spot you sitting in my driveway, you deserve the opportunity to attack my network." Fortunately, Bruce Schneier, author of the insanely-good-reading Crypto-Gram newsletter, is in the same camp as I:

My Open Wireless Network

Whenever I talk or write about my own security setup, the one thing that surprises people -- and attracts the most criticism -- is the fact that I run an open wireless network at home.

There's no password. There's no encryption. Anyone with wireless capability who can see my network can use it to access the internet.

To me, it's basic politeness. Providing internet access to guests is kind of like providing heat and electricity, or a hot cup of tea. But to some observers, it's both wrong and dangerous.

I'm told that uninvited strangers may sit in their cars in front of my house, and use my network to send spam, eavesdrop on my passwords, and upload and download everything from pirated movies to child pornography. As a result, I risk all sorts of bad things happening to me, from seeing my IP address blacklisted to having the police crash through my door.

While this is technically true, I don't think it's much of a risk. I can count five open wireless networks in coffee shops within a mile of my house, and any potential spammer is far more likely to sit in a warm room with a cup of coffee and a scone than in a cold car outside my house. And yes, if someone did commit a crime using my network the police might visit, but what better defense is there than the fact that I have an open wireless network? If I enabled wireless security on my network and someone hacked it, I would have a far harder time proving my innocence.

This is not to say that the new wireless security protocol, WPA, isn't very good. It is. But there are going to be security flaws in it; there always are.

I spoke to several lawyers about this, and in their lawyerly way they outlined several other risks with leaving your network open.

While none thought you could be successfully prosecuted just because someone else used your network to commit a crime, any investigation could be time-consuming and expensive. You might have your computer equipment seized, and if you have any contraband of your own on your machine, it could be a delicate situation. Also, prosecutors aren't always the most technically savvy bunch, and you might end up being charged despite your innocence. The lawyers I spoke with say most defense attorneys will advise you to reach a plea agreement rather than risk going to trial on child-pornography charges.

In a less far-fetched scenario, the Recording Industry Association of America is known to sue copyright infringers based on nothing more than an IP address. The accused's chance of winning is higher than in a criminal case, because in civil litigation the burden of proof is lower. And again, lawyers argue that even if you win it's not worth the risk or expense, and that you should settle and pay a few thousand dollars.

I remain unconvinced of this threat, though. The RIAA has conducted about 26,000 lawsuits, and there are more than 15 million music downloaders. Mark Mulligan of Jupiter Research said it best: "If you're a file sharer, you know that the likelihood of you being caught is very similar to that of being hit by an asteroid."

I'm also unmoved by those who say I'm putting my own data at risk, because hackers might park in front of my house, log on to my open network and eavesdrop on my internet traffic or break into my computers.

This is true, but my computers are much more at risk when I use them on wireless networks in airports, coffee shops and other public places. If I configure my computer to be secure regardless of the network it's on, then it simply doesn't matter. And if my computer isn't secure on a public network, securing my own network isn't going to reduce my risk very much.

Yes, computer security is hard. But if your computers leave your house, you have to solve it anyway. And any solution will apply to your desktop machines as well.

Finally, critics say someone might steal bandwidth from me. Despite isolated court rulings that this is illegal, my feeling is that they're welcome to it. I really don't mind if neighbors use my wireless network when they need it, and I've heard several stories of people who have been rescued from connectivity emergencies by open wireless networks in the neighborhood.

Similarly, I appreciate an open network when I am otherwise without bandwidth. If someone were using my network to the point that it affected my own traffic or if some neighbor kid was dinking around, I might want to do something about it; but as long as we're all polite, why should this concern me? Pay it forward, I say.

Certainly this does concern ISPs. Running an open wireless network will often violate your terms of service. But despite the occasional cease-and-desist letter and providers getting pissy at people who exceed some secret bandwidth limit, this isn't a big risk either. The worst that will happen to you is that you'll have to find a new ISP.

A company called Fon has an interesting approach to this problem. Fon wireless access points have two wireless networks: a secure one for you, and an open one for everyone else. You can configure your open network in either "Bill" or "Linus" mode: In the former, people pay you to use your network, and you have to pay to use any other Fon wireless network.

In Linus mode, anyone can use your network, and you can use any other Fon wireless network for free. It's a really clever idea.

Security is always a trade-off. I know people who rarely lock their front door, who drive in the rain (and, while using a cell phone), and who talk to strangers. In my opinion, securing my wireless network isn't worth it. And I appreciate everyone else who keeps an open wireless network, including all the coffee shops, bars and libraries I have visited in the past, the Dayton International Airport where I started writing this, and the Four Points Sheraton where I finished. You all make the world a better place.

I'll admit that he's gone to far greater lengths to justify the open wireless network than I; frankly, the idea that somebody might try to sit in my driveway in order to hack my desktop machine and store kitty porn on it had never occurred to me. I was always far more concerned that somebody might sit on my ISP's server, hack my desktop machine's IP from there and store kitty porn on it. Which is why, like Schneier, I keep any machine that's in my house as up to date as possible. Granted, that doesn't protect me against a zero-day exploit, but if an attacker is that determined to put kitty porn on my machine, I probably couldn't stop them from breaking down my front door while we're all at work and school and loading it on via a CD-ROM, either.

And, at least in my neighborhood, I can (barely) find the signal for a few other wireless networks that are wide open, too, so I know I'm not the only target of opportunity here.So the prospective kitty porn bandit has his choice of machines to attack, and frankly I'll take the odds of my machines being the more hardened targets over my neighbors' machines any day. (Remember, computer security is often an exercise in convincing the bad guy to go play in somebody else's yard. I wish it were otherwise, but until we have effective response and deterrence mechanisms, it's going to remain that way for a long time.)

I've known a lot of people who leave their front doors unlocked--my grandparents lived in rural Illinois for sixty some-odd years in the same house, leaving the front door pretty much unlocked all the time, and the keys to their cars in the drivers' side sun shade, and never in all that time did any seedy character "break in" to their home or steal their car. (Hell, after my grandfather died a few years ago, the kids--my mom and her siblings--descended on the place to get rid of a ton of the junk he'd collected over the years. I think they would have welcomed a seedy character trying to make off with the stuff at that point.)

Point is, as Schneier points out in the last paragraph, security is always a trade-off, and we must never lose sight of that fact. Remember, dogma is the root of all evil, and should never be considered a substitute for reasoned thought processes.

And meanwhile, friends, when you come to my house to visit, enjoy the wireless, the heat, and the electricity. If you're nice, we may even let you borrow chair for a while, too.

A couple of people have left comments that definitely deserve response, so here we go:

Glenn Vanderberg comments in response to the Larraysaywhut? post, and writes:

Interesting post, Ted ... and for the most part I agree with your comments.  But I have to ask about this one:

Actually, there are languages that do it even worse than COBOL. I remember one Pascal variant that required your keywords to be capitalized so that they would stand out. No, no, no, no, no! You don't want your functors to stand out. It's shouting the wrong words: IF! foo THEN! bar ELSE! baz END! END! END! END!

[Oh, now, that's just silly.]

Seriously?  You don't think Larry has a point there?  That's one of the primary things I always hated about Wirth's languages, for exactly the reason cited here.  Most real-world Pascal implementations relaxed that rule to recognize upper- and lowercase keywords, but he didn't learn, making the same horrible mistake in Modula-2 and Oberon.

Capitalized words draw your attention, and make it hard to see the real code in between.

Rather than disagree with him, I agree with Larry: uppercased keywords, in a language, are just SOOOO last-century. But so is line-numbering, declaration-before-use, and hailing recursion as a feature. It just seems silly to put this out there as a point of language design, when I can't imagine anyone, with the possible exception of the old COBOL curmudgeon in the corner ("In MY day, we wrote code without a shift key, and we LIKED it! Uphill, both ways, I tell you!"), thinks that uppercased keywords is a good idea.

As for Mr. Wirth, well, dude had some good ideas, but even Einstein had his wacky moments. Repeat after me, everybody: "Just because some guy is brilliant and turns out to be generally right doesn't mean we take everything he says as gospel". It's true for Einstein, it's true for Wirth, and it's true even for Dave Thomas (whom I am privileged to call friend, love deeply, and occasionally think is off his rocker... but I digress).

Actually, Glenn, I think case-sensitivity as a whole is silly. Let's face it, all ye who think that the C-family of languages have this one right, when's the last time you thought it was perfectly acceptable to write code like "int Int = 30;" ? Frankly, if anybody chose to overload based on case, I'd force them to maintain that same code for the next five years as punishment.

(I thought about ripping their still-beating hearts out of their chests instead, but honestly, having to live with the mess they create seems worse, and more fitting to boot.)

What's ironic, though, is that to be perfectly frank, I do exactly this with my SQL code, and it DOESN'T! SEEM! TO! SHOUT! to me AT! ALL! For some reason, this

SELECT name, age, favorite_language FROM programmers WHERE age > 25 AND favorite_language != 'COBOL';

just seems to flow pretty easily off the tongue. Err... eyeball. Whatever.

Meanwhile, 'Of Fibers and Continuations' drew some ire from Mark Murphy:

Frankly, this desire to accommodate the nifty feature of the moment smacks a great deal of Visual Basic, and while VB certainly has its strengths, coherent language design and consistent linguistic facilities is not one of them. It's played havoc with people who tried to maintain code in VB, and it's played hell with the people who try to maintain the VB language. One might try to argue that the Ruby maintainers are just Way Smarter than the Visual Basic maintainers, but I think that sells the VB team pretty short, having met some of them.

Conversely, I think you're selling the Ruby guys a bit short. And this is coming from a guy who's old enough to have written code in Visual Basic for DOS several years into his programming experience.

Wow. Next thing you know, Bruce Tate will be in here, talking about the "chuck the baby out the window" game he wrote for QuickBASIC. (True story.) And, FWIW, I too know the love of BASIC, although in this case I did QuickBasic (DOS) for a while, before it became known as QBasic, and Applesoft BASIC even before that. (Anybody else remember lo-res vs. hi-res graphics debates?) Ah, the sweet, sweet memories of PEEK and POKE and.... *shudder* Never mind.

[insert obligatory "get off my lawn!" reference here]

Get off my lawn, ya hooligan!

The death-knell for VB is widely considered to be the move from VB6 to VB.NET. In doing that, they changed significant quantities of the VB syntax. That's why there was so much hue and cry to keep maintaining VB6, because folk didn't want to take the time to port their zillions of lines of VB6 code.

Actually, much of that hue and cry was from a corner of the VB world that really just didn't want to learn something new. It turned out that most of the VB hue'ers and cry'ers were those who'd been hue'ing and cry'ing with every successive release of VB, and in the words of one very popular VB speaker and programmer, "If they don't want to come along, well, frankly, I think we're better off without 'em anyway."

Truthfully? VB seems to have move along just fine since. And, interestingly enough, since its transition to the CLR, VB has had a much stronger "core vision" to the language than it did for many years. I don't know if this is because the CLR helps them keep that vision clear, or if trying to keep up with C# is good intra-corporate competition, or what, but I haven't heard anywhere near the kinds of grousing about new linguistic changes in the two successive revisions of VB since VB.NET's release (VS 2005 and VS 2008) than I did prior to its move to the CLR.

The changes Ruby made in 1.9 had very little syntax impact (colons in case statements, and not much else, IIRC). Fibers, in particular, are just objects, supplied as part of the stock Ruby class library. I'm not aware of new syntax required to use fibers.

Grousing about a language adding to its standard class library seems a little weak. When Microsoft added new APIs to .NET when they released 3.0, I suspect you didn't bat an eye.

Oh, heavens, no. Quite the contrary--when .NET 3.0 shipped with WCF, Workflow and WPF in it, I was actually a little concerned, because the CLR's basic footprint is just ballooning like mad. How long before the CLR installation rivals that of the OS itself? Besides, this monolithic approach has its limitations, as the Java folks have discovered to their regret, and it's not too long before people start noticing the five or six different versions of the CLR all living on their machine simultaneously....

Let's be honest here--an API release is different from changing the execution model of the virtual machine, and that's partly what fibers do.

But of even more interest to this particular discussion, I wasn't really grousing about the syntax, or the addition of fibers, as I was pointing out that this is something that other platforms (notably Win32) has had before, and that it ended up being a "ho-hum, another subject I can safely ignore" topic for the world's programmers. That, and the interesting--and heretofore unrecognized, to me--link between fibers and coroutines and continuations.

In particular, grousing about how Language X adds something to its class library that duplicates a portion of something "baked into" Language Y seems really weak. Does this mean that once something is invented in a language, no other language is supposed to implement it in any way, shape, or form?

Heavens, no! Just like if you want to use objects, you're more than welcome to do so in C, or Pascal, or even assembly!

What if fibers weren't part of the Ruby 1.9 distribution, but rather were done by a third party and released as a popular gem? (I'm not sure if this would have been possible, as there may have been changes needed to the MRI to support fibers, but let's pretend for a moment.) Does this mean that nobody writing class libraries for any programming language are allowed to implement features that are "baked into" some other programming language?

Um... no: witness LINQ, stealing... *ahem* leveraging... a great deal of the concepts that are behind functional languages. Or the Win16 API (or the classic Mac OS API, or the Xt API, or ...), using object concepts from within the C language.

If so, C# should have never been created.

Huh?

Look, I have nothing against Ruby swiping ideas from another language. But let's not pretend that Ruby was built, from the ground up, as a functional language. The concepts that Ruby is putting forth in its 1.9 release are "bolted on", and will show the same leaks in the abstraction model as any other linguistic approach "bolted on" after the fact. This is a large part of the beef with generics in Java, with objects in C, with O/R-Ms, and so on. Languages choose, very precisely, which abstractions they want to make as first-class citizens, and usually when they try to add more of those concepts in after the fact, backwards compatibility and the choices they made earlier trip them up and create a suboptimal scenario. (Witness the various attempts to twist Java into a metaprogramming language: generics, AOP, and so on.)

Besides, if you're going to explore those features, why not go straight to the source? Since when has it become fashionable to discourage people from learning a new concept in the very environment where it is highlighted? Ruby is a phenomenal dynamic language (as is Lisp and Smalltalk, among others), and anybody who wants to grok dynamic languages should learn Ruby (and/or Lisp, and/or Smalltalk). Ditto for functional languages (Haskell and ML/OCaml being the two primary candidates in that camp).

Don't get me wrong -- I agree that there are way better languages for FP than Ruby, even with fibers. That's part of the reason why so many people are tracking JRuby and IronRuby, as having Ruby implementations on common VMs/LRs gives developers greater flexibility for mixing-and-matching languages to fit specific needs (JRuby/Scala/Groovy/Java on JVM, IronEverything/LotsOf# on CLR/DLR).

Which is the same thing I just said. Cool.

I just think you could have spun this more positively and made the same points. The Rails team is having their hats handed to them over the past week or two; casting fibers as a "whither Ruby?" piece just feels like piling on.

Well, frankly, I don't track what's going on in the Rails space at all [and, to be honest, if one more programmer out there invents one more web framework that rhymes with "ails" in any way, so help me God I will SCREAM], so I can honestly say that I wasn't trying to "pile on". What I do find frustrating, however, is the general belief that Ruby is somehow God's Original Scripting Language, and that the Ruby community is constantly innovating while the rest of the programming world is staring on in drooling slack-jawed envy. Most of what Ruby does today is Old Hat to Smalltalkers, and I fully expect that PowerShellers will come along and find most of what the Ruby guys are doing to be interesting experiments in just how powerful the PSH environment really is.

Of deeper concern is the blending of "shell language" and "programming language" that Ruby seems to encourage; the only other language that I think really crosses that line is Perl, and honestly, that's not necessarily good company to be in on this score. When a language tries to hold fealty to too many masters, it loses coherence. Time will tell how well Ruby can chart that narrow course; to my mind, this is what ultimately doomed (and continues to dog) Perl 6.

English is a wonderful language, isn't it? I'm no linguist, but from what little study I've made of other languages (French and German, so far), English seems to have this huge propensity, more so than the other languages, to put multiple meanings behind the same word. Consider, for example, the word "done": it means "completed", as in "Are you finished with your homework? Yes, Dad, I'm done.", or it can mean "wiped out, exhausted, God-just-take-me-now-please", as in "Good God, another open-source Web framework? That's it, I give up. I'm done. Code the damn thing in assembler for all I care."

So is Java "done" like the Patriots, a job well accomplished, or "done" like the Dolphins, the less said, the better?

(For those of you who are not American football fans, the New England Patriots have gone completely undefeated this season, a mark only set once before in the game's history, and the Miami Dolphins almost went completely unvictorious this season, a mark never accomplished. [Update: Hamlet D'Arcy points out, "Actually, a winless season has been accomplished before. Tampa Bay started their first two seasons winless with an overall 0-26 record before finally winning its first game in 1977." Thanks, Hamlet; my fact-checking on that one was lax, as I was trusting the commentary by a sportscaster during the Dolphins-Ravens game, and apparently his fact-checking was a tad lax, as well. ] The playoffs are still going on, but the Patriots really don't look beatable by any of the teams remaining. Meanwhile, the Dolphins managed to eke one out just before the season ended, posting a final record of 1-15, something reserved usually for new teams in the league, not a team with historical greatness behind them. And that's it for Sports, back to you in the studio, Tom.)

Bruce Eckel seems to suggest that Java is somewhere more towards Miami than New England, and that generics were the major culprit. (He also intimates that his criticism of generics has swayed Josh and Neal's opinions to now being critical of generics, something I highly doubt, personally. More on that later.) Now, I'll be the first to admit that I think generics in Java suck, and I've said this before, but the fact remains, no one feature can sink a language. Consider multiple inheritance in C++, something that Stroustrup himself admits (in Design and Evolution of C++) he did before templates or exceptions because he wanted to know how he could do it. Lots of people argued for years (decades, even) over MI and its inclusion in the language, and in the end....

... in the end MI turns out to be a useful feature of the language, but not the way anybody figured they would be. Ditto for templates, by the way. After looking at the Boost libraries, even just the basic examples using them, I feel like I'm looking at Sanskrit or something. As Scott Meyers put it once, "We're a long way from Stack-of-T here, folks."

And that is my principal complaint about generics: the fact that they aren't fully reified down into the JVM means that we lost 90% of the power of generics, and more importantly, we lost all of the emergent behavior and functionality that came out of C++ templates. Nothing new could come out of Java generics, because they were designed to do exactly what they were designed to do: give us type-safe collections. Whee. We're cooking with gas now, folks. Next thing you know, they'll give us printf() back, too.

(Oh, wait, they did that, too.)

Fact is, there's a lot of things that could be done to Java as a language to make it more attractive, but doing so risks that core element that Sun refuses to surrender, that of backwards compatibility. This was evident as far back as JavaPolis 2006, when I interviewed Neal and Josh on the subject; when asked, point-blank, why generics didn't "go all the way down", a la .NET generics do, they both basically said, "that would break backwards compatibility, and that was a core concern from the start". (I disagreed with them, off-camera, mind you, particularly on the grounds that the Collections library, the major source of concern around backwards compatibility, could have been ported over, but then Neal pointed out to me that it wasn't just the library itself but all the places it was used, particularly all those libraries outside of Sun, that was at stake. Perhaps, but I still believe that a happier middle ground could have been eked out.) That is still the message today, from what I can see of Neal's and Josh's public statements.

And the fact is, so far as it goes, Java generics are (ugh) useful. Useful solely as a Java compiler trick, perhaps, and far more verbose than we'd prefer, but useful nonetheless. Using them is about as exciting as using a new hammer, but they can at least get the job done.

There, I've made the obligatory "generics don't completely suck" disclaimer, and I'll be the first one to tell you, I just live with the warnings when I write Java code. Possibly that's because I don't worry too much about type-safe collections in my code, but I know lots of other programmers (particularly those on teams where the team composition isn't perhaps as strong as they'd like it to be) who do, and thus take the extra time to write their code to be generics-friendly and thus warning-free.

The mere fact that we have to work at it to create code that is "generics-friendly" is part of the problem here. For all those who came from C++ years ago, you'll know what I mean when I say that "Java generics are the new C++ const": Writing const-correct code was always a Good Thing To Do, it's just that it was also just such a Damn Hard Thing To Do. Which meant that nobody did it.

Languages should enable you to fall into the pit of success. That's the heart of the Principle of Least Surprise, even if it's not always said that way. (I'm not sure that C# 3 does this, time will tell. I'm reasonably certain that Ruby doesn't, despite the repeated insistence of Ruby advocates, many of whom I deeply respect. I'm nervous that Scala and F# will fall into this same trap, owing to their unusual syntax in places. It will be fun to see how ActionScript 3 turns out.)

Here's a thought: Let's leave Java where it is, and just start creating new JVM languages that cater to specific needs. You can call them Java, too, if you like. Or something else, like Scala or Clojure or Groovy or JRuby or CJ or whatever suits your fancy. Since everybody compiles down to JVM bytecode, it's all really academic--they're all Java, in some fundamental way. Which means that Java can thus rest easy, knowing that it fought the good fight, and that others equally capable are carrying on the tradition of JVM programming.

Eckel makes a good point:

Arguably one of the best features of C is that it hasn't changed at all for decades.

... which completely ignores some of the changes that were proposed and accepted for the C99 standard, but we'll leave that alone for now. The point is, the core C language now is the same core C language that I learned back in my high school days, and most, if not all, C code from even that far back will still compile under today's compilers. (Granted, there's likely to be a ton of warnings if you're using old "K-and-R" C, but the code will still compile.)

What about evolution, though? Don't languages need to evolve in order to stay relevant?

Consider the C case: C++ came along, made a whole bunch of changes to the language, but went zooming off in its own direction, to the point where a standards-compliant C++ compiler won't compile even relatively recent C code.

And how many people have complained about that?

By the way, if you're a C/C++ programmer and you haven't looked at D, you're about to get leaped on the evolutionary ladder again. Just an FYI.

As a matter of fact, if you're a Java or .NET programmer, you'd be well-advised to take a look at D, too. It's one of the more interesting native-compilation languages I've seen in a while, and yet arguably it's just what a C++ compiler author would come up with after studying Java and C# for a while (which, as far as I can tell, is exactly what happened). And because D can essentially mimic C bindings for dynamic libraries, it means that a Java guy can now write a JNI DLL in a garbage-collected language that (mostly) does away with pointer arithmetic for most of its work... just as Java did.

Heck, I'd love to see a D-for-the-JVM variant. And D-for-the-CLR, while we're at it. Just for fun.

Let's do this: somebody take the old, pre-Java5 javac source, and release it as "JWH" (short for Java Work Horse), and maintain it as a separate branch of the Java compiler. Then we can hack on the new Java5 language for years, maybe call it "JWNFF" (short for Java With New-Fangled Features), and everybody can get back to work without complaints.

Well, at least those who want to go back to work can do so; there'll always be people who'd rather complain than Get Stuff Done. *shrug*

Now, on the other hand, let's talk about the JVM, and specifically what needs to change there if the JVM platform is to be the workhorse of the 21st century like it was for the latter half of the last decade of the 20th....

Dave explains Ruby fibers, as they're called in Ruby 1.9. Now, before I get going here, let me explain my biases up front: in the Windows world, we've had fibers for near on to half-decade, I think, and they're basically programmer-managed cooperative tasks. In other words, they're much like threads before threads were managed by the operating system--you decide when to switch to a different fiber, you manage the scheduling, the fiber just gives you a data structure and some basic housekeeping. (I know I'm oversimplifying and glossing over details, but that's the core, as I remember it. It's been a while since I tried to use them.) Legend has it that fibers were introduced into the Win32 API on behalf of the SQL Server team, who need to take that kind of control over thread scheduling in order to best manage the CPU, but here's the rub: they never served much purpose otherwise.

Frankly, nobody could figure out what to do with them. I'm beginning to wonder if it was because our languages of the time (C, C++) didn't have any real idea of freezing execution of a task at a certain point, putting it aside, then coming back to it and restoring it. In other words, the very behavior we see out of a continuation.

In Dave's explanation, Ruby fibers take on a different meaning. According to Dave's explanation:

A fiber is somewhat like a thread, except you have control over when it gets scheduled. Initially, a fiber is suspended. When you resume it, it runs the block until the block finishes, or it hits a Fiber.yield. This is similar to a regular block yield: it suspends the fiber and passes control back to the resume. Any value passed to Fiber.yield becomes the value returned by resume.

They sound a lot like Win32 fibers combined with Python generators, with a touch more by way of API support. (The Win32 API version was codified using C bindings, for starters, not objects.) But Dave quickly points out that fibers can become full-fledged coroutines by allowing fibers to transfer control from one to another, which is interesting, though I suspect lots of people will explore this feature and write lots of bad code as a result. Oh, well: bright shiny new toys have that effect on programmers sometimes.

He then goes on to describe how Ruby can provide pipelines:

As a starting point, let's write two fibers. One's a generator—it creates a list of even numbers. The second is a consumer. All it does it accept values from the generator and print them. We'll make the consumer stop after printing 10 numbers.

    evens = Fiber.new do
      value = 0
      loop do
        Fiber.yield value
        value += 2
      end
    end

    consumer = Fiber.new do
      10.times do
        next_value = evens.resume
        puts next_value
      end
    end

    consumer.resume

Note how we had to use resume to kick off the consumer. Technically, the consumer doesn't have to be a Fiber, but, as we'll see in a minute, making it one gives us some flexibility.

Ah, the classic producer-consumer example. Gotta love it. The interesting thing here, though, is that evens, prior to the call to resume, has done nothing. No execution has taken place. In essence, the fiber here is in deferred execution mode (now, where have I heard that before?), meaning nothing actually fires until asked for. It then runs until it hits the yield, essentially going to sleep again.

Is it me, or does this smell suspiciously like continuations?

More interesting, Dave goes on to define the consumer fiber to take the name of a source to resume, then shows how once can abstract the coupling between producer and consumer away even further by creating a filter that only allows multiples of three through the pipeline:

    def evens
      Fiber.new do
        value = 0
        loop do
          Fiber.yield value
          value += 2
        end
      end
    end

    def multiples_of_three(source)
      Fiber.new do
        loop do
          next_value = source.resume
          Fiber.yield next_value if next_value % 3 == 0
        end
      end
    end

    def consumer(source)
      Fiber.new do
        10.times do
          next_value = source.resume
          puts next_value
        end
      end
    end

    consumer(multiples_of_three(evens)).resume

Running this, we get the output

0
6
12
18
. . .

This is getting cool. We write little chunks of code, and then combine them to get work done. Just like a pipeline.

Actually, instead of calling it a pipeline, let's call it a comprehension and be done with it.

See, Ruby apparently has discovered the joys of functional programming, something that Scala and F# have baked in from the beginning, instead of bolted on from the outside. No offense intended to the Ruby community or to Matz, but I get a little lost as to exactly what Ruby's core concepts are--it's a scripting language, it's a development language, it's a DSL platform, it's object-oriented, it's functional, it's a bird, it's a plane, it's horribly confused.

Dave touches on this point in one of his responses to comments:

The thing that's interesting to me about Ruby in this context is how much is can bend into multiple paradigms. Haskell does FP way better than Ruby. Smalltalk does OO (marginally) better. But Ruby does them all, and in a way that interoperates nicely.

I like a lot of Ruby's core concepts--open classes, mixins, and so on--but I'm worried that Ruby's trying to do too much, much as another language I know and love is. Frankly, this desire to accommodate the nifty feature of the moment smacks a great deal of Visual Basic, and while VB certainly has its strengths, coherent language design and consistent linguistic facilities is not one of them. It's played havoc with people who tried to maintain code in VB, and it's played hell with the people who try to maintain the VB language. One might try to argue that the Ruby maintainers are just Way Smarter than the Visual Basic maintainers, but I think that sells the VB team pretty short, having met some of them.

Don't get me wrong here, I think it's nifty that Ruby has come around to realize the power of atomic components doing one thing well, passing its results on into the pipeline for something else to process, and this is a large part of why PowerShell is, in my mind, the sleeper programming language of 2008/2009. Pipelines also scale very well, since they encourage immutable state, since the results of each processing step are essentially fed in from the outside and the results are passed back out to the next step in the chain--all state is passed from one step to the next, meaning I can run lots of these pipelines in parallel with no fear of deadlocks or bottlenecks, since each processing step is itself essentially state-free. This is also, in fact, a lot of how original transaction-processing systems were designed, which also scaled pretty well, at least until we got the bright idea to store mutable state in them (*cough* EJB *cough*).

Oh, and for what it's worth, this concept is trivial to do in F#, via the pipeline operator ( "|>" ). Ditto for Scala. If you're going to think in pipelines, you may as well work with a language that has the concept baked in a little more deeply, IMHO. And before the Rubyists beat me over the head about this, Dave himself admits this is true in another comment response:

Paolo: I don't think Ruby or Smalltalk really do functional programming to any deep level. However, both can be used to implement particular FP constructs (such as generators).

And maybe, in the end, that's the important thing: recognizing what aspects of functional programming can be easily lifted into your language of choice and used to make your life simpler. Still, I'm always looking for languages that take the concepts that float in my head and let me express them as first-class constructs, not as duck-taped partial implementations thereof. I felt the same way about doing "objects" in C (back in the Win16 programming day, before C++ Windows frameworks emerged), and about doing "aspects" in Java using interception.

If you're going to think in a concept, you generally want a language that expresses that concept as a first-class citizen, or you'll get frustrated quickly. Ruby's fibers may be the gateway drug for developers to learn functional programming, but they're not going to get it at any deep level until they dive into Haskell or ML or one of its derivatives (Scala or F#). For example, once you see the power inherent in Scala's comprehensions, you never look at a simple for loop the same way again.

Oh, and Groovyists? I'm sure they could do this, but I dunno if it's worth it, given that Groovy and Scala, at some level, are fundamentally interoperable as well. (Note to self: must do a blog post about Groovy calling into Scala code, just to show it can be done. Y'all hold me to that, if you don't see it in a week or two.)

Meanwhile, the link between continuations and Ruby fibers (and Win32 fibers, while we're at it) still tickles at the back of my mind.... But that's a thought waiting to be explored another day.

Larry Wall, (in)famous creator of that (in)famous Perl language, has contributed a few cents' worth to the debate over "scripting" languages:

I think, to most people, scripting is a lot like obscenity. I can't define it, but I'll know it when I see it.

Aside from the fact that the original quote reads "pornography" instead of "obscenity", I get what he's talking about. Finding a good definition for scripting is like trying to find a good definition for "object-oriented" or "service-oriented" or... come to think of it, like a lot of the terms that we tend to use on a daily basis. So I'm right there along with him, assuming that his goal here is to call out a workable definition for "scripting" languages.

Here are some common memes floating around:

    Simple language
    "Everything is a string"
    Rapid prototyping
    Glue language
    Process control
    Compact/concise
    Worse-is-better
    Domain specific
    "Batteries included"

...I don't see any real center here, at least in terms of technology. If I had to pick one metaphor, it'd be easy onramps. And a slow lane. Maybe even with some optional fast lanes.

I'm not sure where some of these memes come from, but some of them I recognize (Simple language, Rapid prototyping, glue language, compact/concise), some of them are new to me ("Everything is a string", process control), and some of them I seriously question the sanity of anybody suggesting them (worse-is-better, domain specific, "batteries included"). Fortunately he didn't include the "dynamically typed" or "loosely coupled" memes, which I hear tagged on scripting languages all the time.

But basically, scripting is not a technical term. When we call something a scripting language, we're primarily making a linguistic and cultural judgment, not a technical judgment. I see scripting as one of the humanities. It's our linguistic roots showing through.

I can definitely see the use of the term "scripting" as a term of value judgement, but I'm not sure I see the idea that scripting languages somehow demonstrate our linguistic roots.

We then are treated to one-sentence reviews of every language Larry ever programmed in, starting from his earliest days in BASIC, with some interesting one-liners scattered in there every so often:

On Ruby: "... a great deal of Ruby's syntax is borrowed from Perl, layered over Smalltalk semantics."

On Lisp: "Is LISP a candidate for a scripting language? While you can certainly write things rapidly in it, I cannot in good conscience call LISP a scripting language. By policy, LISP has never really catered to mere mortals. And, of course, mere mortals have never really forgiven LISP for not catering to them."

On JavaScript: "Then there's JavaScript, a nice clean design. It has some issues, but in the long run JavaScript might actually turn out to be a decent platform for running Perl 6 on. Pugs already has part of a backend for JavaScript, though sadly that has suffered some bitrot in the last year. I think when the new JavaScript engines come out we'll probably see renewed interest in a JavaScript backend." Presumably he means a new JavaScript backend for Perl 6. Or maybe a new Perl 6 backend for JavaScript.

On scripting langauges as a whole: "When I look at the present situation, what I see is the various scripting communities behaving a lot like neighboring tribes in the jungle, sometimes trading, sometimes warring, but by and large just keeping out of each other's way in complacent isolation."

Like the prize at the bottom of the cereal box, if you can labor through all of this, though, you get treated to one of the most amazing succinct discussions/point-lists of language design and implementation I've seen in a long while; I've copied that section over verbatim, though I annotate with my own comments in italics:

early binding / late binding

Binding in this context is about exactly when you decide which routine you're going to call for a given routine name. In the early days of computing, most binding was done fairly early for efficiency reasons, either at compile time, or at the latest, at link time. You still tend to see this approach in statically typed languages. With languages like Smalltalk, however, we began to see a different trend, and these days most scripting languages are trending towards later binding. That's because scripting languages are trying to be dwimmy (Do What I Mean), and the dwimmiest decision is usually a late decision because you then have more available semantic and even pragmatic context to work with. Otherwise you have to predict the future, which is hard.

So scripting languages naturally tend to move toward an object-oriented point of view, where the binding doesn't happen 'til method dispatch time. You can still see the scars of conflict in languages like C++ and Java though. C++ makes the default method type non-virtual, so you have to say virtual explicitly to get late binding. Java has the notion of final classes, which force calls to the class to be bound at compile time, essentially. I think both of those approaches are big mistakes. Perl 6 will make different mistakes. In Perl 6 all methods are virtual by default, and only the application as a whole can tell the optimizer to finalize classes, presumably only after you know how all the classes are going to be used by all the other modules in the program.

[Frankly, I think he leaves out a whole class of binding ideas here, that being the "VM-bound" notion that both the JVM and the CLR make use of. In other words, the Java language is early-bound, but the actual linking doesn't take place until runtime (or link time, as it were). The CLR takes this one step further with its delegates design, essentially allowing developrs to load a metadata token describing a function and construct a delegate object--a functor, as it were--around that. This is, in some ways, a highly useful marriage of both early and late binding.

[I'm also a little disturbed by his comments "only the application as a whole can tell the optimizer to finalize classes, presumably only after you know how all that classes are going to be used by all the other modules in the program. Since when can programmers reasonably state that they know how classes are going to be used by all the other modules in the program? This seems like a horrible set-you-up-for-failure point to me.]

single dispatch / multiple dispatch

In a sense, multiple dispatch is a way to delay binding even longer. You not only have to delay binding 'til you know the type of the object, but you also have to know the types of all rest of the arguments before you can pick a routine to call. Python and Ruby always do single dispatch, while Dylan does multiple dispatch. Here is one dimension in which Perl 6 forces the caller to be explicit for clarity. I think it's an important distinction for the programmer to bear in mind, because single dispatch and multiple dispatch are philosophically very different ideas, based on different metaphors.

With single-dispatch languages, you are basically sending a message to an object, and the object decides what to do with that message. With multiple dispatch languages, however, there is no privileged object. All the objects involved in the call have equal weight. So one way to look at multiple dispatch is that the objects are completely passive. But if the objects aren't deciding how to bind, who is?

Well, it's sort of a democratic thing. All the routines of a given name get together and hold a political conference. (Well, not really, but this is how the metaphor works.) Each of the routines is a delegate to the convention. All the potential candidates put their names in the hat. Then all the routines vote on who the best candidate is, and the next best, and the next best after that. And eventually the routines themselves decide what the best routine to call is.

So basically, multiple dispatch is like democracy. It's the worst way to do late binding, except for all the others.

But I really do think that's true, and likely to become truer as time goes on. I'm spending a lot of time on this multiple dispatch issue because I think programming in the large is mutating away from the command-and-control model implicit in single dispatch. I think the field of computation as a whole is moving more toward the kinds of decisions that are better made by swarms of insects or schools of fish, where no single individual is in control, but the swarm as a whole has emergent behaviors that are somehow much smarter than any of the individual components.

[I think it's a pretty long stretch to go from "multiple dispatch", where the call is dispatched based not just on the actual type of the recipient but the caller as well, to suggesting that whole "swarms" of objects are going to influence where the call comes out. People criticized AOP for creating systems where developers couldn't predict, a priori, where a call would end up, how will they react to systems where nondeterminism--having no real idea at source level which objects are "voting", to use his metaphor--is the norm, not the exception?]

eager evaluation / lazy evaluation

Most languages evaluate eagerly, including Perl 5. Some languages evaluate all expressions as lazily as possible. Haskell is a good example of that. It doesn't compute anything until it is forced to. This has the advantage that you can do lots of cool things with infinite lists without running out of memory. Well, at least until someone asks the program to calculate the whole list. Then you're pretty much hosed in any language, unless you have a real Turing machine.

So anyway, in Perl 6 we're experimenting with a mixture of eager and lazy. Interestingly, the distinction maps very nicely onto Perl 5's concept of scalar context vs. list context. So in Perl 6, scalar context is eager and list context is lazy. By default, of course. You can always force a scalar to be lazy or a list to be eager if you like. But you can say things like for 1..Inf as long as your loop exits some other way a little bit before you run into infinity.

[This distinction is, I think, becoming one of continuum rather than a binary choice; LINQ, for example, makes use of deferred execution, which is fundamentally a lazy operation, yet C# itself as a whole generally prefers eager evaluation where and when it can... except in certain decisions where the CLR will make the call, such as with the aforementioned delegates scenario. See what I mean?]

eager typology / lazy typology

Usually known as static vs. dynamic, but again there are various positions for the adjustment knob. I rather like the gradual typing approach for a number of reasons. Efficiency is one reason. People usually think of strong typing as a reason, but the main reason to put types into Perl 6 turns out not to be strong typing, but rather multiple dispatch. Remember our political convention metaphor? When the various candidates put their names in the hat, what distinguishes them? Well, each candidate has a political platform. The planks in those political platforms are the types of arguments they want to respond to. We all know politicians are only good at responding to the types of arguments they want to have...

[OK, Larry, enough with the delegates and the voting thing. It just doesn't work. I know it's an election year, and everybody wants to get in on the whole "I picked the right candidate" thing, but seriously, this metaphor is getting pretty tortured by this point.]

There's another way in which Perl 6 is slightly more lazy than Perl 5. We still have the notion of contexts, but exactly when the contexts are decided has changed. In Perl 5, the compiler usually knows at compile time which arguments will be in scalar context, and which arguments will be in list context. But Perl 6 delays that decision until method binding time, which is conceptually at run time, not at compile time. This might seem like an odd thing to you, but it actually fixes a great number of things that are suboptimal in the design of Perl 5. Prototypes, for instance. And the need for explicit references. And other annoying little things like that, many of which end up as frequently asked questions.

[Again, this is a scenario where smarter virtual machines and execution engines can help with this--in Java, for example, the JVM can make some amazing optimizations in its runtime compiler (a.k.a. JIT compiler) that a normal ahead-of-time compiler simply can't make, such as monomorphic interface calls. One area that I think he's hinting at here, though, which I think is an interesting area of research and extension, is that of being able to access the context in which a call is being made, a la the .NET context architecture, which had some limited functionality in the EJB space, as well. This would also be a good "middle-ground" for multi-dispatch, since now the actual dispatch could be done on the basis of the context itself, which could be known, rather than on random groups of objects that Larry's gathered together for an open conference on dispatching the method call.... I kid, I kid.]

limited structures / rich structures

Awk, Lua, and PHP all limit their composite structures to associative arrays. That has both pluses and minuses, but the fact that awk did it that way is one of the reasons that Perl does it differently, and differentiates ordered arrays from unordered hashes. I just think about them differently, and I think a lot of other people do too.

[Frankly, none of the "popular" languages really has a good set-based first-class concept, whereas many of the functional languages do, and thanks to things like LINQ, I think the larger programming world is beginning to see the power in sets and set projections. So let's not limit the discussion to associative arrays; yes, they're useful, but in five years they'll be useful in the same way that line-numbered BASIC and use of the goto keyword can still be useful.]

symbolic / wordy

Arguably APL is also a kind of scripting language, largely symbolic. At the other extreme we have languages that eschew punctuation in favor of words, such as AppleScript and COBOL, and to a lesser extent all the Algolish languages that use words to indicate blocks where the C-derived languages use curlies. I prefer a balanced approach here, where symbols and identifiers are each doing what they're best at. I like it when most of the actual words are those chosen by the programmer to represent the problem at hand. I don't like to see words used for mere syntax. Such syntactic functors merely obscure the real words. That's one thing I learned when I switched from Pascal to C. Braces for blocks. It's just right visually.

[Sez you, though I have to admit my own biases agree. As with all things, though, this can get out of hand pretty quickly if you're not careful. The prosecution presents People's 1, Your Honor: the Perl programming langauge.]

Actually, there are languages that do it even worse than COBOL. I remember one Pascal variant that required your keywords to be capitalized so that they would stand out. No, no, no, no, no! You don't want your functors to stand out. It's shouting the wrong words: IF! foo THEN! bar ELSE! baz END! END! END! END!

[Oh, now, that's just silly.]

Anyway, in Perl 6 we're raising the standard for where we use punctuation, and where we don't. We're getting rid of some of our punctuation that isn't really pulling its weight, such as parentheses around conditional expressions, and most of the punctuational variables. And we're making all the remaining punctuation work harder. Each symbol has to justify its existence according to Huffman coding.

Oddly, there's one spot where we're introducing new punctuation. After your sigil you can add a twigil, or secondary sigil. Just as a sigil tells you the basic structure of an object, a twigil tells you that a particular variable has a weird scope. This is basically an idea stolen from Ruby, which uses sigils to indicate weird scoping. But by hiding our twigils after our sigils, we get the best of both worlds, plus an extensible twigil system for weird scopes we haven't thought of yet.

[Did he just say "twigil"? As in, this is intended to be a serious term? As in, Perl wasn't symbol-heavy enough, so now they're adding twigils that will hide after sigils, with maybe forgils and fivegils to come in Perl 7 and 8, respectively?]

We think about extensibility a lot. We think about languages we don't know how to think about yet. But leaving spaces in the grammar for new languages is kind of like reserving some of our land for national parks and national forests. Or like an archaeologist not digging up half the archaeological site because we know our descendants will have even better analytical tools than we have.

[Or it's just YAGNI, Larry. Look, if your language wants to have syntactic macros--which is really the only way to have langauge extensibility without having to rewrite your parser and lexer and AST code every n number of years, then build in syntactic macros, but really, now you're just emulating LISP, that same language you said wasn't for mere mortals, waaaay back there up at the top.]

Really designing a language for the future involves a great deal of humility. As with science, you have to assume that, over the long term, a great deal of what you think is true will turn out not to be quite the case. On the other hand, if you don't make your best guess now, you're not really doing science either. In retrospect, we know APL had too many strange symbols. But we wouldn't be as sure about that if APL hadn't tried it first.

[So go experiment with something that doesn't have billions of lines of code scattered all across the planet. That's what everybody else does. Witness Gregor Kiczales' efforts with AspectJ: he didn't go and modify Java proper, he experimented with a new language to see what AOP constructs would fit. And he never proposed AspectJ as a JSR to modify core Java. Not because he didn't want to, mind you, I know that this was actively discussed. But I also know that he was waiting to see what a large-scale AOP system looked like, so we could find the warts and fix them. The fact that he never opened an AspectJ JSR suggests to me that said large-scale AOP system never materialized.]

compile time / run time

Many dynamic languages can eval code at run time. Perl also takes it the other direction and runs a lot of code at compile time. This can get messy with operational definitions. You don't want to be doing much file I/O in your BEGIN blocks, for instance. But that leads us to another distinction:

declarational / operational

Most scripting languages are way over there on the operational side. I thought Perl 5 had an oversimplified object system till I saw Lua. In Lua, an object is just a hash, and there's a bit of syntactic sugar to call a hash element if it happens to contain code. Thats all there is. [Dude, it's the same with JavaScript/ECMAScript. And a few other langauges, besides.] They don't even have classes. Anything resembling inheritance has to be handled by explicit delegation. That's a choice the designers of Lua made to keep the language very small and embeddable. For them, maybe it's the right choice.

Perl 5 has always been a bit more declarational than either Python or Ruby. I've always felt strongly that implicit scoping was just asking for trouble, and that scoped variable declarations should be very easy to recognize visually. Thats why we have my. It's short because I knew we'd use it frequently. Huffman coding. Keep common things short, but not too short. In this case, 0 is too short.

Perl 6 has more different kinds of scopes, so we'll have more declarators like my and our. But appearances can be deceiving. While the language looks more declarative on the surface, we make most of the declarations operationally hookable underneath to retain flexibility. When you declare the type of a variable, for instance, you're really just doing a kind of tie, in Perl 5 terms. The main difference is that you're tying the implementation to the variable at compile time rather than run time, which makes things more efficient, or at least potentially optimizable.

[The whole declarational vs operational point here seems more about type systems than the style of code; in a classless system, a la JavaScript/ECMAScript, objects are just objects, and you can mess with them at runtime as much as you wish. How you define the statements that use them, on the other hand, is another axis of interest entirely. For example, SQL is a declarational language, really more functional in nature (since functional languages tend to be declarational as well), since the interpreter is free to tackle the statement in any sub-clause it wishes, rather than having to start from the beginning and parse right. There's definitely greater distinctions waiting to be made here, IMHO, since there's still a lot of fuzziness in the taxonomy.]

immutable classes / mutable classes

Classes in Java are closed, which is one of the reasons Java can run pretty fast. In contrast, Ruby's classes are open, which means you can add new things to them at any time. Keeping that option open is perhaps one of the reasons Ruby runs so slow. But that flexibility is also why Ruby has Rails. [Except that Ruby now compiles to the JVM, and fully supports open classes there, and runs a lot faster than the traditional Ruby interpreter, which means that either the mutability of classes has nothing to do with the performance of a virtual machine, or else the guys working on the traditional Ruby interpreter are just morons compared to the guys working on Java. Since I don't believe the latter, I believe that the JVM has some intrinsic engineering in it that the Ruby interpreter could have--given enough time and effort--but simply doesn't have yet. Frankly, from having spelunked the CLR, there's really nothing structurally restricting the CLR from having open classes, either, so long as the semantics of modifying a class structure in memory were well understood: concurrency issues, outstanding objects, changes in method execution semantics, and so on.]

Perl 6 will have an interesting mix of immutable generics and mutable classes here, and interesting policies on who is allowed to close classes when. Classes are never allowed to close or finalize themselves, for instance. Sorry, for some reason I keep talking about Perl 6. It could have something to do with the fact that we've had to think about all of these dimensions in designing Perl 6.

class-based / prototype-based

Here's another dimension that can open up to allow both approaches. Some of you may be familiar with classless languages like Self or JavaScript. Instead of classes, objects just clone from their ancestors or delegate to other objects. For many kinds of modeling, it's actually closer to the way the real world works. Real organisms just copy their DNA when they reproduce. They don't have some DNA of their own, and an @ISA array telling you which parent objects contain the rest of their DNA.

[I get nervous whenever people start drawing analogies and start pursuing them too strongly. Yes, this is how living organisms replicate... but we're not designing living organisms. A model is just supposed to represent a part of reality, not try to recreate reality itself. Having said that, though, there's definitely a lot to be said for classless languages (which don't necessarily have to be prototype-based, by the way, though it makes sense for them to be). Again, what I think makes the most sense here is a middle-of-the-road scenario combined with open classes. Objects belong to classes, but fully support runtime reification of types.]

The meta-object protocol for Perl 6 defaults to class-based, but is flexible enough to set up prototype-based objects as well. Some of you have played around with Moose in Perl 5. Moose is essentially a prototype of Perl 6's object model. On a semantic level, anyway. The syntax is a little different. Hopefully a little more natural in Perl 6.

passive data, global consistency / active data, local consistency

Your view of data and control will vary with how functional or object-oriented your brain is. People just think differently. Some people think mathematically, in terms of provable universal truths. Functional programmers don't much care if they strew implicit computation state throughout the stack and heap, as long as everything looks pure and free from side-effects.

Other people think socially, in terms of cooperating entities that each have their own free will. And it's pretty important to them that the state of the computation be stored with each individual object, not off in some heap of continuations somewhere.

Of course, some of us can't make up our minds whether we'd rather emulate the logical Sherlock Holmes or sociable Dr. Watson. Fortunately, scripting is not incompatible with either of these approaches, because both approaches can be made more approachable to normal folk.

[Or, don't choose at all, but combine as you need to, a la Scala or F#. By the way, objects are not "free willed" entities--they are intrinsically passive entities, waiting to be called, unless you bind a thread into their execution model, which then makes them "active objects" or sometimes called "actors" (not to be confused with the concurrency model Actors, such as Scala uses). So let's not get too hog-wild with that "individual object/live free or die" meme, not unless you're going to differentiate between active objects and passive objects. Which, I think, is a valuable thing to differentiate on, FWIW.]

info hiding / scoping / attachment

And finally, if you're designing a computer language, there are a couple bazillion ways to encapsulate data. You have to decide which ones are important. What's the best way to let the programmer achieve separation of concerns?

object / class / aspect / closure / module / template / trait

You can use any of these various traditional encapsulation mechanisms.

transaction / reaction / dynamic scope

Or you can isolate information to various time-based domains.

process / thread / device / environment

You can attach info to various OS concepts.

screen / window / panel / menu / icon

You can hide info various places in your GUI. Yeah, yeah, I know, everything is an object. But some objects are more equal than others. [NO. Down this road lies madness, at least at the language level. A given application might choose to, for reasons of efficiency... but doing so is a local optimization, not something to consider at the language level itself.]

syntactic scope / semantic scope / pragmatic scope

Information can attach to various abstractions of your program, including, bizarrely, lexical scopes. Though if you think about it hard enough, you realize lexical scopes are also a funny kind of dynamic scope, or recursion wouldn't work right. A state variable is actually more purely lexical than a my variable, because it's shared by all calls to that lexical scope. But even state variables get cloned with closures. Only global variables can be truly lexical, as long as you refer to them only in a given lexical scope. Go figure.

So really, most of our scopes are semantic scopes that happen to be attached to a particular syntactic scope.

[Or maybe scope is just scope.]

You may be wondering what I mean by a pragmatic scope. That's the scope of what the user of the program is storing in their brain, or in some surrogate for their brain, such as a game cartridge. In a sense, most of the web pages out there on the Internet are part of the pragmatic scope. As is most of the data in databases. The hallmark of the pragmatic scope is that you really don't know the lifetime of the container. It's just out there somewhere, and will eventually be collected by that Great Garbage Collector that collects all information that anyone forgets to remember. The Google cache can only last so long. Eventually we will forget the meaning of every URL. But we must not forget the principle of the URL. [This is weirdly Zen, and either makes no sense at all, or has a scope (pardon the pun) far outside of that of programming languages and is therefore rendered meaningless for this discussion, or he means something entirely different from what I'm reading.] That leads us to our next degree of freedom.

use Lingua::Perligata;

If you allow a language to mutate its own grammar within a lexical scope, how do you keep track of that cleanly? Perl 5 discovered one really bad way to do it, namely source filters, but even so we ended up with Perl dialects such as Perligata and Klingon. What would it be like if we actually did it right?

[Can it even be done right? Lisp had a lot of success here with syntactic macros, but I don't think they had scope attached to them the way Larry is looking at trying to apply here. Frankly, what comes to mind most of all here is the C/C++ preprocessor, and multiple nested definitions of macros. Yes, it can be done. It is incredibly ugly. Do not ask me to remember it again.]

Doing it right involves treating the evolution of the language as a pragmatic scope, or as a set of pragmatic scopes. You have to be able to name your dialect, kind of like a URL, so there needs to be a universal root language, and ways of warping that universal root language into whatever dialect you like. This is actually near the heart of the vision for Perl 6. We don't see Perl 6 as a single language, but as the root for a family of related languages. As a family, there are shared cultural values that can be passed back and forth among sibling languages as well as to the descendants.

I hope you're all scared stiff by all these degrees of freedom. I'm sure there are other dimensions that are even scarier.

But... I think its a manageable problem. I think its possible to still think of Perl 6 as a scripting language, with easy onramps.

And the reason I think its manageable is because, for each of these dimensions, it's not just a binary decision, but a knob that can be positioned at design time, compile time, or even run time. For a given dimension X, different scripting languages make different choices, set the knob at different locations.

Somewhere in the universe, a budding programming language designer reads that last paragraph, thinks to himself, I know! I'll create a language where the programmer can set that knob wherever they want, even at runtime! Sort of like a "Option open_classes on; Option dispatch single; Option meta-object-programming off;" thing....

And with any luck, somebody will kill him before he unleashes it on us all.

Meanwhile, I just sit back and wonder, All this from the guy who proudly claimed that Perl never had a formal design to it whatsoever?

After pulling down the Solaris Developer Express 9/07 VMWare image, that it would make just too much sense to install Mercurial, grab the OpenJDK sources, and get the OpenJDK build going on that VMWare image and re-release the image back to the world, so those who wanted to build the OpenJDK and have an out-of-the-box ready-to-go experience could do so. (I'd love to do the same for Windows, but there's obvious licensing problems there.) Then, because the VMWare image would already have the Sun Studio 12 and NetBeans IDEs on it, one would have a complete debugging and profiling platform for spelunking the OpenJDK code base.

Thus far,though, I'm running into a significant snag, in that SDX doesn't want to run Sun Studio out of the box: it complains that it can't find CC on the PATH (which is on the PATH, as near as I can tell). Putting it on the PATH and re-launching the IDE (as suggested in the error message) has no effect, nor does modifying my .profile and logging-out-and-back-in-again.

To make matters more interesting, when kicking off Make, it throws a Java exception claiming "out of free space", which shouldn't be the case at all, since the drive the project lives on has a couple of gigs free. I've posted the errors to the Sun Studio 12 forums (after noticing that somebody else posted the exact same problems back in October, with no replies, which is discouraging), but was hoping one of the folks who listen in on the blog has some suggestions to try to fix this. Note that when using "dmake" (Solaris' native make, it seems) from the command-line, it works flawlessly. Help?

Update: Stepen Tilkov comments, "My apologies for pointing out the ridiculously obvious, but you *did* 'export' that PATH, didn't you?" Never apologize for pointing out the ridiculously obvious, Stephen, because not only is it the right answer half the time, the other half of the time, it's not obvious to the guy who needs help, either because he got lazy and forgot to check it (which I'm guilty of a lot), or because they genuinely didn't know it. In this case, though, I don't think that's the case; it appears to be there when I open a Terminal window. That said, though, I have only a vague idea of the scope and lifetime of environment variables under X (compared to within a terminal session), so there's a distinct possibility I'm not getting it set in the GNOME environment around me when I log in. Any good resources to figure that out?

Overall, the SDX environment looks pretty clean, though I can't say I'm comfortable with all the places that Solaris likes to install stuff; why, for example, do they want to put Sun Studio into /opt? It just seems strange to do so, though I guess it's no stranger than Mac OS X's /Applications directory.

Speaking of which.... From the "Why didn't I think of this before now?" Department: Given that the JDK source base is now completely unfettered and free, what holds up the Mac JDK 6 release? I can somewhat understand if Apple doesn't want to pursue the Mac (I said understand, not empathize or agree with, mind you), but why doesn't Sun take the necessary steps to bring a Mac port up to snuff? Or, alternatively, where is the Mac-toting Java-loving crowd? Granted, getting AWT and Swing up to snuff on the Mac might not be a trivial exercise, I'll grant you that, but a large part of the JDK beyond those elements could be ported over without too much difficulty, it would seem to me, particularly given that the JDK compiles with gcc on the Linux platform, and Mac OS has gcc as well. What am I missing here? (Oh, and if you thought of this before me, kudos-and-why-the-hell-didn't-you-say-something-earlier? It's a really good idea, it seems, at least in theory.)

Personally, I think Apple should get off its lazy ass and get Java6 done already. That, or authorize a third party to do it. Java5 is soooo 2006.

My eight-year-old son, a few months ago, asked me what it is I do. I tried to explain to him that Daddy works as a consultant, teaching people how to build computer systems that help people do things. He thought about it a moment, then said, "So you build robots and stuff?" No, not exactly, I build software, which controls the computers. "So you program the robots to do things?" No, I build software like what runs Amazon or eBay. "So you build websites?" At which point, wisdom dawned on me, and I said, "Yes, I build websites."

He thought about it a moment, then said, "Then how come your website is so boring?"

With the coming of the new year comes a change in my professional life. Starting on 11 Feb, I will be working as a technical consultant to Cie Studios, an "interactive and entertainment and marketing company", which is about as far away from my traditional consulting client as I can get without leaving the industry completely.

You see, Cie focuses mostly on front-end, high-gloss kinds of graphical UI things. I focus mostly on back-end, deep-in-the-bowels kinds of plumbing things. They use lots of Flash and other animation tools. I haven't figured out how to draw anything more sophisticated than a stick figure (and believe me, my kids laughed at me last time I drew them in stick figures.) They make things like Nitto 1320 Legends, a free online combination of racing and social networking. I make things like HR systems for big corporations. My parents thought the Cie website was cool and attractive; they barely understand what a "high-scale transactional enterprise system" does, much less why anybody would pay for somebody to help them build it.

Talk about your odd couples.

Nevertheless, I've found a nearly-full-time home for a while, and we're all pretty excited about the partnership. The project I'm working on? Can't say much about it now, but suffice it to say, Cie is looking to leverage my love for programming language design & implementation in a new entertainment project.... which, of course, my kids are excited about, because for the first time they'll actually have something they can look at that Dad built. (Actually, I'm kinda excited about that part, too.)

The tradeoff here is obvious: they teach me about Flash and making user interfaces that are more exciting than my usual console application front-end, and I teach them... uh... I teach them... let's see.... well, anyway, they're happy with the arrangement.

Fortunately, they're also happy with my extracurricular activities (such as NFJS and TechEd, among others), which means, beyond the prospect of being incredibly busy this year, that I may end up doing something a little bit more... flashier... this year on the speaking circuit (pun intended).

Meanwhile, look to the blog for more on programming languages (including but not limited to Clojure, Groovy, Ruby, ES4, F# and Scala), virtual machines (particularly the JVM and CLR), and maybe a little bit on programming the MacOS (as I figure it out myself).

Bruce Schneier has a great blog post on this. I'm joining the movement, with this declaration:

I am not afraid of terrorism, and I want you to stop being afraid on my behalf. Please start scaling back the official government war on terror. Please replace it with a smaller, more focused anti-terrorist police effort in keeping with the rule of law. Please stop overreacting. I understand that it will not be possible to stop all terrorist acts. I accept that. I am not afraid.

In fact, I would amend this a little to include more than just the politically-correct discussion of terrorism and the government:

I am not afraid of security discussions, and I want you to stop being afraid on my behalf. Please start scaling back the draconian requirements on my passwords and connection options. Not everything has to run over HTTPS and require passwords that must be 12 characters long and contain an upper-case letter, a lower-case letter, a number, a punctuation mark, and a letter from the Klingon alphabet. Please replace it with a smaller, more focused security effort in keeping with the risk involved. Please stop overreacting. I understand that it will not be possible to stop all acts of security attack. I accept that. I am not afraid.

I want companies not to abandon their security efforts, but to put the effort into more targeted efforts. Don't spend millions instituting a VPN; instead, spend that time and money getting developers to find and fix all the command injection and/or cross-site scripting attacks that plague web applications.

As has become a blogs.tedneward.com tradition, 'tis time to offer up my predictions for the technical year 2008, and, more importantly, revisit my predictions for the technical year 2007 and see how close to the mark I was. (As usual, I'll simply offer up commentary and my own opinion--you're free to draw whatever conclusions you like about my abilities as a prognosticator.)

First, the predictions from 2007, in retrospect:

• THEN: General: Analysts will call 2007 the Year of the {Something}, where I bet that {Something} will be either "ESB" or "SOA". They will predict that companies adopting {Something} will save millions, if not billions, if only they rush to implement it now. They will tag this with a probability of .8 in order to CYA in case {Something} doesn't pan out. (Yes, I've read far too many of these reports--I'm personally convinced that each of the analyst companies has a template buried away in their basement that they pull out each time they need a new one, and they just do a global search-and-replace of "{Something}" with whatever the technology du jour happens to be.) NOW: Wasn't this the Year of the ESB?
• THEN: .NET: Thousands of developers will horribly abuse WPF in ways that can only be called nightmarish, thus once again proving the old adage that "just because you can doesn't mean you should" still holds. WPF's capabilities with video will prove, in many ways, to be the modern equivalent to the "blink" tag in HTML. This will provide some author with a golden opportunity: "WPF Applications That Suck". Alan Cooper will re-release "About Face", updated to include WPF UI elements. NOW: Well, it wasn't Alan Cooper, but Charles Petzold re-released his classics, updated for WPF. Fact is, WPF hasn't seemed to penetrate all that deeply into the programming world, so maybe this is better as a 2008 prediction, now that Visual Studio 2008 is in the hands of developers.
• THEN: .NET: Thousands of developers will look to Redmond for an answer to the question, "Which should I use? BizTalk, Windows Workflow, or SQL Server Service Broker?", and get no clear answer. NOW: How many presentations at DevConnections, VSLive!, TechEd, and other .NET shows addressed this exact question? How much guidance from Microsoft was there on the subject? 'Nuff said.
• THEN: Windows: Microsoft will try, once again, to kill off the abomination that was called the Windows 95/98/Me line of operating systems, and will once again have to back off as industry outcries of protest (on behalf of little old ladies who are the only ones left running Windows 95/98/Me and probably haven't turned their machine on in months, at least not since the grandkids last visited) go ballistic. NOW: Granny Schottenwilkins in East Duluth, Minnesota died last year, so maybe next year Microsoft will finally have enough momentum to kill Windows 16-bit.
• THEN: Windows: Ditto for Visual Basic 6.0, except now the outcry will be on behalf of developers who aren't capable of learning anything new. Sun will use the resulting PR to announce Project YAVKRWMITT (Yet Another VB Killer Really We Mean It This Time, pronounced "YAV-kermit") on java.net. Meanwhile, efforts to make CLASSPATH into something a VB 6 guy actually has a prayer of understanding will go quietly ignored. NOW: Apparently Microsoft had enough problems getting Vista adoption going, so they didn't bother trying to kill VB 6 this year. Meanwhile, Sun continues to build NetBeans to attract developers from "across the fence", except now its Ruby developers. (Apparently somebody finally showed Simon Phipps what VB 6 code really looks like, and he made the prudent decision to leave it in Microsoft's hands....)
• THEN: Java: JSR 277 will continue to churn along, and once the next draft ships, publicly nobody will like what we produce, though quietly everybody will admit it's a far cry better than what we have now, and when it ships in JDK 7 will be adopted widely and quietly. NOW: We shipped a straw man implementation, which so far as I can tell, nobody bothered to look at, probably because it was a source patch on top of the OpenJDK bits, which nobody else knew how to build....
• THEN: Java: Thousands of new ideas and proposals to extend the Java language in various ways will flood into the community, now that developers can start hacking on it for themselves thanks to the OpenJDK. Only a small fraction of these will ever get beyond the concept stage, and maybe one or two will actually be finished and released to the Web for consideration by the community and the JCP. Thousands more Java developers craving Alpha-Geek status will stick a "Hello, world" message into the compiler's startup sequence, then claim "experienced with modifying the OpenJDK Java compiler" on their resume and roundly criticize Java in one way or another by saying, "Well, I've looked at the code, and let me tell you....". NOW: Numerous language-extension projects emerging on Sourceforge and java.net, none of which appear to have any code behind them... What's it say on your resume, by the way?
• THEN: .NET: Somewhere, a developer will realize that SQL Server 2005 can be a SOAP/WSDL XML service endpoint, and open it up as a private back-channel for his application to communicate with the database through the firewall "for performance reasons" (meaning, "So I can avoid having to talk to the app server in between my web server and my database"). With any luck, the DBA will kill him and hide the body before anybody can find and exploit it. NOW: "Body of dead developer found underneath raised floor in server room. Story, at 11. In related news, WhizzBang Corp filed for bankruptcy, owing to the complete collapse of their stock after news of their entire customer base personal details database being discovered on the Web was made public...."
• THEN: General: Yet Another Virus That's Microsoft's Fault will rip through the Internet, and nobody will notice that the machines affected are the ones that aren't routinely administered or receive updates/patches. Companies will threaten Microsoft with million-dollar lawsuits, yet will fire none of their system administrators who lovingly lavish whole days tuning their Linux IRC servers yet leave the Windows Exchange Server still running Windows NT 4.0. NOW: Surprisingly, this didn't happen. Either we're shockingly bland to the viruses ripping through the Internet now, or else the virus creators are falling behind the anti-virus defenders, or else the viruses have learned to lay low, waiting for the master signal to go off simultaneously.
• THEN: General: Interest in JSON will escalate wildly, hyped as the "natural replacement for XML" in building browser-to-server connections, owing to its incredible simplicity in expressing "object" data. Folks, JSON is a useful format, but it's not a replacement for XML (nor is XML a replacement for it, either). What made XML so popular was not is hierarchical format (Lord above, that's probably the worst part of it, from where we as developers sit), nor its HTML-like simplified-SGML syntax. What made XML interesting was the fact that everybody lined up behind it--Microsoft, Sun, BEA, Oracle, IBM, there's not a big vendor that didn't express its undying love and devotion to XML. I sincerely doubt JSON will get that kind of rallying effect. (And if you're going to stand there and suggest that JSON is better because its simpler and therefore more approachable for developers to build support for themselves, quite honestly, I thought we were trying to get out of developers building all this communications infrastructure--isn't that what the app servers and such taught us?) NOW: Well, JSON continues to be of interest to those who are also hand-writing their JavaScript code on their web pages, which means that to the top-of-the-web-pyramid ubergeek, JSON is "clearly the right way to go", while the rest of the industry just doesn't give a damn so long as the code works.
• THEN: General: Interest in Java/.NET interopability will rise as companies start to realize that (a) the WS-* "silver bullet" isn't, (b) ESB, XML, and SOA are just acronyms and won't, in of themselves, solve all the integration problems, and (c) we have lots of code in both Java and .NET that need to talk to each other. This may be a self-serving prediction, but I got a LOT of interest towards the end of this year in the subject, so I'm guessing that this is going to only get bigger as the WS-* hype continues to lose its shine in the coming years. NOW: It got bigger, then drained away. I guess either the world figured out their story here, or everybody's busy rewriting all their apps in Ruby.
• THEN: Ruby: Interest in Java/Ruby and .NET/Ruby interoperability is going to start quietly making its presence felt, as people start trying to wire up their quick-to-write "stovepipe" RAILS apps against other systems in their production data center, and find that Ruby really is a platform of its own. RubyCLR or JRuby may be part of the answer here, but there's likely some hidden mines there we haven't seen yet. NOW: JRuby now runs faster than the native Ruby interpreter. IronRuby (the logical successor to RubyCLR) looks to do the same for the CLR side, plus we get a dynamic language runtime out of it. Ruby may be the best language running on somebody else's platform in 2008.
• THEN: Languages: A new meme will get started: "JavaScript was that thing, that little toy language, that you used to do stuff in the HTML browser. ECMAScript, on the other hand, is a powerful and flexible dynamic programming language suitable for use in all sorts of situations." Pass it on. If you get it, don't tell anybody else. (Don't laugh--it worked for "The Crying Game".) It's the only way ECMAScript will gain widespread acceptance and shed the "toy" label that JavaScript has. NOW: And now ECMAScript 4 is coming, with a whole new set of features that make even the Ruby guys shiver in anticipation.
• THEN: Languages: Interest in functional-object hybrid languages will grow. Scala, Jaskell, F#, and others not-yet-invented will start to capture developers' attention, particularly when they hear the part about functional languages being easier to use in multi-core systems because it encourages immutable objects and discourages side effects (meaning we don't have to worry nearly so much about writing thread-safe code). NOW: It grew, and it continues to grow. New languages in general are being released (see Clojure, Jaskell, dynalang, and others), and lots of them are incorporating functional language concepts. If you've never programmed in Haskell, now's a good time to learn, because those concepts and syntax are fast making their way towards you...
• THEN: Languages: Interest in Domain-specific languages will reach a peak this year, but a small backlash will begin next year. Meanwhile, more and more developers will realize that one man's "DSL" is another man's "little language", something UNIX has been doing since the early 70's. This will immediately take the shine off of DSLs, since anything that we did in the 70's must be bad, somehow. (Remember disco?) NOW: Apparently people haven't made the connection yet. Give it time, give it time...
• THEN: General: Rails will continue to draw developers who want quick-fix solutions/technologies, and largely that community will ignore the underlying power of Ruby itself. The draw will start to die down once Rails-esque feature ideas get folded into Java toolkits. (Rails will largely be a non-issue with the .NET community, owing to the high-productivity nature of the drag-and-drop interface in Visual Studio.) NOW: In the Java world, see Grails. In the .NET world, see the new MVC container for ASP.NET. In both worlds? Sit back and chortle at how each side chases the other, and quietly look for the best of both. Look like a genius. Retire to Caribbean country on your luxury yacht and supermodel girlfriends.
• THEN: Java: Interface21 is going to start looking like a "big vendor" alongside BEA and IBM. I was talking with some of the I21 folks in Aarhus, Denmark at JAOO, and one of them casually mentioned that they were looking at a Spring 2.1 release somewhere in mid-2008. Clearly Spring is settling into eighteen-month major-version release cycles like all the big (meaning popular), established software systems have a tendency to do. This is both a good thing and a bad thing--it's good in that it means that Spring is now becoming an established part of the Java landscape and thus more acceptable to use in production environments, but it's bad in that Spring is now going to face the inevitable problem all big vendors face: trying to be all things to all people. This is dangerous, both for Interface21 and the people relying on Spring, largely because it means that Spring faces a very real future of greater complexity (and there are those, myself included, who believe that Spring is too complex already, easily on par with the complexity seen in EJB, POJOs notwithstanding). NOW: Spring hit the 2.1 release earlier than 2008... sort of. They actually went one step better, deciding that the next release of Spring isn't a "2.1", but a "2.5". (Apparently Microsoft marketing isn't the only one who likes to jerk around version numbers.) When combined with their re-branding as "SpringSource", they're just a re-org away from being a Real Corporation.
• THEN: General: Marc Fleury will get a golden parachute from Red Hat (at their request and to their immense relief), and hopefully will retire to his own small island (might I suggest Elba, la petite corporal?) to quietly enjoy his millions. A shame that the people who did most of the real work on JBoss won't see a commensurate reward, but that's the way the business world works, I guess. NOW: Has anybody paid any attention to Fleury since his "extended paternity leave"? Maybe it's just me, but the Internet has been blissfully quiet the last half-year or so....
• THEN: General: Some company will get millions to build an enterprise product on the backs of RSS and/or Atom, thus proving that VCs are just as stupid and just as vulnerable to hype now as they were back in the DotCom era. NOW: I think I jumped the gun on this one. Unless, of course, you think about Flickr and Twittr as extended RSS/ATOM feeds....
• THEN: General: Somebody will attempt to use the phrase "Web 2.0" in a serious discussion, and I will be forced to kill them for attempting to use a vague term in a vain effort to sound intelligent. NOW: Tim O'Reilly must be dipped in lemon juice after being subjected to two dozen papercuts for creating this term. I actually heard somebody who should have known better ask, "So how do I log on to this Web 2.0 thing?" Apparently he got it confused with Internet2 (which is a boondoggle in its own right by this point.)
• THEN: Web clients: Ajax will start to lose its luster when developers realize the power of Google Maps isn't in Ajax, but in the fact that it's got some seriously cool graphics and maps. (Or, put another way, when developers realize that Ajax alone won't make their apps as cool as Google Maps, that's it's the same old DHTML from 1998, the hype will start to die down.) NOW:  Have people started building real apps with this stuff yet? Everything so far looks like demos and research prototypes to me....
• THEN: XML: Somebody, somewhere, will realize that REST != HTTP. He will be roundly criticized by hordes of HTTP zealots, and quietly crawl away to go build simpler and more robust systems that use transports other than HTTP. NOW: Anybody? Anybody? Bueller?
• THEN: XML: Somebody, somewhere, will read the SOAP 1.2 specification. H.P. Lovecraft once suggested, loosely paraphrased, the the day Man understands the nature of the universe, he will either be driven into gibbering insanity, or flee back into ignorance in self-preservation. Ditto for the day Man reads the SOAP 1.2 spec and realizes that SOAP is, in fact, RESTful. NOW: Paging Mr. Davis, Mr. Scott Davis, please answer the white courtesy telephone...
• THEN: Security: The US Government will continue its unbelievable quest to waste money on "security" by engaging in yet more perimeter security around airports and other indefensible locations, thus proving that none of them have bothered to read Schneier and learn that real security is a three-part tuple: prevention, detection, and response. NOW: Paging Mr. [DELETED], Mr. [REDACTED] [DELETED], you have a [INSUFFICIENT CLEARANCE]....
• THEN: Security: Thousands of companies will follow in the US Government's footsteps by doing exactly the same thing. (Folks, you can't solve all your problems with cryptography, no matter how big the key size--you just end up with the basic problem of where to store the keys, and no, burying them inside the code isn't going to hide them effectively.) NOW: Anybody had a look at some of these "secure" website approaches recently?
• THEN: Security: More and more rootkits-shipping-with-a-product will be discovered. We used to call it "getting close to the metal", now it's a "rootkit". With great power comes great responsibility... and, as many consumers have already discovered, with great power also comes a tendency to create greater instability... NOW: Rootkits, extensions, call it what you will, but they're growing all over the place....
• THEN: General: Parrot will ship a 1.0 release. Oh, wait, hang on, sorry, I bumped into the crystal ball and accidentally set it to 2017. NOW: Parrot is all but dead, near as I can tell. Too bad, too, but maybe if they'd had a more realistic approach to building a unified type system....
• THEN: .NET: Microsoft will ship Orcas (NetFX 3.5). (Sorry, crystal ball's still set on 2017. Trying to fix it...) NOW: They shipped. 'Nuff said.
• THEN: .NET: Vista will surpass Windows XP in market penetration. (Let's see, almost got it set back to 2007, bear with me... There. Got it.) NOW: They wish. Cue Vista SP1. 'Nuff said.
• THEN: General: I will blog more than I did this year. (Hell, I couldn't blog less, even if I tried.) NOW: Well....
• THEN: General: Pragmatic XML Services, Pragmatic .NET Project Automation and Effective .NET will ship. (Wait, is the crystal ball still on 2017...?) NOW: I wish. 'Nuff said.

Next, my predictions for 2008:

• General: The buzz around building custom languages will only continue to build. More and more tools are emerging to support the creation of custom programming languages, like Microsoft's Phoenix, Scala's parser combinators, the Microsoft DLR, SOOT, Javassist, JParsec/NParsec, and so on. Suddenly, the whole "write your own lexer and parser and AST from scratch" idea seems about as outmoded as the idea of building your own String class. Granted, there are cases where a from-hand scanner/lexer/parser/AST/etc is the Right Thing To Do, but there are times when building your own String class is the Right Thing To Do, too. Between the rich ecosystem of dynamic languages that could be ported to the JVM/CLR, and the interesting strides being made on both platforms (JVM and CLR) to make them more "dynamic-friendly" (such as being able to reify classes or access the call stack directly), the probability that your company will find a need that is best answered by building a custom language are only going to rise.
• General: The hype surrounding "domain-specific languages" will peak in 2008, and start to generate a backlash. Let's be honest: when somebody looks you straight in the eye and suggests that "scattered, smothered and covered" is a domain-specific language, the term has lost all meaning. A lexicon unique to an industry is not a domain-specific language; it's a lexicon. Period. If you can incorporate said lexicon into your software, thus making it accessible to non-technical professionals, that's a good thing. But simply using the lexicon doesn't make it a domain-specific language. Or, alternatively, if you like, every single API designed for a particular purpose is itself a domain-specific language. This means that Spring configuration files are a DSL. Deployment descriptors are a DSL. The Java language is a DSL (since the domain is that of programmers familiar with the Java language). See how nonsensical this can get? Until somebody comes up with a workable definition of the term "domain" in "domain-specific language", it's a nonsensical term. The idea is a powerful one, mind you--creating something that's more "in tune" with what users understand and can use easily is a technique that's been proven for decades now. Anybody who's ever watched an accountant rip an entirely new set of predictions for the new fiscal outlook based entirely on a few seed numbers and a deeply-nested set of Excel macros knows this already. Whether you call them domain-specific languages or "little languages" or "user-centric languages" or "macro language" is really up to you.
• General: Functional languages will begin to make their presence felt. Between Microsoft's productization plans for F# and the growing community of Scala programmers, not to mention the inherently functional concepts buried inside of LINQ and the concurrency-friendly capabilities of side-effect-free programming, the world is going to find itself working its way into functional thinking either directly or indirectly. And when programmers start to see the inherent capabilities inside of Scala (such as Actors) and/or F# (such as asynchronous workflows), they're going to embrace the strange new world of functional/object hybrid and never look back.
• General: MacOS is going to start posting some serious market share numbers, leading lots of analysts to predict that Microsoft Windows has peaked and is due to collapse sometime within the remainder of the decade. Mac's not only a wonderful OS, but it's some of the best hardware to run Vista on. That will lead not a few customers to buy Mac hardware, wipe the machine, and install Vista, as many of the uber-geeks in the Windows world are already doing. This will in turn lead Gartner (always on the lookout for an established trend they can "predict" on) to suggest that Mac is going to end up with 115% market share by 2012 (.8 probability), then sell you this wisdom for a mere price of $1.5 million (per copy).
• General: Ted will be hired by Gartner... if only to keep him from smacking them around so much. .0001 probability, with probability going up exponentially as my salary offer goes up exponentially. (Hey, I've got kids headed for college in a few years.)
• General: MacOS is going to start creaking in a few places. The Mac OS is a wonderful OS, but it's got its own creaky parts, and the more users that come to Mac OS, the more that software packages are going to exploit some of those creaky parts, leading to some instability in the Mac OS. It won't be widespread, but for those who are interested in finding it, they're there. Assuming current trends (of customers adopting Mac OS) hold, the Mac OS 10.6 upgrade is going to be a very interesting process, indeed.
• General: Somebody is going to realize that iTunes is the world's biggest monopoly on music, and Apple will be forced to defend itself in the court of law, the court of public opinion, or both. Let's be frank: if this were Microsoft, offering music that can only be played on Microsoft music players, the world would be through the roof. All UI goodness to one side, the iPod represents just as much of a monopoly in the music player business as Internet Explorer did in the operating system business, and if the world doesn't start taking Apple to task over this, then "justice" is a word that only applies when losers in an industry want to drag down the market leader (which I firmly believe to be the case--nobody likes more than to pile on the successful guy).
• General: Somebody is going to realize that the iPhone's "nothing we didn't write will survive the next upgrade process" policy is nothing short of draconian. As my father, who gets it right every once in a while, says, "If I put a third-party stereo in my car, the dealer doesn't get to rip it out and replace it with one of their own (or nothing at all!) the next time I take it in for an oil change". Fact is, if I buy the phone, I own the phone, and I own what's on it. Unfortunately, this takes us squarely into the realm of DRM and IP ownership, and we all know how clear-cut that is... But once the general public starts to understand some of these issues--and I think the iPhone and iTunes may just be the vehicle that will teach them--look out, folks, because the backlash will be huge. As in, "Move over, Mr. Gates, you're about to be joined in infamy by your other buddy Steve...."
• Java: The OpenJDK in Mercurial will slowly start to see some external contributions. The whole point of Mercurial is to allow for deeper control over which changes you incorporate into your build tree, so once people figure out how to build the JDK and how to hack on it, the local modifications will start to seep across the Internet....
• Java: SpringSource will soon be seen as a vendor like BEA or IBM or Sun. Perhaps with a bit better reputation to begin, but a vendor all the same.
• .NET: Interest in OpenJDK will bootstrap similar interest in Rotor/SSCLI. After all, they're both VMs, with lots of interesting ideas and information about how the managed platforms work.
• C++/Native: If you've not heard of LLVM before this, you will. It's a compiler and bytecode toolchain aimed at the native platforms, complete with JIT and GC.
• Java: Somebody will create Yet Another Rails-Killer Web Framework. 'Nuff said.
• Native: Developers looking for a native programming language will discover D, and be happy. Considering D is from the same mind that was the core behind the Zortech C++ compiler suite, and that D has great native platform integration (building DLLs, calling into DLLs easily, and so on), not to mention automatic memory management (except for those areas where you want manual memory management), it's definitely worth looking into. www.digitalmars.com

See you all next year....

OpenJDK, the open-source JDK 7 release (and no, I don't know if there's any practical difference between the two) has officially opened for business with the promotion of the "real, live" Mercurial repositories. These are the real deal, the same repositories that Sun employees will be working on as they modify the code... which means, in all reality, that there is a very tiny window of opportunity for you to check out code between changesets that are dependent on one another due to the way they've got the forest set up--if you get weird build errors, try re-fetching... but more on that later.

Think about it for a second--now, thanks to the way Mercurial handles source (a distributed source code system is different from a centralized client/server one like SVN or CVS), you can hack away on your own private build of the JDK, and still keep up to date with what Sun's doing. And, if Sun likes what you're doing, you could end up contributing back to the JDK as a whole.

You could make changes to the langtools bundle to start adding new features you think should be in Java.

You could make changes to the hotspot bundle to start exploring ways to optimize GC better.

You could fix bugs you find in the various Java libraries.

You could even add new features to the core classes that you've been saying needed to be there since Day 1.

You can start exploring the new Modules system (JSR-277) strawman implementation.

Then, you can post your diffs, or just tell people where your changesets are, and others can play with them.

Cool.

Getting Started

Meanwhile, for those of you looking to get started with hacking on the JDK, there's a couple of things you need to do:

1. Fetch the prereqs.
2. Fetch the source through Mercurial.
3. Set up your build prompt.
4. Type "make" and go away for an hour or two.

Naturally, things are just a teeny bit more complicated than that, particularly if you're on a Windows box. (Update: Volker Simonis has blogged, similar to this post, his experience in configuring a Suse Enterprise Linux 9.3 box to build the OpenJDK.) Thus, as one who's labored through this process on said platform, I'll show you how I got it all up and running on my machine (well, technically, pseudo-machine, it's a VMWare image with Windows XP SP2 in it). If you have questions, you're free to send me email, but chances are I'll just redirect you to build-dev, where the amazing Kelly O'Hair hangs out and answers build questions. (MAJOR kudos again to Kelly for getting this done!)

Note that Sun is looking at other platforms, and has some good docs on building for Ubuntu 6.x and 7.x in the README already known--I've managed that with very little difficulty, so if you're starting from scratch, I'd suggest grabbing the free VMWare Player, an Ubuntu 7.04 Desktop image, and use that as a JDK build environment.

While we're at it, along the way, I'll be pointing out areas that I think Sun could use and would appreciate some community contributions. (People are always asking me this at conferences, as a way of making their name more well-known and building credibility in the industry.) Note that I have no "pull" with Sun and can't begin to guess where they want the help, I'm simply suggesting where I think the help would be useful--to the community if not to Sun directly.

By the way, the official document for build process requirements is the OpenJDK Build README; I only offer this up as a supplement and to offer up my own experiences. Note that I didn't go "all the way", in that I don't care about building the Java Plug-In or some of the other ancillary stuff, such as the JDK installer itself, so I didn't necessarily install all the bits documented on the OpenJDK README page. I wanted source that I could hack, step into, debug, and so on.

Without further ado....

1. Fetch the prereqs

This sort of goes without saying, but in order to build the JDK, you need to have the necessary tools in place.

• Environment. The build needs to be portable across both Windows and *nix, obviously, so OpenJDK needs a *nix-like environment on the Windows platform. Originally, back in the JDK 1.2 era, this was a commercial toolkit Sun purchased, the MKS Toolkit, but obviously that doesn't work for open source, so they've chosen to use Cygwin. You'll need to make sure you have a couple of tools explicitly documented by Sun (ar.exe, cpio.exe, make.exe, file.exe, and m4.exe), but realistically you'll want the rest of the Cygwin developer tools just to have a complete environment. Cygwin afficionados will (hopefully) weigh in with what the best packages mix to grab are; I just grabbed everything that looked tasty at the time, including stuff that had nothing to do with building OpenJDK. Note that there is a big problem with the make.exe that comes with Cygwin, however...  (Update: Poonam's Weblog notes, "Along with the default installation, we need to install Devel, Interpreters and Utils packages.")
  • On my system, I put this in C:\Prg\cygwin.
  • CONTRIBUTION: There's a lot of MSYS developers who would love it if you could figure out how to make the build work with their toolchain, too.... and don't ask me how, because I have no real idea; I'm totally n00b when it comes to the differences between the two.
• GNU Make 3.78.1 to 3.80. NOT the 3.81 or later version, because apparently there's problems with DOS-like paths, like C:/ (which is obviously going to be a factor on Windows; see some of Kelly's problems with shells for examples of what he's had to wrestle with). Getting this turned out to be a pain, and required some searching; Kelly pointed out a patched make.exe on Cygwin that should work. Make sure this make appears before the regular Cygwin make in your shell prompt (later). (Update: Find the right one here, as well.)
  • On my system, I created a directory for the OpenJDK project, in C:\Prg\OpenJDK, and put GNU make in C:\Prg\OpenJDK\bin.
  • CONTRIBUTION: Either fix the damn PATH issues ("Dear Mr. Gates, Would you please tell your developers to use the right kind of slash, already?"), or else fix Cygwin's make to recognize C:/ (note the forward slash) paths. Not sure what else could be done here.
• Compiler. Sun originally was using the Microsoft Visual C++ 6.0 environment (which made sense at the time, since they were a company and could pay for it); with the open-source release... well... turns out that they're still using the Microsoft tools, only they've also added Microsoft Visual Studio 2003 to the list of supported compilers. (I'm using MSVS2003 myself.) This is a pain because both of those environments are commercial, and the Visual C++ 2005 Express (which is free) doesn't seem to work yet. People have suggested other compilers (Watcom's OpenC++, Cygwin's gcc, and so on), but right now that doesn't sound like a high priority for Sun. Of course, it's fair to suggest that if you're building for Windows, you probably have a MSVS installation somewhere, but still, it'd be nice....
  • On my system, I put MSVS2003 in C:\Prg\MSVS2003. Note that I *just* installed the Visual C++ bits, and left out the rest of the .NET stuff. (I do .NET in another VMWare image, so I don't need it here.) To make our life simpler, reister the environment variables globally. (This is an install option.)
  • CONTRIBUTION: Port the makefiles to use Visual C++ 2005 Express, or one of the other free compilers. I would think the easiest transition would be to VC2005Ex, but there may be some tools missing from the Express editions that the build needs; I haven't spent much time here researching what's working and not. This would likely be (much) harder for other compilers, owing to the differences in toolchains.
  • CONTRIBUTION: Port the makefiles to use Visual Studio 2008.
• DirectX SDK. Yes, you need the DirectX SDK, specifically the Summer 2004 9.X Update, for building some of the advanced graphics stuff. The Build README has it linked there, as well, and the link, as of this writing, is still good. Install it, then take note of the DXSDK environment variable--we'll need it later.
  • On my system, I put DirectX in C:\Prg\DirectX9SDK_062005.
• FreeType 2. This is something to do with fonts, and is needed to remove a commercial dependency that was holding up the OpenJDK from building about a year ago. Grab it, install it, and note the headers and lib directory. The FreeType download page has some links to pre-built stuff, but in the end, you just want freetype.dll, freetype.lib, and the various FreeType headers.
  • On my system, I put these in C:\Prg\OpenJDK\deps\freetype-igor (named after the helpful soul on build-dev who was kind enough to send me his pre-built FreeType bits). Note that underneath that directory, I have windows/freetype-i586/headers and /lib, which I'll use later for environment variables.
  • CONTRIBUTION: Put a "JDK-specific" bundle of FreeType somewhere on the Web for people to download and not have to build.
• A "bootstrap" JDK. Go grab JDK 1.6; you'll need this for building the Java bits. (I would hope this isn't a problem for you; if it is, you may want to quickly go to some other Web page. Any web page.)
  • On my system, this resides in C:\Prg\jdk1.6.0.
• Apache Ant. At least version 1.6.3, I'm using 1.7 without a problem.
  • On my system, this resides in C:\Prg\apache-ant-1.7.0.
• The "binary plugs". As much work has Sun has done to unencumber the JDK, there's still little bits and pieces that are commercial that they can't release the source to, so they put them into a "binary plugs" package and have you install it, then point to it in the build scripts, and copy those files over. They get versioned every time Sun releases a built JDK 7 bundle, but I don't think you need to grab this more than once; just the same, keep an eye on that as time goes on, and if you get weird build errors, check build-dev to see if the plugs changed.
  • On my system, this resides in C:\Prg\OpenJDK\deps\openjdk-binary-plugs. (The .jar file is an executable jar, so just "java -jar" it and tell it to install in C:\Prg\OpenJDK\deps; it adds the rest. It's done in 3 seconds, and consists of 1 file. Now you see why I wouldn't worry too much about this.)
• Mercurial. This is a distributed revision control system, and there's lots more to say about that at the Mercurial website. Its commands look somewhat similar to SVN, though definitely read "Distributed Revision Control with Mercurial" if you're going to be keeping up with the source trees as they go. You *want* the "forest" extension as part of your Mercurial binary, so grab the "Batteries Included" installer version. I went with the non-TortoiseHG version, and had to download all four of the released files off that page and install and uninstall and install and uninstall until I found one that worked (the "win32extrasp1.exe" version in the "dec" release list on Sourceforge).
  • On my system, Mercurial lives in C:\Prg\Mercurial. Put in on the PATH so you have access to "hg.exe".
  • CONTRIBUTION: Figure out what the differences are and post it someplace--how to get the "forest" extension installed and turned on in Mercurial was a pain; Google was of little to no help here. (Tag it as a comment to this blog entry, if you like, and I'll update the entry itself once I see it.)
  • Update: Daniel Fuchs blogs about how to get Mercurial's "forest" extension installed in your installation, in case you don't get the "Batteries Included" version:

    I simply cloned the forest repository in c:\Mercurial. In a cygwin terminal:

      cd c:/Mercurial
      hg clone http://www.terminus.org/hg/hgforest  hgforest
    

    Then I edited c:/Mercurial/Mercurial.ini and added the lines:

      [extensions]
      forest=c:/Mercurial/hgforest/forest.py
    

    as documented in the Mercurial Wiki.

• Optional: FindBugs. The build will start using FindBugs to do source-analysis to find bugs before they happen, so it's not a bad idea to have that as well.
  • On my system, this resides in C:\Prg\findbugs-1.2.1.

At this point, you should be ready to go.

2. Fetch the source.

Ivan's got it (mostly) right: just do this:

cd C:\Prg\OpenJDK

md jdk7

hg fclone http://hg.openjdk.java.net/jdk7/jdk7/ jdk7

(Don't use MASTER as he does, I don't think that works--that was just for the experimental repositories.) Don't forget the trailing slash, either, or you'll get an error saying something along the lines of http://hg.openjdk.java.net/jdk7%5Ccorba is not a valid URL or somesuch.

If your Mercurial doesn't have the "forest" extension, "fclone" won't work; Ivan's got tips on how to pull down the sub-repositories by hand, but I get nervous doing that because what if the list changes and I wasn't paying attention?

Your network will go wild for about twenty minutes or so, maybe longer, pulling lots of stuff down from the URL above. The sources should now reside in C:\Prg\OpenJDK\jdk7. Go browse 'em for a bit, if you feel so inclined. Get a good rush going, because this next step can be the tricky one.

Update: Volker Simonis ran into some issues with using Mercurial and an HTTP proxy, and found it difficult to find assistance across the Web. In the interests of making it easier for others, he's allowed me to republish his experience here:

I just had a real hard time to get the forest extension working and finally found out that it was because the forest extension doesn't honor the "http_proxy" environment variable. So I thought I'll post it here in case anybody else will face the same problem in order to save him some time. (If you're only interested in the solution of the problem, you can skip the next paragraphs and jump right to the end of this post).

I installed Mercurial and the forest extension as described in various places, here on the list and on the Web - that was the easy part:) Afterwards I could do:

/share/software/OpenJDK> hg clone http://hg.openjdk.java.net/jdk7/jdk7/ jdk7
requesting all changes
adding changesets
adding manifests 
adding file changes
added 2 changesets with 26 changes to 26 files
26 files updated, 0 files merged, 0 files removed, 0 files unresolved

So everything worked fine! Note that I'm behind a firewall, but Mercurial correctly picked up my http proxy from the "http_proxy" environment variable!

But now, everytime I tried 'fclone', I got the following error:

/share/software/OpenJDK> hg fclone http://hg.openjdk.java.net/jdk7/jdk7/ jdk7 
[.]
abort: error: Name or service not known

That was not amusing. First I thought I got something wrong during the installation of the forest extension. I than used the '--traceback' option to "hg" which told me that the error was in keepalive.py:

/share/software/OpenJDK> hg --traceback fclone http://hg.openjdk.java.net/jdk7/jdk7/ jdk7 
[.] 
Traceback (most recent call last):
...
File "/share/software/Python-2.5.1_bin/lib/python2.5/site-packages/mercurial/keepalive.py", 
line 328, in _start_transaction raise urllib2.URLError(err)
URLError: <urlopen error (-2, 'Name or service not known')>
abort: error: Name or service not known 

So I enabled the debugging output in keepalive.py and realized, that the first two connections to hg.openjdk.java.net where made trough the proxy, while the third one (the first that actually fetches files), wants to go directly to hg.openjdk.java.net, which badly fails:

/share/software/OpenJDK> hg fclone http://hg.openjdk.java.net/jdk7/jdk7/ jdk7
keepalive.py - creating new connection to proxy:8080 (1078835788)
keepalive.py - STATUS: 200, OK
keepalive.py - re-using connection to proxy:8080 (1078835788)
keepalive.py - STATUS: 200, OK 
[.]
keepalive.py - creating new connection to hg.openjdk.java.net (1078970092)
abort: error: Name or service not known

The problem can be fixed by adding the proxy settings to your .hgrc file, like this:

[http_proxy]
host=proxy:8080

where you have to replace "proxy:8080" with the name and the port of your real proxy host!

Volker's original email came from the buid-dev list, and if you have further questions about Mercrial and HTTP proxies, I'd suggest that as a resource.

3. Set up your environment.

This is where you'll spend a fair amount of time, because getting this right can be ugly. There's some environment variables that tell the build script where to find things, and we have to point out those things, like compiler location and such. If you installed everything in the same places I did, then the following, which I put into C:\Prg\OpenJDK\build_shell.sh, should work for you:

#!/bin/sh

# "External" bits (outside of OpenJDK path structure)
#
export ALT_BOOTDIR=C:/Prg/jdk1.6.0
export ANT_HOME=c:/Prg/apache-ant-1.7.0
export FINDBUGS_HOME=/cygdrive/c/Prg/findbugs-1.2.1

# OpenJDK flag (to make FreeType check pass)
#
export OPENJDK=true

export OPENJDK_HOME=C:/Prg/OpenJDK
openjdkpath=$(cygpath --unix $OPENJDK_HOME)

# OpenJDK-related bits
# (ALT_JDK_IMPORT_PATH fixes a corba bug; remove it later)
#
export ALT_JDK_IMPORT_PATH=$(cygpath --unix C:/Prg/jdk1.6.0)
export ALT_BINARY_PLUGS_PATH=$OPENJDK_HOME/openjdk-binary-plugs
export ALT_UNICOWS_DLL_PATH=$OPENJDK_HOME/deps/unicows
export ALT_FREETYPE_LIB_PATH=$OPENJDK_HOME/deps/freetype-igor/windows/freetype-i586/lib
export ALT_FREETYPE_HEADERS_PATH=$OPENJDK_HOME/deps/freetype-igor/windows/freetype-i586/include/freetype2
. $openjdkpath/jdk7/jdk/make/jdk_generic_profile.sh

# Need GNU make in front of Cygwin's; this is the only practical way to do it
#
PATH=$openjdkpath/bin:$PATH
export PATH

# Let people know this is an OpenJDK-savvy prompt
#
export PS1='OpenJDK:\[\e]0;\w\a\]\[\e[32m\]\u@${COMPUTERNAME}:\[\e[33m\]\w\[\e[0m\]\n\$ '

Note the UNICOWS_DLL thing in the middle; this was necessary in earlier builds, and I don't know if it still is. For now I'm not messing with it; if you discover that you need it, the Build README has links. (Update: Kelly confirmed that this is no longer necessary in the OpenJDK build. Yay!)

Note that I set the COMPILER_VERSION flag to tell the build script which compiler I'm using--if that's not set, the build fails pretty quickly, complaining that "COMPILER_VERSION" cannot be empty. (Update: Kelly mentions, "I suspect the reason you are having the COMPILER_VERSION problem is that the makefiles are trying to run cl.exe to get the version, and if the PATH/LIB/INCLUDE env variables are not setup right, cl.exe fails. Several people have run into that problem.")

Note that OPENJDK must be set, or the build process thinks this is a commercial build, and an early sanity-check to see what version of FreeType is running will fail. (Specfically, Sun builds a tool just to see if the code compiles; if it fails to compile, chances are you forgot to set this flag. That's been my problem, each and every time I tried to rebuild the OpenJDK build space. Hopefully I never forget it again.)

Note that I call into jdk_generic_profile.sh to do some more setup work; this gets all the MSVS2003 stuff into the PATH as well.

Be very careful with which path you use; sometimes the build wants C:/Prg style paths, and sometimes it wants /cygdrive/c/Prg style paths. Don't assume the script above is perfect--I'm still testing it, and will update this entry as necessary as I find bugs in it.

(Update: Kelly mentions, "Be careful putting cygwin before VS2003 in the PATH, cygwin has a link.exe and so does VS2003, you need the one from VS2003.")

From a Cygwin bash prompt,

cd /cygdrive/c/Prg/OpenJDK

. ./build_shell.sh

cd jdk7

make sanity

It will churn, think, text will go flying by, and you will (hopefully) see "Sanity check passed". If not, look at the (voluminous) output to figure out what paths are wrong, and correct them. Note that certain paths may be reported as warnings and yet the buld will still succeed, that's OK, as far as I can tell.

And no, I don't know what all of those environment variables are for. Kelly might, but I suspect there's a lot of built-up cruft from over the years that they'd like to start paring down. Let's hope.

4. Type "make" and go away for a while.

Specifically, type "make help" to see the list of targets.

OpenJDK:Ted@XPJAVA:/cygdrive/c/Prg/OpenJDK/jdk7
$ make help
Makefile for the JDK builds (all the JDK).

--- Common Targets ---
all -- build the core JDK (default target)
help -- Print out help information
check -- Check make variable values for correctness
sanity -- Perform detailed sanity checks on system and settings
fastdebug_build -- build the core JDK in 'fastdebug' mode (-g -O)
debug_build -- build the core JDK in 'debug' mode (-g)
clean -- remove all built and imported files
clobber -- same as clean

--- Common Variables ---
ALT_OUTPUTDIR - Output directory
OUTPUTDIR=./build/windows-i586
ALT_PARALLEL_COMPILE_JOBS - Solaris/Linux parallel compile run count
PARALLEL_COMPILE_JOBS=2
ALT_SLASH_JAVA - Root of all build tools, e.g. /java or J:
SLASH_JAVA=J:
ALT_BOOTDIR - JDK used to boot the build
BOOTDIR=C:/Prg/jdk1.6.0
ALT_JDK_IMPORT_PATH - JDK used to import components of the build
JDK_IMPORT_PATH=c:/Prg/JDK16~1.0
ALT_COMPILER_PATH - Compiler install directory
COMPILER_PATH=C:/Prg/MSVS2003/Common7/Tools/../../Vc7/Bin/
ALT_CACERTS_FILE - Location of certificates file
CACERTS_FILE=/lib/security/cacerts
ALT_DEVTOOLS_PATH - Directory containing zip and gnumake
DEVTOOLS_PATH=/usr/bin/
ALT_DXSDK_PATH - Root directory of DirectX SDK
DXSDK_PATH=C:/Prg/DIRECT~1
ALT_MSDEVTOOLS_PATH - Root directory of VC++ tools (e.g. rc.exe)
MSDEVTOOLS_PATH=C:/Prg/MSVS2003/Common7/Tools/../../Vc7/Bin/
ALT_MSVCRT_DLL_PATH - Directory containing mscvrt.dll
MSVCRT_DLL_PATH=C:/WINDOWS/system32
WARNING: SLASH_JAVA does not exist, try make sanity
WARNING: CACERTS_FILE does not exist, try make sanity

--- Notes ---
- All builds use same output directory unless overridden with
ALT_OUTPUTDIR=<dir>, changing from product to fastdebug you may want
to use the clean target first.
- JDK_IMPORT_PATH must refer to a compatible build, not all past promoted
builds or previous release JDK builds will work.
- The fastest builds have been when the sources and the BOOTDIR are on
local disk.

--- Examples ---
make fastdebug_build
make ALT_OUTPUTDIR=/tmp/foobar all
make ALT_OUTPUTDIR=/tmp/foobar fastdebug_build
make ALT_OUTPUTDIR=/tmp/foobar all
make ALT_BOOTDIR=/opt/java/jdk1.5.0
make ALT_JDK_IMPORT_PATH=/opt/java/jdk1.6.0

OpenJDK:Ted@XPJAVA:/cygdrive/c/Prg/OpenJDK/jdk7
$

The one I want is "make fastdebug_build" or "make debug_build" (so I have debug symbols and can go spelunking). Do it.

Watch the stuff go flying by.

Get bored.

Get lunch.

Come back, it might be done.

If it's a successful build, you'll have "stuff" in the C:\Prg\OpenJDK\jdk7\build directory, corresponding to the kind of build you kicked off; for a "fastdebug" build, for example, there'll be a "windows-i586-fastdebug" directory in which you'll find a "j2sdk-image" directory in which there should be a complete JDK environment. Try running Java and see if it works (make sure your PATH is updated to point to the right place before you do!)

If not, it's debugging time. Note that the "build" directory is completely built from scratch, so if you get a partial build and want to start over from scratch, just "rd /s build" from the jdk7 directory. (It's easier than "make clean" or "make clobber", I've found.)

More About Builds

When building the JDK, you may want to build bits "underneath" the top-level directory, but doing this is a tad tricky. I asked about this on the build-dev list, and Kelly responded with a great email about the build process, specifically about launching "sub" makes within the system:

Due to history, a build directly from the jdk/make directories uses a default OUTPUTDIR of jdk/build/* but if FASTDEBUG=true, it's jdk/build/*-fastdebug, or if a plain debug build with just VARIANT=DBG it would be jdk/build/*-debug The variant builds leave results in a completely separate outputdir.

If you used the very top level makefile (which came from the now defunct control/make area) the default OUTPUTDIR is ./build/* (at the very top of the repositories). When this top level Makefile runs the jdk/make Makefiles, it passes in a ALT_OUTPUTDIR to refer to this top level build result area because it's default outputdir is not the same place.

I don't know the complete history as to why this was done this way, but my tendency is to try and get us back to a single default OUTPUTDIR for all the repositories. Someday...

This is what I do when I work on just the jdk repository:
cd jdk/make && gnumake

That primes the outputdir area, then I can drop down in:
cd jdk/make/java && gnumake

Or even drop in and clean an area and re-build it:
cd jdk/make/jpda && gnumake clean && gnumake

Or just repeat the entire build (incremental build)
cd jdk/make && gnumake

If I wanted the jdk image (j2sdk-image), I would need to:
cd jdk/make && gnumake image

But the output by default will go to jdk/build/* and a different directory if VARIANT=DBG or FASTDEBUG=true.

This should help having to go through the whole process for incremental updates. (Note that on my system, I had to call it "make" instead of "gnumake".)

Futures

As time goes on, I'll hopefully find the time to blog about how to find various little things in the JDK and make source-modifications to prove that they work, and use that as a springboard from which to let you start hacking on the JDK. In the meantime, please, if you run into trouble and find fixes to any of the above, comment or email me, and I'll correct this.

Contributions/Suggested TODOs

• Have the make system automatically capture the results of the build in a log file, for easier debugging. Granted, you can "make >| build.output", but that seems tedious when it could easily be captured automagically for you each time. ("make sanity" does this, capturing the results in build/windows-i586-fastdebug/sanityCheckMessages.txt and sanityCheckWarnings.txt.
• Documentation, documentation, documentation. This thing does a lot of recursive makes and invokes a lot of tools (some of them built as part of the build process itself), and it would be much easier to debug and understand if the process were better documented. Even a simple flowchart/tree of the various Make invocations and what each does (in practice) would be helpful when trying to track down issues.
• Add support for all output to be captured into a build log file. This can obviously be done at the command-line via "make |& tee build.log" or ">& build.log", but I think it'd be nice if it were somehow folded in as part of the build process so it happened automatically.

Updates

1. Kelly OHair sent me some updated information, such as the right link to use for the README file (from the HG repository instead of the SVN one).
2. Kelly also mentioned that the plugin and installers are not part of the OpenJDK build yet, so that's not something I could build even if I wanted to. Which is fine, for me.
3. Kelly confirmed that UNICOWS is not needed in OpenJDK.
4. Kelly mentions that link.exe on the PATH must be VS2003's, not Cygwin's.
5. Kelly mentions the COMPILER_VERSION problem might be PATH issues.
6. Kelly notes, "On the C:\ vs C:/ vs. /cyg*/ paths, I try and use the C:/ all the time everywhere, and anywhere in the Makefiles where I know I need C:\ or the /cyg*/ paths, I try and use cygpath or MKS dosname to force them into the right form. NMAKE.EXE only likes C:\ paths, and cygwin PATH only wants /cyg*/ paths, and Windows/Java/MKS never want /cyg*/ paths. :^( I wish we had a better solution on Windows for this shell/path mania."
7. Poonam's Weblog has a good page on building the OpenJDK on Windows with NetBeans, from which I've stolen... ahem, leveraged... some links. His webpage has a link for the UNICOWS download page, but that only includes the DLL, not the .LIB, which you will also need. (It's an import library for the DLL; you need both.) The only place I know to get the .LIB is from the Platform SDK, and you need an old one, circa November 2001 or so. I don't know if it's kosher to redistribute any other way (meaning, Microsoft won't let us, as far as I know).
8. Added Volker Simonis' experiences with HTTP proxies in Mercurial
9. Added the "sub" make build discussion from Kelly on build-dev
10. Added the link to Volker's blog about building on Suse Linux

From the cover of Dr. Dobb's Journal (Jan/2008):

PHP: The Power Behind Web 2.0

Only because you're saying "Web 2.0", mind you. *shudder* All those in favor of "alt.death.to.tim.oreilly.for.coining.that.phrase" newsgroup, say Aye....

You know, for years now, I've had just a terrible time making out Christmas lists.

The fact is, I'm fortunate enough to be in a position where if I really want something, I'm usually able to just go out and get it. If it's more expensive than what a single trip to the store armed with a credit card can obtain, then it's probably something I don't really need. (High-end gaming desktop machines do not fall into that category, no matter what my family might suggest.) So, it's rather difficult to come up with a list of things for friends and family to buy for me, particularly since impulse control is... well... almost entirely lacking in my case.

One year, I just said, "Make a charitable donation in my name to your favorite charity." I thought this was a nice solution, but it didn't go over incredibly well, probably because my family likes to see the whole "wow, thanks!" while we're all sitting underneath the tree. I ended up with the usual range of CDs, books, games, and so on. Now, mind you, I didn't return any of those things, but still, I felt vaguely guilty at my rampant consumerism when it's apparent with just a glance through any world-facing publication that there's a lot of people who could've used that money to survive for the year on what one CD cost.

I just figured out what I want for Christmas.

Those few of you who are non-geeks may not have heard of the "One Laptop Per Child" project: it's a charitable group that has created a (relatively) inexpensive laptop designed for use by underprivileged children to help close the gap between the technology "haves" and "have-nots". Now, in of itself, this is a Good Thing, but for a limited time, the foundation is sweetening the pot significantly:

From now through December 31, 2007, OLPC is offering a Give One Get One program in the United States and Canada. This is the first time the revolutionary XO laptop has been made available to the general public. For a donation of $399, one XO laptop will be sent to empower a child in a developing nation and one will be sent to the child in your life in recognition of your contribution. Up to $200 of your donation may be tax-deductible (your $399 donation minus the fair market value of the XO laptop you will be receiving).

For all U.S. donors who participate in the Give One Get One program, T-Mobile is offering one year of complimentary HotSpot access.

If you participate in Give One Get One today, you will receive your XO laptop(s) in early 2008. Your donated laptop will reach a child in Afghanistan, Cambodia, Haiti, Mongolia or Rwanda in the same early 2008 timeframe.

Aside from the fact that the T-Mobile access is a useful thing for a guy who's one the road close to 75% of his life, and aside from the fact that my family can take the "get one" laptop and give it to my 8-year-old as a starter laptop (or take it down to the local women's shelter and give it to a child there who probably would otherwise never set fingers on a working computer keyboard), the simple fact is, it's a nifty little gadget. Granted, I'm not going to give up my MacBookPro anytime soon over it, but it's got enough functionality to access wireless networks, and provides a Smalltalk-based environment and Python-based environment in which people can learn to program, and what's not to like about that?

Dear Santa--

I think I've been a good boy this year, but there are others in the world who've been better than me in circumstances that I can't even begin to imagine. So please, this year, make a donation to the One Laptop Per Child initiative, in either my name or yours.

Thanks, from both myself and someone neither of us has met yet,

Ted

I grieve with him. I can't imagine how the babysitters feel right now--at age 13 and 11, to have to make the hard choices any babysitter has to make in a the event of a fire, and to have to live with that choice for the rest of your life. To be the parents, to have to live with the "what-might-have-beens" and the "if-onlys".

And to have to learn, as Brian talks about, how to go on.

No parent should ever have to bury their child.

From wikipedia:

The Turing test is a proposal for a test of a machine's capability to demonstrate intelligence. Described by Alan Turing in the 1950 paper "Computing machinery and intelligence," it proceeds as follows: a human judge engages in a natural language conversation with one human and one machine, each of which try to appear human; if the judge cannot reliably tell which is which, then the machine is said to pass the test.

From C.NET: "Warning sounded over 'flirting robots'"

Those entering online dating forums risk having more than their hearts stolen.

A program that can mimic online flirtation and then extract personal information from its unsuspecting conversation partners is making the rounds in Russian chat forums, according to security software firm PC Tools.

The artificial intelligence of CyberLover's automated chats is good enough that victims have a tough time distinguishing the "bot" from a real potential suitor, PC Tools said. The software can work quickly too, establishing up to 10 relationships in 30 minutes, PC Tools said. It compiles a report on every person it meets complete with name, contact information, and photos.

"As a tool that can be used by hackers to conduct identity fraud, CyberLover demonstrates an unprecedented level of social engineering," PC Tools senior malware analyst Sergei Shevchenko said in a statement.

Among CyberLover's creepy features is its ability to offer a range of different profiles from "romantic lover" to "sexual predator." It can also lead victims to a "personal" Web site, which could be used to deliver malware, PC Tools said.

Daaaaaamnnnnnn... you can offer a range of different profiles. From "romantic lover" to "sexual predator".

Can the "Make You Happy All Night Long" Web Service be far behind?

I would love to get my hands on this code, though. Think about what the natural language parsing routines must be like, particularly considering they have to be able to parse smileys, too.

If they can mimic flirting, how far away are we from code that mimics online help agents? It'd be like voicemail, but worse, since you could never know if you were talking to a real person or not....

This is, without a doubt, the most accurate quote ever about the "fun" of writing a book:

Writing a book is an adventure. To begin with, it is a toy and an amusement; then it becomes a mistress, and then it becomes a master, and then a tyrant. The last phase is that just as you are about to be reconciled to your servitude, you kill the monster, and fling him out to the public. (Source: Winston Churchill)

Keep that in mind, all you who are considering authoring as a career or career supplement.

Were I to offer my own, it would be like so:

Writing a book is like having a child.

Trying is the best part, in some ways. You have this idea, this burning sensation in your heart, that just has to get out into the world. But you need a partner, a publisher who will help you bring your vision to life. You write proposals, you write tables of contents, you imagine the book cover in your mind. Then, YES! You get a publisher to agree. You sign the contract, fax it in, and you are on the way! We are authoring!

At first, it is wonderful and exciting and full of potential. You run into a few hangups, a few periods of nausea as you realize the magnitude of what you're really doing. You resolve to press on. As you continue, you begin to feel like you're in control again, but you start to get this sense like it's an albatross, a weight around your neck. Before long, you're dragging your feet, you can't seem to muster the energy to do anything, just get this thing done. The deadline approaches, the sheer horror of what's left to be done paralyzes you. You look your editor in the eye (literally or figuratively) and say, "I can't do this." The editor says, "Push". You whimper, "Don't make me do this, just cancel the contract." The editor says, "Push". You scream at them, "This is YOUR fault, you MADE me do this!" The editor says, "Push". Then, all of a sudden, it's done, it's out, it's on the shelf, and you take photos and show it off to all the friends, neighbors and family, who look at you a little sympathetically, and don't mention how awful you really look in that photo.

As the book is out in the world, you feel a sense of pride an joy at it. You imagine it profoundly changing the way people look at the world. You imagine it reaching bestseller lists. You're already practicing the speech for the Nobel. You're sitting in your study, you reach out and grab one of the free copies still sitting on your desk, and you open to a random page. Uh, oh. There's a typo, or a mistake, or something that clearly got past you and the technical reviewers and the copyeditors. Damn. Oh, well, one mistake can't make that much difference.

Then the reviews come in on Amazon. People like it. People post good reviews. One of them is not positive. You get angry: this is your baby they are attacking. How DARE they. You make plans to find large men with Italian names and track down that reviewer. You suddenly realize your overprotectiveness. You laugh at yourself weakly. You try to convince yourself that there's no pleasing some people.

Then someone comes up to you at a conference or interview or other gathering, and says, "Wow, you wrote that? I have that book on my shelf!" and suddenly it's all OK. It may not be perfect, but it's yours, and you love it all the same, warts and all.

Nearly a dozen books later, it's always the same.

Michael Nygard (author of the great book Release It!), writes that "[his] definition of 'done' continues to expand". Currently, his definition reads:

A feature is not "done" until all of the following can be said about it:

1. All unit tests are green.
2. The code is as simple as it can be.
3. It communicates clearly.
4. It compiles in the automated build from a clean checkout.
5. It has passed unit, functional, integration, stress, longevity, load, and resilience testing.
6. The customer has accepted the feature.
7. It is included in a release that has been branched in version control.
8. The feature's impact on capacity is well-understood.
9. Deployment instructions for the release are defined and do not include a "point of no return".
10. Rollback instructions for the release are defined and tested.
11. It has been deployed and verified.
12. It is generating revenue.

Until all of these are true, the feature is just unfinished inventory.

As much as I agree with the first 11, I'm not sure I agree with #12. Not because it's not important--too many software features are added with no positive result--but because it's too hard to measure the revenue a particular program, much less a particular software feature, is generating.

My guess is that this is also conflating the differences between "features" and "releases", since they aren't always one and the same, and that not all "features" will be ones mandated by the customer (making #6 somewhat irrelevant). Still, this is an important point to any and all development shops:

What do you call "done"?

For those who've seen me give presentations at the No Fluff Just Stuff symposia over the year who've also asked to get the demo code that I write during the presentations, I've put it all up online. Simply find the show you attended below, and the hyperlink downloads the associated .zip file:

• Boulder JUG
• Denver JUG
• Phoenix JUG
• Atlanta
• Austin
• Boston
• Chicago
• Columbus
• Dallas
• Denver
• Green Bay
• London
• Miami
• Milwaukee
• Minneapolis (1st show)
• Minneapolis (2nd show)
• Oklahoma City
• Omaha
• Orlando
• Phoenix
• Portland
• Princeton, NJ
• Raleigh, NC
• Reston, DC
• Seattle
• St Louis
• Toronto

Usual disclaimer applies, as with all demo code: I don't guarantee the code for any particular purpose except as a supplement to the lecture that was given at that time in that city, the code isn't guaranteed to do anything except soak up space on your hard drive, I can't be held responsible if you use it in production, and so on.

If any of the links are broken, email me and let me know, but I think they should all work.

... for Ruby, or PowerShell/.NET? I'm looking for something to make it easier to use WebDAV from a shell scripting language on Windows; Ruby and PowerShell are the two that come to mind as the easiest to use on Windows. For some reason, Google doesn't yield much by way of results, and I've got to believe there's better WebDAV support out there than what I'm finding.

(Yes, I could write one, but why bother, if one is out there that already exists? DRY!)

BTW, anybody who finds one and wants credit for it, I'll be happy to post here. If you're a commercial vendor and you send me a license to go with it, I'll post that, too, with some code and explanation on how I'm using it, though I doubt it's going to be all that different from how anybody else would use it.

Some of you knew me way back when I started the domain name "javageeks.com". Obviously, it's been a while since I updated that site, and in fact, if you go try to read some of the papers there, you'll get redirected to "neward.net", which has been on hiatus ever since the ISP there decided that my use of JSP was somehow trashing their server. (This is what happens when college students run an ISP in the spare time, I guess.)

Anyway, for a variety of reasons, I've since decided that my online presence should coalesce into one place, this domain (tedneward.com), and I've been slowly moving things over to this site in recognition of that fact. As a result, a bunch of the papers that I'd put up for reading on javageeks.com are now here on tedneward.com, in the "Writing" section of the website.

For those of you too tied to RSS to go visit the site itself, here's a list of the papers I've moved over thus far:

• Using the Boot Classpath
• Finding Loaded Classes
• Using Java Policy
• When is a Static Not Static?
• Multiple Java Homes
• X-Power: Using XML to Write Papers

It's not the complete list, since I'm also (trying to) move the authoring format over from my own home-grown XML+XSLT system into DocBook, but that means either developing an XSLT to transform my home-grown format into DocBook, or else just manually switching them over. I'm not convinced yet which is the faster way to go.

Part of it isn't just transmuting their form, though. Some of them could use with a touch of updating (such as the Java Policy one--there's been a few changes there since I wrote that paper, back in Java 1.2 days), so I think I'm going to have to labor through and transform and update them manually. Still, it could be worse: I could have written them in Lotus Ami Pro....

Now, I've had my issues with corporate management before (some of you may recall my run-in with Sun lawyers over the domain "javageeks.com"), and I've seen corporations behave badly with cease-and-desist letters (Hey, Microsoft... you and RedHat can come out of the corner now...), but this action on Sun's part to effectively muscle out the four project leads on the open-source OpenDS project definitely deserves some kind of award.

(For those of you who want the short version: four Sun employees, including Neil Wilson, the poster, have been working on the OpenDS project for the last n number of years. Sun decides to offshore its directory services development to Europe (WTF?!?), and pink-slips the four project owners. The project owners decide to continue working on the project, and decide that there should be a fifth owner, a Sun employee, in order to make sure Sun keeps a hand in a project it vested time & energy & money into. Sun decides instead to require that ALL the project owners be Sun employees. Project owners, a tad distrubed, circle back to discuss things. Sun management threaten to yank all four ex-employees' severance benefits if they fight this. Project owners basically decide to bail entirely, and not anger the Sun Gods any further, until after the benefits have been paid out. Project owner Wilson blogs the aforementioned story, which annoys a few dark corners of the blogosphere, including yours truly.)

In fact, in honor of the book "In Search of Stupidity", I hereby nominate Sun for a Stupidity Award, in this case, a two-parter: one, "For having the incredible temerity to threaten with termination the very core group of people who built the product into what it was", and two, "For having the absolute chutzpah to effectively backstab the very community-oriented efforts that Sun upper management are seeking to build as a way to make the company once again relevant in the computing industry".

I dunno if anybody at Sun reads my blog, but if you do, forward this over to Simon Phipps and Jonathan Schwartz, because if they want to be taken seriously as an open-source company, they need to find the morons who acted with this kind of heavy-handedness and defenestrate them at the earliest opportunity. An open letter of apology and response should be posted on one or both blogs. Then, they need to go back to Mr. Wilson, apologize profusely, and ask him and the other three--very nicely--if they'll agree to take up the reins of the OpenDS project again and allow Sun to demonstrate--by actions, not words--that they are a community-focused company.

Or, I suppose, they could do none of this, and prove--by actions, not words--that they don't give a sh*t. (Frankly, I could care less about directory services projects, but I would think Sun cares far more about its reputation at this point than the project itself.)

'Tis your move, Sun. Are you "profoundly involved in the open source world", or not?

Update: Simon Phipps writes, in a comment here (which may not be accessible to some RSS readers, so I will quote in full):

Ted: I have investigated this situation. It's very regerttable that Sun had to let some great engineers go as part of a staff reduction, unrelated to the governance issue, but sadly that happens in every company. The real issue I was concerned about here was the discussion of governance. I discovered that Neil did not tell the whole story.

The original governance of OpenDS said this:

<blockquote>The OpenDS project has single, overall Project Lead. This Project Lead, who is appointed by Sun Microsystems, is responsible for managing the entire project, and is the final arbiter of all decisions.</blockquote>

At some point, without any community discussion I can find, and without management discussion, while still Sun employees, the leaders of OpenDS changed the governance to say this:

<<The OpenDS project has single, overall Project Lead. This Project Lead, who is appointed and removed by a majority vote of the Project Owners, is responsible for managing the entire project, and is the final arbiter of all decisions in the development process.>>

You can see the change in the Subversion system for OpenDS here.

When Neil says:

<<On November 14, 2007, a member of executive management within Sun’s software division contacted one of the recently-laid-off OpenDS project owners and demanded that the owners approve a governance change that would grant Sun full control of the OpenDS project.>> that is misleading. What actually happened is the leaders were asked to revert their arbitrary change of the governance (they also removed the phrase "Project Owners will always attempt to reach consensus before making decisions") so that the matter could be openly discussed.

I'm continuing to investigate and monitor, but right now the evidence I see is that Sun has not acted in bad-faith towards the governance. I'd welcome evidence to the contrary if it exists, but that SVN dif seems pretty conclusive to me.

There are, I think, two issues here that are getting conflated, and I want to tease them apart. What is not an issue, as far as I'm concerned, was Sun's decision to let these engineers go in the first place. Business decisions that involve letting people go are always hard decisions to make, and I won't second-guess the thought process that led up to it. I'm willing to believe that if Sun management felt they could have kept them in place, they would have. (You may choose to disagree but that's your own value judgement to make.) That doesn't take away from the two issues I want to discuss here.

First, there is the issue of whether or not Sun's actions were in accordance with the project governance model for the OpenDS project, which Simon seems to focus on above. Frankly, I could care less about this aspect of things; whether that change was appropriate, necessary, legal, whatever, is not the issue I want to discuss or point out. Yes, there is the possibility that Neil-and-company's actions were regarded as being contrary to the charter of the project by the legitimately-selected Project Owner, and Sun felt it necessary to retaliate legally. Even in that scenario, it strikes me as a heavy-handed reaction.

(Quite footnote: on a whim, after posting my update, I checked the SVN revisions beyond what Simon posted. The revision change in question, from 1622 to 1739, occured on April 28th, 2007, which is close enough to have been part of this whole fiasco, but obviously I'm wasn't there, I can't know that for certain. Needless to say, if all these changes occurred without any community input, that's clearly not playing to the spirit of the open source community, IMHO.)

Secondly, there is the issue of Sun's threat to revoke their severance benefits. To me, this is unconscionable. Severance is a way of providing valuable employees with a "cover period" while they seek new employment ("we were encouraged to use this time to find a new job", Neil reports). This is a nice gesture, a way of saying "thank you" for the valuable efforts of a valued employee. But to turn around and threaten to remove those benefits if the employees don't play ball on a job-related issue when those employees are seeking to further the company's stated cause seems heavy-handed and unnecessary.

Look, in order to be seen as effective without being heavy-handed, keep the scope of the response within the frame of the supposed offense: If they are violating the governance model, then threaten to remove them from the project. If they are seeking to use the code in a way that is violating the license of the original, then threaten legal action over the licensing model and those actions. To go straight for the severance benefits here seems akin to Pres. Bush reaching for the nuclear codes when a foreign national jaywalks. The punishment must fit the crime. And the crime has to be proven first, in any event. The risks of a "preventative first strike" are only acceptable when the liabilities are equally as high, if not higher.

Look at it this way: the way this played out, Sun...

1. ... kept control over the OpenDS project... which they were going to have anyway, under either side's story.
2. ... still paid out the severance benefits for those engineers.
3. ... earned a negative PR hit among the very people they're trying to convince, the open-source community.
4. ... earned the enmity of the core developers working on the OpenDS project, which I presume they would have preferred to have continue working on the project, even if they no longer paid their salary.
5. ... forced their Chief Open Source Officer to spend time investigating this whole sordid mess.

What might have happened if Sun had simply approached the engineers in question and instead of threatening to yank their benefits, had said "Look, we understand what you're trying to do, but as we understand it, the project's governance statement says otherwise... We value you guys, we value what you did, we want you to continue to contribute, how do we resolve this amicably?" Sun might have had to compromise--noting, by the way, that they might have done this even when legally or morally they didn't have to--and would have instead...

1. ... kept control over the OpenDS project.
2. ... still paid out the severance benefits for those engineers.
3. ... earned a positive PR boost at best, no PR at worst, among the community.
4. ... kept the goodwill of the developers who represented Sun's efforts on the project.
5. ... had a "feel-good" story their Chief Open Source Officer could use when addressing the OpenDS group at the next directory services conference. ("Look, Sun is committed to open source, even when we can't contribute the resources financially, we can prove this by...")

Sometimes, even when you're right, you get more from taking the position that there is room to compromise on your end. Flies, honey and vinegar, and all that. I applaud Simon's efforts to get involved and track this one down, but I don't think this is an open-source issue.

I think this is a management issue.

When you can't do something simple with it.

Check out this absurdly simple Groovy script:

import javax.management.ObjectName
import javax.management.MBeanServerConnection
import javax.management.remote.JMXConnectorFactory as JMXFactory
import javax.management.remote.JMXServiceURL as JMXUrl

def serverUrl = 'service:jmx:rmi:///jndi/rmi://127.0.0.1:9004/jmxrmi'
def serv = JMXFactory.connect(new JMXUrl(serverUrl))
def on = new ObjectName('Catalina:type=Server')
def gmb = new GroovyMBean(serv, new ObjectName('Catalina:type=Server')).serverInfo

For those of you not up on your JMX, this is a simple connection via an RMI connector to the JMX server running at port 9004 (which happens to be my local Tomcat installation). This is straight off the Groovy-JMX documentation page, but slimmed down because the GroovyMBean constructor throws an exception, claiming that the desired constructor cannot be found:

C:\Projects\Exploration\Groovy>groovy GroovyJMX.groovy
Caught: groovy.lang.GroovyRuntimeException: Could not find matching constructor
for: groovy.util.GroovyMBean(javax.management.remote.rmi.RMIConnector, javax.management.ObjectName)
at GroovyJMX.run(GroovyJMX.groovy:9)
at GroovyJMX.main(GroovyJMX.groovy)
C:\Projects\Exploration\Groovy>

C'mon, folks. I have no idea what the problem is, and debugging this is a nightmare. I've looked around various Groovy forums, and nobody appears to have any real idea what's going on. Either nobody is really using Groovy as a JMX client (in which case, just remove the GroovyMBean from the library), or else Groovy has a bug within it (thus reducing its efficicacy as a production-ready language).

I'm fully willing to accept that the problem is with me or my environment somehow. The challenge, however, is for somebody to take a stock JDK 1.6 and Groovy 1.0 download (oh, and 1.1-rc2 fails with the same error, so that's not the issue, either), run the above 8-line script, and tell me why mine isn't working. (I've already done the suggested step of removing the mx4j jar out of the groovy-1.0/lib directory, so that doesn't help, either.)

Oh, and if you're going to write in claiming that this is a ClassLoader issue or something, you'd be wrong--all of the JMX types being loaded are coming out of rt.jar (which I verified using -verbose:class, doing which required me to edit the Groovy launcher scripts, which I find to be just silly--don't make it hard for me to use the basic management & monitoring facilities of the JVM). The only ClassLoader player I don't know for certain is the org.codehaus ClassLoader that's established by Groovy itself, so if the problem is in ClassLoaders, it's inside of the Groovy implementation, which means it's a Groovy problem, not mine.

I've even taken the step to compile the Groovy code into .class files, and run those:

C:\Projects\Exploration\Groovy>groovyc -d compiled GroovyJMX.groovy
C:\Projects\Exploration\Groovy>cd compiled

C:\Projects\Exploration\Groovy\compiled>java -classpath .;libg\groovy-1.0.jar;li
bg\asm-2.2.jar GroovyJMX
Exception in thread "main" groovy.lang.GroovyRuntimeException: Could not find ma
tching constructor for: groovy.util.GroovyMBean(javax.management.remote.rmi.RMIC
onnector, javax.management.ObjectName)
at groovy.lang.MetaClassImpl.invokeConstructor(MetaClassImpl.java:776)
at groovy.lang.MetaClassImpl.invokeConstructor(MetaClassImpl.java:688)
at org.codehaus.groovy.runtime.Invoker.invokeConstructorOf(Invoker.java:
163)
at org.codehaus.groovy.runtime.InvokerHelper.invokeConstructorOf(Invoker
Helper.java:140)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeNewN(ScriptBy
tecodeAdapter.java:243)
at GroovyJMX.run(GroovyJMX.groovy:9)
at gjdk.GroovyJMX_GroovyReflector.invoke(Unknown Source)
at groovy.lang.MetaMethod.invoke(MetaMethod.java:115)
at org.codehaus.groovy.runtime.MetaClassHelper.doMethodInvoke(MetaClassH
elper.java:713)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:560)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:450)
at org.codehaus.groovy.runtime.Invoker.invokeMethod(Invoker.java:131)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.
java:111)
at org.codehaus.groovy.runtime.InvokerHelper.runScript(InvokerHelper.jav
a:408)
at gjdk.org.codehaus.groovy.runtime.InvokerHelper_GroovyReflector.invoke
(Unknown Source)
at groovy.lang.MetaMethod.invoke(MetaMethod.java:115)
at org.codehaus.groovy.runtime.MetaClassHelper.doMethodInvoke(MetaClassH
elper.java:713)
at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:664)
at org.codehaus.groovy.runtime.Invoker.invokeMethod(Invoker.java:111)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.
java:111)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(Scrip
tBytecodeAdapter.java:187)
at GroovyJMX.main(GroovyJMX.groovy)

C:\Projects\Exploration\Groovy\compiled>

And here we can obviously see, by the light of the .jars passed in, that the mx4j jars are nowhere to be found (unless, of course, Groovy is explicitly searching the hard drive for them, because I deleted them entirely out of Groovy's lib directory).

I've asked a couple of the Groovy heavyweights (not mentioning anybody by name) if they know what's up. Silence. Not a good sign.

I'm about to go off and try the same thing using JRuby. If it works, out of the box, then the problem is with Groovy, not with me. If you're a Groovy expert, or know someone who is, then have them email me the solution (assuming they can find one), because this is a point where I'm about to close the door on Groovy forever: if you can't do something simple with it, it's not ready for prime-time. It may be OK to whip up dirt-simple websites where 90% of the stuff is pre-generated, but if I can't use it for something like being a JMX client, then it ain't worth my time.

'Nuff said.

Update: OK, I may have to eat my words.

Playing around with some JRuby/JMX stuff (which is absurdly simple, even without jmx4r, which I'm trying to get gem to install right now), I went back and decided to try a little JMX without using GroovyMBean:

import javax.management.*
import javax.management.remote.*
import java.lang.management.*

def serverUrl = 'service:jmx:rmi:///jndi/rmi://127.0.0.1:9004/jmxrmi'
def connector = JMXConnectorFactory.connect(new JMXServiceURL(serverUrl))
def mbsc = connector.mBeanServerConnection

def memory_mbean =
ManagementFactory.newPlatformMXBeanProxy(mbsc, "java.lang:type=Memory", MemoryMXBean.class)

This, by the way, is almost a straight port of Jeff's corresponding JRuby/JMX example code. Works just fine. From there, I thought, "Let's see if I can get past the problem I was having with GroovyMBean a few minutes ago." So I add the one-line GroovyMBean constructor (taking the mBeanServerConnection as the first parameter):

def gmb = new GroovyMBean(mbsc, new ObjectName('Catalina:type=Server')).serverInfo

Voila! Success! A quick execution test verifies that I'm all good:

def query = new ObjectName('Catalina:*')
String[] allNames = mbsc.queryNames(query, null)
def modules = allNames.findAll{ name ->
    name.contains('j2eeType=WebModule')
}.collect{ new GroovyMBean(mbsc, it) }

println "Found ${modules.size()} web modules. Processing ..."

modules.each{ m ->
    println "Found ${m.name()} at ${m.path} (${m.processingTime})"
}

returns:

C:\Projects\Exploration\Groovy>groovy GroovyJMX.groovy
Found 5 web modules. Processing ...
Found Catalina:j2eeType=WebModule,name=//localhost/docs,J2EEApplication=none,J2E
EServer=none at /docs (0)
Found Catalina:j2eeType=WebModule,name=//localhost/host-manager,J2EEApplication=
none,J2EEServer=none at /host-manager (0)
Found Catalina:j2eeType=WebModule,name=//localhost/examples,J2EEApplication=none
,J2EEServer=none at /examples (0)
Found Catalina:j2eeType=WebModule,name=//localhost/,J2EEApplication=none,J2EESer
ver=none at (0)
Found Catalina:j2eeType=WebModule,name=//localhost/manager,J2EEApplication=none,
J2EEServer=none at /manager (0)
C:\Projects\Exploration\Groovy>

Now if that ain't cool, I don't know what is.

Now for the hard part: What happened? Why'd my earlier code fail?

Looking back at my example script above, I clearly see that somewhere along the way in my debugging/exploration, I accidentally left out the call to obtain the MBeanServerConnection from the Connector and pass that in to the GroovyMBean constructor. Ugh.

Formal apologies to the Groovy crowd. However, I stipulate, for your own consideration, that

1. Originally my script was not so--I copied it line-for-line from the Groovy/JMX page, and I got the GroovyCastException that led me down this path. (You decide whether you believe me or not. )
2. The GroovyMBean constructor could (and should) be overloaded to take a Connector and extract the MBeanServerConnection from it and proceed. In fact, I'd suggest that the GroovyMBean should be written to take a JMXServiceURL as a parameter and handle all the details of connecting to the remote server.
3. Exploration tests would have yielded a better record of my efforts, and maybe found the point where my coding got led astray. It would also help track down the problem for others, if there was a language issue at stake here. (There may be--one noticeable difference is that in the working version, I don't use the import-redeclaration feature, whereas in the non-working version I did. More research is required.) At the time, I was just trying to debug a simple script problem, but at some point, greater rigor on my part should have kicked in so I could have better results to work from. Sigh.
4. Debugging this was way too hard. As we move into an era of more languages-atop-VMs, better debugging/spelunking facilities has to be a top priority.
5. Groovy (and JRuby) need to make sure they honor--somehow--the various flags I want to pass to the Java VM, such as -verbose or -client/-server, or even -ms or -mx. Java does have the JAVA_TOOLS_OPTIONS environment variable that will be picked up by the Java launcher (java.exe), but that only works if the language in question uses the launcher itself as part of its startup script. Both Groovy and JRuby seem to fail this test (though I admit I didn't look super-hard for undocumented/underdocumented ways to do it).

"Hi. My name's Ted, and I write shitty code."

With this opening, a group of us earlier this year opened a panel (back in March, as I recall) at the No Fluff Just Stuff conference in Minneapolis. Neal Ford started the idea, whispering it to me as we sat down for the panel, and I immediately followed his opening statement in the same vein. Poor Charles Nutter, who was new to the tour, didn't get the whispered-down-the-line instruction, and tried valiantly to recover the panel's apparent collective discard of dignity--"Hi, I'm Charles, and I write Ruby code"--to no avail. (He's since stopped trying.)

The reason for our declaration of impotent implementation, of course, was, as this post states so well, a Zen-like celebration of our inadequacies: To be a Great Programmer, you must admit that you are a Terrible Programmer.

To those who count themselves as the uninitiated into our particular brand of philosophy (or religion, or just plain weirdness), the declaration is a statement of anti-Perfectionism. "I am human, therefore I make mistakes. If I make mistakes, then I cannot assume that I will write code that has no mistakes. If I cannot write code that has no mistakes, then I must assume that mistakes are rampant within the code. If mistakes are rampant within the code, then I must find them. But because I make mistakes, then I must also assume that I make mistakes trying to identify the mistakes in the code. Therefore, I will seek the best support I can find in helping me find the mistakes in my code."

This support can come in a variety of forms. The Terrible Programmer cites several of his favorites: use of the Statically-Typed Language (in his case, Ada), Virulent Assertions, Testing Masochism, the Brutally Honest Code Review, and Zeal for the Visible Crash. Myself, I like to consider other tools as well: the Static Analysis Tool Suite, a Commitment to Confront the Uncomfortable Truth, and the Abject Rejection of Best Practices.

By this point in time, most developers have at least heard of, if not considered adoption of, the Masochistic Testing meme. Fellow NFJS'ers Stuart Halloway and Justin Gehtland have founded a consultancy firm, Relevance, that sets a high bar as a corporate cultural standard: 100% test coverage of their code. Neal Ford has reported that ThoughtWorks makes similar statements, though it's my understanding that clients sometimes put accidental obstacles in their way of achieving said goal. It's amibtious, but as the ancient American Indian proverb is said to state, If you aim your arrow at the sun, it will fly higher and father than if you aim it at the ground.

In fact, on a panel this weekend in Dallas, fellow NFJS'er David Bock attributed the rise of interest in dynamic languages to the growth of the unit-testing meme, since faith in the quality of code authored in a dynamic language can only follow if the code has been rigorously tested, since we have no tool checking the syntactic or semantic correctness before it is executed.

Among the jet-setting Ruby crowd, this means a near-slavish devotion to unit tests. Interestingly enough, I find this attitude curiously incomplete: if we assume that we make mistakes, and that we therefore require unit tests to prove to ourselves (and, by comfortable side effect, the world around us) that the code is correct, would we not also benefit from the automated--and in some ways far more comprehensive--checking that a static-analysis tool can provide? Stu Halloway once stated, "In five years, we will look back upon the Java compiler as a weak form of unit testing." I have no doubt that he's right--but I draw an entirely different conclusion from his statement: we need better static analysis tools, not to abandon them entirely. 

Consider, if you will, the static-analysis tool known as FindBugs. Fellow NFJS'er (and author of the JavaOne bestselling book Java Concurrency in Practice) Brian Goetz offered a presentation last year on the use of FindBugs in which he cited the use of the tool over the existing JDK Swing code base. Swing has been in use since 1998, has had close to a decade of debugging driven by actual field use, and (despite what personal misgivings the average Java programmer may have about building a "rich client" application instead of a browser-based one) can legitimately call itself a successful library in widespread use.

If memory serves, Brian's presentation noted that when run over the Swing code base, FindBugs discovered 70-something concurrency errors that remained in the code base, some since JDK 1.2. In close to a decade of use, 70-something concurrency bugs had been neither fixed nor found. Even if I misremember that number, and it is off by an order of magnitude--7 bugs instead of 70--the implication remains clear: simple usage cannot reveal all bugs.

The effect of this on the strength of the unit-testing argument cannot be understated--if the quality of dynamic-language-authored code rests on the exercise of that code under constrained circumstances in order to prove its correctness, then we have a major problem facing us, because the interleaving of execution paths that define concurrency bugs remain beyond the ability of most (arguably all) languages and/or platforms to control. It thus follows that concurrent code cannot be effectively unit tested, and thus the premise that dynamic-language-authored code can be proven correct by simple use of unit tests is flawed.

Some may take this argument to imply a rejection of unit tests. Those who do would be seeking evidence to support a position that is equally untenable, that unit-testing is somehow "wrong" or unnecessary. No such statement is implied; quite the opposite, in fact. We can neither reject the necessitary of unit testing any more than we can the beneficence of static analysis tools; each is, in its own way, incomplete in its ability to prove code correct, at least in current incarnations. In fact, although I will not speak for them, many of the Apostles of UnitTestitarianism in fact indirectly support this belief, arguing that unit tests do not obviate the need for a quality-analysis phase after a development iteration, because correct unit tests cannot imply completely correct code.

Currently much research is taking place in the statically-typed languages to make them more typesafe without sacrificing the productivity enhancements seen in the dynamically-typed language world. Scala, for example, makes heavy use of type-inferencing to reduce the burden of type declarations by the programmer, requiring such declarations only when the inferencing yields ambiguity. As a result, Scala's syntax--at a first glance--looks remarkably similar in places to Ruby's, yet the Scala compiler is still fully type-safe, ensuring that accidental coercion doesn't yield confusion. Microsoft is pursuing much the same route as part of their LINQ strategy for Visual Studio 2008/.NET 3.5 (the "Orcas" release), and some additional research is being done around functional languages in the form of F#.

At the end of the day, the fact remains that I write shitty code. That means I need all the help I can get--human-centric or otherwise--to make sure that my code is correct. I will take that help in the form of unit tests that I force myself (and others around me force myself) to write. But I also have to accept the possibility that my unit tests themselves may be flawed, and that therefore other tools--which do not suffer from my inherent human capacity to forget or brush off--are a powerful and necessary component of my toolbox.

ROTFL!

It seems that starting up a conversation about architecture is the topic of the month. Another email, this time from anattendee at a recent Denver Java Users Group meeting I spoke at a few weeks ago:

I just finished another interview to act as a consultant on a enterprise Java project at a large (Fortune 500) Denver company.  It went pretty well from a technical perspective, but I couldn't convince the interviewer of a few points.  I wanted your perspective so that I can share it at the Denver Open Source Users Group this Thursday with our members...

- Interviewer said that neither him nor his team attend any events in person (DJUG, Boulder Java Users Group, DOSUG, NFJS, ApacheCon).

- Interviewer said the budget wasn't there for these events.

- I pointed out that ones like DJUG were free.  He said "OH" like he didn't even know. (warning bells ringing in my head)

- I noted the value that (as you also stated) an attendee gets from not just the presentation, but the interaction with the speaker AND audience.  He was not impressed.

- I asked if he or anyone on his team had ever contributed to anything open source.  He said that none of his current employees or himself do.  But a guy that left last year did.  (warning bells ringing again)

- He said that they are primarily a consumer of open source, not a producer and that if they really wanted to change code, they would grow the tool in-house anyway.

- They are still on Junit 3.8 because they haven't had time to upgrade (this was a greenfield project as of 6 months ago)

- They are sticking with Ant because Maven looks daunting.

I made some attempts to probe why he thought this way, but of course, didn't make much headway.  I think I'll stay with my current contracts for a while longer.  This didn't appear to be "the place" I would want to work.

Probably a smart idea; as a general rule, if you strongly disagree with the software development practices in place at a particular company (or any part of the company's culture, for that matter), it's often not worth it to jump in--you just have to jump back out within six months or so. Granted, there are those who can stomach that kind of mismatch between internal values and external culture, but generally only by "checking out" of the environment around them and pursuing other things on the side. (As a side note, it seems to me that this "checking out" at work is probably the number one reason and source for developers working on open-source projects: if you're mentally and emotionally engaged at work, your desire to write code when you get home seems significantly diminished. It's almost like a drug addict's wet dream: getting high at work, so you don't have to bother with it at home.)

Anyway, the answers I'd love to hear your quick opinion on are:

1) Is this the norm for Fortune 500 sized companies (I've only got experience of a dataset of 10, and I'm sure you have more)

There's a conflation of topics here, so let's take them in order.

1. Do many/most F500 companies act as open-source consumer? Absolutely.
2. Do many/most F500 companies act as open-source producer? Not even close, but this makes sense. Most F500 companies are not in the business of contributing software; in fact, a significant majority of them don't make money on IT in the slightest, so it's not to their competitive advantage to offer up contributions to open source. (That's going to piss off the open-source zealots to no end, I'm sure, but in the end, which would you rather a drug manufacturer focus on: finding a cure to cancer, or patching security bugs in Tomcat? The benefit of an economy of specialization, is specialization. From each according to their abilities, and all....)
3. Do many/most developers at F500 companies not go to events like local JUGs or conferences like NFJS? Unfortunately, far too many developers don't treat their careers as a work-in-progress, but believe that once they've landed the job, they're on a track to moderate financial success for life. In fact, lots of developers fit the demographic of "male, 18-30, and single", which is the absolute worst demographic for future planning. Lots of these guys think that "Hey, I'm smart, the money will always flow in, right? Java/.NET/C++/COBOL will always be the tool of choice, right? What, me worry?" Unfortunately, it's my experience that it doesn't get better with age. Developers in the age range of 31 and up have seen one (or two) generations of languages/platforms go by, and have had to re-tool themselves, and are still pissed about it. The ones who realize that no matter how much they learn, there's still a lot more to take on, those are the ones going to JUGs and NFJS and TechEd and JavaOne and whatever else comes their way. Those are the same ones who see training classes as opportunities for advancement, not opportunities to play 8 hours of uninterrupted Solitaire. And, unfortunately, those developers are the exception, not the rule, it seems. Which means, if you take the time to invest in yourself, you will never be in the bottom of the candidate pool for your next job. Period.
4. Do many/most F500 projects stay with outdated things like JUnit 3.8? Honestly, this is the least of their sins. Over half of the code being written out there is being done without benefit of unit tests--frankly, the fact that they're using JUnit at all is a point in their favor, not against them. But I see your concern, the "we're frozen in time" syndrome when it comes to technology choices can often atrophy over long periods of time, and yes, that is a concern. But not in this case, no. If they're still using JDK 1.4 because they're still on WebSphere and have no plans to move forward until IBM makes them do it (because of end-of-support concerns--are you listening out there, VB6 world?), then yes, it's a concern. But only if they refuse to do the risk analysis of upgrading; if they do that analysis (and the architect or project manager should be able to tell you, if not show you, that analysis), and still decide to stay with what they've got for reasons they can explicate, then that's just being smart.
5. Do many/most F500 projects stick with Ant because Maven looks daunting? Dear Lord, man, I don't use Maven, for the same reason! Be careful not to project your own value judgements ("Maven is better than Ant!" "Maven is just a pile of crap on top of the goodness that is Ant!" and so on) on the company's decisions around build process. I've seen a few teams that used Maven and loved it, and I've seen a few teams that used Maven and hated it. I've seen some teams use Ant (or MSBuild) in ways that were simple, clean, and elegant, and I've seen Ant used in ways that would keep you up at night, shivering. (I was an accomplice to one of those latter Ant-trocides. Therapist says with time, I may be able to sleep through the night.) So long as a company can offer an articulate reason for their technology decisions, you can disagree with them if you wish, but you have to honor them. If you have serious concerns still, subvert them from within. Once you've lived their build process for a while, you'll know where the pain points are and can raise suggestions, perhaps suggesting Maven as a solution, perhaps not, to help address those pain points. But never forget, as the newly-installed outsider, your credibility will be at its lowest on the very first day you start there[1].

2) Would you also hear warning bells going off in your head after hearing this guy reply as he did above?

I hear warning bells all the time, but that's because I don't expect any place I go into to be perfect on all three axes of the triangle (People, Process, Technology). I'm looking for where the red flags are, because nobody's perfect. Having said that, it's critical for your sanity as a developer to know which of those three axes you need to be on the "high" end of the scale in order to preserve your own sanity, and which ones you can live without (or work to change).

3) Is it worth being an evangelist to folks like this convincing them that events AND contributing to open source are personal-improvement exercises that lead to better code and architecture at your main job?

Depends on a couple of factors. One, how much do you want to be an influencer? Some people derive great joy from changing the way a company does business, others see that as an impediment to their larger goal of "building cool stuff". And two, how much do you want to emotionally invest in this group? If you're just there for the short-term, certainly tell them and quote the statistics, but don't lose sleep over it. If you're really invested in this company (because of its location, the work its doing, you own the firm, whatever), then obviously put your heart out there on the line.

4) How far back on that continuum that we talked about at DJUG do the big companies have to be to start missing out on the early-adopter advantage? (1, 3 or 5 years back on mature technologies?)

That's a hard one to answer. "That continuum" is the technology-adoption continuum I mentioned in the JUG talk: every company is pegged somewhere on the technology adoption continuum, despite our tendencies to classify companies as either "(b)leading-edge/early-adopters" or "legacy players". Frankly, I think the hard part of that question is the phrase "mature technologies"--what's "mature" these days? What does the definition of "mature" mean for a technology? It's a hard thing to nail down, and until we do, I can't really answer the question. In the spirit of the question, though, I'd say that any firm who isn't at the very least watching forums like TheServerSide or InfoQ, and/or building a corporate bookshelf on a yearly/quarterly/monthly basis (and Safari, the joint Pearson/OReilly online book project is a HUGE win here for firms seeking to do this) is missing on golden opportunities to keep their ear to the ground. I'm not saying they need to adopt them right away, but every firm that writes software should have prototyped a Ruby/Rails app at this point, in order to get an idea of what Ruby and Rails bring to the playground, and whether or not they can play nicely with the rest of the firm's Java/.NET/C++/whatever apps.

[1] Ironically, the reverse is true of consultants--their credibility is at its highest on day one, and generally drops off from there, unless the consultant can prove his worth and earn more credibility. 'Tis horribly unfair, and exactly the reverse of what it should be, but I don't make these rules, I just observe them.

By the way, just a reminder, if you've got questions on architecture or other software-development-related topics that you'd love to see answered here, feel free to drop me a line: "ted@" this domain name. Or, blog them, and shoot me an email with the blog link.

This is not a title I convey lightly, but Michael Nygard's Release It! deserves the honor. It's the first book I've ever seen that addresses the issues of building software that's Production-friendly and sysadmin-approachable. He describes a series of antipatterns describing a variety of software failures, and offers up a series of solutions (patterns, if you will) to building software systems designed to combat said failures.

From the back cover:

Every website project is really an enterprise integration project: the stakes are high and the projects complex. In this world where good marketing can be fatal to yor website, where networks are unreliable, and where astronomically unlikely coincidences happen daily, you need all the help you can get.

...

You're a whiz at development. But 80% of typical project lifecyle cost can occur in production--not in development.

Although Michael's personal experience stems mostly from the Java space, the lessons and stories he offers up are equally relevant to Java, .NET, C++, Ruby, PHP, and any other language or platform you can imagine. Michael Nygard not only knows the Ten Fallacies of Enterprise Development, he breathes them.

Go. Now. Buy. Read. Don't write another line of code until you do.

I'm certain I'm not the first one to blog this, but I wanted to help fan the information out to the world: this email crossed my virtual desk today, and it indicates a subtle shift that many probably didn't see coming:

This morning we announced that later this year, with the final release of Visual Studio 2008, we will make available the source to much of the .NET Framework Libraries under the Microsoft Reference License.  This means that “anyone” who accepts the license will be able to browse and view source code.  The set of libraries initially includes  the Base Class Libraries (System namespace, IO, Text, Collections, CodeDom, Regular Expressions, etc), ASP.NET, WinForms, and WPF .  Microsoft will add to this list as time goes on.

There are two ways people will access the source code:

1) They will download a package with all the source, and then they will be able to install and browse locally

2) VS 2008 integration will enable developers to debug from their own source code into the .NET Framework source code.  We’ll provide symbols for our source on an internet-accessible source-server; to enable this experience, the developer needs to set up the URI for the server.  This second option is really cool, and enables a kick-ass developer experience.

We’re announcing this via Scott Guthrie’s blog, with screenshots of the VS experience.

Distributed with the email was a Q&A document, which (in the interests of minimizing the accusations of FUD on my part) I will paste in its entirety:

Reference Source Code Availability for the .NET Framework

--Microsoft Confidential until October 3^rd, 9:00am Pacific time--

Key Points

· Microsoft is releasing the source code for .NET Framework libraries under the Microsoft Reference License.  This license allows viewing of source code, but not modification or redistribution. The source code will be downloadable and viewable by anyone who accepts the license agreement. 

· Microsoft will introduce a capability in Visual Studio 2008 to allow .NET developers who are debugging applications, to debug not only into their own source code, but also into .NET Framework source code using Visual Studio. 

· This release falls under Microsoft’s Shared Source Initiative, which encompasses a spectrum of source code offerings, complementing the company’s other activities around sharing source code.  This is another example of Microsoft’s continued commitment to increasing transparency and addressing developer needs. 

Q&A

Q. What are you announcing?

Microsoft is releasing the source code for .NET Framework base class libraries under the Microsoft Reference License, which is available here: http://www.microsoft.com/resources/sharedsource/licensingbasics/referencelicense.mspx .  This license allows viewing of source code, but not copying or re-compiling. The source code will be downloadable and viewable by anyone who accepts the license agreement. 

In addition, Microsoft will introduce a capability in Visual Studio 2008 to allow .NET developers who are debugging applications, to debug not only into their own source code, but also debug into the .NET Framework source code using Visual Studio.  With this capability, when developers are stepping through code, if they wish, they will be able to step into the source code for the .NET base class libraries.

Q. When will this become available to developers?

We expect the debugging capability to be available to .NET developers who use the RTM version of Visual Studio 2008, when Visual Studio 2008 releases. Our current plan is to release Visual Studio 2008 before the end of the 2007.

Developers will be able to download the source code package for the .NET libraries around the same time, or soon after.

Q: Which libraries are included in the source-code release?

This release will include the Base Class Libraries (BCL), Windows Forms, ASP.NET, System.Data, and WPF. BCL includes many of the basic classes in the framework including collections, string and text handling, IO,serialization, remoting, and others. We plan to include additional libraries into the set as time goes on.

Q: Why are you releasing only a subset of the .NET Framework libraries under the Microsoft Reference License?

Our intention is to make .NET Framework library source code available broadly, in response to customer demand. Before release under the Microsoft Reference License, each library is subject to a code review. As time allows, we will complete additional code reviews and release additional modules under this license.

Q: Will the libraries distributed under the Microsoft Reference License be limited to “managed code” libraries – in other words, .NET Framework libraries?

Microsoft’s intention in releasing developer libraries under the Microsoft Reference License (Ms-RL) is to provide transparency and allow developers to more deeply understand the inner workings of the source code. Developers who better understand the source code will be more effective in writing software that makes use of the licensed source code. Microsoft may release additional libraries under the license, potentially including unmanaged libraries, and these additional libraries would also deliver the same debugging experience described previously. Microsoft has no announcement at this time regarding additional libraries being released under the Microsoft Reference License.

Q: How does this relate to CodePlex, and other source-code sharing programs offered by Microsoft?

Microsoft has an initiative called the Shared Source Initiative which encompasses a spectrum of source code offerings.  At one end of the spectrum, Microsoft shares source code for commercial products like Windows and Office through programs offered to governments, enterprises, MVPs, OEMs, SIs and others.  On the other end of the spectrum, Microsoft encourages open source development through the CodePlex project management site, as well as direct contribution of source code to community projects on CodePlex.  Today, we’re announcing a new offering: the source code for the .NET Framework libraries will be available under the Microsoft Reference License. This offering will enable developers to view and even debug into the source code for the .NET Framework libraries.

Q: Allowing anyone to view the source code of a library – does this present a security risk?

The security of the .NET Framework does not depend on the obscurity of the .NET Framework source code. A realistic threat model must assume that any bad actor who wishes to subvert the security of any library, for practical purposes, has full access to the source code. This release will not change that.

Q: What do the developers need in order to take advantage of the debugging capability?

The debugging capability will be available to developers who use the RTM version of the Professional or Standard editions of Visual Studio 2008. Microsoft will also publish instructions for how to enable other .NET development or debugging tools to take advantage of the same capability. The Express versions of Visual Studio will not include this capability.

Q: What do the developers need in order to simply view the .NET source code?

Developers will need to download the source code package, and accept the license agreement before unpacking the archive. At that point, the source files will be viewable in any text editor.

Q. Why is Microsoft making this available now?

This is another step along the path of increasing transparency and addressing customer needs. Generally speaking, we are always looking for ways to continue to improve and enhance the developer experience for people who build applications using Microsoft tools. Delivering this particular capability for our customers requires changes in the developer tool itself, and the introduction of Visual Studio 2008 gives us the opportunity to do that.

Q. Why didn’t you do this sooner?

Our primary goal was to enable developers to build better applications. A new capability like this requires changes in the developer tool, and the release of Visual Studio 2008 presents us with an opportunity to deliver the right experience for developers. Maintaining the right developer experience, while preserving the security stability and serviceability of the .NET Framework, were paramount requirements for us. We believe we’ll have a simple and easy developer experience that will empower developers to build better more reliable applications. Having said this, the debugging experience is very transparent, and developers won’t even notice the change.

Q. Can developers modify and rebuild the .NET Framework source code, and redistribute the results of that action?

The source code of the developer libraries being made available, including the libraries in the .NET Framework, will be released under the Microsoft Reference License. http://www.microsoft.com/resources/sharedsource/licensingbasics/referencelicense.mspx

The Microsoft Reference License allows viewing of source code for reference purposes, but does not allow editing, copying, or rebuilding. Microsoft’s intention in releasing developer libraries under this license is to provide transparency and allow developers to more deeply understand the inner workings of the source code. Developers who better understand the source code will be more effective in writing software that makes use of the licensed source code.

Q. How is the license accepted?

The license is accepted differently depending on how the person accesses the source. If it is through the VS2008 experience, there is a popup “do you accept this license?” dialog. If it is through the source package, then the MSI installer will require the person to accept the license before installing.

Q. Is this an open-source release?

Microsoft believes that a consistent framework for releasing source code – one the community can rely on – delivers the best value for customers and minimizes confusion. This is why use a set of source licenses, with clearly differentiated purposes, for source releases. (See http://www.microsoft.com/resources/sharedsource/licensingbasics/sharedsourcelicenses.mspx ) This particular release is being offered under the Microsoft Reference License.

Q. {Java,Ruby, Python} is truly open source and lets me rebuild the source code. Why can I not do this with .NET?

We’re focused on delivering value to developers. With the .NET Framework library source code, our goal is to enable developers to view the source of the platform, learn from that, and potentially provide feedback into Microsoft. (This would commonly be done via the Product Feedback center Microsoft has used for years – see http://connect.microsoft.com/feedback/default.aspx?SiteID=210 ) The Visual Studio debugging experience extends that view capability to a debugging session. With this capability, we think we’re delivering an easy, seamless experience for developers, and good value to the community.

Allowing developers to rebuild a framework, any framework, from source, and then redistribute the modified result, can introduce problems with providing support, serviceability, integrity and security of the framework itself. This is true with .NET, Java, Ruby, and other frameworks. Multiple independent redistributed modified versions of a framework can decrease the reliability and dependability of the common platform, which is not desirable.

In the path we’ve taken here, we are seeking to balance the requirements we hear from the developer community for transparency and of reliability and dependability of the platform. In striking that balance, we believe the Microsoft Reference License is the right license for this release.

Q. How does the debugging experience really work? Will the source code remain resident on the machine?

Visual Studio will request source from a Microsoft (MSDN) server as needed when a developer steps into .NET Framework library code during in a debugging session. In more detail, the debugger queries the remote server for the version of symbols that matches the binaries used in the application being debugged. If the debugger finds the correct version of symbols, it downloads the source files on-demand as the developer debugs into them. In all cases, the first download of source code requires explicit acceptance of the license. This is the same flow works for all source code that we will add in the future to this program, whether managed, unmanaged, 32 bit, 64 bit, desktop, mobile, and so on.

Q. What if a company does not want to allow the .NET Framework source code or other library source code, to be viewed by its developers?

Viewing or debugging into the Microsoft library source code is an action that will require an explicit acceptance of the license, regardless of the mechanism by which developers view the code (whether in a developer tool like Visual Studio, or as a separate archive download). Companies that do not wish to view the source code should instruct their employees to not accept this license.

Q. What are the terms of support with this source code? What if the developer finds a bug in the code?

The license spells out what the developer should expect. Essentially, people can look at the source code, but they may not copy it. If a person finds what they believe to be a bug in the code, Microsoft would encourage the person to submit feedback via the product feedback center, see http://connect.microsoft.com/feedback/default.aspx?SiteID=210 .

As you can well see, there's a lot of opportunities for FUD here--the Q&A document makes it clear that the source license is a "read-only" license, in that you can't recompile the code yourself. However, for those who are hell-bent on doing so, two options are available to you:

1. Build the code without telling Microsoft. This is not a trivial undertaking, and I'd be shocked if it turns out to be as simple as "create a solution, import the files, press the big green button". It may not turn out to be worth the effort.
2. Download Rotor Whidbey, also known as SSCLI 2.0, and build that, instead. It won't have ASP.NET, WinForms, or WPF, but it will have much of the "core" FCL, and the core CLR (modulo GC and JIT, which Microsoft re-wrote for Rotor v1) to boot.

Speaking of the latter, Joel Pobar and I are doing a revision of the SSCLI Essentials book to bring it up to date with the Rotor Whidbey sources; I'm doing an editing pass on the chapters now, and we'll be hunting up reviewers to take a look probably before the end of the month. (If you're interested, you know how to get a hold of me.)

Some additional comments to go along with the Q&A from above:

• "Allowing anyone to view the source code of a library – does this present a security risk? The security of the .NET Framework does not depend on the obscurity of the .NET Framework source code. A realistic threat model must assume that any bad actor who wishes to subvert the security of any library, for practical purposes, has full access to the source code." Amen, brother! Sing it loud, sing it proud: obscurity is not security!
• "Why is Microsoft making this available now? This is another step along the path of increasing transparency and addressing customer needs. Generally speaking, we are always looking for ways to continue to improve and enhance the developer experience for people who build applications using Microsoft tools. Delivering this particular capability for our customers requires changes in the developer tool itself, and the introduction of Visual Studio 2008 gives us the opportunity to do that." There's a small amount of shuffle-and-dance going on here; to allow developers to debug .NET in the manner Microsoft describes above--Visual Studio fetching symbols and/or source from MSDN when the developer steps-in to the source--did, probably, require a tool change. Just using the existing facility to talk to a symbol server over the Internet would most likely not be a great experience, though I'd defer to John Robbins on that one. But nothing prevented them from making sources available in the traditional (a la SSCLI) manner (a big tarball) long before this. Instead, I think what we're seeing is something Stu Halloway predicted three years ago at a No Fluff Just Stuff speaker panel: "In five years, Microsoft will be the biggest open-source company on the planet". I think Microsoft just needed to figure out how it wanted to embrace open source, because at the end of the day, Microsoft needs to make money, and it wasn't sure how to do open source and make money at the same time--after all, the "golden bailout" that both Red Hat and JBoss both used won't be available to Microsoft.
• "Can developers modify and rebuild the .NET Framework source code, and redistribute the results of that action? The source code of the developer libraries being made available, including the libraries in the .NET Framework, will be released under the Microsoft Reference License. http://www.microsoft.com/resources/sharedsource/licensingbasics/referencelicense.mspx

  The Microsoft Reference License allows viewing of source code for reference purposes, but does not allow editing, copying, or rebuilding. Microsoft’s intention in releasing developer libraries under this license is to provide transparency and allow developers to more deeply understand the inner workings of the source code. Developers who better understand the source code will be more effective in writing software that makes use of the licensed source code." In other words, no, you can't make a forked copy of the library and ship it yourself. At least, not for now. Mark my words, though, Microsoft will pay very close attention to this issue, and if customers find themselves wanting/needing to do so, Microsoft will find a way to make this work. It just may not be in the same manner that traditional open-source projects would do so. (Remember, Microsoft is very aware of their bottom line, and they need to tread lightly anywhere the goals of "making money" might possibly be threatened or thrown for a loop. They're not risk-avoidant, but they aren't going to take risks unnecessarily, either.)

• "{Java,Ruby, Python} is truly open source and lets me rebuild the source code. Why can I not do this with .NET?" This is an interesting and tricky subject. On the one hand, allowing developers "full reign" over the source provides the greatest amount of power and flexibility, and opens up the possibility of customizing the .NET Framework to behave exactly the way you want it to. On the other hand, being able to do that also means that the versioning problem just took on a whole new order of magnitude of complexity, because now you have some serious problems regarding the FCL assemblies. Remember, the FCL assemblies are signed with Microsoft's private key, so either Microsoft has to release that in order to allow you to rebuild the assemblies to match up with the ones already built by Microsoft (and we already know that releasing a private key is a Bad Thing, right?), or your custom-built FCL assemblies would be built with your own private key, which means now that your library is only used when code is compiled directly against it--and what happens when you build your application with a third-party library which itself was compiled against Microsoft's code? Trust me on this--the resulting picture is not pretty. Don't imagine that you have answers to this problem, either: it's a tricky problem, one that no programming platform has solved in a transparent and universal manner. Where a solution might work for your particular situation, be assured that it'll fail for somebody else's.

 The one thing I wish they'd do, and I can't tell from the announcement or the Q&A document if they will, is release the sources in a single tarball for download and offline use; I don't want to have to be tethered to an Internet connection all the time while debugging code. (Remember that the first Fallacy, "The network is always available", doesn't apply at 39,000 feet.)

All in all, this makes the latest marker in Microsoft's slow-but-steady shift towards a more open-source-friendly position. Whether or not the developer community meets them halfway, of course, resides entirely with your point of view on the statement, "Microsoft is an evil empire that does nothing good for anybody but themselves". If you agree with that statement, you and the other Microsoft Conspiracy Theorists will probably conclude that Lee Harvey Oswald was not only not alone, but he was paid by Microsoft. If you dsiagree with that statement, then you and the other Microsoft-sponsored Microsoft Certified Partners, Microsoft-sponsored Regional Directors, Microsoft-sponsored Most Valuable Professionals, and Microsoft-sponsored development teams will be happy to note that the checks still cash despite this move to open source. (And yes, there's likely to be a few of you out there who are somewhere in the middle on this, and you will probably get killed in the crossfire between those two rival gangs as a result of all this.)

I just got back from the No Fluff Just Stuff show in St Louis, where I gave my "Why the Next Five Years Will Be About Programming Languages" keynote, and a fellow speaker emailed me to point out the Code To Joy blog, which says some things that were... um... well, rather than try and select an adjective, I'll let you look for yourself:

Ted Neward talked about how the next 5 years will be about languages. (Fellow speaker Alex Miller has a post which contains a link to a similar talk and some of his own commentary).

Ted's thesis was that the Big Runtimes (the JVM and CLR) act as a bridge between the academic language lawyers and the problem solvers in the field: the academics can experiment with language syntax and exotic semantics; the problem solvers can use these languages and rely on the venerable runtime libraries to get things done.

I enjoyed the talk thoroughly. Ted paced to-and-fro like a giant cat: if NFJS is Narnia, then Ted is Aslan, except very hungry, and with a mild-case of rabies. Many good jabs toward, well, everybody. He's an equal-opportunity offender (and a big softie underneath it all.)

A giant cat. A very hungry giant cat. A very hungry giant cat with a mild case of rabies.

*sniff* It's like I can retire now. *sniff* I'd like to thank the Academy, and my parents, and ...

I get e-mail from blog readers, and this one--literally--stopped me in my tracks as I was reading. Rather than interpret, I'll just quote (with permission) the e-mail and respond afterwards

Hi Ted,

I had a job interview last Friday which I wanted to share with you. It was for a “Solutions Architect” role with a large Airline here in New Zealand. I had a preliminary interview with the head Architect which went extremely well, and I was called in a few days later for an interview with the other three guys on the Architecture team.

The second interview started off with the usual pleasantries, and then the technical grilling began with:

“What are the best practices that you would put in place on a project?”

I replied “I’ve come to realise that there is no such thing as ’Best Practice’ in architecture – everything is contextual.”

Well, it went down like a sh*t sandwich! The young German guy who asked the question looked at me like some sort of heretic and said – “ I disagree”. I thought to myself, no damn it – I’m going to push it to see where this guys argument goes, so I said “I can take any best practice you can think of, and by changing the context, I can render that best practice a worst practice”. He didn’t like that very much, but I thought, ‘bugger him, I know I’m right’.

He then came out with the next question “What do you do when your architectural principles are compromised”. I asked “what do you mean”, and he indicated that an application he designed recently, was found to be far too expensive to implement So he asked again ‘what would you do’?

I replied “redesign it”. He scoffed at this answer, and reiterated “but what if the redesign was compromising your architectural principles”? So I asked “what is more important. Your principles, or achieving a business objective?”. I don’t remember exactly what his answer was, but it was along the lines of “you have to maintain a corporate standard”.

His next attack was at extreme programming. Having seen that I had used extreme programming one of my recent projects (in addition to also using waterfall, more recently, on others), he asked “don’t you think that the very risky nature of extreme programming is at odds with it’s ability to deliver software consistently?”. This was a bit of a stunner. I indicated that, once again, it was contextual. XP is appropriate on some projects, but it is not on others. On the XP project I worked on, it was entirely appropriate. We delivered early, within budget, the client got what he wanted, and we got a few million dollars worth of work out of it. Not surprisingly, he didn’t have a lot to say to me after that.

Having worked as a consultant for a number of years now, I have been entirely focused on adding business value. I was stunned to hear first hand, how divorced from the business this “architect” was. Clearly he has to maintain some sort of structure with their corporate systems, but surely each business solution should be assessed primarily in the context of it’s own business objectives.

The interview was good in the respect that I was able to quickly establish that it wasn’t the place for me, but it did leave me with some unanswered questions:

1. How could their idea of an architect (being the policemen of corporate best practice) be so far removed from someone like myself, who aims to make case by case judgements based on pragmatism and experience?
2. Is architecture supposed to be facilitative or restrictive?
3. What relevance do architects have today? Are they just overpaid, out of touch developers?

Regards,

Shane Paterson

Hands on Architect Type

(for lack of a more relevant title)

Wow.

For starters, Shane, kudos to you for sticking to your guns, and for figuring out really quickly that this was clearly not a place you wanted to work--a lot of developers have a mentality that says that they need the company more than the company needs them, sort of a "job at any price" mindset. Interviews aren't supposed to be the place where candidates grovel and say whatever the company wants them to hear--an interview is supposed to be a vetting process for both sides.

But on to your questions:

How could their idea of an architect be so far removed from someone like myself? I can't answer this one solidly, but I can say that the definition of an architect seems to be vague and indiscriminate a term, only exceeded in opacity by the term "software" itself. For some companies I've worked for, the "architect" was as you describe yourself, someone whose hands were dirty with code, acting as technical lead, developer, sometimes-project-manager, and always focused on customer/business value as well as technical details. At other places, the architect (or "architect team") was a group of developers who had to be promoted (usually due to longevity) with no clear promotion path available to them other than management. This "architect team" then lays down "corporate standards", usually based on "industry standards", with little to no feedback as to the applicability of their standards to the problems faced by the developers underneath them. A friend of mine on the NFJS tour, Brian Sletten, tells a story of how he consulted on a project, implementing the (powerful) 1060 Netkernel toolkit at the core of the system, to resounding success. Then, on deployment, the "architecture team" took a look, pronounced the system to be incompatible with their "official standards", and forced new development of a working product. In other words, the fact that it worked (and could easily be turned to interoperate with their SOAP-based standard, of which there were zero existing services) was in no way going to stand as an impediment to their enforcement of the corporate standard.

Is architecture supposed to be facilitative or restrictive? Ah, this is a harder one to answer. In essence, both. Now, before the crowd starts getting out their torches and pitchforks to have a good old-fashioned lynching, hear me out.

Architecture is intended to be facilitative, of course, in that a good architecture should enable developers to build applications quickly and easily, without having to spend significant amounts of time re-inventing similar infrastructure across multiple projects. A good architecture will also facilitate interoperability across applications, ensure a good code quality, ensure good maintainability, provide for future extensibility, and so on. All of this, I would argue, falls under the heading of "facilitation".

But an architecture is also intended to be restrictive, in that it should channel software developers in a direction that leads to all of these successes, and away from potential decisions that would lead to prolems later. In other words, as Microsoft's CLR architect Rico Mariani put it, a good architecture should enable developers to "fall into the pit of success", where if you just (to quote the proverbial surfer) "go with the flow", you make decisions that lead to all of those good qualities we just discussed.

This is asking a lot of an architecture, granted. But that's the ideal.

What relevance do architects have today? Well, this is a dangerous question, in that you're asking it of one who considers himself an architect and technologist, so take this with the usual grain of salt. Are we just overpaid out-of-touch developers? God, I hope not. Fowler talks about architecture being irrelevant in an agile project, but I disagree with that notion pretty fundamentally: an architect is the captain of the ship, making the decisions that cross multiple areas of concern (navigation, engineering, and so on), taking final responsibility for the overall health of the ship and its crew (project and its members), able to step into any station to perform those duties as the need arises (write code for any part of the project should they lose a member). He has to be familiar with the problem domain, the technology involved, and keep an eye out on new technologies that might make the project easier or answer new customers' feature requests.

And if anybody stands up at this point and says, "Hey, wait a minute, that's a pretty tall order for anybody to fill!", then you start to get an idea of why architects do, frequently, get paid more than developers do. Having to know the business, the technology at a high and low level of detail, keeping your hands in the code, and watching the horizon for new developments in industry, is a pretty good way to burn out any free time you might have thought you'd have.

Granted, all of these answers notwithstanding, there's a large number of "architects" out there whose principal goal is to simply remain employed. To do that, they cite "best practices" established by "industry experts" as a cover for making decisions of their own, because nobody ever gets fired for choosing what industry "best practices" dictate. That's partly why I hate that term: it's a cop-out. It's basically relying on articles on popular websites and magazines to do your thinking for you. Inevitably, when somebody at a conference says the word, "Best Practice", listeners' minds turn off, their pens turn on, and they dutifully enscribe this bit of knowledge into their projects at home, without considering the applicability to their project or corporate culture. Nothing, not a single technology, not a single development methodology, not even a single tool, is always the right answer.

In the end, I think what Shane ran into was an "architect" with an agenda and an alpha-geek complex. He refused to consider somebody with a competing point of view, because God forbid somebody show him not to be the expert he's hoodwinked everybody else at work to think he is. Unfortunately I've run across this phenomenon too often to call it statistical error, and the only thing you can do is to do exactly what you did, Shane: get the hell out of Dodge.

Well, those of you who've seen me at the various No Fluff Just Stuff shows probably already knew this was coming, but today I finally made the switch, bought a Mac, and started the (rather scary) experiment of converting my IT life over to a brand new operating system, something I haven't done in close to two decades. (Not since I bought my first 286 PC, in fact, back in high school.) I am now the new owner of a 17" MacBookPro (4GB, 1900x1200 matte, for those who want such details), and I'm slowly starting the transition.

Thus far, the Mac experience isn't all that exciting--I unwrapped the box, plugged the thing into the wall outlet (hey, now there's a switch--Macs come with pre-powered batteries, it seems), booted and walked through the opening steps of Mac OS X. (Don't ask me which version it is--I honestly don't know yet, nor do I know how to find out. I will, in time, but for now...) Vaguely reminiscent of Vista's installation (or maybe that should be the other way around), if a touch smoother.

Next steps, install iWork, so I can see if it's the Office-killer that some of my Mac-zealot friends say it is, install Firefox (because Safari just doesn't work for me, not even for the fifteen minutes I spent using it), and purchase VMWare Fusion. This is the secret to my $4000 experiment--I don't have to switch over completely in one day, because I have everything I do bundled up in VMWare images (one for Java work, one for .NET 2005 work, one for NetFX 3 work, one for playing-around-with-new-programming-languages work, and so on), so as long as VMWare images are cross-compatible between Intel Macs and Intel PCs, I'm good.

Unfortunately, this isn't going to be quite as smooth as I'd hoped; I'd forgotten that the Mac OS and Windows don't exactly share the same filesystem--Windows prefers NTFS (and has for years now), but apparently the Mac OS doesn't have write capabilities to NTFS. This presents a small problem, as my external 160GB USB drives are all NTFS-formatted. So I can suck the images off the drives, but can't write to them. Ugh. Possible salvation lies in the Ext2FS filesystem, which apparently is supported (through open-source efforts) on both Mac OS and Windows. I'll pick up a new 160GB drive, format it Ext2FS, and see how well it works on both machines.

(Curious readers may wonder why I need a filesystem that's writable in both directions; frankly, I need the full-fidelity because I'm not entirely convinced of the sanity of this move just yet, and I want to be able to go back if I need or want to. In fact, I'm going to be carrying the trusty T42p around with me, snuggled right up against the MBP, for a few months yet.)

So, as I write this, I'm running my "Work" guest image on top of VMWare Fusion. (Had to buy the upgrade to VMWare Workstation 6 in order to support the move to Fusion, since Fusion uses the hardware format of Workstation 6. Besides, Fusion/6 has some support for DirectX, and yes, I'm also interested in getting some games installed in here and playing around that way too. ) I have to say, I really don't care for the touchpad, having spent the last half-decade training my index finger to use the nipple on the Thinkpads, but I picked up a Bluetooth mouse for serious work sessions, and like all good mice, it has two buttons. (Anybody know how to emulate a right-mouse-button-click on a MacBookPro?)

Fortunately as well, I have friends who are serious Macophiles, and I'm sure they will take my n00b questions with patience and understanding... since it's their fault I'm here anyway.

Wish me luck....

I know many of you, over the past couple of years, have sent me LinkedIn invitations that I've turned down (mostly because I've been nervous about what LinkedIn would do with the personal information), but a recent email discussion between myself and Van Meyer (of Meyer Strategic, Inc) has finally convinced me to jump in. The profile is public, in case anybody wanted to have a look.

So... to all those who sent me LinkedIn invitations that I turned down, I'm now online, and will accept invitations offered me.

Assuming, of course, I know you.

Software is endless battle and conflict, and you cannot develop effectively unless you can identify the enemies of your project. Obstacles are subtle and evasive, sometimes appearing to be strengths and not distractions. You need clarity. Learn to smoke out your obstacles, to spot them by the signs and patterns that reveal hostility and opposition to your success. Then, once you have them in your sights, have your team declare war. As the opposite poles of a magnet create motion, your enemies--your opposites--can fill you with purpose and direction. As people and problems that stand in your way, who represent what you loathe, oppositions to react against, they are a source of energy. Do not be naive: with some problems, there can be no compromise, no middle ground.

In the early 1970s, Margaret Thatcher shook up British politics by refusing to take the style of the politicians before her: where they were smooth and conciliatory, she was confrontational, attacking her opponents directly. She bucked the conventional wisdom, attacking her opponents mercilessly where historically politicians sought to reassure and compromise. Her opponents had no choice but to respond in kind. Thatcher's approach polarized the population, seizing the attention, attracting the undecided and winning a sizable victory.

But now she had to rule, and as she continued down her obstinate, all-in style of "radicalism" politics, she seemed to gain more enemies than any one politician could hold off. Then, in 1982, Argentina attacked the British Falkland Islands in the South Atlantic. Despite the distance--the Falklands were almost diametrically opposite the globe from the British home islands, off the tip of South America--Thatcher never hesitated, dispatching the British armed forces to respond to the incursion with deadly force, and the Argentinians were beaten. Suddenly, her obstinacy and radicalism were seen in a different light: courage, nobility, resolute and confident.

Thatcher, as an outsider (a middle-class woman and a right-wing radical), chose not to try and "fit in" with the crowd, but to stake her territory clearly and loudly. Life as an outsider can be hard, but she knew that if she tried to blend in, she could easily be replaced. Instead, she set herself up at every turn as one woman against an army of men.

As Greene notes,

We live in an era in which people are seldom directly hostile. The rules of engagement--social, political, military--have changed, and so must your notion of the enemy. An up-front enemy is rare now and is actually a blessing. ... Although the world  is more competitive than ever, outward aggression is discouraged, so people have learned to go underground, to attack unpredictably and craftily. ... Understand: the word 'enemy'--from the Latin inimicus, "not a friend"--has been demonized and politicized. Your first task as a strategist is to widen your concept of the enemy, to include in that group those who are working against you, thwarting you, even in subtle ways.

Software projects face much the same kind of problem: there are numerous forces that are at work trying to drag the project down into failure. While agile books love to assume an environment in which the agile methodology is widely accepted and embraced, reality often intrudes in very rude ways. Sometimes management's decision to "go agile" is based not to try and deliver software more successfully, but to simply take the latest "fad" and throw it into the mix, such that when it fails, management can say, "But we followed industry best practices, it clearly can't be management at fault." (This is the same idea behind the statement, "Nobody ever got fired for buying IBM (or Microsoft or Java or ...)." Sometimes the users are not as committed to the project as we might hope, and at times, the users are even contradictory to one another, as each seeks to promote their own agenda within the project.

As a software development lead (or architect, or technical lead, or project manager, or agile coach, or whatever), you need to learn how to spot these enemies to your project, identify them clearly, and make it clear that you see them as an enemy that will not be tolerated. This doesn't mean always treat them openly hostile--sometimes the worst enemies can be turned into the best friends, if you can identify what drives them to the position that they take, and work to meet their needs. Case in point: system administrators frequently find themselves at odds with developers, because the developer seeks (by nature) to change the system, and sysadmins seek (by nature) to keep everything status quo. Recognizing that sysadmins have historically been blindsided by projects that essentially ignored their needs (such as a need to know that the system is still running, or the need to be able to easily add users, change security policies, or diagnose failures quickly at 3AM in the morning) means that we as developers can either treat them as enemies to be overcome, or win them as friends by incorporating their needs into the project. But they are either "for us" or "against us", and not just a group to be ignored.

Other enemies are not to be tolerated at any level: apathy, sloth, or ignorance are all too common among developer teams. Ignorance of how underlying technologies work. Apathy as to the correctness of the code being created. Sloth in the documentation or tests. These are enemies that, given enough time and inattention, will drag the project down into the tar pits of potential failure. They cannot be given any quarter. Face them squarely, with no compromise. Your team, if they hold these qualities, must be shown that there is no tolerance for them. Hold brown-bag lunches once a week to talk about new technologies, and their poential impact on the team or company or project. Conduct code reviews religiously, during active development (rather that at the end as a token gesture), with no eye towards criticizing the author of the code, but the code itself. Demand perfection in the surrounding artifacts of the project: the help files, the user documentation, the graphics used for buttons and backdrops and menus.

Do not wait for the enemies of your project to show themselves, either--actively seek them out and crush them. Take ignorance, for example. Do not just "allow" each of your developers to research new technologies, but demand it of them: have each one give a brown-bag presentation in turn. In a four-man team, this means that each developer will have a month in which to find something new to discover, analyze, discuss and present. They do not have to have all the answers to the technology, and in fact, if the technology is sufficiently large or interesting, they can spend the next month investigating a new element or new use of the technology. Demanding this of your developers means they are forced to educate themselves, and forced to raise their own awareness of the changing world. (Naturally, developers must be given time to do this research; anecdotally, giving them Friday afternoons to do this experimentation and research, when energy and interest in the work week is already typically at an ebb, works well.)

Wherever possible, avoid enemies that are large and hard to pinpoint. Simply saying "We need to have better quality code" is too amorphous and too vague. Developers have nothing to measure against. Personalize your enemies, eyeball to eyeball. Put names to them, make them clearly visible to all involved. "We will have 100% code coverage in our unit tests" is a clearly-defined goal, and anything that prevents that goal from being reached will be clearly visible. "We will not ship code that fails to pass any unit test" is another clear goal, but must be paired with something that avoids the natural "Well, then, we'll not write any unit tests, and we'll have met that goal!" response. Demanding a ratio of unit-test-lines-to-lines ratio is a good start: "We will have three lines of unit test code per line of code" now offers a measurable, identifiable enemy that can be stared in the face. Go so far as to make a ceremony out of it: call the developers into a room, put a poster on a wall, and make your intentions clear. Motivate them. "When we presented the release of the payroll system to the HR department last year, the users called it 'barely acceptable' and 'hard to use'. I refuse to allow that to happen again. The system we build for them this year will be amazing. It will be reliable. It will have those features they need to get their job done (and be specific here), and we will accept no excuse otherwise."

One of the world's most successful software companies, Microsoft is no stranger to the polarity strategy. The company as a whole as declared war on its enemies in a variety of fields, and with few exceptions, has won in almost every conflict. Microsoft actively courts conflict and confrontation. The presence of a well-established competitor in a particular field is no barrier to entry; Microsoft has routinely entered fields with dominant competitors and come out ahead in the end. Witness their entries into the word-processor and spreadsheet markets held at the time by dominant competitors WordPerfect and Lotus 1-2-3, their entry into the video-game console market against well-established competitors Sega and Nintendo, and more recently, the mobile entertainment device market (the Zune) against the iPod. In the latter, the battle has just begun, and the market remains firmly in the hands of the iPod, but let it not be forgotten that Microsoft is not one to retreat quickly from a battle.

Microsoft is also known to do this within their projects; developers who are committed to a project yet seem hesitant or lax in their work are asked if they are really "on board" with this project. The legends of Microsoft developers putting in 80-plus hours a week on a project are somewhat true, but not because Microsoft management expects it of them, but because developers have been willing to put that kind of time into the project in order to succeed.

And Microsoft management itself has declared war on its enemies, time and again, looking to eliminate any and all opposition the successful release of software. Distractions? Every developer gets his own office. Household chores? Microsoft has been known to offer their developers laundry services at work. Computing resources? It's not at all uncommon to walk into a Microsoft developers' office and see multiple CPUs (under desks, in corners, laptops, and so on) and monitors all over the room. Bodily needs? Refrigerators on every floor, stocked with sodas of every variety, water, juices, anything that the thirsty developer could need, all free for the taking. Even fatigue is an enemy: Microsoft buildings have video game consoles, foosball tables, bean bag chairs, and more tools of relaxational activities to help the developer take a break when necessary, so that they can resume the fight refreshed.

Greene notes further,

There are always hostile people and destructive relationships. The only way to break out of a negative dynamic is to confront it. Repressing your anger, avoiding the person threatening you, always looking to conciliate--these common strategies spell ruin. Avoidance of conflict becomes a habit, and you lose the taste for battle. Feeling guilty is pointless; it is not your fault you have enemies. Feeling wronged or victimized is equally futile. In both cases you are looking inward, concentrating on yourself and your feelings. Instead of internalizing a bad situation, externalize it and face your enemy. It is the only way out.

To adapt this to software, instead of simply talking about the hopeless situation in which you find yourself--your company has no interest in agile, your team is just too "inexperienced" to tackle the kinds of projects you are being given, and so on--externalize it. Face the enemy. Your company has no interest in agile? Fine--instead of trying to talk them into it, take the radical approach, do a project in an agile fashion (even without upper management's knowledge if necessary), and show them the results. Can't get the IT budget to allow for a source-control server or continuous integration server? Use your desktop machine instead. Face the enemy, confront it, and defeat it.

Enemies are not evil, and should not be seen as something to avoid. Enemies give us something against which to measure ourselves, to use as a foil against which to better ourselves. They motivate and focus your beliefs. Have a co-worker who refuses to see the benefits of dynamic languages? Avoiding him simply avoids an opportunity for you to learn from him and to better your arguments and approaches. Have a boss who doesn't see what the big deal behind a domain-specific language is? Have conversations on the subject, to understand her reluctance and opposition, and build a small DSL to show her the benefits of doing so. Don't avoid these people, for they offer you opportunities to better yourself and your understanding.

Enemies also give you a standard against which to judge yourself. It took Joe Frazier to make Muhammad Ali a truly great boxer. The samurai of Japan had no guage of their excellence unless they challenged the best swordsmen they could find. For the Indianapolis Colts of last year, each victory was hollow unless they could beat their arch-rivals, the New England Patriots. The bigger the opponent, the greater your reward, even in defeat, for if you lose, you have opportunities to analyze the results and discover how and why you lost, then correct your strategy for the next time. For there will always be a next time.

Don't seek to eschew enemies, either. Enemies give us options and purpose. Julius Caesar identified Pompey as his enemy early in his rise to the throne. Everything he did from then on was measured against Pompey, to put him in a stronger position to meet his enemy. Once he defeated Pompey, however, Caesar lost his way, and eventually fell because he viewed himself as a god and above all other men. Enemies force on you a sense of realism and humility.

Remember, enemies surround you and your project, and sometimes even within your project. Keep a sharp eye out, so that once spotted, they can be identified, analyzed, and handled. Show no quarter to those enemies: they must either join you to help you in your quest to build great software, or be ruthlessly eliminated from your path. They can either benefit from the project, or they can be removed from the battlefield entirely. Some enemies--ignorance, apathy, sloth--are not easily defeated, nor once defeated will they remain so. Never lay down your arms against them or trust your arms to someone else--you are the last line of your own defense.

I've been doing a series of video interviews for Pearson (the group behind the publishers Addison-Wesley and Prentice-Hall, among other titles), starting with a series of about a dozen or so we took at the SDWest show back in March. At said show... well... Barbara's blog says it best. (Warning--partial nudity here. Not suitable for work. )

And yes, it really did happen that way--Bjarne and Herb weren't entirely sure if having a T-shirt emblazoned with "I love C#" on it would go well with their fans, so... *shrug* I reversed it and we went on.

Of course, they *were* probably half-joking, and we *could* probably have done it without a problem... but when you're interviewing one of your childhood heroes (actually, two of them), you don't look to let obstacles stand in the way.

Meanwhile, the entire series is now up on iTunes, under "On Software" (or subscribe), or subscribe to an RSS feed. More are coming, and unfortunately I can't tell you who I was interviewing this week at OSCon, or a few months ago at Microsoft TechEd, but they're all looking pretty good....

Oh, and if you have any suggestions of questions to ask these kinds of folks for these video podcasts, by all means, drop me a line--Pearson's planning a bunch more, and I can always use good ideas for questions....

For those who aren't familiar with the term, "yellow journalism" was a moniker applied to journalism (newspapers, at the time) articles that were written with little attention to the facts, and maximum attention to gathering attention and selling newspapers. Articles were sensationalist, highly incorrect or unvalidated, seeking to draw at the emotional strings the readers would fear or want pulled. Popular at the turn of the last century, perhaps the most notable example of yellow journalism was the sinking of the Maine, a US battleship that exploded in harbor while visiting Cuba (then, ironically, a very US-friendly place). Papers at the time attributed the explosion to sabotage work by Spain, despite the fact that no cause or proof of sabotage was ever produced, leading the US to declare war on the Spanish, seize several Spanish colonies (including the Phillipines in the Pacific, which would turn out to be important to US Pacific Naval interests during World War Two), and in general pronouce anything Spanish to be "enemies of the state" and all that.

Vaguely reminiscent of Fox News, now that I think of it.

In this case, however, yellow journalism meets the Web in two recent "IT magazine" pieces that have come to my attention: this one, which blasts Sun for not rolling out updates in a more timely fashion to its consumers, despite the many issues that constant update rollouts pose for those same consumers, but more flagrantly, this one, which states that Google researchers have found a vulnerability in the Java Runtime Environment that "threatens the security of all platforms, browsers, and even mobile devices". As if that wasn't enough, check out these "sky-is-falling" quotes:

" 'It’s a pretty significant weakness, which will have a considerable impact if the exploit codes come to fruition quickly. It could affect a lot of organizations and users.'

"... anyone using the Java Runtime Environment or Java Development Kit is at risk.

" 'Delivery of exploits in this manner is attractive to attackers because even though the browser may be fully patched, some people neglect to also patch programs invoked by browsers to render specific types of content.'

"... the bugs threaten pretty much every modern device.

" '... this exploit is browser independent, as long as it invokes a vulnerable Java Runtime Environment.'

"... the problem is compounded by the slim chance of an enterprise patching Java Runtime vulnerabilities.

Now, I have no problems with the media reporting security vulnerabilities; in fact, I encourage it (as any security professional should), because consumers and administrators can only take action to protect against vulnerabilities when we know about them. But here's the thing: nowhere, not one place in the article, describes what the vulnerability actually is. Is this a class verifier problem? Is this a buffer overflow attack? A luring attack? A flaw in the platform security model? A flaw in how Java parses and consumes image formats (a la the infamous "picture attachment attack" that bedevils Outlook every so often)?

No details are given in this article, just fear, uncertainty and doubt. No quote, no vague description of how the vulnerability can be exploited, not even a link to the original report from Google's Security team.

Folks, that is sensationalist journalism at its best. Or worst, if you prefer.

Mr. Tung, who authored the article, should have titled it "The Sky is Falling! The Sky is Falling!" instead. Frankly, if I were Mr. Tung's editor, this drivel would never have been published. If I were given the editor's job tomorrow, I'd thank Mr. Tung for his efforts and send him over to a competitor's publication. Blatant, irresponsible, and reckless.

Now, if you'll excuse me, I'm going to try and find some hard data on this vulnerability. Any vulnerability that can somehow strike across every JVM ever written (according to the article above) must be some kinda doozy. After all, I need to learn how to defend myself before al Qaeda gets hold of this and takes over "pretty much every modern device" and uses them to take over the world, which surely must be next....

At a software conference not too long ago, I was asked what book I was currently reading that I'd recommend, and I responded, "Robert Greene's The 33 Strategies of War". When asked why I'd recommend this, the response was pretty simple: "Because I believe that there's more parallels to what we do in military history than in constructing buildings."

Greene's book is an attempt at a distillation of what all the most successful generals and military leaders throughout history used to make them so successful. A lot of these concepts and ideas are just generally good practices, but a fair amount of them actually apply pretty directly to software development (whether you call it "agile" or not). Consider this excerpt from the Preface, for example:

The war [that exists in the real world] exists on several levels. Most obviously, we have our rivals on the other side. The world has become increasingly competitive and nasty. In politics, business, even the arts, we face opponents who will do almost anything to gain an edge. More troubling and complex, however, are the battles we face with those who are supposedly on our side. There are those who outwardly play the team game, who act very friendly and agreeable, but who sabotage us behind the scenes, ues the group to promote their own agenda. Others, more difficult to spot, play subtle games of passive aggression, offering help that never comes, instilling guilt as a secret weapon. On the surface everything seems peaceful enough, but just below it, it is every man and woman for him- or herself, this dynamic infecting even families and relationships. The culture may deny this reality and promote a gentler picture, but we know it and feel it, in our battle scars.

Without trying to paint a paranoid picture, this "dynamic of war" frequently infects software development teams and organizations; developers vs. management, developers vs. system adminstrators, developers vs. DBAs, even developers vs. architects or developers vs. developers. His book, then, suggests that we need to face this reality and learn how to deal with it:

What we need are not impossible and inhuman ideals of peace and cooperation to live up to, and the confusion that brings us, but rather practical knowledge on how to deal with conflict and the daily battles we face. And this knowledge is not about how to be more forceful in getting what we want or defending ourselves but rather how to be more rational and strategic when it comes to conflict, channeling our aggressive impulses instead of denying or repressing them. If there is an ideal to aim for, it should be that of the strategic warrior, the man or woman who manages difficult situations and people through deft and intelligent maneuver.

... and I want that man or woman heading up my project team.

It may seem incongruous to draw parallels between war and software development, because in war there is an obvious "enemy", an obvious target for our aggression and intentions and strategies and tactics. It turns out, however, that the "enemy" in software development is far more nebulous and amorphous, that of "failure", which can be just as tenacious and subversive. This enemy won't ever try to storm your cubicles and kill you or try to hold you for ransom, but a lot of the strategies that Greene talks about aren't so much about how to kill people, but how to think strategically, which is, to my mind, something we all of us have to do more of.

Consider this, for example; Greene suggests "six fundamental ideals you should aim for in transforming yourself into a strategic warrior in daily life":

• Look at things as they are, not as your emotions color them. Too often, it's easy to "lose your head" and see the situation in emotional terms, rather than rational ones. "Fear will make you overestimate the enemy and act too defensively"; in other words, fear will cause you to act too conservatively and resist taking the necessary gamble on a technology or idea that will lead to success. "Anger and impatience will draw you into rash actions that will cut off your options"; or, anger and impatience will cause you to act rashly with respect to co-workers (such as DBAs and sysadmins) or technology decisions that may leave you with no clear path forward. "The only remedy is to be aware that the pull of emotion is inevitable, to notice it when it is happening, and to compensate for it."
• Judge people by their actions. "What people say about themselves [on resumes, in meetings, during conversations] does not matter; people will say anything. Look at what they have done; deeds do not lie." Which means, you have to have a way by which to measure those deeds, meaning you have to have a good "feel" for what's going on in your department--simply listening to reports in meetings is often not enough. "In looking back at a defeat [failed project], you must identify the things you could have done differently. It is your own bad strategies, not the unfair opponent [or management decisions or unhelpful IT department, or whatever], that are to blame for your failures. You are responsible for the good and bad in your life."
• Depend on your own arms. "... people tend to rely on things that seem simple and easy or that have worked before. ... But true strategy is psychological--a matter of intelligence, not material force. ... But if your mind is armed with the art of war, there is no power that can take that away. In the middle of a crisis, your mind will find its way to the right solution. ... As Sun-tzu says, 'Being unconquerable lies with yourself.' "
• Worship Athena, not Ares. This one probably doesn't translate directly; Athena was the goddess of war in its form seen in guile, wisdom, and cleverness, whereas Ares was the god of war in its direct and brutal form. Athena always fought with the utmost intelligence and subtlety; Ares fought for the sheer joy of blood. Probably the closest parallel here would be to suggest that we seek subtle solutions, not brute force ones, meaning look for answers that don't require hiring thousands of consultants and developers. But that's a stretch.
• Elevate yourself above the battlefield. "In war, strategy is the art of commanding the entire miliary operation. Tactics, on the other hand, is the skill of forming up the army for battle [project] itself and dealing with the immediate needs of the battlefield. Most of us in life are tacticians, not strategists." Too many project managers (and team members) never look beyond the immediate project in front of them to consider the wider implications of their actions. "To have the power that only strategy can bring, you must be able to elevate yourself above the battlefield, to focus on your long-term objectives, to craft an entire campaign, to get out of the reactive mode that so many battles in life lock you into. Keeping your overall goals in mind, it becomes much easier to decide when to fight [or accept a job or accept a project] and when to walk away."
• Spiritualize your warfare. "... the greatest battle is with yourself--your weaknesses, your emotions, your lack of resolution in seeing things through to the end. You must declare unceasing war on yourself. As a warrior in life, you welcome combat and conflict as ways to prove yourself, to better your skills, to gain courage, confidence and experience." That means we should never let fear or doubt stop us from tackling a new challenge (but, similarly, we shouldn't risk others' welfare on wild risks). "You want more challenges, and you invite more war [or projects]."

Granted, it's not a complete 1-to-1 match, but there's a lot that the average developer can learn from the likes of Sun-Tzu, MacArthur, Julies Caesar, Genghis Khan, Miyamoto Musashi, Erwin Rommel, or Carl von Clausewitz.

Just for reference purposes, the original 33 strategies (some of which may not be easy or even possible to adapt) are:

1. Declare war on your enemies: The Polarity Strategy
2. Do not fight the last war: The Guerilla-War-of-the-Mind Strategy
3. Amidst the turmoil of events, do not lose your presence of mind: The Counterbalance Strategy
4. Create a sense of urgency and desperation: The Death-Ground Strategy
5. Avoid the snares of groupthink: The Command-and-Control Strategy
6. Segment your forces: The Controlled-Chaos Strategy
7. Transform your war into a crusade: Morale Strategies
8. Pick your battles carefully: The Perfect-Economy Strategy
9. Turn the Tables: The Counterattack Strategy
10. Create a threatening presence: Deterrence Strategies
11. Trade space for time: The Nonengagement Strategy
12. Lose battles but win the war: Grand Strategy
13. Know your enemy: The Intelligence Strategy
14. Overwhelm resistance with speed and suddenness: The Blitzkrieg Strategy
15. Control the dynamic: Forcing Strategies
16. Hit them where it hurts: The Center-of-Gravity Strategy
17. Defeat them in detail: The Divide-and-Conquer Strategy
18. Expose and attack your opponent's soft flank: The Turning Strategy
19. Envelop the enemy: The Annihiliation Strategy
20. Maneuver them into weakness: The Ripening-for-the-sickle Strategy
21. Negotiate while advancing: The Diplomatic-War Strategy
22. Know how to end things: The Exit Strategy
23. Weave a seamless blend of fact and fiction: Misperception Strategies
24. Take the line of least expectation: The Ordinary Extraordinary Strategy
25. Occupy the moral high ground: The Righteous Strategy
26. Deny them targets: The Strategy of the Void
27. Seem to work for the interests of others while furthering your own: The Alliance Strategy
28. Give your rivals enough rope to hang themselves: The One-Upmanship Strategy
29. Take small bites: The Fait Accompli Strategy
30. Penetrate their minds: Communication Strategies
31. Destroy them from within: The Inner-Front Strategy
32. Dominate while seeming to submit: The Passive-Aggression Strategy
33. Sow uncertainty and panic through acts of terror: The Chain-Reaction Strategy

What I'm planning to do, then, is go through the 33 strategies of war, analogize as necessary/possible, and publishthe results. Hopefully people find it useful, but even if you don't think it's going to help, it'll help me internalize the elements I want to through the process just for my own use. And, in the end, that's the point of "spiritualize your warfare": trying to continuously enhance yourself.

Naturally, I invite comment and debate; in fact, I'd really encourage it, because I'm not going to promise that these are 100%-polished ideas or concepts, at least as how they apply to software. So please, feel free to comment, either publicly on the blog or privately through email, whether you agree or not. (Particularly if you don't agree--the more the idea is tested, the better it stands, or the sooner it gets refactored.)

David Chappelle, a man I greatly respect and admire, recently blogged that

... the war between REST and WS-* is over. The war ended in a truce rather than crushing victory for one side--it's Korea, not World War II. The now-obvious truth is that both technologies have value, and both will be used going forward.

While I agree with his conclusion (that both technologies have a place and can we please just move along here?), I think the analogy is a bit misplaced. I'll get to that in a second.

Elliott Rusty Harold, never one to let a conclusion go by without putting his name on it somewhere, then tries to take David's conclusion and, in his own unique and partisan style, subvert the discussion and David's conclusion entirely:

That’s a nice analogy. Take it one step further though. WS-* is North Korea and REST is South Korea. While REST will go on to become an economic powerhouse with steadily increasing standards of living for all its citizens, WS-* is doomed to sixty+ years of starvation, poverty, tyranny, and defections until it eventually collapses from its own fundamental inadequacies and is absorbed into the more sensible policies of its neighbor to the South.

The analogy isn’t as silly as it sounds either. North Korean/Soviet style “communism” fails because it believes massive central planning works better than the individual decisions of millions of economic actors. WS-* fails because it believes massive central planning works better than the individual decisions of millions of web sites. It’s no coincidence that the WS-* community constantly churns out volume after volume of specification and one tool after another. The WS-* community really believes that developers are too stupid to be allowed to manage themselves. Developers have to be told what to do and kept from getting their grubby little hands all over the network protocols because they can’t be trusted to make the right choices.

By contrast you don’t see a lot of complicated REST frameworks or specifications. You could read all the relevant REST specifications in a slow afternoon (mostly the HTTP spec and a couple of subsidiary RFCs, plus XML and Namespaces in XML. Maybe Atom syntax and Atom-pub too if you feel ambitious.). REST/HTTP sets up a simple economic system based on a few clear rules, and then pretty much gets out of the way to let people do their own thing. It doesn’t even get too upset when people break the rules, and start tunneling everything through POST or deleting items with GET. The main people harmed by such bad decisions will be the sites themselves, and they will be dealt with by the RESTful market.

Let's get a few things straight: as any student of modern political theory will tell you, communism is a political system built around an economic model. The economic model, in many ways, is an idealistic one, somewhat reminiscent of the open source model: everybody produces what they can, contributes it to the whole, and partakes of that collection only as much as they need to in order to meet their needs/requirements. "From each, according to their abilities, to each, according to their needs". In a perfect world, it's a far more sensible system than this capitalistic system in which we find ourselves--it's a system that's predicated on cooperation rather than competition, looking to avoid the extremes of the capitalistic system: no poverty, and no blindingly rich.

The political system, however, is where communist theory breaks down completely. The governing body, as any group invested with absolute power will, in Hobbesian fashion, becomes corrupt and looks to ensure that it receives the lion's share of the benefits. This distorts the entire basis of the system, leaving those who are producing at their fullest potential to wonder, "What's in it for me?" and look to get away with doing as little as possible in order to obtain the maximum possible. In short, the commonly-attributed failure of communism is "human nature".

(Sort of puts a rather dark horizon on the whole open-source community, if you ask me.)

I don't think anyone is going to hold up North Korea as a model Communist society; in fact, of all those governments still following the communist political doctrine (of which the current count, by most sources, is two: Cuba and North Korea), it's fair to say that none of them are exactly winning admirers either among their populations, the political watchdog groups, or the academics. So drawing any kind of analogy in which "communists" are one end of the analogy is almost guaranteed to make your partisaned point of view pretty clear. In fact, politically, the problem with North Korea isn't so much with its economic system as its totalitarian form of government, which is almost entirely orthogonal to its centrally-planned economy; several European states--Sweden being one of them--have high degrees of central planning and they seem to get along quite well with the rest of the world. So let's put the "North Korea is bad because it's communist" argument on the shelf, shall we? It's the fact that (a) they're totalitarian, and more importantly, (b) their leader doesn't like us, that makes them one of the "Axis of Evil" states. Saudi Arabia is also totalitarian, and their populace doesn't exactly rank at the top of most standards-of-living charts, either. In fact, the Saudis (76) rank below Cuba (50) on one measurement of standard of living. (Data drawn from the Human Development Report 2006.)

All that said, it's probably obvious that (a) I don't think the communistic argument here is relevant, and (b) I don't think the WS-* set of agreements are at all communistic or totalitarian. In fact, I'd argue they much more closely follow the model of the European Union, rather than communistic practice at either the economic or political levels.

Look at it this way: the WS-* set of documents intend to provide points of agreed interoperability, not a set of practices that platform developers must follow. Assume you, a Ruby shop, and I, a Java shop, want to integrate systems. Assume we need to ensure that the communication cannot be hijacked, must have certain atomic guarantees, and can't just "disappear" into the ether of the Internet. I don't particularly care how you write your code, and you don't particularly care how I write mine, but in order for us to get our work done between us, we have to agree on how the stuff across the wire will look.

Consider, for a moment, the issue of security: We can certainly agree on using HTTP/S, which requires that we both establish digital certificates that will be validated by code every time the communication is established. (We can't rely on the Web's traditional one-sided certificate approach, because that only verifies that the client knows who the server is; the server has no mathematical guarantees that the client is who they claim.) HTTP/S also means a new shared session key must be exchanged between the principals, meaning that no intermediaries (a fundamental feature of TCP/IP, and of the RESTful architecture itself) are possible, since now the intermediary has to have its own certificate, if it's going to be able to influence the communication somehow. That means now that you and I don't trust each other directly, but a third party, which is a wholly different set of discussions.

WS-Security (or, more appropriately, WS-SecureConversation) addresses this.

On the issue of reliable communication, we can either assume that the TCP/IP "best delivery" semantics are sufficient, but given how many of us have seen the "ghost request", the HTTP request that gets sent from the browser but never gets a response, or the "phantom email" that get sent and never received, most of us are going to be somewhat skeptical about the idea of our bank using TCP/IP for transfers and deposits without some degree of additional reliability requirements. Using REST, then, we build our own ACK system, identifying messages with unique values and requiring recipients to respond when a message is received, just to make sure the message was received.

WS-ReliableMessaging addresses this.

These are not requirements, by the way--you do not have to use these facilities if you are building a SOAP-based service. They are opt-in, if your payload is wrapped in an SOAP envelope (which consists of two additional elements around your POX data, by the way). In fact, if you build your own REST/POX service, chances are you're going to end up re-designing something very similar to the SOAP envelope anyway, when you build in some kind of structure for handling out-of-band data (like authentication information). You're free to use HTTP headers, but they're limited to simple name-value pairs, which can be somewhat constraining at times. (Look at what cookies look like sometime; clearly they could benefit in a big way from some larger structure.)

The key thing here is, when you want these facilities, people have already agreed on how they should look so that you don't have to work out all those details for yourself. In this way, it's something like the Euro: a nation doesn't have to use the Euro as its currency (look at Great Britain), but it benefits from doing so. Or, for a different analogy, the WS-* set of specifications are like NATO: an agreed-upon set of protocols and understandings designed to allow for maximum interoperability during a time of crisis (namely, the presumed invasion of Germany by the Warsaw Pact nations). It didn't mean that each nation wasn't able to purchase whatever arms it desired, or train their soldiers in whatever fashion they desired, it was simply an established logistical structure to help everybody "get along" and work together effectively in the face of the crushing numbers of Soviet, Polish, and East German (among others) tanks and soldiers.

In essence, the goal of REST was to create a loose-as-possible coupling between client and server for the purpose of distributed hypermedia. Fielding's thesis makes this eminently clear from the beginning. Trying to create a generalized distributed communication system was never part of the goal, at least according to the thesis itself. More importantly, there are numerous places where tighter coupling, or greater interaction, or some kind of transactional capability, are necessary. Not in all situations, but as Michael Nygard points out in Release It!, a lot of the integration scenarios that developers face aren't across HTTP boundaries, but across departmental boundaries.

Oh, and it's not just the big vendors who get to propose these SOAP-based standards (though if you want yours to gain traction with the industry as a whole, you're obviously going to have to elicit their support); you're more than welcome/free to use the SOAP Envelope/Header area for your own purposes, just as we do with HTTP headers. Just make sure your tags don't conflict with anybody else's (ah, yes, that's what XML namespaces were for), and party on.

Put that way, the WS-* stack doesn't seem so bad, does it? Clearly the Euro and NATO function a lot better than communism does...

Meanwhile, back to Korea.

It may seem odd to many that I'm criticizing the use of a war as an analogy to Computer Science; after all, I was the one who let the genie out of the bottle in the first place. My issue here isn't with the use of a war as an analogy to what we do (I think there's a lot more analogy to war in what we do than there is to building a house, in fact), but with the particular analogy in question. The SOAP-vs-REST debates helped frame a large part of how we view distributed systems. Korea, "the forgotten war", gave us a legacy of...

... well, nothing.

Apologies to all who served in Korea (as did my uncles), but the basic fact was that Korea really didn't change anything. At the close of World War Two, the Koreans, who'd lost their national independence to Japan in the mid-30's, were eager to re-form their nation and resume governance. Unfortunately, the two superpowers had their own ideas: the Soviet Union (and, more notably, China) refused to consider a united Korea under Western influence, and the US refused to consider a Korea under Communist influence, regardless of what the people themselves might have wanted. (Truthfully, it's hard to tell now what the general populace wanted at the time--sources are pretty biased and partisan and entirely conflicting.) This was how things were left, the Korean peninsula split at the 38th parallel, until 1950.

On June 25th, 1950, the North invaded the South in a surprise attack, pushed the ROK (Republic of Korea) and allied forces back down the peninsula until General MacArthur, the Pacific War hero of World War Two, dared an amphibious landing at Seoul, far to the rear of the front lines, and the Allies were able to push the North's forces back up the peninsula. Then, pursuing a strategy of "total war", MacArthur chose to continue north, pushing the Communist forces all the way back to their border with China. MacArthur, in fact, wanted to push even further, taking the war into China if necessary, to achieve the kind of total victory he'd been able to create in World War Two. President Truman, mindful of the fact that the Soviet Union was presumably willing to go to war globally to prevent the fall of its (presumably) close ally, refused to allow Truman to slip his leash, and ultimately dismissed him, an entirely unpopular move with the US population. (MacArthur was widely supported in the US, and there were very real fears he would stage a coup and remove Truman from office. MacArthur turned out to be more patriot than populist leader, however, and wrote one of the most stirring speeches ever delivered instead.) China then dispatched troops to aid the North, pushing the Allies back to the 38th parallel, where things remained until an armistice (and later a peace) was signed in 1953.

Militarily, nothing much changed. The front lines remained as they are today, fifty years later, and the war did nothing to change the balance of power in the region, or in the world. The US still feared and distrusted Communist China and Communist USSR (who were in fact very dubious allies, though it wasn't known at the time), ultimately leading in part to the Red Scare and the Committee on Un-American Activities, what came to be known as "McCarthyism". For most of the domestic US population, once the troops came home, the desire was simply to "go back to the way things were", and forget that the conflict ever happened.

This is why I disagree with the analogy: for many of us who've participated in this debate, the "REST-vs-SOAP" discussion (which really was more of a "REST-vs-RPC" discussion) helped frame several key concepts: the idea of coarse-grained communication, the desire for loose coupling, the discussion around XML as a lingua franca for data, and so on. Clearly the REST-vs-RPC debate helped frame a great deal of the conversation around distributed systems for decades to come, and in that sense, it makes zero sense whatsoever to analogize that debate as being similar to the Korean War.

Those who know me or who've seen me speak know that I don't pull any punches; this is a deliberate stance on my part, as I'm generally way too busy to bother with soft-shoeing around topical areas that might be sensitive to certain groups or teams. I call 'em as I see 'em, and if people don't like the results, I'm always open to being convinced otherwise. (Strong opinionation and high open-mindedness have to go hand-in-hand, if you're going to work to avoid being proven a complete idiot repeatedly in your life.)

That's why I have to give a huge shout out to the Sun build and source-repository engineers who've been working over the last half-year or so (probably longer, but I don't know for certain) to make the OpenJDK project a reality. Where they could have simply tossed the source and build state into a Subversion or CVS respository, washed their hands and said, "There, the source is out there, enjoy", they've instead slowly-and-steadily taken what was a pretty ugly setup and build process and whittled it down to a pretty dirt-simple set of instructions to get a JDK build up and running on your local machine.

If you ever took a moment to pull down the SCSL or JCL sources for JDK 1.2 or 1.3, particularly if you were on a Windows box (as I am), you probably fled screaming from the room (as I did, more than once). The old builds required out-of-date versions of Microsoft Visual C++ (5.0!), and a commercial UNIX-like toolset (MKS, and not even a version you could purchase anywhere, from what I could see). Clearly, the build process in those days was geared specifically around the environment that was existing inside of Sun, and if you weren't a Sun employee with access to those specific versions of those tools, forget it. You still had the source, but...

I pulled down the OpenJDK bits again last night (fresh SVN checkout), and I realized as I was going through the steps to rebrick the build environment that it's been getting steadily simpler and simpler. First, the build tools for Windows now need nothing more complicated than a few tools out of Cygwin and the GNU Make utility (largely because the Windows NMAKE utility is pretty weak compared to GNU's, not to mention the fact that NMAKE doesn't really exist for non-Windows platforms... the SSCLI-built version being the only exception I know). Second, the version of the Microsoft compiler needed has been upgraded to Visual Studio 2003 (not 2005, as building native apps under 2005 took a left turn, as anyone who's ever wrestled with manifests and DLLs can tell you), and a version of the DirectX9 SDK (which is a free download from MSDN). As a matter of fact, if you just want to build the various flavors of Hotspot and not the rt.jar bits from sources, you can even skip GNU make and the DirectX SDK. It's almost turnkey from there.

If you're any kind of plumbing wonk, as I aspire/desire/claim to be, this is a huge step in the right direction, and it's easy to repeat: fire up your Subversion client, point it to the OpenJDK SVN respository, pull down the trunk, and start building. Particularly fun is to build a 'debug' build of Hotspot in order to get the symbols, then build a custom Java launcher, and step through it in the debugger. And I mean, right out of your launcher and into the JDK itself. Or, drop the compiled 'debug' JVM.DLL into your JRE's bin\client or bin\server (or, as I do, create new subdirectories in the bin dir and create some custom -debugclient and -debugserver options, so you can switch back and forth as desired). Or, take the compiled 'fastdebug' build, and run it with the '-Xprintflags' option, in case you needed to be convinced that you really don't know all the -XX options that are available to you...

Kudos, Sun build team. Major, major kudos.

And yes, for those in the .NET space that were wondering, SSCLI Essentials, 2nd Ed, is under way as I write this...

It's not often that I meet another "Ted" in the world, much less in my own industry. The last one I knew (besides my great-uncle Ted, who unfortunately passed away a number of years ago) was Ted Rallis, a buddy of mine from my college days who was equal parts philosopher and contemplator-of-navel lint and computer scientist (whom I haven't seen in years, if you're out there Mr. Rallis, drop me a line).

I'd heard rumors of an evil twin brother, though. People claimed that they'd met me at other shows, and while I won't claim to remember everybody I've ever met at a conference (much as I'd like to, my memory just isn't that good), I was pretty convinced that at least a percentage of those folks claiming to have met me before were somehow mistaken. Now I'm sure of it.

While at TSSJS 2007 in Barcelona, I met Ted Goddard, the architect of ICEFaces, and damn if we don't look alike. One of his compatriots snapped a photo of the two of us, and Ted blogged it.

Oh, and by the way? If you're using JSF (which a rising number of people appear to be doing, according to the NFJS polls we take during the speaker panel), you should check out the fruit of Ted's labors. I'm no JSF expert, but people I trust tell me it's pretty good stuff....