Since a number of people have been connecting to my blog via my last post on installing Windows 7 into a VMWare image, I thought since the Windows7 RC is now available, I'd update my experiences with installing it.

I downloaded the Windows7 RC ISO image (a freakishly hideous name containing every character on my US keyboard, plus a few in Klingon, I think.... if you can stand it, the full name of the ISO is 7100.0.090421-1700_x86fre_client_en-us_Retail_Ultimate-GRC1CULFRER_EN_DVD) from the Microsoft CONNECT website, not bothering with any of the other images (x64, ia64, and a "server" image I've not explored yet), using Microsoft's File Transfer Manager. (I know, I know, somebody's going to complain again about the ISOs not being available via a straight HTTP download or Torrent, but this is just an RC release, folks, and this is ostensibly to Microsoft-friendly customers who already have the FTM utility installed.) Took about 3+ hours to download on my home connection... or so it claimed. I went to bed after starting it last night. It was done when I woke up. What more do you want from me?

I created a new VMWare image, as a "Windows Vista" VM with 1GB RAM, a 60GB IDE hard disk (by default Fusion wants to create a 40 GB SCSI disk, but IDE seems to play nicer with the early betas of Microsoft OS'es, and I made it all one file rather than Fusion's default "Split into 2GB files" option), with the experimental 3D graphics turned on, battery status turned off, and (this is HUGE) the "Allow your Mac to open applications in the virtual machine" option turned OFF. Can't repeat this enough, for ANY VMWare VM containing Windows inside of it, turn off that option—leaving it on sucks up HUGE amounts of CPU time. (It's barely documented, and only determined Googling found that this was what was rendering my VMWare Fusion 2 images all but unusable.)

I attached the ISO to the VMWare CD and turned the thing loose. It takes a while, but so long as the ISO file and the VMWare VMDK disk file are on separate drives, the perf isn't too bad—roughly twenty minutes later (or, as I measure things, one randomly-generated map game of Pax Galaxia later), the image had installed all the core files on the VM disk, restarted itself, finished the installation, and restarted itself again. (I have no idea why Win7 wants to reboot itself twice during the install—if I remember the Vista installs correctly, it only restarts once). As I write this, I'm starting at the "Setup is preparing your computer for first use" screen with the funky Cylon-like flashing bar underneath the text (I'm serious, it really looks like the graphic artists at Microsoft are paying homage to BSG during that Setup screen). Whoops, I take it back—got through that screen rather quickly, and now we're into the username/password/product key stage. Plug that in, set the Update policy, the date and time, the network defaults (Public Location for all my VMs, just because), and.... "Welcome".

There's no Step Four. Although, according to Windows Update, there's already an update for Windows7 that should be downloaded and installed. *grin* Actually, it seems like the driver it installed was for the VMWare virtual sound device, which normally doesn't kick in until I install the VMWare Tools. It tells me that this is an "Unsupported Creative Sound Device", however, so maybe it's an older driver. *shrug* Not sure, don't care, because my next step is....

Install the VMWare Tools. I install VMWare Tools in the image, after the Update is complete. (No restart was required, so why not?) Actually, let me rephrase that—I tried to install the VMWare tools, but when I selected it from the Fusion menu bar... nothing happened. Hmm. OK, let's do the restart and see what happens. VM shuts down quickly enough (no having to wait for updates to finish, which was somewhat annoying with Vista), and when I restart, it seems to restart quickly enough (again, no obvious updates to be installed), so I get to a working desktop (640x480, how did we ever think this was reasonable?!?), and try the Install VMWare Tools option from the Fusion menubar again. It thinks for a bit, and the cursor flashes to the "pointer-with-CD" icon for a second before flashing back, but after a few seconds, the "What do you want to do?" (Autoplay) menu pops up as if I'd slipped the CD into the drive, so all looks good. Go through the UAC "Continue/Cancel" dialog (see below), choose "Complete" for the VMWare Tools install options, and let 'er rip. Disks spin, lights flicker, and a "VMWare Shared Tools" network folder shows up on the Win7 desktop, indicating that it's suddenly discovered the Shared Folder (to my MacOS user account) is there. But now we're back to the Windows-display-exercise program, which leads me to believe that it's the VMWare driver that's doing the exercise, not Windows itself. (VMWare? Anybody listening and care to comment?)

And now I'm into Win7 desktop customization steps, things like display sizing and desktop icon selection, background image, and all that other jazz that you probably don't care about. (If you do, then I'm a bit worried about you—be an individual! Choose your own settings!) All in all, pretty flawless and smooth.

Thoughts on the process:

• It feels like we're getting away from the "minimal install" process that Vista tried to create. For a while, there was a meme that said that installing Windows was too hard for the average person, and Microsoft promised to reduce the number of steps it had to go through to install the OS. Take the date/time screen, for example: it picked up the defaults from the underlying (virtual) hardware, why not just assume those and skip that step? Users can always change it later.
• I still have to set a Administrator password. I know that Microsoft is trying to find that sweet-spot balance between "too secure" and "unsecure" for desktop operating systems, but I have to hand it to the Ubuntu folks here—the "passwordless root" idea that they use is pretty slick. MacOS uses it (for the most part) in places, as well. I like the balance that approach achieves: it forces the user to enter "superuser" mode to do something sensitive, but it isn't challenging for a password (unless the superuser installs one) every time.
• It's not going through display-screen calesthenics on each startup with this build. My previous Win7 image, every time I restart the VM, goes through every possible video/monitor size combination before settling in on the resolution I established in the last session. That was a bit disconcerting, until I realized that it's Windows trying to get some exercise in to be less overweight. *grin*
• What, no PowerShell installed by default? Either it's not there, or it's buried pretty deeply. Command Prompt (cmd.exe) is right where it's always been, under Accessories, but no PowerShell.... Whoops, no, I take it back, it's in a folder underneath Accessories, forcing one more click to get to it. Hey Microsoft: do me a favor and pin that guy to the Start Menu. Make it easy for me to use, if you really want me to believe that this is supposed to replace Command Prompt someday.
• On that note, though, the PowerShell "ISE" (Interactive Scripting Environment) is an interesting and new toy to play with.
• "Pin to Taskbar" is an interesting option that I'm going to have to play around with. Not being a huge MacOS Dock fan (which is pretty clearly the inspiration for the new Taskbar), I'm not sure how well I'll like the new "it's the QuickLaunch and the Taskbar combined" idea.

Overall, I'm looking forward to putting a few things into this image (VS 2008, VS 2010, Office, and so on) and seeing how it reacts. As always, your mileage may vary, no implied warranties with this blog post, blah blah blah, but if you do anything with the Windows OS, you really should get hold of the RC (build 7100) and put it into a Virtual PC, VMWare, VirtualBox, Xen or some other virtualized box to play with. Like it or not, it's entirely reasonable to believe that Windows7 is going to win a few folks back from the Vista "less-than-I-expected" crowd.

As always, caveat emptor, and feel free to comment....

It's been going around in developer circles now for a few days, this whole controversy about the "Perform like a pr0n star" presentation from the Golden Gate Ruby Conference and the related accusations of misogyny and sexism and overblown accusations and double-standardisms and what-all else, and I've deliberately waited to let opinions in my head settle out before blogging on the whole thing. Sara J Chipps reacts on her blog, and the comments to her comments are also somewhat... interesting... to note.

Without any particular implied importance or order:

• Matt Aimonetti, you are an idiot. You had to know that this was going to generate more than a few strong reactions. I'll admit, it's a funny title, and it definitely generated a ton of buzz around your name, but for the rest of your life, you're going to be "the porno Rails guy", and in about a year or so, it's not going to be funny anymore. You've touched off a firestorm, and you can't very well hide from it, and frankly, I think the short-term boost to your public recognizance is going to be more than outweighed by the long-term judgments that will be levied against you. "Wait, this is the guy who did that talk? Wow. I bet he's a good developer, but can I risk him pulling the same kind of stunt at a meeting with our VP or clients? Nah, I'll go for this other guy...."
• Clearly we have a lot of issues to work out in the programming industry. I'm not going to go into the rights or wrongs of putting those images into his talk. I'm talking about the discussion that followed (one comment here says, "Matt Aimonetti is obviously an antisocial twerp still living in his mothers basement at the age of 35 who has never even been able to muster up the courage to actually talk to a real-life woman, let alone respect one.", and a follow-up comment says, "Great presentation, nevermind the jackasses, keep up the good work!"), and the fact that at no point in the time leading up to this presentation did anybody pull Mr. Aimonetti off to one side and say, "Dude, it was funny when we thought of it, sure, but it's time to stop." If ever we wanted to convince the rest of the world that the programming industry wasn't populated by a bunch of 13-year-olds giggling over the fact that somebody said, "Boobies".... well, maybe next year.
• Ruby community, you have a long way to go if you want to convince people to spend money on you. Maybe you don't mind that corporations think that you guys are clearly unstable and immature. If/when you want to gain some degree of corporate acceptance, and maybe make it out of your parents' basement someday, you're going to have to learn that how you handle yourself in public goes a long way towards establishing peoples' attitudes towards you as professionals, and right now, you all collectively look like a bunch of 13-year-olds, what between this and DHH's famous "FUCK YOU" presentation of a few years ago. If you're OK with not being taken seriously, then cool, more power to you. But personally, I like the idea of making money at things I like to do and have fun doing, and you're not helping yourselves.
• Why are we such prudes? Whether you agree or not with the rightness of the "porn" metaphor, you have to admit that there is factual basis in the bones of this particular comment: "This is probably the least offensive thing I've seen in 3 weeks." Glance at the billboards in the airport next time you're walking to the gate. Glance at the racks of magazines in the grocery store as you prepare to check out. Glance at the beer commercials on TV during prime-time. In every case, sexy, young, attractive, scantily-clad men and women seek to create an instinctive emotional reaction inside your head to subconsciously create a feel-good link between whatever product is being hawked and your id. Honestly, the photos in the presentation are hardly all that titillating—and a very long ways from the kind of commercials you can see on TV in Europe—so why are we getting up in arms over this?
• Matt Aimonetti, you are an idiot. Notice how nobody's talking about the actual subject of your presentation? A good presenter knows that the message should never outstrip the delivery mechanism, just like a sauce should never overpower the flavor of the dish it accompanies. For all that the content of your presentation might have been spot-on, the lessons that might have been learned from the presentation have drowned in the "He's a pig!" "No he's not!" that has followed. Great job there, mate. Way to get your message across.
• To the commenter on the presentation page who said, "ps [sic] feminism is dead", get a clue. Women still, on average, get paid less than men do for an equivalently-skilled employee in the same job. Maybe it's not $.50 to every $1 as it used to be, but so long as it's even measurable, there's work to be done. This industry in particular has absolutely no reason for gender discrimination in any form, since there's absolutely nothing "physical" about what we do. (Ditto for medicine and law, for that matter.)
• Presentations reach far beyond just the attendees. One commenter on Sara's blog notes, "What an over reaction, there was nothing wrong with that presentation, i wouldn't show it to a board room but as far as showing it to a ruby developers conference then no probs." Frankly, that's a short-sighted attitude, making the presumption that someone of the suit-and-tie set (those supposedly inhabiting the "board room" where this kind of presentation isn't appropriate) wouldn't actually be in the audience at a ruby developers conference. Oh, granted, when in Rome, one has to expect Romans to act like Romans, but that just means that the Ruby community isn't welcome inside the board room, right? (Somehow I doubt this is what the numerous people who are trying to make money off of Ruby really want.) Fact is, that presentation is now captured by the Internet for all time, and it will forever be known as "The Ruby Porno Presentation", and it's an even money bet that somebody in that board room has seen the presentation (and the video, and the play-by-play from the people who had friends who had friends that were there....).
• To the commenters who say, "You asked for it", get a clue. Commenters have suggested that the title should have clued people into what was coming: "I'm totally flabbergasted no one has stated the obvious here: if you see a presentation labeled "CouchDB: Perform like a pr0n star" and you choose to go to it, don't act all surprised when R-rated images are used as props." Sorry, no biscuit. Presenters use analogies and imagery all the time in their titles in order to "sell" their talks. Recently I was part of a talk that was labeled as a "smackdown"—did that mean the audience should have expected to see images of physical violence? If I title my next talk as something that's "hard-core", should you expect to see images of ball gags and snuff film clips? This is what happens when we co-opt terms like "smackdown" and "hard-core"—you can't fall back to the original meanings and then claim ignorance when people misunderstand how you're going to use them. (God only knows what Mr. Aimonetti would have done for a presentation on "Naked Objects". *shudder*)
• Matt Aimonetti, you are an idiot. You could have had your joke and keep it tasteful too. You do, in fact, from time to time in the early part of the presentation: the photo of the "little blue pills" was perfect, offering a hint as to what you meant while keeping the double-entendre alive. Every single "objectionable" photo in that presentation could have been replaced by a more subtle one that kept everybody's mind on the subject and still got the point across. The fact that you resorted to the heavy-handed imagery only proves that you wanted to beat the audience's head with it.
• Please, let the one-ups-manship stop. Can we please agree that moving and powerful presentations can be done without having to resort to cheap tricks? They almost always come off badly, particularly when you have to keep the gag running for a full hour or so. Anybody remember Marc Fleury's "Joker" retinue at TheServerSide a half-decade ago? Can you tell me what his presentation was about? Now, consider Dave Thomas' "Cargo Cults" talk from NFJS around the same time—what was he covering? If you were there for both talks, chances are you remember Dave's talk far better than you remember the Fleury keynote beyond the fact that he wore Joker face paint the entire time. Good presentations are about using humor to underscore and support the message, and not making humor the central point of the message. Think about that before you start reaching for the bad innuendo.
• Is this really the kind of industry we want? Granted, it may seem like all of this is way overblown if you're a 25-year-old guy recently graduated from college and hacking on your first or second Rails project. "What do these grumpy idiots not understand about 'it's a joke'? My God, is everybody nuts? Are they trying to say that we can't have fun at work or with what we do?" To which all I can say is two things: one, check in with yourself five or ten years from now, when your daughters are learning about body images by staring at pictures of women who are entirely artificial (and yes, guys, those pictures you see are entirely artificial, having been touched up and enhanced in many ways), and two, you're more than welcome to have whatever jokes you like at whomever's expense you like, in private. This wasn't in private. A developer conference is not a private locale. More importantly, though, think about it—when you bring your girlfriend to work, do you want her hearing those same jokes that buddies toss off back and forth? What seems like "harmless fun" now, may have a very different feel to it for you a few years from now.

I'll freely admit, I drop profanity from time to time in my presentations. And to everyone who comes up afterwords (figuratively and literally) saying I shouldn't use such offensive language, I apologize, and point out that I did so in order to underscore the point, knowing that I'm taking that risk, and knowing that I may be required to offer up apologies after the fact for having offended them. (To date, those apologies still number in the single digits.) So perhaps I am no better than Mr. Aimonetti in the final accounting of things.

But all of this loses sight of a core principle. Regardless of the efficacy of his presentation, regardless of your feelings about the subject matter, regardless of your thoughts around the overblown-or-not nature of this discussion, a deeper principle is at stake here, that of professional presentation etiquette: Mr. Aimonetti, you owe an apology to anyone and everyone that was offended by your presentation (for whatever reason). Failure to deliver that, in my mind, equates to a personal and professional FAIL on your part.

When you stand up on stage, and you say something that somebody finds offensive, you owe that person an apology, even if you think their reasoning or rationale is bogus.

It's simple common courtesy.

Recently, NFJS alum and buddy Dion Almaer questioned the widespread, almost default, usage of a relational database for all things storage related:

Ian Hickson: “I expect I’ll be reverse-engineering SQLite and speccing that, if nothing better is picked first. As it is, people are starting to use the database feature in actual Web apps (e.g. mobile GMail, iirc).”

When I read that comment to Vlad’s post on HTML 5 Web Storage I gulped. This would basically make SQLite the HTML 5 for storage in the browser. You would have to be a little crazy to re-write the exact semantics (including bugs) of SQLite and its dialect. What if you couldn’t use the public domain code?

Gears lead out strong with making a relational database part of the toolbox for developers. It embedded its own SQLite, in fact one that was customized to have the very cool full text search ability. However, this brings up the point of “which SQLite do you standardize on?”

The beauty of using SQL and SQLite is that many developers already know it. RDBMS has been mainstream for donkey’s years; we have tools to manage SQL, to view the model, and to tweak for performance. It has gone through the test of time.

However, SQL has always been at odds with many developers. Ted Neward brought up ORM as the vietnam of computer science (which is going a touch far ;). I was just lamenting with a friend at Microsoft on how developers spend 90% of their time munging data. Our life is one of transformations, and that is why I am interested in a world of JavaScript on client and server AND database. We aren’t there yet, but hopefully we can make progress.

One of Vlad’s main questions is “Is SQL the right API for Web developers?” and it is a valid one. I quickly found that for most of my tasks with the DB I just wanted to deal with JSON and hence created a wrapper GearsDB to let me insert/update/select/delete the database with a JSON view of the world. You probably wouldn’t want to do this on large production applications for performance reasons, but it works well for me.

Now a days, we have interesting APIs such as JSONQuery which Persevere (and other databases) use. I would love to see Firefox and other browsers support something like this and let us live in JSON throughout the stack. It feels so much more Webby, and also, some of the reasons that made us stay with SQL don’t matter as much in the client side world. For example, when OODBMS took off in some Enterprises, I remember having all of these Versant to Oracle exports just so people could report on the darn data. On the client the database is used for a very different reason (local storage) so lets use JSON!

That being said, at this point there are applications such as Gmail, MySpace search, Zoho, and many iPhone Web applications that use the SQL storage in browsers. In fact, if we had the API in Firefox I would have Bespin using it right now! We had a version of this that abstracted on top of stores, but it was a pain. I would love to just use HTML 5 storage and be done.

So, I think that Firefox should actually support this for practical reasons (and we have SQLite right there!) but should push JSON APIs and let developers decide. I hope that JSON wins, you? I also hope that Hixie doesn’t have to spec SQLite :/

Dion's right when he says "developers spend 90% of their time munging data" and that "Our life is one of transformations", but I think he's being short-sighted and entirely narrow-minded when he says, "I am interested in a world of JavaScript on client and server AND database." Dion, I love you, man, but you're falling prey to the Fallacy of the One True Language. JavaScript (or ECMAScript, as its official name is given) is an interesting and powerful language, but why do you want to force your biases and perceptions on the rest of the world, man? You're being just as bad as the C++ or Java guys were in their heyday—remember when Java stored procedures were all the rage because "everybody knows that Java is the wave of the future"?

The fact is, from where I stand, there is no one storage solution or language solution or user-interface solution that is the Right Thing To Do in all situations. Not even inside the browser. There will be situations where a SQLite is the Right Thing, and other situations where a document-oriented JSON-like or CouchDB-like thing will be the Right Thing, and trying to force-feed one into a situation that's best solved by the other is a bad idea.

Dion alludes to my article about the Vietnam of Computer Science, but in fact, his suggestion charges right into another quagmire—how long before somebody starts trying to create a JSON-to-RDBMS adaption layer? Or JSON-to-CouchDB? Or things equally ridiculous? The fact is, data has three fundamentally different "shapes" to it, and trying to pound data from one shape into the other has all the efficacy and elegance to it just as much as pounding round pegs into square holes does. Dion even alludes to this with this paragraph:

One of Vlad’s main questions is “Is SQL the right API for Web developers?” and it is a valid one. I quickly found that for most of my tasks with the DB I just wanted to deal with JSON and hence created a wrapper GearsDB to let me insert/update/select/delete the database with a JSON view of the world. You probably wouldn’t want to do this on large production applications for performance reasons, but it works well for me.

JSON is certainly an attractive representation format for ECMAScript objects, thanks to its fundamental roots in ECMAScript's object literal syntax, and the powerful/dangerous eval() functionality offered by ECMAScript environments, but JSON also lacks a number of things a SQL-based dialect has, including a powerful query syntax for selecting individual and subsets of entities from the whole, which only becomes more and more necessary as the data base itself gets larger and larger. (Anybody who suggests that a local browser store would only remain within a certain size is clearly not thinking further ahead than the current day. Look at how cookies are outrageously abused as local storage for a lot of sites, or how Viewstate was abused in early ASP.NET apps—if you give the HTML/front-end developer a local storage mechanism, they will use it, and use it as far and as long and as hard as they can.) On top of which, JSON simply doesn't have the years of solid backing behind it than a SQL-based storage format does. And so on, and so on, and so on.

Ironically, just as JSON is a scheme for representing native objects in some kind of data format (in this case, a plain-text one), developers casually ignore the idea of storing objects in a native data format with all of the other bells-and-whistles that a database provides. Naturally, I'm referring to the idea of an object database—if JSON is appropriate for storing certain kinds of data in certain scenarios, then why isn't it appropriate to consider a native object database for some of those same certain kinds of scenarios? Not that I have anything against a JSON-based database scenario—in fact, I can easily imagine a JSON database that indexes the properties of the stored objects and takes ECMAScript functions as "native queries" in the same way that db4o doe. But let's stop with the repeated attempts at "one size fits all", and just accept that the world is a polyglot world, and that no one language—or data storage format, or data access API—will be the Right Thing To Do for all scenarios. Each language, format, API or tool has a reason to exist, a particular way it looks at the world, and optimizes itself to work best when used in that particular style. Trying to force one into the terms of the other is the road to another Computer Science quagmire.

Viva la Polyglot!

A friend of mine, from Canada, recently decided not to come to the US anymore.

Today was my final time trying to enter the US to do what many other people have done in my industry before: go and speak at a conference.

The reason I was given this time was that although I had forfeit the speaking fee they were going to pay me, I was still going to be speaking at a conference where other speakers were getting paid, and that there was no reason an American couldn’t fill that spot. When I asked if there would have been any issue if the conference was a free one and nobody was getting paid, I didn’t get an answer.

D'Arcy's experience at the border control reflects a growing dilemma that other speakers in my industry have also been facing: when you travel overseas to speak at a conference, and you get the dreaded "What are you here for?" question, should you tell them the truth and face the battery of questions that boil down to "Are you taking any money out of the country?", or should you lie, claim you're on vacation, and point out how you're putting money into the country in question?

Particularly when the organizers of the conference have every reason to prefer people at home—financial, lack of cultural barriers, reduced language barriers, and more—and invite me to come speak, anyway?

Note that because the US Border Patrol apparently Googles people when they stop at the border,

This all started of course when I was up-front and honest about the speaking engagement the first time I went through, which flagged me in their system. This became very obvious this past weekend when I attended the Twin Cities Code Camp and was at the border for an hour. On that entry I specified that I was going for a shopping weekend, which I was; I was also planning on going to the Twin Cities Code Camp, a free event and one that I was volunteering at. I didn’t mention that because why confuse the issue trying to explain what a code camp was, that it was free, and why I would consider speaking for free. This was a mistake for two reasons…

For one, they do have internet at CBP offices. So if you’re flagged, and you have to go for secondary interviewing, realize that you may be Googled. And as such, blog posts talking about said code camp or eating a Chipotle Burrito may appear as well (“So how was the burrito?” was a question I was asked).

... and because there's no reason to assume other nations' border patrol won't do the same thing, I'm not going to answer that question. I don't want my views aired on a public forum and in the context of a particular discussion acting as a convenient reason for a bureaucrat to create heartache for the citizens of his country that are expecting me to come and help them be more useful and productive and competitive.

D'Arcy's spot-on right on one point, and I applaud him for saying it:

Canadians have long taken for granted our border with the USA. If there’s one thing this experience has taught me, its that there is an air of entitlement that we’ve had in regards to being able to cross over and do whatever we want in the US. We assume that we’ll be as welcome as we were in the past, and that there really isn’t that much difference between us: we drive the same cars, watch the same television and movies, listen to the same music, read the same books.

That "entitlement" isn't limited to just Canadians—other citizens of other Western nations, including my own, feel that same sense of entitlement. Border control is just a hassle, just another annoying obstacle keeping me from my travel destination, just like airport security and agricultural inspections. (Having lived in Stamford, CT in the 70s when entire forests were being depopulated by some sort of caterpiller/moth infestation, and in LA in the 80s when we had to stay indoors at night as the authorities did overhead spraying of Malathion over our house at night to kill off the fruit fly infestation, I'm really kinda sensitive to the need for those ag inspections.)

But the fact is, you are leaving your country, and the laws you grew up with, and entering a new country, one which owes you nothing.

But we are different. We are separate, independent entities with different history, values, and morals. So to the second reason why that was a mistake: I, as a Canadian, have no right to make a call as to whether I’m of a benefit to a neighbouring country. I can rationalize all I want that the event is free, and that I’m actually trying to help other Americans by sharing my knowledge, but that’s not my call to make.

The US is in a state of protectionism right now whether they admit it or not. When you continue to hear about the vast number of jobs being lost, it makes sense that they want to ensure their people are being protected first and foremost. Many of those people include friends of mine whose companies are laying off people.

(By the way, D'Arcy, you misspelled "neighboring".)

As much as D'Arcy has the right attitude about the ways in which nations get to make decisions for their little plots of land upon the earth, and our ability to argue with them, I still want to point out that the whole economic protectionist argument has been used before, and it's pretty much been debunked at a number of levels. (I'm not going down the path of talking about border security, which is a different issue entirely and not what stops D'Arcy from coming to the US.)

The debate around protectionism has been around as long as people have studied economics as a formal "science", and the end results are pretty clear: everybody benefits when the borders are open and unrestricted. The "multiplier effect" that macroeconomists talk about more than makes up for whatever "drain" a foreigner imposes on the local economy.

Note: For those of you who haven't heard of the multiplier effect, it works like this: while in the US to speak at whatever conference he wants to speak at, D'Arcy spends a dollar at a hotel gift shop, of which the hotel uses $.95 to pay its local worker's hourly wage, of which the worker spends $.90 on a hot dog for lunch, of which the hot dog stand operator uses $.85 to buy buns for tomorrow's customers.... And so on. Why aren't we spending the full dollar each time? Mostly because people will often save some portion of that dollar (unless you're American, because we don't save anything, it seems), and because the government will take some portion of that dollar each time in taxes. What this means, though, is that the US$1 that D'Arcy spent turned into US$4 or US$5 or more towards the total GDP of the country. Econ is a fascinating subject sometimes.

And, of course, ask any three economists a question, and you'll get five different answers. This subject is no different: protectionism has its proponents, too, usually when the local economy is taking a hit... like now. It feels right, protecting those who are "close to home" (and believe me, I'm sympathetic, I've had friends who've pinged me about finding a new job within the last six months), but in the end, everything it does is artificial—in much the same way that unions artificially keep wages high for union workers, and impose some serious constraints on the companies that employ them. (I don't think it's an accident that industries being hammered mercilessly by the soft economy—the auto manufacturers and the airlines—are also ones with large union populations.) Protectionism is almost always a short-term gain, long-term loss kind of operation. The "perennial gale of creative destruction" (from Alan Greenspan's Age of Turbulence) isn't always gentle, but it is necessary.

D'Arcy, in the end, closes his piece with a fond wish:

My hope is that at some point the US and Canada will be able to get back to where our countries were before 9/11. At the same time though, I hope that Canada realizes during this time that it has its own identity; that we are more than just who we border against. Maybe locking down the border will become a good thing after all.

Frankly, my wish would be for Canadians to realize their own identity (and I think Canadians are pretty aware of this in the same way that Americans don't even realize that it's a problem), as well, assuming that's even a problem. What's more, I think that Canadians will find that they don't need the US nearly as much as Americans like to think they do.

But locking down the border is going to affect more than just Canadians—my fear is that this protectionist attitude will in fact deter other really bright people from coming to the US and sharing their knowledge and wisdom, or even just participating in our economy for a while. Assume for just a moment that the million or so H-1B visas currently allocated are suddenly all revoked and their holders must return to their countries of origin—how many rent checks, car payments, utility bills, movie nights, dinners at local restaurants and bank accounts are going to be exiled with them? And this doesn't even begin to touch the potentials for racism that lurk hidden within the system—granting visas and citizenship more easily to "Westerners" (Brits, Germans, Australians, whatever) than "foreigners" (Hispanics, Indians, Chinese).

The fact is, this "locking down the border" won't help us, in the long-term. Whatever benefits we as Americans accrue from keeping our jobs intact will be lost when those barriers finally come down and we find we can't compete on the global scale. The "perennial gale of creative destruction" can't be bought off, it can only be delayed. (Ask the Soviets Russians about their success with the high-protectionist tactic the next time you're in Moscow or St. Petersburg.)

At some point, the borderless Internet is going to come crashing against the bordered "real world", and it's not going to be a pretty fight. And we, those of us who define and shape and act as the primary consumer and producer of the Internet's benefits, are going to find ourselves facing some uncomfortable choices.

In the meantime, however this story ends, I want to be able to say that my country acted in its own defense, but without prejudice, malice, or ignorance. But I'm very worried that I won't be able to say that... and I'm worried what damage we will do to ourselves in the interim.

(Editor's note: It will be fascinating to see how many people call me an ignorant racist based on nothing more than the blog title. You want to disagree with me, that's fine, just do so on a material basis from the body of the post, not just the title.)

The Simple-Talk newsletter is a monthly e-zine that the folks over at Red Gate Software (makers of some pretty cool toys, including their ANTS Profiler, and recent inheritors of the Reflector utility legacy) produce, usually to good effect.

But this month carried with it an interesting editorial piece, which I reproduce in its entirety here:

When the market is slack, nothing succeeds better at tightening it up than promoting serial group-panic within the community. As an example of this, a wave of multi-core panic spread across the Internet about 18 months ago. IT organizations, it was said, urgently had to improve application performance by an order of magnitude in order to cope with rising demand. We wouldn't be able to meet that need because we were at the "end of the road" with regard to step changes in processor power and clock speed. Multi-core technology was the only sure route to improving the speed of applications but, unfortunately, our current "serial" programming techniques, and the limited multithreading capabilities of our programming languages and programmers, left us ill-equipped to exploit it. Multi-core mania gripped the industry.

However, the fever was surprisingly short-lived. Intel's "largest open-source effort ever" to provide a standard tool for writing multi-threaded code, caused little more than a ripple of interest. Various books, rushed out while the temperature soared, advocated the urgent need for new "multi-core-friendly" programming models, involving such things as "software pipelines". Interesting as they undoubtedly are, they sit stolidly on bookshelves, unread.

The truth is that it's simply not a big issue for the majority of people. Writing truly "concurrent" applications in languages such as C# is difficult, as you get very little help from the language. It means getting involved with low-level concurrency primitives, such as lock statements and so on.

Many programmers lack the skills to do this, but more pertinently lack the need. Increasingly, programmers work in a web environment. As long as these web applications are deployed to a load-balanced web farm, then page requests can be handled in parallel so all available cores will be used efficiently without the need for the programmer to be concerned with fine-grained parallelism.

Furthermore, the SQL Server engine behind these web applications is intrinsically "parallel", and can handle and use effectively about as many cores as you care to throw at it. SQL itself is a declarative rather than procedural language, so it is fundamentally concurrent.

A minority of programmers, for example games programmers or those who deal with "embarrassingly parallel" desktop applications such as Photoshop, do need to start working with the current tools and 'low-level' coding techniques that will allow them to exploit multi-core technology. Although currently perceived to be more of "academic" interest, concurrent languages such as Erlang, and concurrency techniques such as "software transactional memory", may yet prove to be significant.

For most programmers and for most web applications, however, the multi-core furore is a storm in a teacup; it's just not relevant. The web and database platforms already cope with concurrency requirements. We are already doing it.

My hope is that this newsletter, sent on April 1st, was intended to be a joke. Having said that, I can’t find any verbage in the email that suggests that it is, in which case, I have to treat it as a legitimate editorial.

And frankly, I think it’s all crap.

It's dangerously ostrichian in nature—it encourages developers to simply bury their heads in the sand and ignore the freight train that's coming their way. Permit me, if you will, a few minutes of your time, that I may be allowed to go through and demonstrate the reasons why I say this.

To begin ...

When the market is slack, nothing succeeds better at tightening it up than promoting serial group-panic within the community. As an example of this, a wave of multi-core panic spread across the Internet about 18 months ago. IT organizations, it was said, urgently had to improve application performance by an order of magnitude in order to cope with rising demand. [...] Multi-core mania gripped the industry.

Point of fact: The “panic” cited here didn’t start about 18 months ago, it started with Herb Sutter’s most excellent (and not only highly recommended but highly required) article, “The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software”, appeared in the pages of Dr. Dobb’s Journal in March of 2005. (Herb’s website notes that “a much briefer version under the title “The Concurrency Revolution” appeared in C/C++ User’s Journal” the previous month.) And the panic itself wasn’t rooted in the idea that we weren’t going to be able to cope with rising demand, but that multi-core CPUs, back then a rarity and reserved only for hardware systems in highly-specialized roles, were in fact becoming commonplace in servers, and worse, as they migrated into desktops, they would quickly a fact of life that every developer would need to face. Herb demonstrated this by pointing out that CPU speeds had taken an interesting change of pace in early 2003:

Around the beginning of 2003, [looking at the website Figure 1 graph] you’ll note a disturbing sharp turn in the previous trend toward ever-faster CPU clock speeds. I’ve added lines to show the limit trends in maximum clock speed; instead of continuing on the previous path, as indicated by the thin dotted line, there is a sharp flattening. It has become harder and harder to exploit higher clock speeds due to not just one but several physical issues, notably heat (too much of it and too hard to dissipate), power consumption (too high), and current leakage problems.

Joe Armstrong, creator of Erlang, noted in a presentation at QCon London 2007 that another of those physical limitations was the speed of light—that for the first time, CPU signal couldn't get from one end of the chip to the other in a single clock cycle.

Quick: What’s the clock speed on the CPU(s) in your current workstation? Are you running at 10GHz? On Intel chips, we reached 2GHz a long time ago (August 2001), and according to CPU trends before 2003, now in early 2005 we should have the first 10GHz Pentium-family chips.

Just to (re-)emphasize the point, here, now, in early 2009, we should be seeing the first 20 or 40 GHz processors, and clearly we’re still plodding along in the 2 – 3 GHz range. The "Quake Rule" (when asked about perf problems, tell your boss you'll need eighteen months to get a 2X improvement, then bury yourselves in a closet for 18 months playing Quake until the next gen of Intel hardware comes out) no longer works.

For the near-term future, meaning for the next few years, the performance gains in new chips will be fueled by three main approaches, only one of which is the same as in the past. The near-term future performance growth drivers are:

• hyperthreading
• multicore
• cache

Hyperthreading is about running two or more threads in parallel inside a single CPU. Hyperthreaded CPUs are already available today, and they do allow some instructions to run in parallel. A limiting factor, however, is that although a hyper-threaded CPU has some extra hardware including extra registers, it still has just one cache, one integer math unit, one FPU, and in general just one each of most basic CPU features. Hyperthreading is sometimes cited as offering a 5% to 15% performance boost for reasonably well-written multi-threaded applications, or even as much as 40% under ideal conditions for carefully written multi-threaded applications. That’s good, but it’s hardly double, and it doesn’t help single-threaded applications.

Multicore is about running two or more actual CPUs on one chip. Some chips, including Sparc and PowerPC, have multicore versions available already. The initial Intel and AMD designs, both due in 2005, vary in their level of integration but are functionally similar. AMD’s seems to have some initial performance design advantages, such as better integration of support functions on the same die, whereas Intel’s initial entry basically just glues together two Xeons on a single die. The performance gains should initially be about the same as having a true dual-CPU system (only the system will be cheaper because the motherboard doesn’t have to have two sockets and associated “glue” chippery), which means something less than double the speed even in the ideal case, and just like today it will boost reasonably well-written multi-threaded applications. Not single-threaded ones.

Finally, on-die cache sizes can be expected to continue to grow, at least in the near term. Of these three areas, only this one will broadly benefit most existing applications. The continuing growth in on-die cache sizes is an incredibly important and highly applicable benefit for many applications, simply because space is speed. Accessing main memory is expensive, and you really don’t want to touch RAM if you can help it. On today’s systems, a cache miss that goes out to main memory often costs 10 to 50 times as much getting the information from the cache; this, incidentally, continues to surprise people because we all think of memory as fast, and it is fast compared to disks and networks, but not compared to on-board cache which runs at faster speeds. If an application’s working set fits into cache, we’re golden, and if it doesn’t, we’re not. That is why increased cache sizes will save some existing applications and breathe life into them for a few more years without requiring significant redesign: As existing applications manipulate more and more data, and as they are incrementally updated to include more code for new features, performance-sensitive operations need to continue to fit into cache. As the Depression-era old-timers will be quick to remind you, “Cache is king.”

Herb’s article was a pretty serious wake-up call to programmers who hadn’t noticed the trend themselves. (Being one of those who hadn’t noticed, I remember reading his piece, looking at that graph, glancing at the open ad from Fry’s Electronics sitting on the dining room table next to me, and saying to myself, “Holy sh*t, he’s right!”.) Does that qualify it as a “mania”? Perhaps if you’re trying to pooh-pooh the concern, sure. But if you’re a developer who’s wondering where you’re going to get the processing power to address the ever-expanding list of features your users want, something Herb points out as a basic fact of life in the software development world ...

There’s an interesting phenomenon that’s known as “Andy giveth, and Bill taketh away.” No matter how fast processors get, software consistently finds new ways to eat up the extra speed. Make a CPU ten times as fast, and software will usually find ten times as much to do (or, in some cases, will feel at liberty to do it ten times less efficiently).

...  then eking out the best performance from an application is going to remain at the top of the priority list. Users are classic consumers: they will always want more and more for the same money as before. Ignore this truth of software (actually, of basic microeconomics) at your peril.

To get back to the editorial, we next come to ...

However, the fever was surprisingly short-lived. Intel's "largest open-source effort ever" to provide a standard tool for writing multi-threaded code, caused little more than a ripple of interest. Various books, rushed out while the temperature soared, advocated the urgent need for new "multi-core-friendly" programming models, involving such things as "software pipelines". Interesting as they undoubtedly are, they sit stolidly on bookshelves, unread.

Wow. Talk about your pretty aggressive accusation without any supporting evidence or citation whatsoever.

Intel's not big into the open-source space, so it doesn't take much for an open-source project from them to be their "largest open-source effort ever". (What, they're going to open-source the schematics for the Intel chipline? Who could read them even if they did? Who would offer up a patch? What good would it do?) The fact that Intel made the software available in the first place meant that they knew the hurdle that had yet to be overcome, and wanted to aid developers in overcoming it. They're members of the OpenMP group for the same reason.

Rogue Wave's software pipelines programming model is another case where real benefits have accrued, backed by case studies. (Disclaimer: I know this because I ghost-wrote an article for them on their Software Pipelines implementation.) Let's not knock something that's actually delivered value. Pipelines aren't going to be the solution to every problem, granted, but they're a useful way of structuring a design, one that's curiously similar to what I see in functional programming languages.

But simply defending Intel's generosity or the validity of an alternative programming model doesn't support the idea that concurrency is still a hot topic. No, for that, I need real evidence, something with actual concrete numbers and verifiable fact to it.

Thus, I point to Brian Goetz’s Java Concurrency in Practice, one of those “books, rushed out while the temperature soared”, which also turned out to be the best-selling book at Java One 2007, and the second-best-selling book (behind only Joshua Bloch’s unbelievably good Effective Java (2nd Ed) ) at Java One 2008. Clearly, yes, bestselling concurrency books are just a myth, alongside the magical device that will receive messages from all over the world and play them into your brain (by way of your ears) on demand, or the magical silver bird that can wing its way through the air with no visible means of support as it does so. Myths, clearly, all of them.

To continue...

The truth is that it's simply not a big issue for the majority of people. Writing truly "concurrent" applications in languages such as C# is difficult, as you get very little help from the language. It means getting involved with low-level concurrency primitives, such as lock statements and so on.

Many programmers lack the skills to do this, but more pertinently lack the need. Increasingly, programmers work in a web environment. As long as these web applications are deployed to a load-balanced web farm, then page requests can be handled in parallel so all available cores will be used efficiently without the need for the programmer to be concerned with fine-grained parallelism.

He’s right when he says you get very little help from the language, be it C# or Java or C++. And getting involved with low-level concurrency primitives is clearly not in anybody’s best interests, particularly if you’re not a concurrency guru like Brian. (And let’s be honest, even low-level concurrency gurus like Brian, or Joe Duffy, who wrote Concurrent Programming on Windows, or Mike Woodring, who co-authored Win32 Multithreaded Programming, have better things to do.) But to say that they “pertinently lack the need” is a rather impertinent statement. “As long as these web applications are deployed to a load-balanced web farm", which is very likely to continue to happen, “then page requests can be handled in parallel so all available cores will be used …”

Um... excuse me?

Didn’t you just say that programmers didn’t need to learn concurrency constructs? It would strike me that if their page requests are being handled in parallel that they have to learn how to write code that won’t break when it’s accessed in parallel or lead to data-corruption problems or race conditions when their pages are accessed in parallel. If parallelism is a fundamental part of the Web, don’t you think it’s important for them to learn how to write programs that can behave correctly in parallel?

Look for just a moment at the average web application: if data is stored in a per-user collection, and two simultaneous requests come in from a given user (perhaps because the page has AJAX requests being generated by the user on the page, or perhaps because there’s a frameset that’s generating requests for each sub-frame, or ...), what happens if the code is written to read a value from the session, increment it, and store it back? ASP.NET can save you here, a little, in that it used to establish a per-user lock on the entirety of the page request (I don’t know if it still does this—I really have lost any desire to build web apps ever again), but that essentially puts an artificial throttle on the scalability of your system, and makes the end-users’ experience that much slower. Load-balancer going to spray the request all over the farm? So long as the user session state is stored on every machine in the farm, that’ll work... But of course if you store the user’s state in the SQL instance behind each of those machines on the farm, then you take the performance hit of an extra network round-trip (at which point we’re back to concurrency in the database) ...

... all because the programmer couldn’t figure out how to make “lock” work? This is progress?

The Java Servlet specification specifically backed away from this "lock on every request" approach because of the performance implications. I heard a fair amount of wailing and gnashing during the early ASP.NET days over this. I heard the ASP.NET dev team say they made their decision because the average developer can't figure out concurrency correctly anyway.

And, by the way folks, this editorial completely ignores XML services. I guess "real" applications don't write services much, either.

The next part is even better:

Furthermore, the SQL Server engine behind these web applications is intrinsically "parallel", and can handle and use effectively about as many cores as you care to throw at it. SQL itself is a declarative rather than procedural language, so it is fundamentally concurrent.

True… and false. SQL is fundamentally “parallel” (largely because SQL is a non-strict functional language, not just a “declarative” one), but T-SQL isn’t. And how many developers actually know where the line is drawn between SQL and T-SQL? More importantly, though, how many effective applications can be written with a complete ignorance of the underlying locking model? Why do DBAs spend hours tuning the database’s physical constructs, establishing where isolation levels can be turned down, establishing where the scope of a transaction is too large, putting in indexed columns where necessary, and figuring out where page, row, or table locking will be most efficient? Because despite the view that a relational database presents, these queries are being executed in parallel, and if a developer wants to avoid writing an application that requires a new server for each and every new user added to the system, they need to learn how to maximize their use of the database’s parallelism. So even if the language is "fundamentally concurrent" and can thus be relied upon to do the right thing on behalf of the developer, the implementation isn't, and needs to be understood in order to be implemented efficiently.

He finishes:

For most programmers and for most web applications, however, the multi-core furore is a storm in a teacup; it's just not relevant. The web and database platforms already cope with concurrency requirements. We are already doing it.

This is one of those times I wish I had a time machine handy—I'd love to step forward five years, have a look around, then come back and report the findings. I'm tempted to close with the challenge to just let’s come back in five years and see what the programming language landscape and hardware landscape looks like. But that's too easy an "out", and frankly, doesn't do much to really instill confidence, in my opinion.

To ignore the developers building "rich" applications (be they being done in Flex/Flash, Cocoa/iPhone, WinForms, Swing, WPF, or what-have-you) is to also ignore a relatively large segment of the market. Not every application is being built on the web and is backed by a relational database—to simply brush those off and not even consider them as part of the editorial reveals a dangerous bias on the editor's part. And those applications aren't hosted in an "intrinsically 'parallel'" container that developers can just bury their head inside.

Like it or not, folks, the path forward isn't one that you get to choose. Intel, AMD, and other chip manufacturers have already made that clear. They're not going to abandon the multicore approach now, not when doing so would mean trying to wrestle with so many problems (including trying to change the speed of light) that simply aren't there when using a multicore foundation. That isn't up for debate anymore. Multicore has won for the forseeable future. And, as a result, multicore is going to be a fact of the developer's life for the forseeable future. Concurrency is thus also a fact of the developer's life for the forseeable future.

The web and database platforms “cope” with concurrency requirements by either making "one-size-fits-all" decisions that almost always end up being the wrong decision for high-scale systems (but I'm sure your new startup-based idea, like a system that allows people to push "micro-entries" of no more than 140 characters in length to a publicly-trackable feed would never actually take off and start carrying millions and millions of messages every day, right?), or by punting entirely and forcing developers to dig deeper beneath the covers to see the concurrency there. So if you're happy with your applications running no faster than 2GHz for the rest of the forseeable future, then sure, you don't need to worry about learning concurrency-friendly kinds of programming techniques. Bear in mind, by the way, that this essentially locks you in to small-scale, web-plus-database systems for the forseeable future, and clearly nothing with any sort of CPU intensiveness to it whatsoever. Be happy in your niche, and wave to the other COBOL programmers who made the same decision.

This is a leaky abstraction, full stop, end of story. Anyone who tells you otherwise is either trolling for hits, trying to sell you something, or striving to persuade developers that ignorance isn't such a bad place to be.

All you ignorant developers, this is the phrase you will be forced to learn before you start your next job: "Would you like fries with that?"

While playing around with a recent research-oriented project for myself (more on that later), I discovered something that I haven't seen mentioned anywhere in the Scala universe before. (OK, not really--as you'll see towards the end of this piece, it really is documented, but allow me my brief delusions of grandeur as I write this. They'll get deflated quickly enough.)

So the core of the thing was a stack-oriented execution engine; essentially I'm processing commands delivered in a postfix manner. Since some of these commands are relational operators, it's important that there be two things to relationally operate on the execution stack, after which I want to evaluate the relational operation and push its result (1 if true, 0 if false) back on the stack; this is pretty easily done via the following:

def compareOp(op : (Int, Int) => Boolean) =
{
  checkStack(2)
  val v1 = (execStack.pop()).asInstanceOf[Int]
  val v2 = (execStack.pop()).asInstanceOf[Int]
  val vr = op(v1, v2)
  execStack.push(if (vr) 1 else 0)
} 

where "execStack" is a mutable.Stack[Any] held in an enclosing function.

Interestingly enough, however, when I wrote this the first time, I wrote it like this, which is a very different sequence of operations:

def compareOp(op : (Int, Int) => Boolean) =
{
    checkStack(2)
    def v1 = (execStack.pop()).asInstanceOf[Int]
    def v2 = (execStack.pop()).asInstanceOf[Int]
    def vr = op(v1, v2)
    execStack.push(if (vr) 1 else 0)
}

See the difference? Subtle, is it not? But the actual code is significantly different, something that's more easily seen with a much simpler (and standalone) example:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        def v1 = (stack.pop()).asInstanceOf[Int]
        def v2 = (stack.pop()).asInstanceOf[Int]
        def vr = v1 + v2
        System.out.println(vr)
    }
} 

When run, the console prints out "36", as we'd well expect.

But suppose we want to look at those values of v1 and v2 along the way, perhaps as part of a logging operation, or perhaps because you're just screwing around with some ideas in your head and you don't want to bother to fire up an IDE with Scala support in it. So you decide to spit those values to a console:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        def v1 = (stack.pop()).asInstanceOf[Int]
        def v2 = (stack.pop()).asInstanceOf[Int]
        System.out.println(v1)
        System.out.println(v2)
        def vr = v1 + v2
        System.out.println(vr)
    }
} 

And then something *very* different happens; you get "24", "12", and then a NoSuchElementException.

If you're like me the first time I ran into this, your first reaction is, "Eh?". Actually, if you're like me, when you're programming, your profanity filters are probaby at an ebb, so your first reaction is "WTF?!?", said with great gusto and emphasis. Which has a tendency to get some strange looks when you're at a Denny's doing your research, I will admit. Particularly when it's at 3 AM in the morning. And the bar crowd is in full alcoholic haze and slightly nervous about the long-haired, goatee-sporting guy in his headphones, wearing his black leather jacket and swearing like a drunken sailor at his laptop. But I digress.

What is Scala doing here?

Turns out this is exactly as the language designers intended, but it's subtle. (Or maybe it's just subtle to me at 3AM when I'm pumped full of caffeine.)

Let's take this a different way:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        def v1 = (stack.pop()).asInstanceOf[Int]
        def v2 = (stack.pop()).asInstanceOf[Int]
        System.out.println(stack)
    }
} 

When run, the console prints "Stack(12, 24)", which *really* starts to play with your mind when you're a little short on sleep and a little high on Diet Coke. At first glance, it looks like Scala is broken somehow--after all, those "pop" operations are supposed to modify the Stack against which they're operating, just as the push()es do. So why is the stack convinced that it still holds the values of 12 and 24?

Because Scala hasn't actually executed those pop()s yet.

The "def" keyword, it turns out, isn't what I wanted here--what I wanted (and in retrospect it’s painfully obvious) was a "val", instead, in order to force the execution of those statements and capture the value into a local value (an immutable local variable). The "def" keyword, instead, creates a function binding that waits for formal execution before evaluating. So that when I previously said

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        def v1 = (stack.pop()).asInstanceOf[Int]
        def v2 = (stack.pop()).asInstanceOf[Int]
        def vr = v1 + v2
        System.out.println(vr)
    }
} 

… what in fact I was saying was this:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        def v1 = (stack.pop()).asInstanceOf[Int]
        def v2 = (stack.pop()).asInstanceOf[Int]
        System.out.println(v1 + v2)
    }
} 

… which is the same as:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        System.out.println((stack.pop()).asInstanceOf[Int] + (stack.pop()).asInstanceOf[Int])
    }
} 

… which, when we look back at my most recent "debugging" version of the code, substituting the "def"ed versions of v1 and v2 (and vr) where they're used, makes the reason for the NoSuchElementException become entirely more clear:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        System.out.println((stack.pop()).asInstanceOf[Int])
        System.out.println((stack.pop()).asInstanceOf[Int])
        System.out.println((stack.pop()).asInstanceOf[Int] + (stack.pop()).asInstanceOf[Int])
    }
} 

Now, normally, this would probably set off all kinds of alarm bells in your head, but the reaction that went off in mine was "COOL!", the reasons for which revolve around the concept of "laziness"; in a functional language, we frequently don't want to evaluate the results right away, instead preferring to defer their execution until actually requiring it. In fact, many functional languages—such as Haskell—take laziness to new heights, baking it directly into the language definition and assuming laziness everywhere, so much so that you have to take special steps to avoid it. There’s a variety of reasons why this is advantageous, but I’ll leave those discussions to the Haskellians of the world, like Matt Podwysocki and Simon Peyton-Jones.

From a Scalist’s perspective, laziness is still a useful tool to have in your toolbox. Suppose you have a really powerful function that calculates PI to a ridiculous number of decimal places. In Java, you might be tempted to do something like this:

class MyMath
{
  public static final double PI = calculatePiToARidiculousNumberOfPlaces();
  private static double calculatePiToARidiculousNumberOfPlaces()
  {
    // implementation left to the reader's imagination
    // imagine it being "really cool"
  }
} 

The problem with this is that if that method takes any length of time to execute, it's being done during class initialization during its ClassLoading phase, and aside from introducing a window of time where the class *could* be used before that initialization is finished (it's subtle, it's not going to happen very often, but it can, according to older versions of the JVM Spec), the problem is that the time required to do that initialization is paid for *regardless of whether you use PI*. In other words, the classic Stroustrup-ian "Don't pay for it if you don't use it" principle is being completely tossed aside.

In Scala, using the "def" keyword here, aside from avoiding the need for the additional decorators, completely eliminates this cost--people won't need the value of PI until it becomes used:

object App
{
    def PI = calculatePiToARidiculousNumberOfPlaces()
    def calculatePiToARidiculousNumberOfPlaces() =
    {
        System.out.println("Calculating PI")
        3 + 0.14
    }
    def main(args : Array[String]) =
    {
        System.out.println("Entering main")
        System.out.println("PI = " + PI)
    }
} 

(In fact, you'd probably just write it without the calculating method definition, since it's easier that way, but bear with me.)

When you run this, of course, we see PI being calculated after main()'s been entered, thus proving that PI is being calculated only on demand, not ahead of time, as a public-static-final-constant would be.

The problem with this approach is, you end up calculating PI on each access:

object App
{
    def PI = calculatePiToARidiculousNumberOfPlaces()
    def calculatePiToARidiculousNumberOfPlaces() =
    {
        System.out.println("Calculating PI")
        3 + 0.14
    }
    def main(args : Array[String]) =
    {
        System.out.println("Entering main")
        System.out.println("PI = " + PI)
        System.out.println("PI = " + PI)
            // prints twice! Not good!
    }
} 

Which sort of defeats the advantage of lazy evaluation.

This got me wondering--in F#, we have lazy as a baked-in concept (sort of), such that when I write

#light
let sixty = lazy (30 + 30)
System.Console.WriteLine(sixty) 

What I see on the console is not 60, but a Lazy<T> type instance, which effectively defers execution until it's Force() method is invoked (among other scenarios). This means I can write things like

let reallyBigList = lazy ([1..1000000000000] |> complexCalculation |> anotherComplexCalcuation)

without fear of blowing the stack or heap apart, since laziness means the list won't actually be calculated until it's forced; we can see this from the following (from the F# interactive console):

> let sixtyWithSideEffect = lazy (printfn "Hello world"; 30+30);;
val sixtyWithSideEffect: Lazy<int>
> sixtyWithSideEffect.Force();;
Hello world
val it : int = 60
> sixtyWithSideEffect.Force();;
val it : int = 60 

(Examples taken from the excellent Expert F# by Syme/Granicz/Cisternino; highly recommended, if a touch out-of-date to the current language definition. I expect Chris Smith’s Programming F#, from O’Reilly, to correct that before too long.)

It would be nice if something similar were doable in Scala. Of course, once I start looking for it, it makes itself visible, in the wonderful Venners/Odersky/Spoon book, Programming In Scala, p. 444:

You can use pre-initialized fields to simulate precisely the initialization behavior
of class constructor arguments. Sometimes, however, you might prefer
to let the system itself sort out how things should be initialized. This can
be achieved by making your val definitions lazy. If you prefix a val definition
with a lazy modifier, the initializing expression on the right-hand side
will only be evaluated the first time the val is used.

[...]

This is similar to the situation where x is defined as a parameterless
method, using a def. However, unlike a def a lazy val is never evaluated
more than once. In fact, after the first evaluation of a lazy val the result of the
evaluation is stored, to be reused when the same val is used subsequently.

Perfect! The key, then, is to define PI like so:

object App
{
    lazy val PI = calculatePiToARidiculousNumberOfPlaces()
    def calculatePiToARidiculousNumberOfPlaces() =
    {
        System.out.println("Calculating PI")
        3 + 0.14
    }
    def main(args : Array[String]) =
    {
        System.out.println("Entering main")
        System.out.println("PI = " + PI)
        System.out.println("PI = " + PI)
            // prints once! Awesome!
    }
} 

That means, if I apply it to my Stack example from before, I should get the same deferred-execution properties of the "def"-based version ...

def main(args : Array[String]) =
{
    import scala.collection.mutable.Stack
    var stack : Stack[Any] = new Stack()
    stack.push(12)
    stack.push(24)
    lazy val v1 = (stack.pop()).asInstanceOf[Int]
    lazy val v2 = (stack.pop()).asInstanceOf[Int]
    System.out.println(stack)
        // prints out "Stack(12,24)
} 

... but if I go back to the version that blows up because the stack is empty, using lazy val works exactly the way I would want it to:

def main(args : Array[String]) =
{
    import scala.collection.mutable.Stack
    var stack : Stack[Any] = new Stack()
    stack.push(12)
    stack.push(24)
    lazy val v1 = (stack.pop()).asInstanceOf[Int]
    lazy val v2 = (stack.pop()).asInstanceOf[Int]
    System.out.println(v1)
    System.out.println(v2)
    lazy val vr = v1 + v2
    System.out.println(vr)
        // prints 12, 24, then 36
        // and no exception!
} 

Nice.

So, it turns out that my accidental use of "def" inside the compareOp function behaves exactly the way the language designers wanted it to, which is not surprising, and that Scala provides nifty abilities to defer processing or extraction of values until called for.

Curiously, the two languages differ in how laziness is implemented; in F#, the lazy modifier defines the type to be a Lazy<T> instance, an ordinary type that we can pass around from F# to C# and back again as necessary (in much the same way that C# defined nullable types to be instances of Nullable<T> under the hood). We can see that from the interactive console output above, and from the fact that we call Force() on the instance to evaluate its value.

In Scala, however, there is no corresponding Lazy[T] type; instead, the PI() method is defined to determine whether or not the value has already been evaluated:

public double PI();
  Code:
   0:   aload_0
   1:   getfield        #135; //Field bitmap$0:I
   4:   iconst_1
   5:   iand
   6:   iconst_0
   7:   if_icmpne       48
   10:  aload_0
   11:  dup
   12:  astore_1
   13:  monitorenter
   14:  aload_0
   15:  getfield        #135; //Field bitmap$0:I
   18:  iconst_1
   19:  iand
   20:  iconst_0
   21:  if_icmpne       42
   24:  aload_0
   25:  aload_0
   26:  invokevirtual   #137; //Method calculatePiToARidiculousNumberOfPlaces:()D
   29:  putfield        #139; //Field PI:D
   32:  aload_0
   33:  aload_0
   34:  getfield        #135; //Field bitmap$0:I
   37:  iconst_1
   38:  ior
   39:  putfield        #135; //Field bitmap$0:I
   42:  getstatic       #145; //Field scala/runtime/BoxedUnit.UNIT:Lscala/runtime/BoxedUnit;
   45:  pop
   46:  aload_1
   47:  monitorexit
   48:  aload_0
   49:  getfield        #139; //Field PI:D
   52:  dreturn
   53:  aload_1
   54:  monitorexit
   55:  athrow
  Exception table:
   from   to  target type
    14    48    53   any

If you look carefully at the bytecode, the implementation of PI is checking a bitmask field (!) to determine if the first bit is flipped (!) to know whether or not the value is held in the local field PI, and if not, calculate it and store it there. This means that Java developers will just need to call PI() over and over again, rather than have to know that the instance is actually a Lazy[T] on which they need to call Value or Force (such as one would from C# in the F# case). Frankly, I don’t know at this point which approach I prefer, but I’m slightly leaning towards the Scala version for now. (If only Java supported properties, then the syntax “MyMath.PI” would look like a constant, act lazily, and everything would be great.)

(It strikes me that the F# developer looking to write something C#-accessible need only tuck the Lazy<T> instance behind a property accessor and the problem goes away, by the way; it would just be nicer to not have to do anything special on either side, to have my laziness and Force() it, too. Pipe dream, perhaps.)

In retrospect, I could wish that Scala weren't *quite* so subtle in its treatment of "def" vs "val", but now that I'm aware of it, it'll (hopefully) not bite me quite so subtly in the sensitive spots of my anatomy again.

And any experience in which you learn something is a good one, right?

An interesting blog post was forwarded to me by another of my fellow ThoughtWorkers, which suggests a new software stack for building an enterprise system, acronymized as “JOSH”:

The Book Of JOSH

Through a marvelous, even devious, set of circumstances, I'm presented with the opportunity to address my little problem without proscribed constraints, a true green field opportunity.

Json OSGi Scala HTTP

Json delivers on what XML promised. Simple to understand, effective data markup accessible and usable by human and computer alike. Serialization/Deserialization is on par with or faster then XML, Thrift and Protocol Buffers. Sure I'm losing XSD Schema type checking, SOAP and WS-* standardization. I'm taking that trade.

OSGi a standardized dynamic, modular framework for versioned components and services. Pick a logger component, a HTTP server component, a ??? component, add your own internal components and you have a dedicated application solution. Micro deployment with true replacement. What am I giving up? The monolithic J2EE application servlet loaded with 25 frameworks, SCA and XML configuration hell. Taking the trade.

HTTP is simple, effective, fast enough, and widely supported. I'm tired of needlessly complex and endless proprietary protocols to move simple data from A to B with all the accompanying firewall port insanity. Yes, HTTP is not perfect. But I'm taking this trade where I can as well.

All interfaces will be simple REST inspired APIs based on HTTP+JSON. This is an immediate consequence of the JOSH stack.

Scala is by far the toughest, yet the easiest selection in the JOSH stack. I wrestled far more with the JSON or XML or Thrift or Protocol Buffers decision.

And, let’s be honest, the stack sounds a lot better than what he was working with before....

[...] Yes, you see, I have a small problem.

So whats the issue, you say? I write a whole blog about nothing, you say? We all know the right answer, you're pointing out? Yea, I know, its intuitively obvious to the casual observer.

We'll rewrite it from scratch.

Course we'll need a cluster of WebSphere Application Servers, and an Oracle RAC cluster for all that data. Don't forget the middleware needed to transition over from the legacy systems, so toss in an ESB cluster, and what heck a couple of BPEL servers too.

Need a SOA Center of Excellence of course too. Can't integrate without some common XML Business Object Schemas. Also need to roll the Rational RUP suite and some beefy IDE environments and for that shiny look, sprinkle the works with lots of WS-* sparkly dust. Bake 3-5 years or until done, whenever.

My presentation slides for all this will be killer. I can sell this stuff. I'm good at it. I'll look like a bloody genius. I'll have Vendors fawning all over me. And the best part is the bubble on this mess won't pop for YEARS, when I'll have plenty of plausible deniability. "Hey the plan was perfect, the business, IT managers and their people were incapable of executing it."

I feel like the enterprise IT equivalent of an AIG trader pocketing ill gotten gains from writing Credit Default Swaps that we can't pay off.

Ewww... even thinking about all that makes me want to go upstairs, step into the shower, turn the water as hot as it will go, and wash. Scrub my skin raw with soap and sponge until the top five layers of epidermis are gone, and still not feel clean.

On the surface of things, the stack sounds pretty good. OSGi is a pretty solid spec for managing versioning and modularity within a running Java system, and more importantly, it’s well-known, relatively reliable, and pretty well-proven to handle the classic problems well. And of course, anybody who knows me knows that I’m a fan of the Scala language as a potential complement or supplement to the Java programming language, so that’s hardly in debate.

But there are a few concerns. JSON is a simple wire protocol, granted, but that is both a good thing and a bad thing (it’s object-centric, for one, and will run into some of the same issues as objects do with certain relationships), and it lacks the ubiquity that XML provides. Granted, XML clearly suffered from an overabundance of adoption, but it still doesn’t take away the fact that ubiquity is really necessary if you’re building a baseline for something that will talk to a variety of different systems. Which, I admit, may not be in his list of requirements, I don’t know. And HTTP is great for long-haul, client-initiated communication, but it definitely has its limitations (which he acknowledges, openly, to his credit), at least to internal-facing consumers. There is no peer for external-facing consumers, that’s a given.

And the stack is clearly also missing something else...

The JOSH stack is lacking a letter, because a solution for persisted data is missing in the stack.

A great deal of what needs to be done does not require a ACID RDB cluster. Some of it does and I'm kicking that can down the road.

For the rest, either the data is ReadOnly and loaded a 1-3 times a day or is best persisted by a distributed Key-Value storage system. A number of these are now available as open source solutions and at the right moment I'll need to pick one and add that letter to the JOSH stack.

As a commenter suggested, CouchDB might be a solution here, or I’ll even throw db4o into the ring for discussion as an option. Again, it’ll depend on how far-and-wide the data will be seen by other systems—the more other systems need to see it, the less further away from a “regular” RDBMS we can go.

Certainly, it’s a great start for discussion, even if the acronym is likely to give those named Joshua an unhealthy ego boost.

Part of me wonders, though... what would the equivalent on .NET look like? JSON + Assemblies + F# + HTTP = JAFH?

This crossed my Inbox:

I read your article entitled: The Polyglot Programmer. How about the thought that rather than becoming a polyglot-software engineer; pick a polyglot-language. For example, C# is borrowing techniques from functional and dynamic languages. Let the compiler designer worry about mixing features and software engineers worry about keep up with the mixture. Is this a good approach? [From Phil, at http://greensoftwareengineer.spaces.live.com/]

Phil, it’s an interesting thought you’ve raised—which is the better/easier approach to take, that of incorporating the language features we want into a single language, rather than needing to learn all those different languages (and their own unique syntaxes) in order to take advantage of those features we want?

After all, we’re starting to see this taking place within a certain number of languages already, particularly C#; first, in 3.0, they introduced a number of features in support of LINQ that make C# a useful starting point for working with a functional language. Extension methods, for example, allow us to add a number of different methods to the collection classes that provide some functional capabilities (Select<>, GroupBy<>, and so on), as Matt Podwysocki demonstrates, generics contribute the type-safety that most functional languages embrace, anonymous methods and delegates provide better functions-as-first-class-constructs (including lambdas), and anonymous types make it vastly easier to return and pass tuples. And now, in 4.0, we’re getting the “dynamic” keyword, which will add support for invoking methods and properties dynamically, in the grand tradition of most dynamic languages (like Python and Ruby), and 3.0’s local variable type inference allows us to write “var x = ...”, which feels pretty dynamic (even if it’s not, under the hood).

Unfortunately, I think for the most part, the answer’s going to be, “Yes, it would be nice, if it weren’t for the fact that there are very few languages that won’t collapse underneath their own weight if they did so.”

Consider, for example, the C# language. Already, with the C# 3.0 definition, the language specification weighs in at close to a thousand pages. The additional features in 4.0 could easily push it over a thousand and possibly, with all the places where “dynamic” behavior will need to be factored into the existing specification, could push that well into the 1200 to 1300 page range. What’s the upper limit on a language’s complexity to maintain and enhance, much less for its programmers to comprehend?

(By comparison, the C++ specification, as I can best remember, didn’t weigh in at more than a thousand pages, but given that the current working draft is under password protection, and I can’t find the prior spec as a freely-available download, I can’t see if memory is correct or not.)

Or, consider the various edge cases that came up around the introduction of nullable types in C# 2.0. What started out as a fairly simple suggestion—“let’s let T? represent the idea that this instance of T could be nullable, and at runtime it’ll be a Nullable<T> instance behind the scenes”—turned into a pretty ugly morass of edge cases at the language level that resulted in some serious bug-fixing right up until the final ship date.

Thing is, languages that aren’t written deliberately to allow their own modification and evolution tend to fail over time. C++ was one such example, and I think both Java and C# will stand as successor examples before long.

Right now, in C# 3.0, type inference is limited entirely to local variables because the language isn’t syntactically set up to leave out type names wherever possible—the “var” token is a type placeholder, largely because the parser has to have a type first. (This is the same purpose the “dynamic” keyword seems to be playing for 4.0, though I can’t say so for certain.) In F# and Scala, this syntax is deliberately written Pascal-style, with the name first, optionally followed by a colon and the type, because the parser can see the colon and realize the type is already specified, or see no colon and realize the type should be inferred. That syntax is used consistently throughout the F# and Scala languages, and that means it’s pretty easy, lexically speaking, for the languages to recognize when type inference should kick in.

What’s more, both F# and Scala don’t really support the O-O notion of method overloading, because again, it gets confusing when trying to kick in type inference—something about too many possibilities confusing the type-inferencer. (I’m not entirely positive of this point, by the way, it’s based on some conversations I’ve had with language designers over the last few years. I could be wrong, and would love to see a language that supports both.) Instead, they force developers to be more explicit about parameters being passed—F# won’t even do implicit widening conversions, in fact, such as automatically widening ints to longs.

But both F# and Scala have a very interesting facility to allow definitions of methods/functions using very flexible syntactic rules, such that they look like operators or keywords built into the language; F# defines its pipeline operator ( |> ) in its library definitions, for example. Scala defines numerous “keywords”, like synchronized or transient, as classes in the Scala package extending “StaticAnnotation”—in other words, their syntax and behavior is defined as an annotation, rather than as a built-in part of the language. Ditto for Scala’s XML support.

Lisp, of course, was one of the first (if not the first) language to do this, and it’s my understanding that this has been one of the principal reasons it has survived all these years as a language—because it’s an abstraction built on top of an abstraction built on top of an abstraction, et al, it makes it easier to change those underlying abstractions when the context changes.

This doesn’t mean those “polyactivist” languages like C# are bad things, it just means that there’s a danger that they’ll eventually collapse from too many moving parts all trying to talk to each other at the same time. As an exercise, open the C# 3.0 spec, and start checking off all the sections that will need to be touched by the introduction of the “dynamic” keyword as a new type.

Or, to put it analagously, yes, for a lot of work, a single multifunction tool can be useful, but for a lot of other work, you want tools that are specialized to the task at hand. Let’s not minimize the usefulness of that multifunction tool, but let’s not try to use a Swiss Army knife where a jeweler’s screwdriver is really needed.

This email crossed my Inbox last week while I was on the road:

Due to the current economic situation, TechWeb has made the difficult decision to discontinue the Software Development events, including SD West, SD Best Practices and Architecture & Design World. We are grateful for your support during SD's twenty-four year history and are disappointed to see the events end.

This really bums me out, because the SD shows were some of the best shows I’ve been to, particularly SD West, which always had a great cross-cutting collection of experts from all across the industry’s big technical areas: C++, Java, .NET, security, agile, and more. It was also where I got to meet and interview Bjarne Stroustrup, a personal hero of mine from back in my days as a C++ developer, where I got to hang out each year with Scott Meyers, another personal hero (and now a good friend) as well as editor on Effective Enterprise Java, and Mike Cohn, another good friend as well as a great guy to work for. It was where I first met Gary McGraw, in a rather embarrassing fashion—in the middle of his presentation on security, my cell phone went off with a klaxon alarm ring tone loud enough to be heard throughout the entire room, and as every head turned to look at me, he commented dryly, “That’s the buffer overrun alarm—somewhere in the world, a buffer overrun attack is taking place.”

On a positive note, however, the email goes on to say that “Cloud Connect [will] take over SD West's dates in March 2010 at the Santa Clara Convention Center”, which is good news, since it means (hopefully) that I’ll still get a chance to make my yearly pilgrimage to In-N-Out....

Rest in peace, SD. You will be missed.

Interesting little tidbit crossed my Inbox today...

Only 8% members of the Scientific Research Society agreed that "peer review works well as it is". (Chubin and Hackett, 1990; p.192).

"A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research." (Horrobin, 2001)

Horrobin concludes that peer review "is a non-validated charade whose processes generate results little better than does chance." (Horrobin, 2001). This has been statistically proven and reported by an increasing number of journal editors.

But, "Peer Review is one of the sacred pillars of the scientific edifice" (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and "Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process" (Horrobin, 2001).

...

Chubin, D. R. and Hackett E. J., 1990, Peerless Science, Peer Review and U.S. Science Policy; New York, State University of New York Press.

Horrobin, D., 2001, "Something Rotten at the Core of Science?" Trends in Pharmacological Sciences, Vol. 22, No. 2, February 2001. Also at http://www.whale.to/vaccine/sci.html and http://post.queensu.ca/~forsdyke/peerrev4.htm (both pages were accessed on February 1, 2009)

Goodstein, D., 2000, "How Science Works", U.S. Federal Judiciary Reference Manual on Evidence, pp. 66-72 (referenced in Hoorobin, 2000)

I know that we don't generally cite the scientific process as part of the rationale for justifying code reviews, but it seems to have a distinct relationship. If the peer review process is similar in concept to the code review process, and the scientific types are starting to doubt the efficacy of peer reviews, what does that say about the code review?

(Note: I'm not a scientist, so my familiarity with peer review is third-hand at best; I'm wide open to education here. How are the code review and peer review processes different, if in fact, they are different?)

The Horrobin "sacred pillars" quote, in particular, makes me curious: Don't we already apply "scientific [and engineering] methods" to the peer review process? And can we honestly say that we in the software industry apply "scientific [and engineering]" methods to the code review process? Can we iterate the list? Or do we just trust that intuition and "more eyeballs" will help spot any obvious defects?

The implications here, when tied up next to the open source fundamental principle that states that "more eyeballs is better", are interesting to consider. If review is not a scientifically-proven or "engineeringly-sound" principle, then the open source folks are kidding themselves in thinking they're more secure or better-engineered. If we conduct a scientific measurement of code-reviewed code and find that it is "a non-validated charade whose processes generate results little better than does chance", we've at least conducted the study, and can start thinking about ways to make it better. (I do wish the email author had cited sources that provide the background to the statement, "This has been statistically proven", though.)

I know this is going to seem like a trolling post, but I'm genuinely curious--do we, in the software industry, have any scientifically-conducted studies with quantifiable metrics that imply that code-reviewed code is better than non-reviewed code? Or are we just taking it as another article of faith?

(For those who are curious, the email that triggered all this was an invitation to a conference on peer review.

This is the purpose of the International Symposium on Peer Reviewing: ISPR (http://www.ICTconfer.org/ispr) being organized in the context of The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009 (http://www.ICTconfer.org/kgcm), which will be held on July 10-13, 2009, in Orlando, Florida, USA.

I doubt it has any direct relevance to software, but I could be wrong. If you go, let me know of your adventures and conclusions. )

Just got this email from Chris Sells:

For twelve 45-minute slots at this year’s DSL DevCon (April 16-17 in Redmond, WA), we had 49 proposals. You have been selected as speakers for the following talks. Please confirm that you’ll be there for both days so that I can put together the schedule and post it on the conference site. This DevCon should rock. Thanks!

Martin Fowler - Keynote

Paul Vick + Gio - Mgrammar Deep Dive

Tom Rodgers - Domain Specific Languages for automated testing of equity order management systems and trading machines

Paul Cowan - DSLs in the Horn Package Manager

Guillaume Laforge - How to implement DSLs with Groovy

Markus Voelter - Eclipse tooling for Model-Driven stuff

Dionysios G. Synodinos - JavaScript DSLs for the Client Side

Ted Neward, Bradford Cross - Functional vs. Dynamic DSLs: The Smackdown

Gilad Bracha - embedding EBNF in a general purpose language

Umit Yalcinalp, Tilman Giese - RUMBA: RUby Managed Business data for Applications

Bob Archer - A DSL for Cool Effects in Adobe Pixel Blender

Chance Coble - Language Oriented Programming in F#

As my 15-year-old son Michael has grown fond of saying... w00t! The list of topics is fascinating, and I'm really looking forward to most, if not all, of them. Chance's talk on LOP in F# should be good, I'm really curious to see Gilad's discussion of EBNF (and wondering if this is Newspeak we'll be seeing), and Guillaume is always fun to watch when he's going on about Groovy. Of course, I'm also excited to be paired up with Brad, who's an insanely smart guy--I have a feeling I'll learn a lot just by standing next to him. (Sort of a speakers' osmosis.)

If you're not planning to be here for this (and the Lang.NET Symposium), either you have life-saving surgery scheduled that can't be pushed back, or you're clearly not interested in DSLs. For your own sake, I hope it's the latter.

Seriously, come for the full week. The Lang.NET Symposium last year was an amazing event, for a number of reasons, not the least of which is that it saw Sun celebrities John Rose, Charlie Nutter and Brian Goetz step on to the Microsoft campus, deliver a great presentation on the JVM, MLVM/invokedynamic, and JRuby, and get good feedback and discussion from Microsoft engineers and other notables. You don't get to see that every day.

Answer: "I don't know, but I'm damn well going to find out!"

(Now I really wish I were in London. Ah, well, will just have to go see Ward Cunningham speak at Alt.NET Seattle, instead.)

Matt Podwysocki makes it all clear:

Hey, I'd have learned Haskell a LONG time ago if I'd known it could yield up a beer!

Last year I had the opportunity to return to the land of my roots, Poland, and speak at Java Developer Days (JDD). Just today, the organizers from JDD sent me a link with a nice little photo montage from the conference. (I did notice a few photos from the after-party were selectively left out of the montage, however, which is probably a good thing because that was the first time I'd ever met a Polish Mad Dog, and boy did they all go down easy...)

If you're anywhere in the area around Krakow in March, you definitely should swing by for their follow-up conference, 4Developers--it sounds like it's going to be another fun event, and this time it's going to reach out to more than just the Java folks, but also the .NET crowd (and a few others), as well.

(I don't really expect any of the readers of this blog living outside Poland to really pack up and head over to Krakow for a weekend, mind you, but if you're a technology speaker and you're interested in hanging with an extremely good group of people, the people who put these shows on--ProIdea--are top-notch, take great care of the speakers, and overall make the entire experience well worth the trip.)

Because I’m about to start my third part in the WCF/F# series, I realized that I’ve now hit the “rule of three” mark: in this particular case, this will mark the third project I’m creating that unifies WCF and F#, and frankly, it’s a pain in the *ss to do it all by hand each time: create an F# Library, add the System.ServiceModel and System.Runtime.Serialization assemblies, go create an App.config file and add it to the project as an Existing Item…. Painful.

So… as a brief interlude, I decided to go re-acquaint myself with the Visual Studio project template system, and sure enough, it’s basically what I remember: a collection of files with some template-style functionality, bundled into a .zip file and stored in the Visual Studio directory, under <VSDir>\Common7\IDE\ProjectTemplates. What was new to me, however, was the highly useful “File | Export Template…” menu option, allowing me to take an existing F#/WCF project and use it as a template to create the .zip bundle. (Naturally, I didn’t discover this until I’d built the silly thing by hand.)

Sara Ford has more on creating a VS template on her Visual Studio Tools blog/column, number 336 to be precise. (You should read all of them, by the way—start with #1 and work your way there. When you’re done, you’ll have a much better appreciation of everything Visual Studio can do, and you’ll be able to find a ton of ways to save yourself and your team some time and effort.)

You can always take a .zip bundle like this and drop it into the Visual Studio 2008 “My Exported Templates” directory, but quite frankly, I didn’t want that. I wanted my template to appear in a subcategory of Visual F# in the New Project dialog box, under “WCF”, just as the C# versions do. The easiest way to do this is to manually create the “WCF” directory (full path thus being <VSDir>\Common7\IDE\ProjectTemplates\FSharp\WCF), and drop the .zip file there. Note that if you restart Visual Studio at this point, you won’t see the new template; it builds a cache of the .zip templates in a sister directory (ProjectTemplatesCache), so instead, you have to tell Visual Studio to reset that cache by firing “devenv /setup” from the command-line. (This will require admin privileges, by the way.)

After that, you have an F#/WCF project template, and you’re good to go.

If you’ve not read the first part in the series, take a look there first.

While it’s always easier to build WCF services with nothing but primitive types understood by all the platforms to which you’re communicating (be it Java through XML services or other .NET systems via WCF’s more efficient binding types), this gets old and limiting very quickly. The WCF service author will want to develop whole composite types that can be exchanged across the wire, and this is most often done via the DataContract attribute applied to the types that will be exchanged.

In Michele Leroux Bustamente’s Learning WCF examples, this is covered in Chapter #2, and the corresponding code I’m using as a basis for conversion to F# is in Labs\Chapter2\DataContracts_Part1.

One notable difference between this example and the previous one is that the type definitions are stored in a separate assembly, ContentTypes.dll. There’s two basic choices to use here: one, to use the C# types as defined, from a service written in F#, or two, to define the types in F# and use them from the service. A third choice, defining the types in F# and using them from C#, also presents itself, but is uninteresting to us from a purely instructional standpoint—if you know how to write C#, then you can take the types defined in F# and use them just as you would have the C# types.

For instructional purposes, I’m going to take the second approach: I’m first going to convert the ContentTypes.dll assembly over to F#, again to show how to create types in F# that are structurally equivalent to the types defined in C#, since that’s something that has changed since Nick Holmes blogged about this last year), then I’m going to flip the service over to F# as well.

Defining the Data Types

The first step, for many service authors, is to define the interfaces for the service(s) and the types that will be exchanged; in this case, since I’m building from Michele’s example, these have already been defined as:

   1: using System;

   2: using System.ServiceModel;

   3: using System.Runtime.Serialization;

   4:  

   5: namespace ContentTypes

   6: {

   7:     

   8:    [DataContract(Namespace="http://schemas.thatindigogirl.com/samples/2006/06")]

   9:     public class LinkItem

  10:     {

  11:  

  12:         [DataMember(Name = "Id", IsRequired = false, Order = 0)]

  13:         private long m_id;

  14:         [DataMember(Name = "Title", IsRequired = true, Order = 1)]

  15:         private string m_title;

  16:         [DataMember(Name = "Description", IsRequired = true, Order = 2)]

  17:         private string m_description;

  18:         [DataMember(Name = "DateStart", IsRequired = true, Order = 3)]

  19:         private DateTime m_dateStart;

  20:         [DataMember(Name = "DateEnd", IsRequired = false, Order = 4)]

  21:         private DateTime m_dateEnd;

  22:         [DataMember(Name = "Url", IsRequired = false, Order = 5)]

  23:         private string m_url;

  24:  

  25:         public DateTime DateStart

  26:         {

  27:             get { return m_dateStart; }

  28:             set { m_dateStart = value; }

  29:         } 

  30:  

  31:         public DateTime DateEnd

  32:         {

  33:             get { return m_dateEnd; }

  34:             set { m_dateEnd = value; }

  35:         }

  36:        

  37:         public string Url

  38:         {

  39:             get { return m_url; }

  40:             set { m_url = value; }

  41:         }

  42:         

  43:         public long Id

  44:         {

  45:             get { return m_id; }

  46:             set { m_id = value; }

  47:         }

  48:  

  49:         public string Title

  50:         {

  51:             get { return m_title; }

  52:             set { m_title = value; }

  53:         }

  54:  

  55:         public string Description

  56:         {

  57:             get { return m_description; }

  58:             set { m_description = value; }

  59:         }

  60:     }

  61: }

Note that now, in a C#3-friendly world, we can slim the definition of the LinkItem down to a much smaller level thanks to the power of automatic properties:

   1: using System;

   2: using System.ServiceModel;

   3: using System.Runtime.Serialization;

   4:  

   5: namespace ContentTypes

   6: {    

   7:     [DataContract(Namespace="http://schemas.thatindigogirl.com/samples/2006/06")]

   8:     public class LinkItem

   9:     {

  10:         [DataMember(Name = "Id", IsRequired = false, Order = 0)]

  11:         public long Id { get; set; }

  12:         [DataMember(Name = "Title", IsRequired = true, Order = 1)]

  13:         public string Title { get; set; }

  14:         [DataMember(Name = "Description", IsRequired = true, Order = 2)]

  15:         public string Description { get; set; }

  16:         [DataMember(Name = "DateStart", IsRequired = true, Order = 3)]

  17:         public DateTime DateStart { get; set; }

  18:         [DataMember(Name = "DateEnd", IsRequired = false, Order = 4)]

  19:         public DateTime DateEnd { get; set; }

  20:         [DataMember(Name = "Url", IsRequired = false, Order = 5)]

  21:         public string Url { get; set; }

  22:     }

  23: }

… but either way, the type ends up looking the same. Converting this over to F# is relatively easy, if not any shorter or more convenient than the C# 3.0 version, owing to the fact that, by default, F# will not generate mutable properties by default:

   1: #light

   2:  

   3: namespace ContentTypes

   4:     

   5: open System

   6: open System.Runtime.Serialization

   7: open System.ServiceModel

   8:  

   9: [<DataContract(Namespace="http://schemas.thatindigogirl.com/samples/2006/06")>]

  10: type LinkItem() =

  11:     let mutable id : int64 = 0L

  12:     let mutable title : string = String.Empty

  13:     let mutable description : string = String.Empty

  14:     let mutable dateStart : DateTime = DateTime.Now

  15:     let mutable dateEnd : DateTime = DateTime.Now

  16:     let mutable url : string = String.Empty

  17:  

  18:     [<DataMember(Name = "Id", IsRequired = false, Order = 0)>]

  19:     member public l.Id

  20:         with get() = id

  21:         and set(value) = id <- value

  22:     [<DataMember(Name = "Title", IsRequired = true, Order = 1)>]

  23:     member public l.Title

  24:         with get() = title

  25:         and set(value) = title <- value

  26:     [<DataMember(Name = "Description", IsRequired = true, Order = 2)>]

  27:     member public l.Description

  28:         with get() = description

  29:         and set(value) = description <- value

  30:     [<DataMember(Name = "DateStart", IsRequired = true, Order = 3)>]

  31:     member public l.DateStart

  32:         with get() = dateStart

  33:         and set(value) = dateStart <- value

  34:     [<DataMember(Name = "DateEnd", IsRequired = false, Order = 4)>]

  35:     member public l.DateEnd

  36:         with get() = dateEnd

  37:         and set(value) = dateEnd <- value

  38:     [<DataMember(Name = "Url", IsRequired = false, Order = 5)>]

  39:     member public l.Url

  40:         with get() = url

  41:         and set(value) = url <- value

Notice that I have to create a mutable backing field, and define the properties in the F# LinkItem type to explicitly access and mutate those values. This is a bit frustrating, because it seems like F# should be able to infer what I want from a simple property declaration, in the same way that C# can, but perhaps that’s asking too much from the language right now, considering the silly thing hasn’t even shipped yet.

(Psssst, Luke, Don, if you’re listening, automatic property generation in F# would be a nifty feature to add between now and then, if you guys can ninja it in there before the next CTP…)

Notice, by the way, the namespace directive at the top of the F# code; this is necessary to set the prefix around the LinkItem type. Without it, remember, the F# code is going to be slipped inside an outer class declaration matching the filename, effectively naming the class Module1+LinkItem, which would not be structurally equivalent to the C# type.

Lesson #4: Always put a namespace or module declaration around the types exported from a service.

Notice that LinkItem also has a default constructor, as per Lesson #2; this is necessary because the DataContract-related code inside of WCF is going to need to be able to construct one of these and set its properties. If we want to set any reasonable defaults, that’s easily done in the mutable member definitions.

One principal difference between the F# version and the C# version is that the DataMember attributes are applied to the properties, instead of the fields, largely because the F# language wants to keep a layer of encapsulation between the code you write as an F# programmer, and the actual code generated. So, for example, the “field” id, above, doesn’t actually get generated exactly as described—in truth, it turns into a field called “id@11”. This is a marked difference from C# (or even VB), which deliberately gives us more control over how the physical structure of classes looks. This is even more obvious in a basic F# program where a top-level declaration reads, “let x = 12”; where it might be tempting to assume that x will be a static field on the class surrounding the declaration, the F# compiler actually generates a property.

In this particular case, whether the attribute applies to the fields or the property declarations isn’t going to make a large difference, but in more sophisticated classes, it might, so it’s better to apply the attribute to the property and not the field, at least, from what I’ve found so far.

Lesson #5: Put DataMember attributes on the properties of the DataContract, not the fields.

Defining the Service

The definition of the service is actually pretty straightforward. Add either the C# ContentTypes.dll or the F# ContentTypes.dll as an assembly reference, and where the C# code (GigManagerService.cs) reads:

   1: using System;

   2: using System.Collections.Generic;

   3: using System.Text;

   4: using System.ServiceModel;

   5: using ContentTypes;

   6:  

   7: namespace GigManager

   8: {

   9:     [ServiceContract(Name = "GigManagerServiceContract", Namespace = "http://www.thatindigogirl.com/samples/2006/06", SessionMode = SessionMode.Required)]

  10:     public interface IGigManagerService

  11:     {

  12:         [OperationContract]

  13:         void SaveGig(LinkItem item);

  14:  

  15:         [OperationContract]

  16:         LinkItem GetGig();

  17:     }

  18:  

  19:     [ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession)]

  20:     public class GigManagerService : IGigManagerService

  21:     {

  22:  

  23:         private LinkItem m_linkItem;

  24:  

  25:         public void SaveGig(LinkItem item)

  26:         {

  27:             m_linkItem = item;

  28:         }

  29:  

  30:         public LinkItem GetGig()

  31:         {

  32:             return m_linkItem;

  33:         }

  34:     }

  35: }

… the corresponding F# code (Program.fs) reads like so:

   1: #light

   2:  

   3: module GigManager =

   4:     open System

   5:     open System.Runtime.Serialization

   6:     open System.ServiceModel

   7:     

   8:     open ContentTypes

   9:     

  10:     [<ServiceContract(Name = "GigManagerServiceContract", 

  11:         ConfigurationName = "IGigManagerService",

  12:         Namespace = "http://www.thatindigogirl.com/samples/2006/06", 

  13:         SessionMode = SessionMode.Required)>]

  14:     type IGigManagerService =

  15:         [<OperationContract>]

  16:         abstract SaveGig: item : LinkItem -> unit

  17:         [<OperationContract>]

  18:         abstract GetGig: unit -> LinkItem

  19:         

  20:     [<ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession)>]

  21:     type GigManagerService() =

  22:         let mutable li : LinkItem = LinkItem()

  23:         interface IGigManagerService with

  24:             member gms.SaveGig(item) = li <- item                

  25:             member gms.GetGig() = li

Careful readers will notice that there’s one additional element in the F# version that isn’t in the C# version; specifically, on line 11, I’ve added a “ConfigurationName” element to the IGigManagerService’s ServiceContract attribute. I do this because, again, the F# compiler is doing some interesting things to the code under the hood. In particular, the interface IGigManagerService is actually exposed under a slightly different name—remember, F# likes to use nested classes, not namespaces, so where the C# version of IGigManagerService is formally known as “GigManager::IGigManagerService”, the F# version is “Program/GigManager/GigManagerService”, where Program is the name of the .fs file. This seems to cause WCF some heartache when it starts looking through the App.config file and matching it up against the names exported from the actual class—it won’t match up correctly. So, by giving it a ConfigurationName that matches the human-readable interface name, WCF is happy again.

Lesson #5: Use ConfigurationName on ServiceContract to avoid having to learn F#’s naming bindings to the CLR.

The rest of the code in Program.fs is the hosting code, which structurally is no different than that of the previous post.

One key thing to remember, however, is that the host “service” element will also be looking at type names, so if you forget to set the name of the service, you’ll need to use a type-investigation tool (ILDasm or Reflector) to figure out what the host class name is; in the case above, it would be “Program+GigManager+GigManagerService”, forcing the App.config file to read as follows:

   1: <?xml version="1.0" encoding="utf-8" ?>

   2: <configuration>

   3:   <system.serviceModel>

   4:     <services>

   5:       <service name="Program+GigManager+GigManagerService" 

   6:                behaviorConfiguration="serviceBehavior">

   7:         <host>

   8:           <baseAddresses>

   9:             <add baseAddress="http://localhost:8000"/>

  10:             <add baseAddress="net.tcp://localhost:9000"/>

  11:           </baseAddresses>

  12:         </host>

  13:         <endpoint address="GigManagerService"

  14:                   binding="netTcpBinding"

  15:                   contract="IGigManagerService" />

  16:         <endpoint address="mex"

  17:                   binding="mexHttpBinding"

  18:                   contract="IMetadataExchange" />

  19:       </service>

  20:     </services>

  21:       <behaviors>

  22:           <serviceBehaviors>

  23:               <behavior name="serviceBehavior">

  24:                   <serviceMetadata httpGetEnabled="true"/>

  25:               </behavior>

  26:           </serviceBehaviors>

  27:       </behaviors>

  28:     <!-- This <diagnostics> section should be placed inside the <system.serviceModel> section. In addition, you'll need to add the <system.diagnostics> snippet to specify service model trace listeners and a file for output. -->

  29:     <diagnostics performanceCounters="All" wmiProviderEnabled="true" >

  30:       <messageLogging logEntireMessage="true" logMalformedMessages="true" logMessagesAtServiceLevel="true" logMessagesAtTransportLevel="true" maxMessagesToLog="100000"  />

  31:     </diagnostics>

  32:   </system.serviceModel>

  33:   <!-- This <system.diagnostics> section illustrates the use of a shared listener for service model output. It requires you to also add the <diagnostics> snippet for the <system.serviceModel> section. -->

  34:   <system.diagnostics >

  35:     <sharedListeners>

  36:       <add name="sharedListener" 

  37:                  type="System.Diagnostics.XmlWriterTraceListener"

  38:                  initializeData="c:\logs\servicetrace.svclog" />

  39:     </sharedListeners>

  40:     <sources>

  41:       <source name="System.ServiceModel" switchValue="Verbose, ActivityTracing" >

  42:         <listeners>

  43:           <add name="sharedListener" />

  44:         </listeners>

  45:       </source>

  46:       <source name="System.ServiceModel.MessageLogging" switchValue="Verbose">

  47:         <listeners>

  48:           <add name="sharedListener" />

  49:         </listeners>

  50:       </source>

  51:     </sources>

  52:   </system.diagnostics>

  53: </configuration>

Caveat emptor. In all honesty, despite the motivation of Lesson #5, I don’t think there’s any way around learning at least a little bit of F#’s name-mapping scheme, but at least we can be selective about where and when we apply it.

From Scott Hanselman's blog:

Are you in King County/Seattle/Redmond/Bellevue Washington and surrounding areas? Are you a huge nerd? Perhaps a geek? No? Maybe a dork, dweeb or wonk. Maybe you're in town for an SDR (Software Design Review) visiting BillG. Quite possibly you're just a normal person.

Regardless, why not join us for some Mall Food at the Crossroads Bellevue Mall Food Court on Monday, January 19th around 6:30pm?

...

NOTE: RSVP by leaving a comment here and show up on January 19th at 6:30pm! Feel free to bring friends, kids or family. Bring a Ruby or Java person!

Any of the SeaJUG want to attend? (Anybody know of a Ruby JUG in the Eastside area, by the way?) I'm game....

For a while now, I’ve held the opinion that the “sweet spot” for functional languages on the JVM and CLR will be in the services space, since services and functions seem pretty similar to one another in spirit—a given input produces a given output, with (ideally) no shared state, high concurrency expectations, idempotent processing, and so on. This isn’t to say that a functional language is going to make a non-trivial service trivial, but I think it will make it simpler and more likely to scale better over time, particularly as the service gets more complicated.

As part those explorations into the union of services and functional languages, I’ve been taking some of Michele Leroux Bustamente’s excellent labs from Learning WCF and flipping the services over to F#. Along the way, I’ve discovered a few “quirks” of F# that make building a WCF service a tad more complicated than it needs to be, so I’ve decided to blog what’s going on so others can find it easier.

(Many thanks to Nick Holmes’ blog, which helped identify one of the first problems I ran into, though a few things have changed since he blogged back in February, so I thought I’d catch everything up to the Sep 08 CTP of F#.)

This isn’t intended to be a tutorial on WCF, so if you’re not familiar with WCF, I strongly suggest you go get Michele’s book. I’m assuming you’ll know the WCF basics (address, binding, contract, config files, behaviors, etc), and I just want to show the deltas necessary to make F# work. Note that I’m just doing the service side of things—I believe clients will probably continue to be written in C# or VB or some other OO language, in keeping with the theory that OO will remain the predominant way of developing client-facing stuff. (Note that this also neatly avoids the basic problem that svcutil.exe only generates either C# or VB proxy code, and that “Add Service Reference” isn’t available inside an F# project, as of this writing.)

Defining Contracts in F#

The first step in any straight-up WCF service is, of course, to define the contract that both sides will agree to. (Yes, I know, we could do everything in terms of picking Message types apart; I’ll get to that in a later piece.) First things first: taking Michele’s HelloIndigo_Part1 solution, I add a new project to it, “FHost”, an F# application. Add the System.ServiceModel and System.Runtime.Serialization assemblies, and we’re good to get going.

Michele’s “HelloIndigo_Part1” solution defines the contract between client and service this way:

   1: namespace Host

   2: {

   3:     [ServiceContract(Namespace = "http://www.thatindigogirl.com/samples/2006/06")]

   4:     public interface IHelloIndigoService

   5:     {

   6:         [OperationContract]

   7:         string HelloIndigo();

   8:     }

   9:     // ...

  10: }

This contract can be consumed in two ways; one is to build this interface into its own assembly that’s linked to both the WCF service host and to the WCF client, but in her example (as is perfectly reasonable in a WCF project), she repeats the interface in both the client and the service, so to be faithful to that, let’s define the interface in the F# code:

   1: #light

   2:  

   3: open System

   4: open System.ServiceModel

   5:  

   6: [< ServiceContract(Namespace = "http://www.thatindigogirl.com/samples/2006/06") >]

   7: type IHelloIndigoService =

   8:     [< OperationContract >]

   9:     abstract HelloIndigo: unit -> string

(The color syntax highlighting is off because I’m using the C# mode of the “Code Snippet” plugin in Windows Live Writer to post the code, and it doesn’t have an F# mode. Yet.)

Pay very close attention to the interface definition in F#, because there is a subtle WCF “bug” that F# exposes by accident. When F# compiles an interface, if a method in the interface has parameters, if no name is specified for that parameter, then WCF will throw an ArgumentNullException when you try to run svcutil.exe over the compiled assembly, or when you pass the type in to the ServiceHost constructor, claiming “Value cannot be null. Parameter name: name”. The problem is that F#, unlike C# or VB, allows methods to have parameters without names, and WCF can’t handle this. Verifying this is a b*tch; if you use ILDasm to view the F#-compiled assembly, it looks like there are parameter names there, because ILDasm generates them as placeholders for display. (Reflector is your friend here.)

The WCF team has basically said that this behavior is by design—SOAP, which is a key concept to the WCF stack, doesn’t really have great support for unnamed parameters (and yes, I know, this is not exactly true, but I’m not going to get into that debate here), so the WCF team has basically said there’s really nothing they can do but maybe issue a better error message than ArgumentNullException.

Lesson #1: Always name your WCF contract interface params.

Caveat emptor.

Defining the Service Implementation

Next step is to define the service implementation. Again, Michele’s code looks like so:

   1: public class HelloIndigoService : IHelloIndigoService

   2: {

   3:     public string HelloIndigo()

   4:     {

   5:         return "Hello Indigo";

   6:     }

   7: }

Not a really difficult operation, so converting that to F# is pretty straightforward:

   1: [< ServiceBehavior(ConfigurationName="HIS") >]    

   2: type HelloIndigoService() =

   3:     interface IHelloIndigoService with

   4:         member s.HelloIndigo() : string =

   5:             "Hello Indigo"

There are two things important to this definition. First, the parentheses at the end of the “type” declaration line create a default no-argument constructor for the HelloIndigoService, which is required—without it, WCF is going to complain about being unable to construct an instance of this type.

Lesson #2: Always provide the default type constructor in the service implementation.

Second, the ServiceBehavior attribute is one I’ve added, because F# does some funky things with the type names during compilation; for example, since my F# code is in a file called “Host.fs”, the F# compiler synthesizes a class called “Host” which acts as a nesting wrapper around everything else in the file, so technically the typename of HelloIndigoService is now “Host+HelloIndigoService”, which will cause some chaos when WCF tries to match up the service name with the appropriate entry in the App.config file. You can either make sure the App.config matches the CLR-level type names generated by the F# compiler, or you can explicitly specify the configuration names; I choose the latter, so that it’s a bit more clear what’s going on.

Lesson #3: Always specify the configuration name on the service implementation.

The App.config file, by the way, now looks like this, the only change from Michele’s labs being the changes to the configuration name of the service behavior (line 13):

   1: <?xml version="1.0" encoding="utf-8" ?>

   2: <configuration>

   3:     <system.serviceModel>

   4:         <behaviors>

   5:             <serviceBehaviors>

   6:                 <behavior name="serviceBehavior">

   7:                     <serviceMetadata httpGetEnabled="false" />

   8:                 </behavior>

   9:             </serviceBehaviors>

  10:         </behaviors>

  11:         <services>

  12:             <service behaviorConfiguration="serviceBehavior" 

  13:                      name="HIS">

  14:                 <clear />

  15:                 <endpoint address="HelloIndigoService" 

  16:                           binding="basicHttpBinding"

  17:                           name="basicHttp" 

  18:                           contract="Host+IHelloIndigoService" />

  19:                 <endpoint binding="mexHttpBinding" 

  20:                           name="mex" 

  21:                           contract="IMetadataExchange" />

  22:                 <host>

  23:                     <baseAddresses>

  24:                         <add baseAddress="http://localhost:8000/HelloIndigo" />

  25:                     </baseAddresses>

  26:                 </host>

  27:             </service>

  28:         </services>

  29:     </system.serviceModel>

  30: </configuration>

Still with me? One last part to go, defining the (self-hosting) host.

Defining the Self-Hosting Host

In simple examples, frequently the service code self-hosts, meaning it doesn’ t need to be deployed into IIS. Michele uses a wrapper class to defer some of the hosting details, a la:

   1: internal class MyServiceHost

   2: {

   3:     internal static ServiceHost myServiceHost = null;

   4:  

   5:     internal static void StartService()

   6:     {

   7:         // Instantiate new ServiceHost 

   8:         myServiceHost = new ServiceHost(typeof(HelloIndigoService));

   9:         myServiceHost.Open();

  10:     }

  11:  

  12:     internal static void StopService()

  13:     {

  14:         // Call StopService from your shutdown logic (i.e. dispose method)

  15:         if (myServiceHost.State != CommunicationState.Closed)

  16:             myServiceHost.Close();

  17:     }

  18: }

  19:  

  20: class Program

  21: {

  22:     static void Main(string[] args)

  23:     {

  24:         try

  25:         {

  26:             MyServiceHost.StartService();

  27:             Console.WriteLine("Press <ENTER> to terminate the host application");

  28:             Console.ReadLine();

  29:         }

  30:         finally

  31:         {

  32:             MyServiceHost.StopService();

  33:         }

  34:     }

  35: }

I don’t quite think the wrapper is necessary, so I simplified it down to:

   1: let main() =

   2:     Console.WriteLine("IHelloIndigoService = " + typeof<IHelloIndigoService>.ToString() )

   3:  

   4:     let hisType = typeof<HelloIndigoService>

   5:     let host = new ServiceHost(hisType, ([| |] : Uri[] ) )

   6:     host.Open()

   7:     Console.WriteLine("Press <ENTER> to terminate the host application")

   8:     Console.ReadLine() |> ignore

   9:     host.Close()

  10:  

  11: main()

One quirk of the current (Sept 08) F# CTP is that when working with variable-argument parameters (like the second argument of the ServiceHost constructor), F# doesn’t have a great syntax. We have to explicitly specify, in this case, an empty array of Uri objects, but simply specifying an empty array (“[| |]”) will be interpreted as an empty array of objects, and thus generate a compile error. We have to explicitly set the type of the array to be an array of Uri, hence the type specifier.

Oh, and don’t forget, if you’re running as a non-Administrator on Vista or XP, you’ll need to create a URL ACL to allow a non-Administrator user to create an HTTP endpoint; the relevant command for the example above is this:

netsh http add urlacl url=http://+:8000/HelloIndigo user=devtop-t42p\ted

(Obviously, you substitute in your own domain and username for mine.) Make sure to do this from an Administrator-enabled command prompt, or you’ll just get another security error.

The beautiful thing about this example is that if it works, you can use the Client written in C# without a hitch of a problem, thus demonstrating quite clearly that WCF isn’t sharing assemblies between client and service. Given that this service also sets up the MEX endpoint, you should also be able to run svcutil against the running service and generate proxy code if you want to prove that it’s doable; I didn’t do it for this example, since I trust that the App.config-specified MEX endpoint will still be there, and because I was more interested in taking the existing Client and making it work as-is.

More to come, but this should get you started, anyway. Thanks again to Michele for letting me scaffold off of her!

Chris Sells, an acquaintance (and perhaps friend, when he's not picking on me for my Java leanings) of mine from my DevelopMentor days, has a habit of putting on a "DevCon" whenever a technology seems to have reached a certain maturity level. He did it with XML a few years ago, and ATL before that, both of which were pretty amazing events, filled with the sharpest guys in the subject, gathered into a single room to share ideas and shoot each others' pet theories full of holes.

He's at it again, this time with DSLs; from the announcement on his blog:

Are you interested in presenting a 45-minute talk on some Domain Specific Language (DSL) related topic? It doesn't matter which platform or OS you're targeting. It also doesn't matter whether you're an author, a vendor, a professional speaker or a developer in the trenches (in fact, I tend to be biased toward the latter). We're after interesting and unique applications of DSL technology and if you're doing good work in that area, then I need you to send me a session topic and 2-4 sentence abstract along with a little bit about yourself.

I'll be taking submissions 'til February 9th, 2009, but don't delay. Passion and a burning story to tell count twice as much as anything else.

And don't be shy about spreading this announcement around! I've got good coverage in the .NET and Windows communities, but don't know very many folks in the Java or Unix or hardcore modeling worlds, so if you're in that world, let those guys know! Thanks.

The DSL DevCon itself will be in Redmond, WA on the Microsoft campus April 16-17, 2009, right after the Lang.NET conference. Lang.NET will be focused on general-purpose languages, whereas the DSL DevCon will focus on domain-specific languages. The idea is that if you want to attend one or the other or both, that's totally fine. We'll have 2.5 days of Lang.NET on April 14-16 and then 1.5 days of DSL DevCon content.

Oh, and the cost for both conferences is the same: $0.

We're only accepting 150 attendees to either conference. Every one of the five previous DevCons have sold out, so when we open registration, you'll want to be quick about getting your name on the list.

Submit your DSL-related talk idea!

For those of you who are deep in the Java or Ruby space, I really urge you to take a chance here and come to the event--just because it's being held on the Microsoft campus doesn't mean you're going to be forcibly plugged into the Matrix; the same goes for the Lang.NET event in the earlier part of the week, too. Don't believe me? I have proof: Brian Goetz, John Rose, and Charlie Nutter, Sun employees all, attended last years Lang.NET event, talked about the JVM and JRuby, and not only did they not have to give up their "sun.com" email addresses, but they came away with some new appreciations for the CLR, the ecosystem there, and even a few insights about their own platform in comparison to the JVM. (I won't say this as an absolute fact, but I think a lot of John's work on method handles for Java7 came out of conversations he'd had with some of the CLR guys that week.)

This is a DevCon, not a MarCon or a SaleCon. If you're a dev, you're welcome to come here. Frankly, I'd love to see the Java and Ruby (and LLVM and Parrot and ...) guys storm the castle, so to speak, if for no other reason than so Chris will stop teasing me about being a Java guy.

I'm getting *hammered* by the Google "Windows7 VMware" hits, which I can only assume is from people looking for hints and advice on installing Windows7 into a VMWare image, and I feel compelled to point out that there's already a pre-built VMWare VM available from the "Virtual Appliance" pages at VMware.com; currently, it resides here. Note that you will need to BitTorrent it down, I haven't found a straight HTTP download link from that (off-vmware.com) site.

I wish I knew the full legalities of making said VM available; I only hope that the guys who are doing it have checked, but if you're at all concerned about such things, trust me, it's pretty painless to install Win7 into your own VM of your own making.

A friend of mine and fellow NFJS speaker, Ken Sipe, blogged about his experiences with Windows 7, and unfortunately, they're not positive. In fact, they're downright painful to read.

And he hasn't even begun the installation process yet:

First I went to the public beta site... and selected the 64-bit version in english and got this [screen shot]. WTF?? Repeated attempts resulted in the same. An oops page with a pre-canned search. Where did I go wrong? Well as you can tell, I'm on my Mac. So I pulled out fusion to launch Windows XP for round 2 of the attempt.

I thought this is just wrong, but determined to get a look, I switch to windows and my suspicions were confirmed when I got one page further. I got the download page with a couple of large buttons on the bottm of the page and one read "Download Now". Hey, that's what I want... I want to download now. I clicked the button and... nothing. Click... Nothing... No way... they didn't. Round 2 was in XP, but with firefox.

Round 3 as you would expect is XP with IE. That combination was successful and I'm now 29% into my download.

BTW... In the process of testing a few more times in writing up this blog, the round 1 mac failure was fixed to the point where you will get download page (nice response time msft), however the download button fails.

Why is it necessary to be like this? Why is it so hard to put up a link to a download which is platform neutral? Wouldn't Microsoft want to attract customers from other platforms? Does it always have to be all or nothing?

Ken, for whatever it's worth, I ran into exactly the same roadblocks you did, in almost precisely the same sequence you did. The only saving grace for me, personally, was that after Firefox (on the Mac instead of inside the VM) couldn't download the image, I thought that maybe Microsoft wanted to use their custom "File Transfer Manager" utility (that allows for multiple connections, suspends and restarts, etc) to do the download, so I fired up the VM that has that utility installed, and surfed to the MSDN Subscriber Download page instead of the public download page.

Now, I could go into spin/defense mode and try to point out that the vast majority of the people interested in working with Windows 7 are, in all likelihood, going to be that same community of users that use IE, and that Microsoft is only really beholden to those folks, or that Microsoft knows that the beta images will scream through the Internet over BitTorrent streams anyway, or that Microsoft wants to make sure that it's available to those IE users first, or .... But that would all be a pretty slippery slope, and quite frankly, I don't really believe in any of those arguments, anyway.

Why does Microsoft do this? Honestly, in the spirit of "Never attribute to malice that which can be explained by stupidity or ignorance" (one of another NFJS speaker's favorite quote), I think the causation here is pretty simple to explain: I doubt anybody at Microsoft tested it with any other browser beyond IE. I could be wrong, of course, but I'm guessing that the conversation went something like this:

Manager: "Dilbert!"

Dilbert-the-website-dev: "Yes, boss?"

Manager: "Steve Ballmer, you remember him? He wants a public web page for downloading the Windows 7 beta, and he wants it yesterday. Make it happen!"

Dilbert: "Yes, boss. But what about--"

Manager: "No buts! This is TOP PRIORITY. Make it happen!"

Stupid? Yep. An attempt to exclude anybody except those on IE from downloading it? I doubt it.

Stay strong, Ken. It really does get better after this. Really.

This is more a continuation of my earlier Windows7 post, but I've installed the new Windows7 beta into a VMWare Fusion VM with zero difficulties, and I just finished putting VS2008 (and the SP1 patch) on it, then the latest F# CTP on top of that, and so far it all looks pretty smooth. Put in the DDK and the SDK, and I've got a nice Windows7 development image to play with.

I've had a few people ask me if I've still had problems with the mouse, but to be honest I installed it without the driver installed in the VMWare Tools install, so as soon as I copy off the .vmdk and .vmss files to a quiet little corner of the hard drive as backup, I'll try installing the mouse driver to see if it works, and report back here soon.

An open message to the Visual Studio installation team: One thing I'd like to see changed for VS2010--instead of giving me a "cmd.exe" environment for using VS from the command-line, can you at least give me a PowerShell .ps1 shell link to go alongside it? And why does the VS2008 SP1 patch require me to put Visual Studio in the CD tray to reference the vs_setup.msi about halfway through?

Update: Mouse driver works flawlessly. Dunno if it was a bug they fixed, or just random good VM karma, but the entire VMWare Tools package now works perfectly, as far as I can tell. Note: I haven't heard any sound out of it, but sometimes the sound driver in Fusion cuts out for reasons beyond my understanding, and after a reboot, sound is back without a problem. Besides, sound is not as important to me in a work VM as mouse or network, anyway, so....

For a couple of years now, I've been going around the world and giving a talk entitled "Pragmatic Architecture", talking both about what architecture is (and what architects really do), and ending the talk with my own "catalog" of architectural elements and ideas, in an attempt to take some of the mystery and "cloud" nature of architecture out of the discussion. If you've read Effective Enterprise Java, then you've read the first version of that discussion, where Pragmatic Architecture was a second-generation thought process.

Recently, the patterns & practices group at Microsoft went back and refined their Application Architecture Guide, and while there's a lot about it that I wish they'd done differently (less of a Microsoft-centric focus, for one), I think it's a great book for Microsoft-centric architects to pick up and have nearby. In a lot of ways, this is something similar to what I had in mind when I thought about the architectural catalog, though I'll admit that I'd prefer to go one level "deeper" and find more of the "atoms" that make up an architecture.

Nevertheless, I think this is a good PDF to pull down and put somewhere on your reference list.

Notes and caveats: Firstly, this is a book for solution architects; if you're the VP or CTO, don't bother with it, just hand it to somebody further on down the food chain. Secondly, if you're not an architect, this is not the book to pick up to learn how to be one. It's more in the way of a reference guide for existing architects. In fact, my vision is that an architect faced with a new project (that is, a new architecture to create) will think about the problem, sketch out a rough solution in his head, then look at the book to find both potential alternatives (to see if they fit better or worse than the one s/he has in her/his head), and potential consequences (to the one s/he has in her/his head). Thirdly, even if you're a Java or Ruby architect, most of the book is pretty technology-neutral. Just take a black Sharpie to the parts that have the Microsoft trademark around them, and you'll find it a pretty decent reference, too. Fourthly, in the spirit of full disclosure, the p&p guys brought me in for a day of discussion on the Guide, so I can't say that I'm completely unbiased, but I can honestly say that I didn't write any of it, just offered critique (in case that matters to any potential readers).

People are used to the idea of phishing attacks showing up in their email, but in glowing testament to the creativity of potential attackers, Twitter recently has seen a rash of phishing attacks through Twitter's "direct messaging" feature.

The attack plays out like this: someone on your Twitter followers list sends you a direct message saying, "hey! check out this funny blog about you... " with a hyperlink to a website, "http://jannawalitax.blogspot.com/" . Clicking on the hyperlink takes you to a website that redirects to a webpage containing what looks like the Twitter login page. This is an attempt to get you to fill in your username, and more importantly, your password.

Needless to say, I'd avoid it. If you do get suckered in (hey, I admit it, I did), make sure to change your password immediately after.

What I find fascinating about this attack is that the direct messages come from people that are on my followers list--unless Twitter somehow has a hole in it that allows non-followers to direct-message you, it means that this is a classic security Ponzi scheme: I use the attack to gather the credentials for the people that I'm following directly, then log in and use those credentials to attack their followers, then use those gathered credentials to attack their followers, and so on. Fixing this is also going to be a pain--literally, everybody on Twitter has to change their password, or the scheme can continue with the credentials of those who didn't. (Assuming Twitter doesn't somehow lop the attack off at the knees, for example, by disallowing hyperlinks or something equally draconian.)

We won't even stop to consider what damage might be done if a Twitter-user uses the same password and login name for their Twitter account as they do for other accounts (such as email, banking websites, and so on). If you're one of those folks, you seriously might want to reconsider the strategy of using the same password for all your websites, unless you don't care if they get spoofed.

There's two lessons to be learned here.

One, that as a user of a service--any service--you have to be careful about when and how you're entering your credentials. It's easy to simply get into the habit of offering them up every time you see something that looks familiar, and if supposed "computer experts" (as most of the Twitterverse can be described) can be fooled, then how about the casual user?

Two, and perhaps the more important lesson for those of us who build software, that any time you build a system that enables people to communicate, even when you put a lot of energy into making sure that the system is secure, there's always an angle that attackers will find that will expose a vulnerability, even if it's just a partial one (such as the gathering of credentials here). If you don't need to allow hyperlinks, don't. If you don't need to allow Javascript, don't. Start from the bare minimum that people need to make your system work, and only add new capabilities after they've been scrutinized in a variety of ways. (YAGNI sometimes works to our advantage in more ways than one, it turns out.)

Kudos, by the way, to the Twitter-keepers, who had a message describing the direct-message phishing attack on the Twitter Home page within hours.

Thanks again to the folks at Microsoft who've been gracious enough to award me MVP Architect status again this year, and to the INETA Speakers Bureau, who've decided that I'm to remain an INETA speaker for another twelve months.

What's more impressive is the list of new speakers that INETA has added, including Rachel Appel, Alan Stevens, and Steve Andrews, among others. Congratulations to all three of you, you deserve it. Looking forward to seeing you guys on the road in 2009!

It's once again that time of year, and in keeping with my tradition, I'll revisit the 2008 predictions to see how close I came before I start waxing prophetic on the coming year. (I'm thinking that maybe the next year--2010's edition--I should actually take a shot at predicting the next decade, but I'm not sure if I'd remember to go back and revisit it in 2020 to see how I did. Anybody want to set a calendar reminder for Dec 31 2019 and remind me, complete with URL? )

Without further preamble, here's what I said for 2008:

• THEN: General: The buzz around building custom languages will only continue to build. More and more tools are emerging to support the creation of custom programming languages, like Microsoft's Phoenix, Scala's parser combinators, the Microsoft DLR, SOOT, Javassist, JParsec/NParsec, and so on. Suddenly, the whole "write your own lexer and parser and AST from scratch" idea seems about as outmoded as the idea of building your own String class. Granted, there are cases where a from-hand scanner/lexer/parser/AST/etc is the Right Thing To Do, but there are times when building your own String class is the Right Thing To Do, too. Between the rich ecosystem of dynamic languages that could be ported to the JVM/CLR, and the interesting strides being made on both platforms (JVM and CLR) to make them more "dynamic-friendly" (such as being able to reify classes or access the call stack directly), the probability that your company will find a need that is best answered by building a custom language are only going to rise. NOW: The buzz has definitely continued to build, but buzz can only take us so far. There's been some scattershot use of custom languages in a few scattershot situations, but it's certainly not "taken the world by storm" in any meaningful way yet.
• THEN: General: The hype surrounding "domain-specific languages" will peak in 2008, and start to generate a backlash. Let's be honest: when somebody looks you straight in the eye and suggests that "scattered, smothered and covered" is a domain-specific language, the term has lost all meaning. A lexicon unique to an industry is not a domain-specific language; it's a lexicon. Period. If you can incorporate said lexicon into your software, thus making it accessible to non-technical professionals, that's a good thing. But simply using the lexicon doesn't make it a domain-specific language. Or, alternatively, if you like, every single API designed for a particular purpose is itself a domain-specific language. This means that Spring configuration files are a DSL. Deployment descriptors are a DSL. The Java language is a DSL (since the domain is that of programmers familiar with the Java language). See how nonsensical this can get? Until somebody comes up with a workable definition of the term "domain" in "domain-specific language", it's a nonsensical term. The idea is a powerful one, mind you--creating something that's more "in tune" with what users understand and can use easily is a technique that's been proven for decades now. Anybody who's ever watched an accountant rip an entirely new set of predictions for the new fiscal outlook based entirely on a few seed numbers and a deeply-nested set of Excel macros knows this already. Whether you call them domain-specific languages or "little languages" or "user-centric languages" or "macro language" is really up to you. NOW: The backlash hasn't begun, but only because the DSL buzz hasn't materialized in much way yet--see previous note. It generally takes a year or two of deployments (and hard-earned experience) before a backlash begins, and we haven't hit that "deployments" stage yet in anything yet resembling "critical mass" yet. But the DSL/custom language buzz continues to grow, and the more the buzz grows, the more the backlash is likey.
• THEN: General: Functional languages will begin to make their presence felt. Between Microsoft's productization plans for F# and the growing community of Scala programmers, not to mention the inherently functional concepts buried inside of LINQ and the concurrency-friendly capabilities of side-effect-free programming, the world is going to find itself working its way into functional thinking either directly or indirectly. And when programmers start to see the inherent capabilities inside of Scala (such as Actors) and/or F# (such as asynchronous workflows), they're going to embrace the strange new world of functional/object hybrid and never look back. NOW: Several books on F# and Scala (and even one or two on Haskell!) were published in 2008, and several more (including one of my own) are on the way. The functional buzz is building, and lots of disparate groups are each evaluating it (functional programming) independently.
• THEN: General: MacOS is going to start posting some serious market share numbers, leading lots of analysts to predict that Microsoft Windows has peaked and is due to collapse sometime within the remainder of the decade. Mac's not only a wonderful OS, but it's some of the best hardware to run Vista on. That will lead not a few customers to buy Mac hardware, wipe the machine, and install Vista, as many of the uber-geeks in the Windows world are already doing. This will in turn lead Gartner (always on the lookout for an established trend they can "predict" on) to suggest that Mac is going to end up with 115% market share by 2012 (.8 probability), then sell you this wisdom for a mere price of $1.5 million (per copy). NOW: Can't speak to the Gartner report--I didn't have $1.5 million handy--but certainly the MacOS is growing in popularity. More on that later.
• THEN: General: Ted will be hired by Gartner... if only to keep him from smacking them around so much. .0001 probability, with probability going up exponentially as my salary offer goes up exponentially. (Hey, I've got kids headed for college in a few years.) NOW: Well, Gartner appears to have lost my email address and phone number, but I'm sure they were planning to make me that offer.
• THEN: General: MacOS is going to start creaking in a few places. The Mac OS is a wonderful OS, but it's got its own creaky parts, and the more users that come to Mac OS, the more that software packages are going to exploit some of those creaky parts, leading to some instability in the Mac OS. It won't be widespread, but for those who are interested in finding it, they're there. Assuming current trends (of customers adopting Mac OS) hold, the Mac OS 10.6 upgrade is going to be a very interesting process, indeed. NOW: Shhh. Don't tell anybody, but I've been seeing it starting to happen. Don't get me wrong, Apple still does a pretty good job with the OS, but the law of numbers has started to create some bad upgrade scenarios for some people.
• THEN: General: Somebody is going to realize that iTunes is the world's biggest monopoly on music, and Apple will be forced to defend itself in the court of law, the court of public opinion, or both. Let's be frank: if this were Microsoft, offering music that can only be played on Microsoft music players, the world would be through the roof. All UI goodness to one side, the iPod represents just as much of a monopoly in the music player business as Internet Explorer did in the operating system business, and if the world doesn't start taking Apple to task over this, then "justice" is a word that only applies when losers in an industry want to drag down the market leader (which I firmly believe to be the case--nobody likes more than to pile on the successful guy). NOW: Nothing this year.
• THEN: General: Somebody is going to realize that the iPhone's "nothing we didn't write will survive the next upgrade process" policy is nothing short of draconian. As my father, who gets it right every once in a while, says, "If I put a third-party stereo in my car, the dealer doesn't get to rip it out and replace it with one of their own (or nothing at all!) the next time I take it in for an oil change". Fact is, if I buy the phone, I own the phone, and I own what's on it. Unfortunately, this takes us squarely into the realm of DRM and IP ownership, and we all know how clear-cut that is... But once the general public starts to understand some of these issues--and I think the iPhone and iTunes may just be the vehicle that will teach them--look out, folks, because the backlash will be huge. As in, "Move over, Mr. Gates, you're about to be joined in infamy by your other buddy Steve...." NOW: Apple released iPhone 2.0, and with it, the iPhone SDK, so at least Apple has opened the dashboard to third-party stereos. But the deployment model (AppStore) is still a bit draconian, and Apple still jealously holds the reins over which apps can be deployed there and which ones can't, so maybe they haven't learned their lesson yet, after all....
• THEN: Java: The OpenJDK in Mercurial will slowly start to see some external contributions. The whole point of Mercurial is to allow for deeper control over which changes you incorporate into your build tree, so once people figure out how to build the JDK and how to hack on it, the local modifications will start to seep across the Internet.... NOW: OpenJDK has started to collect contributions from external (to Sun) sources, but still in relatively small doses, it seems. None of the local modifications I envisioned creeping across the 'Net have begun, that I can see, so maybe it's still waiting to happen. Or maybe the OpenJDK is too complicated to really allow for that kind of customization, and it never will.
• THEN: Java: SpringSource will soon be seen as a vendor like BEA or IBM or Sun. Perhaps with a bit better reputation to begin, but a vendor all the same. NOW: SpringSource's acquisition of G2One (the company behind Groovy just as SpringSource backs Spring) only reinforced this image, but it seems it's still something that some fail to realize or acknowledge due to Spring's open-source (?) nature. (I'm not a Spring expert by any means, but apparently Spring 3 was pulled back inside the SpringSource borders, leading some people to wonder what SpringSource is up to, and whether or not Spring will continue to be open source after all.)
• THEN: .NET: Interest in OpenJDK will bootstrap similar interest in Rotor/SSCLI. After all, they're both VMs, with lots of interesting ideas and information about how the managed platforms work. NOW: Nope, hasn't really happened yet, that I can see. Not even the 2nd edition of the SSCLI book (by Joel Pobar and yours truly, yes that was a plug) seemed to foster the kind of attention or interest that I'd expected, or at least, not on the scale I'd thought might happen.
• THEN: C++/Native: If you've not heard of LLVM before this, you will. It's a compiler and bytecode toolchain aimed at the native platforms, complete with JIT and GC. NOW: Apple sank a lot of investment into LLVM, including hosting an LLVM conference at the corporate headquarters.
• THEN: Java: Somebody will create Yet Another Rails-Killer Web Framework. 'Nuff said. NOW: You know what? I honestly can't say whether this happened or not; I was completely not paying attention.
• THEN: Native: Developers looking for a native programming language will discover D, and be happy. Considering D is from the same mind that was the core behind the Zortech C++ compiler suite, and that D has great native platform integration (building DLLs, calling into DLLs easily, and so on), not to mention automatic memory management (except for those areas where you want manual memory management), it's definitely worth looking into. www.digitalmars.com NOW: D had its own get-together as well, and appears to still be going strong, among the group of developers who still work on native apps (and aren't simply maintaining legacy C/C++ apps).

Now, for the 2009 predictions. The last set was a little verbose, so let me see if I can trim the list down a little and keep it short and sweet:

• General: "Cloud" will become the next "ESB" or "SOA", in that it will be something that everybody will talk about, but few will understand and even fewer will do anything with. (Considering the widespread disparity in the definition of the term, this seems like a no-brainer.)
• Java: Interest in Scala will continue to rise, as will the number of detractors who point out that Scala is too hard to learn.
• .NET: Interest in F# will continue to rise, as will the number of detractors who point out that F# is too hard to learn. (Hey, the two really are cousins, and the fortunes of one will serve as a pretty good indication of the fortunes of the other, and both really seem to be on the same arc right now.)
• General: Interest in all kinds of functional languages will continue to rise, and more than one person will take a hint from Bob "crazybob" Lee and liken functional programming to AOP, for good and for ill. People who took classes on Haskell in college will find themselves reaching for their old college textbooks again.
• General: The iPhone is going to be hailed as "the enterprise development platform of the future", and companies will be rolling out apps to it. Look for Quicken iPhone edition, PowerPoint and/or Keynote iPhone edition, along with connectors to hook the iPhone up to a presentation device, and (I'll bet) a World of Warcraft iPhone client (legit or otherwise). iPhone is the new hotness in the mobile space, and people will flock to it madly.
• .NET: Another Oslo CTP will come out, and it will bear only a superficial resemblance to the one that came out in October at PDC. Betting on Oslo right now is a fools' bet, not because of any inherent weakness in the technology, but just because it's way too early in the cycle to be thinking about for anything vaguely resembling production code.
• .NET: The IronPython and IronRuby teams will find some serious versioning issues as they try to manage the DLR versioning story between themselves and the CLR as a whole. An initial hack will result, which will be codified into a standard practice when .NET 4.0 ships. Then the next release of IPy or IRb will have to try and slip around its restrictions in 2010/2011. By 2012, IPy and IRb will have to be shipping as part of Visual Studio just to put the releases back into lockstep with one another (and the rest of the .NET universe).
• Java: The death of JSR-277 will spark an uprising among the two leading groups hoping to foist it off on the Java community--OSGi and Maven--while the rest of the Java world will breathe a huge sigh of relief and look to see what "modularity" means in Java 7. Some of the alpha geeks in Java will start using--if not building--JDK 7 builds just to get a heads-up on its impact, and be quietly surprised and, I dare say, perhaps even pleased.
• Java: The invokedynamic JSR will leapfrog in importance to the top of the list.
• Windows: Another Windows 7 CTP will come out, and it will spawn huge media interest that will eventually be remembered as Microsoft promises, that will eventually be remembered as Microsoft guarantees, that will eventually be remembered as Microsoft FUD and "promising much, delivering little". Microsoft ain't always at fault for the inflated expectations people have--sometimes, yes, perhaps even a lot of times, but not always.
• Mac OS: Apple will begin to legally threaten the clone market again, except this time somebody's going to get the DOJ involved. (Yes, this is the iPhone/iTunes prediction from last year, carrying over. I still expect this to happen.)
• Languages: Alpha-geek developers will start creating their own languages (even if they're obscure or bizarre ones like Shakespeare or Ook#) just to have that listed on their resume as the DSL/custom language buzz continues to build.
• XML Services: Roy Fielding will officially disown most of the "REST"ful authors and software packages available. Nobody will care--or worse, somebody looking to make a name for themselves will proclaim that Roy "doesn't really understand REST". And they'll be right--Roy doesn't understand what they consider to be REST, and the fact that he created the term will be of no importance anymore. Being "REST"ful will equate to "I did it myself!", complete with expectations of a gold star and a lollipop.
• Parrot: The Parrot guys will make at least one more minor point release. Nobody will notice or care, except for a few doggedly stubborn Perl hackers. They will find themselves having nightmares of previous lives carrying around OS/2 books and Amiga paraphernalia. Perl 6 will celebrate it's seventh... or is it eighth?... anniversary of being announced, and nobody will notice.
• Agile: The debate around "Scrum Certification" will rise to a fever pitch as short-sighted money-tight companies start looking for reasons to cut costs and either buy into agile at a superficial level and watch it fail, or start looking to cut the agilists from their company in order to replace them with cheaper labor.
• Flash: Adobe will continue to make Flex and AIR look more like C# and the CLR even as Microsoft tries to make Silverlight look more like Flash and AIR. Web designers will now get to experience the same fun that back-end web developers have enjoyed for near-on a decade, as shops begin to artificially partition themselves up as either "Flash" shops or "Silverlight" shops.
• Personal: Gartner will still come knocking, looking to hire me for outrageous sums of money to do nothing but blog and wax prophetic.

Well, so much for brief or short. See you all again next year....

When I was in college, at the University of California, Davis, I lived in the International Relations building (D Building in the Tercero dorm area, for any other UCD alum out there), and got my first real glimpse of the feminist movement up front. It seemed like it was filled with militant, angry members of the female half of the species, who insisted that their gender was spelled "womyn", so that it wasn't somehow derived from "man" (wo-man, wo-men, get it?), who blamed most of the world's problems on the fact that men were running the show, and that therefore, because of my own gender, I was to share equally in the blame for its ills.

Maybe I was--Lord knows I certainly wasn't an entirely nice guy back then (and some will chirp from the back of the theater, "back then?!?")--but it still left the whole "feminist" thing as something I couldn't really be around, much less support.

My sister, it turned out, had a different experience at University of California, Santa Cruz, and became one.

Needless to say, this made family get-togethers somewhat awkward.

Then, a few years later, she asked my help in purchasing a new computer for herself. Basically, she just wanted me along to help explain any of the technical terms that she wasn't entirely familiar with, and to give her some advice on whether they were important to her needs. Not an unreasonable request, and not something I wouldn't do for anybody else, male or female alike. (I sometimes wish my father would ask my help before buying, but that's another story.)

We went to the store, and I got my first lesson in sexual discrimination.

The entire time we were in the store, despite the fact that it was my sister asking the questions, despite the fact that I only answered questions that she asked of me directly (in other words, I was there to help her, not to help the sales guy sell to her), almost the entire conversation was spent with the sales guy talking to me, even if he was answering her question. His body language was unquestionably that of, "She's clearly not capable of making this decision herself", and addressed everything to me, despite her repeated attempts to catch his eye and have him talk to her, the actual purchaser with the question.

I was a bit taken aback. I don't think the sales guy even noticed. That bothered me more than anything.

Ever since that time, I've been curiously and cautiously trying to figure out why there aren't more women in IT.

Several theories have presented themselves over the years:

1. Women, aside from a statistical minority, are structurally incapable of mastering IT. This is the "math is hard" argument, and I think we can all pretty much agree where this one belongs.
2. Women are encouraged/forced down an educational path that leads them away from IT until much later in life. I've heard this from a couple of women my age, and while I think there may be some validity to it, at least back in the day, I don't know if there still is. I'd love to get some feedback from recent high school or college grads who can weigh in with some anecdotal evidence one way or the other.
3. Women are entering IT, but not in the areas that I hang out in. This is definitely possible, and I think is happening, to some degree. At an Adobe "Flex Camp" last night (I was Chet Haase's roadie for the evening), I noticed a far more even split of gender than I'd ever seen at a Java or .NET user group, and when I mentioned this to one of the other speakers, he nodded and said that women were far more prevalent in the "web design" space, which is clearly not a space I play much in. I've also heard that the system admin space is much more "female-friendly", too.
4. Women get in to IT, then out of it or stay "hidden" in it. I've heard the theory that some women choose to get out of IT because they're not willing to put the same kind of time and energy into it as some men are, or they choose to remain at the software developer level instead of trying to advance the corporate ladder into management or other more visible positions.
5. There are exactly the number of women in IT that want to be there. Hey, let's face it, maybe women just don't like software development, and that's OK, because there's a lot of jobs I don't like, either.

My concern is with theories 1 and 2. There should be no reason whatsoever that a woman cannot succeed every bit as much as a man can. This is one of those (few?) industries where the principal qualifications are entirely intellectual/mental, and that means there's absolutely no reason why one gender should be favored over another. (Nursing and teaching are others, for example.)

So, without further ado, those of you who are interested, check out Dana Coffey's link on the Lego Build event at the MSDN events coming up.

It amazes me how insular and inward-facing the software industry is. And how the "agile" movement is reaping the benefits of a very simple characteristic.

For example, consider Jeff Palermo's essay on "The Myth of Self-Organizing Teams". Now, nothing against Jeff, or his post, per se, but it amazes me how our industry believes that they are somehow inventing new concepts, such as, in this case the "self-organizing team". Team dynamics have been a subject of study for decades, and anyone with a background in psychology, business, or sales has probably already been through much of the material on it. The best teams are those that find their own sense of identity, that grow from within, but still accept some leadership from the outside--the classic example here being the championship sports team. Most often, that sense of identity is born of a string of successes, which is why teams without a winning tradition have such a hard time creating the esprit de corps that so often defines the difference between success and failure.

(Editor's note: Here's a free lesson to all of you out there who want to help your team grow its own sense of identity: give them a chance to win a few successes, and they'll start coming together pretty quickly. It's not always that easy, but it works more often than not.)

How many software development managers--much less technical leads or project managers--have actually gone and looked through the management aisle at the local bookstore?

Tom and Mary Poppendieck have been spending years now talking about "lean" software development, which itself (at a casual glance) seems to be a refinement of the concepts Toyota and other Japanese manufacturers were pursuing close to two decades ago. "Total quality management" was a concept introduced in those days, the idea that anyone on the production line was empowered to stop the line if they found something that wasn't right. (My father was one of those "lean" manufacturing advocates back in the 80's, in fact, and has some great stories he can tell to its successes, and failures.)

How many software development managers or project leads give their developers the chance to say, "No, it's not right yet, we can't ship", and back them on it? Wouldn't you, as a developer, feel far more involved in the project if you knew you had that power--and that responsibility?

Or consider the "agile" notion of customer involvement, the classic XP "On-Site Customer" principle. Sales people have known for years, even decades (if not centuries), that if you involve the customer in the process, they are much more likely to feel an ownership stake sooner than if they just take what's on the lot or the shelf. Skilled salespeople have done the "let's walk through what you might buy, if you were buying, of course" trick countless numbers of times, and ended up with a sale where the customer didn't even intend to buy.

How many software development managers or project leads have read a book on basic salesmanship? And yet, isn't that notion of extracting what the customer wants endemic to both software development and basic sales (of anything)?

What is it about the software industry that just collectively refuses to accept that there might be lots of interesting research on topics that aren't technical yet still something that we can use? Why do we feel so compelled to trumpet our own "innovations" to ourselves, when in fact, they've been long-known in dozens of other contexts? When will we wake up and realize that we can learn a lot more if we cross-train in other areas... like, for example, getting your MBA?

Dustin Campbell, a self-professed "IDE guy", is speaking at the .NET Developer's Association of Redmond this evening, on the future of Visual Basic in Visual Studio 2010, and I feel compelled, based on my earlier "dissing" of VB in my thoughts of PDC post, to give VB a little love here.

First of all, he notes publicly that the VB and C# teams have been brought together under one roof, organizationally, so that the two languages can evolve in parallel to one another. I have my concerns about this. Frankly, I think the Managed Languages team at Microsoft is making a mistake by making these two languages mirror images of one another, no matter what their customers are telling them; it's creating an artificial competition between them, because if you can't differentiate between the two on a technical level, then the only thing left to differentiate them on is an aesthetic level (do you prefer curly braces and semicolons, or keywords?). Unfortunately, the market has already done so, to the tune of "C# developers make more than VB developers do (on average)", leaving little doubt in the minds of VB developers where they'd rather be... and even less doubt in the minds of C# developers where they'd rather the VB developers remain, lest the supply and demand curves shift and move the equilibrium point of C# developer salaries further south.

Besides, think about this for a moment: how much time and energy has Microsoft (and other .NET authors) had to invest in making sure that every SDK and every article ever written has both C# and VB sample code? All because Microsoft refuses to simply draw a line in the sand and say, once and for all, "C# is the best statically-typed object-oriented language for the CLR on the planet, and Visual Basic is the best dynamically-typed object-oriented language for the CLR on the planet", and run with it. Then at least there would be solid technical reasons for using one or the other, and at least we could take this out of the realm of aesthetics.

Or, contrarily, do the logical thing and create one language with two parsers, switching between them based on the file extension. That guarantees that the two evolve in parallel, and releases resources from the languages team to work on other things.

Next, he shows some simple spin-off-a-thread code, with the Thread constructor taking a parameter to a function name, traditional delegate kinds of stuff, then notes the disjoint nature of referencing a method defined elsewhere in the class but only to be used once. Yes, he's setting up for the punchline: VB gets anonymous methods, and "VB's support for lambda (expressions) reaches parity with C#'s" in this next release. I don't know if this was a feature that VB really needed to get, since I don't know that the target audience for VB is really one that cares about such things (and, before the VB community tries to lynch me, let me be honest and say that I'm not sure the target audience for C# does, either), but at least it's nice that such a powerful feature is now present in the VB language. Subject to the concerns of last paragraph, of course.

Look, at the end of the day, I want C# and VB to be full-featured languages each with their own raison d'etre, as the French say, their own "reason to be". Having these two "evolve in parallel" or "evolve in concert" with one another is only bound to keep the C#-vs-VB language wars going for far too long.

Along the way, he's showing off some IDE features, which presumably will be in place for both C# and VB (since the teams are now unified under a single banner), what he's calling "highlights": they'll do the moral equivalent of brace matching/highlighting, for both method names (usage as well as declaration/definition) and blocks of code. There's also "pretty listing", where the IDE will format code appropriately, particularly for the anonymous methods syntax. Nice, but not something I'm personally going to get incredibly excited about--to me, IDE features like this aren't as important as language features, but I realize I'm in something of the minority there, and that's OK.

He demonstrates VB calling PLINQ (Parallel LINQ), pointing out some of the inherent benefits (and drawbacks) to parallelism. This isn't really a VB "feature" per se. <<MORE>>

Now he gets into some more interesting stuff: he begins by saying, "Now let's talk about the Dynamic Language Runtime (DLR)." He shows some VB code hosting the IronPython runtime, simple boilerplate to get the IronPython bits up and running inside this CLR process. (See the DLR Hosting Spec for details, it's pretty straightforward stuff: call IronPython.Hosting.Python.CreateRuntime, then call GetEngine("python") and SetSearchPaths() to tell IPy where to find the Python libs and code.) Where he's going with this is to demonstrate using VB's late-binding capabilities to get hold of a Python file ("random.py", using the DLR UseFile() call), and he dynamically calls the "shuffle" function from that Python file against the array of Ints he set up earlier.

(We get into a discussion as to why the IDE can't give Intellisense on the methods he's calling in the Python code. I won't go into the details, but essentially, no, VS isn't going to be able to do that, at least not for this scenario, any time soon. Maybe if the Python code was used directly from within VS, but not in this hosted sense--that would be a bit much for the IDE to analyze and understand.)

Next he points out some of the "ceremony" remaining in Visual Basic, essentially showing how VB's type inferencing is getting better, such as with array literals, including a background compilation warning where the VB compiler finds that it can't find a common type in the array literal declaration and assumes it to be an array of Object (which is a nice "catch" when the wrong type shows up in the array by accident or typo). He shows off multidimensional array literal and jagged array literal syntax (which requires the internal array literals in the jagged array to be wrapped up in parentheses, a la "{({1,2,3}), ({1, 2, 3, 4, 5})}", which I find a touch awkward and counterintuitive, quite frankly), while he's at it.

(We get into a discussion of finer-granularity color syntax highlighting options, such as colorizing different keywords differently, as well as colorizing different identifiers based on their type. Now that's an interesting idea.)

By the way, one thing that I've always found interesting about VB is its "With" keyword, a la "New Student With {.Id=101, .Name="bart", .Score=53, .Gender="male"}".

He then shows how VB 10 has auto-implemented properties: "Property Gender As String" does exactly what .NET programmers have had to do by hand for so long: create a field, generate simple Get and Set blocks and so on. Another nice feature of this: the autogenerated properties can have defaults, as in, "Public Property Age As Integer = 1". That's kinda nice, and something that VB should have had years ago.

And wahoo! THE UNDERSCORE IS (almost) HISTORY! "Implicit line completion" is a feature of VB 10. This has always plagued me like... well... the plague... when writing VB code. It's not gone completely, there's a few cases where ambiguity would reign without it, but it appears to be gone for 95% of the cases. Because this is such a radical change, they've even gone out and created a website to help the underscores that no longer find themselves necessary: www.unemployedunderscores.com .

He goes into a bit about co- and contravariance in generic types, which VB now supports more readily. (His example is about trying to pass a List(Of Student) into a method taking a List(Of Person), which neither he nor I can remember if it's co- or contra-. Sorry.) The solution is to change the method to take an IEnumerable(Of Person), instead. Not a great solution, but not a bad one, either.

... Corey Vidal, you have outdone every YouTube video I've ever seen, and I was a huge fan of "White and Nerdy".

John Williams, if you don't call this kid, you are missing out on some serious talent. To sing all four of those parts a capella and stitch them together like that, that's crazy.

Having freshly converted both the Visual Studio 2010 and Oslo SDK VPC images that we received at PDC 2008 last month to VMWare images, I figure it's time to dive into M.

At PDC, the Addison-Wesley folks were giving away copies of "The 'Oslo' Modeling Language" book, which is apparently official canon of the "M" language for Oslo, so I flip to page 1 and start reading:

The "Oslo" Modeling Language (M) is a modern, declarative language for working with data. M lets users write down how they want to structure and query their data using a convenient textual syntax that is convenient to both author and read.

M does not mandate how data is stored or accessed, nor does it mandate a specific implementation technology. Rather, M was designed to allow users to write down what they want from their data without having to specify how those desires are met against a given technology or platform. That stated, M in no way prohibits implementations from providing rich declarative or imperative support for controlling how M constructs are represented and executed in a given environment.

Hmm... I have to admit, all kinds of warning bells and alarm flags are going off in my head, and we're just two sentences into this thing. This sounds like something we've all done before; in fact, though I've not tried it, I have a feeling that if we were to go back through those two paragraphs and replace every instance of "M" with "SQL", we'd find a paragraph that could easily slip into the opening chapter of any introductory SQL or RDBMS book.

The goals of "separation of declaration from intent" have been around for that long, probably longer, and even the fiercest and staunchest defenders of SQL find themselves sometimes wandering through SQL declarations and code that clearly violate Chris Date's politely-worded commands around normal form and separation of declaration from intent and implementation.

I keep reading, though, and a few paragraphs later, find something intriguing.

Another important aspect of data management that M does not address is that of update. M is a functional language that does not have constructs for changing the contents of an extent. (Author's note: an "extent", defined a few paragraphs earlier, is that "an extent provides dynamic storage for values.") How data changes is outside the scope of the language. That said, M anticipates that the contents of an extent can change via external (to M) stimuli. Subsequent versions of M are expected to provide declarative constructs for updating data.

Wow. So the first question becomes, when are those "subsequent versions" expected? Is this simply a state of the PDC Preview bits, or something that's not in scope for v1 of the Oslo SDK?

I flip through the rest of the first chapter, which seems like a decent overview, and what I see there is an interesting type-declaration language; in many ways, it's highly reminiscent of XML Schema Descriptions (XSD) more than SQL declarations, but I suppose that's to be expected, at least for now. I'm sure they're going to cherry-pick a lot of the best data-declarative constructs from XSD, SQL, and any other metadata-based formats/languages, and that the semantics will change as they explore what works well and what doesn't. For now, though, "M" exists essentially as a data-descriptor language, and this is reinforced when I start playing with "m.exe", the "M compiler" (?).

First thing, I simply fire up "m.exe" to see what the options are. And... nothing. Huh? I wait for a bit, then Ctrl-C it, and start hunting through the documentation to see if I'm missing something here. I try a few different tests, like "m /?" or "m -help", and each time, the compiler just seems to wander off into the weeds, requiring a Ctrl-C to kill it.

What the heck? I know that these are PDC pre-alpha CTP "nothing is guaranteed to work" bits, but this seems a bit on the excessive side--I have every faith that Microsoft wouldn't hand these out if you can't even run the compiler! So acting on a hunch, I fire up "m /?" again, and tab away to look at something else. Sure enough, my hunch is rewarded--after a long pause, eventually the help screen comes up. So, apparently, the m.exe tool just takes fricken forever to run, is all.

Currently, the only targets M can compile to is their internal Repository for storing types, and a generic "T-SQL" target for any T-SQL-compliant database (which I presume for now means only SQL Server of various versions, but theoretically, I suppose, Sybase could work too, given those two systems' shared ancestry. And, given a pretty simple sample to work with, m.exe produces a pretty-easily-anticipated result; this:

module Ted
{
  type Person
  {
    Id : Integer32 = AutoNumber();
    Name : Text;
  } where identity Id;
  People : Person*;    
}

turns into this:

set xact_abort on;
go

begin transaction;
go

set ansi_nulls on;
go

create schema [Ted];
go

create table [Ted].[People]
(
  [Id] int not null identity,
  [Name] nvarchar(max) not null,
  constraint [PK_People] primary key clustered ([Id])
);
go

commit transaction;
go

... which, when you look at it, is pretty much what you'd want.

Interestingly enough, there's no reason why people in the Java or Ruby space couldn't use "M" just as easily, so long as the database targeted is one that M understands. (It also wouldn't be a terribly difficult exercise to build an M compiler in Java or Ruby, for that matter. Might be a fun off-time project, in fact.)

One thing that's also pretty clear is that M is very collection-centric, as the first chapter spends probably 50% of its time describing all the various ways that collections in M (written as "{a, b, c}") interact with one another (they can be compared for equality directly, for example, and have some neat projection/filter capabilities that were clearly drawn from the relational algebra and LINQ syntax). Having said that, though, one thing that is obviously missing is the traditional object "reference"-style connection, where A OWNS-A B.

What this seems to imply, then, is that the object/relational-mapping horrors of the past two decades aren't yet over. What's not clear is how M will make it easier (or if it will at all) to access those extents from the languages we traditionally use in the .NET space (C#, VB, C++/CLI, etc), specifically, what the mechanism for conducting a query will be like, and what it's return types will be when it cross the boundary back into C#.

If you're not sure what I mean by that, consider it this way: ADO.NET has a simple mechanism for taking the query--a raw string as a parameter--and executing it, and when it returns, it's handed back to your C# code as a DataSet, or else as an IDataReader for row-based/column-based firehose-style consumption. Much of the criticism of ADO.NET stems around two parts: the untyped nature of the query string, leading to potential typos and errors, and the relative awkwardness for extracting the data from the results, either the DataSet or the IDataReader, at least when compared to languages that have built-in set/tuple constructs.

The one sample that does show any sort of C# -> M kinds of interaction is in the MParserDemo sample, and here, when it queries the database, it does so using traditional ADO.NET API calls, so I'm not sure it's to be taken as a good indicator of the plans around M yet.

If all there was to Oslo was "M", I'd say it was an interesting little side-note at PDC, something that maybe a few folks might find interesting and otherwise not worth studying, but this is not the sum total of the Oslo bits; there is also Mg, the MGrammar language, a language specifically for building DSLs, and that's where my attention (and next blog post) is going next.

Roy Fielding has weighed in on the recent "buzzwordiness" (hey, if Colbert can make up "truthiness", then I can make up "buzzwordiness") of calling everything a "REST API", a tactic that has become more en vogue of late as vendors discover that the general programming population is finding the WSDL-based XML services stack too complex to navigate successfully for all but the simplest of projects. Contrary to what many RESTafarians may be hoping, Roy doesn't gather all these wayward children to his breast and praise their anti-vendor/anti-corporate/anti-proprietary efforts, but instead, blasts them pretty seriously for mangling his term:

I am getting frustrated by the number of people calling any HTTP-based interface a REST API. Today’s example is the SocialSite REST API. That is RPC. It screams RPC. There is so much coupling on display that it should be given an X rating.

Ouch. "So much coupling on display that it should be given an X rating." I have to remember that phrase--that's a keeper. And I'm shocked that Roy even knows what an X rating is; he's such a mellow guy with such an innocent-looking face, I would've bet money he'd never run into one before. (Yes, people, that's a joke.)

What needs to be done to make the REST architectural style clear on the notion that hypertext is a constraint? In other words, if the engine of application state (and hence the API) is not being driven by hypertext, then it cannot be RESTful and cannot be a REST API. Period. Is there some broken manual somewhere that needs to be fixed?

Go Roy!

For those of you who've not read Roy's thesis, and are thinking that this is some kind of betrayal or trick, let's first of all point out that at no point is Roy saying that your nifty HTTP-based API is not useful or simple. He's simply saying that it isn't RESTful. That's a key differentiation. REST has a specific set of goals and constraints it was trying to meet, and as such prescribes a particular kind of architectural style to fit within those constraints. (Yes, REST is essentially an architectural pattern: a solution to a problem within a certain context that yields certain consequences.)

Assuming you haven't tuned me out completely already, allow me to elucidate. In Chapter 5 of Roy's thesis, Roy begins to build up the style that will ultimately be considered REST. I'm not going to quote each and every step here--that's what the hyperlink above is for--but simply call out certain parts. For example, in section 5.1.3, "Stateless", he suggests that this architectural style should be stateless in nature, and explains why; the emphasis/italics are mine:

We next add a constraint to the client-server interaction: communication must be stateless in nature, as in the client-stateless-server (CSS) style of Section 3.4.3 (Figure 5-3), such that each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client.

This constraint induces the properties of visibility, reliability, and scalability. Visibility is improved because a monitoring system does not have to look beyond a single request datum in order to determine the full nature of the request. Reliability is improved because it eases the task of recovering from partial failures [133]. Scalability is improved because not having to store state between requests allows the server component to quickly free resources, and further simplifies implementation because the server doesn't have to manage resource usage across requests.

Like most architectural choices, the stateless constraint reflects a design trade-off. The disadvantage is that it may decrease network performance by increasing the repetitive data (per-interaction overhead) sent in a series of requests, since that data cannot be left on the server in a shared context. In addition, placing the application state on the client-side reduces the server's control over consistent application behavior, since the application becomes dependent on the correct implementation of semantics across multiple client versions.

In the HTTP case, the state is contained entirely in the document itself, the hypertext. This has a couple of implications for those of us building "distributed applications", such as the very real consideration that there's a lot of state we don't necessarily want to be sending back to the client, such as voluminous information (the user's e-commerce shopping cart contents) or sensitive information (the user's credentials or single-signon authentication/authorization token). This is a bitter pill to swallow for the application development world, because much of the applications we develop have some pretty hefty notions of server-based state management that we want or need to preserve, either for legacy support reasons, for legitimate concerns (network bandwidth or security), or just for ease-of-understanding. Fielding isn't apologetic about it, though--look at the third paragraph above. "[T]he stateless constraint reflects a design trade-off."

In other words, if you don't like it, fine, don't follow it, but understand that if you're not leaving all the application state on the client, you're not doing REST.

By the way, note that technically, HTTP is not tied to HTML, since the document sent back and forth could easily be a PDF document, too, particularly since PDF supports hyperlinks to other PDF documents. Nowhere in the thesis do we see the idea that it has to be HTML flying back and forth.

Roy's thesis continues on in the same vein; in section 5.1.4 he describes how "client-cache-stateless-server" provides some additional reliability and performance, but only if the data in the cache is consistent and not stale, which was fine for static documents, but not for dynamic content such as image maps. Extensions were necessary in order to accomodate the new ideas.

In section 5.1.5 ("Uniform Interface") we get to another stinging rebuke of REST as a generalized distributed application scheme; again, the emphasis is mine:

The central feature that distinguishes the REST architectural style from other network-based styles is its emphasis on a uniform interface between components (Figure 5-6). By applying the software engineering principle of generality to the component interface, the overall system architecture is simplified and the visibility of interactions is improved. Implementations are decoupled from the services they provide, which encourages independent evolvability. The trade-off, though, is that a uniform interface degrades efficiency, since information is transferred in a standardized form rather than one which is specific to an application's needs. The REST interface is designed to be efficient for large-grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction.

In order to obtain a uniform interface, multiple architectural constraints are needed to guide the behavior of components. REST is defined by four interface constraints: identification of resources; manipulation of resources through representations; self-descriptive messages; and, hypermedia as the engine of application state. These constraints will be discussed in Section 5.2.

In other words, in order to be doing something that Fielding considers RESTful, you have to be using hypermedia (that is to say, hypertext documents of some form) as the core of your application state. It might seem like this implies that you have to be building a Web application in order to be considered building something RESTful, so therefore all Web apps are RESTful by nature, but pay close attention to the wording: hypermedia must be the core of your application state. The way most Web apps are built today, HTML is clearly not the core of the state, but merely a way to render it. This is the accidental consequence of treating Web applications and desktop client applications as just pale reflections of one another.

The next section, 5.1.6 ("Layered System") again builds on the notion of stateless-server architecture to provide additional flexibility and power:

In order to further improve behavior for Internet-scale requirements, we add layered system constraints (Figure 5-7). As described in Section 3.4.2, the layered system style allows an architecture to be composed of hierarchical layers by constraining component behavior such that each component cannot "see" beyond the immediate layer with which they are interacting. By restricting knowledge of the system to a single layer, we place a bound on the overall system complexity and promote substrate independence. Layers can be used to encapsulate legacy services and to protect new services from legacy clients, simplifying components by moving infrequently used functionality to a shared intermediary. Intermediaries can also be used to improve system scalability by enabling load balancing of services across multiple networks and processors.

The primary disadvantage of layered systems is that they add overhead and latency to the processing of data, reducing user-perceived performance [32]. For a network-based system that supports cache constraints, this can be offset by the benefits of shared caching at intermediaries. Placing shared caches at the boundaries of an organizational domain can result in significant performance benefits [136]. Such layers also allow security policies to be enforced on data crossing the organizational boundary, as is required by firewalls [79].

The combination of layered system and uniform interface constraints induces architectural properties similar to those of the uniform pipe-and-filter style (Section 3.2.2). Although REST interaction is two-way, the large-grain data flows of hypermedia interaction can each be processed like a data-flow network, with filter components selectively applied to the data stream in order to transform the content as it passes [26]. Within REST, intermediary components can actively transform the content of messages because the messages are self-descriptive and their semantics are visible to intermediaries.

The potential of layered systems (itself not something that people building RESTful approaches seem to think much about) is only realized if the entirety of the state being transferred is self-descriptive and visible to the intermediaries--in other words, intermediaries can only be helpful and/or non-performance-inhibitive if they have free reign to make decisions based on the state they see being transferred. If something isn't present in the state being transferred, usually because there is server-side state being maintained, then they have to be concerned about silently changing the semantics of what is happening in the interaction, and intermediaries--and layers as a whole--become a liability. (Which is probably why so few systems seem to do it.)

And if the notion of visible, transported state is not yet made clear in his dissertation, Fielding dissects the discussion even further in section 5.2.1, "Data Elements". It's too long to reprint here in its entirety, and frankly, reading the whole thing is necessary to see the point of hypermedia and its place in the whole system. (The same could be said of the entire chapter, in fact.) But it's pretty clear, once you read the dissertation, that hypermedia/hypertext is a core, critical piece to the whole REST construction. Clients are expected, in a RESTful system, to have no preconceived notions of structure or relationship between resources, and discover all of that through the state of the hypertext documents that are sent back to them. In the HTML case, that discovery occurs inside the human brain; in the SOA/services case, that discovery is much harder to define and describe. RDF and Semantic Web ideas may be of some help here, but JSON can't, and simple XML can't, unless the client has some preconceived notion of what the XML structure looks like, which violates Fielding's rules:

A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API). From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations. The transitions may be determined (or limited by) the client’s knowledge of media types and resource communication mechanisms, both of which may be improved on-the-fly (e.g., code-on-demand). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]

An interesting "fuzzy gray area" here is whether or not the client's knowledge of a variant or schematic structure of XML could be considered to be a "standardized media type", but I'm willing to bet that Fielding will argue against it on the grounds that your application's XML schema is not "standardized" (unless, of course, it is, through a national/international/industry standardization effort).

But in case you'd missed it, let me summarize the past twenty or so paragraphs: hypermedia is a core requirement to being RESTful. If you ain't slinging all of your application state back and forth in hypertext, you ain't REST. Period. Fielding said it, he defined it, and that settles it.

Before the hate mail comes a-flyin', let me reiterate one vitally important point: if you're not doing REST, it doesn't mean that your API sucks. Fielding may have his definition of what REST is, and the idealist in me wants to remain true to his definitions of it (after all, if we can't agree on a common set of definitions, a common lexicon, then we can't really make much progress as an industry), but...

... the pragmatist in me keeps saying, "so what"?

Look, at the end of the day, if your system wants to misuse HTTP, abuse HTML, and carnally violate the principles of loose coupling and resource representation that underlie REST, who cares? Do you get special bonus points from the Apache Foundation if you use HTTP in the way Fielding intended? Will Microsoft and Oracle and Sun and IBM offer you discounts on your next software purchases if you create a REST-faithful system? Will the partisan politics in Washington, or the tribal conflicts in the Middle East, or even the widely-misnamed "REST-vs-SOAP" debates come to an end if you only figure out a way to make hypermedia the core engine of your application state?

Yeah, I didn't think so, either.

Point is, REST is just an architectural style. It is nothing more than another entry alongside such things as client-server, n-tier, distributed objects, service-oriented, and embedded systems. REST is just a tool for thinking about how to build an application, and it's high time we kick it off the pedastal on which we've placed it and let it come back down to earth with the rest of us mortals. HTTP is useful, but not sufficient, so solve our problems. REST is as well.

And at the end of the day, when we put one tool from our tool belt "above all others", we end up building some truly horrendous crap.

Recently, a blog reader asked me if I wasn't doing any speaking any more since I'd joined ThoughtWorks, and that's when I realized I'd been bad about updating my speaking calendar on the website. Sorry, all; no, ThoughtWorks didn't pull my conference visa or anything, I've just been bad about keeping it up to date. I'll fix that ASAP, but in the meantime, three events that I'll be at in the coming wintry months include:

Øredev 2008: 19 - 21 November, Malmoe, Sweden

Øredev will be a first for me, and I've ben invited to give a keynote there, along with a few technical sessions. I'm also told that .NET Rocks! will be on hand, and that they want to record a session, on whichever topic happens to cross the curious, crafty and cunning Carl, or the uh... the uh... sorry, Richard, there's just no good "R" adjectives I can use here. I mean, "rough" and "ready" don't exactly sound flattering in this context, right? Sorry, man.

In any event, I'm looking forward to this event, because it's a curious mix of technologies and ideas (agile, ALT.NET, Java, core .NET, languages, and so on), and because I've never been to Sweden before. One more European country, off my bucket list!

(Yes, I had to cut-and-paste the Ø wherever I needed it. *grin*)

DevTeach 2008: 1 - 5 December, Montreal, Quebec (Canada)

This has been one of my favorite shows since it began, way back in 2003, and a large part of that love has to do with the cast and crew of characters that I see there every year: Julie Lerman, Peter DeBetta, Carl and Richard (again!), Beth Massi, "Yag" Griver, Mario Cardinal and the rest of the Quebecois posse, Ayende, plus some new faces and friends, like Jessica Moss and James Kovacs. (Oh, and for the record, folks, for those of you who are still talking about it, the O/R-M smackdown of a year ago was staged. It was all fake. Ayende and I are really actually friends, we were paid a great deal of money by Carl and Richard to make it sound good, and in fact, we both agree that the only place anybody should really ever store their data is in an XML database.)

If you're near Montreal, and you're a .NET dev, you really owe it to yourself to check this show out.

Update: I just got this email from Jean-Rene, the guy who runs DevTeach:

Every attendees will get Visual Studio 2008 Pro, Expression Web 2 and Tech-Ed DEV set in their bag!

DevTeach believe that all developers need the right tool to be productive. This is what we will give you, free software, when you register to DevTeach or SQLTeach. Yes that right! We’re pleased to announce that we’re giving over a 1000$ of software when you register to DevTeach. You will find in your conference bag a version of Visual Studio 2008 Professional, ExpressionTM Web 2 and the Tech-Ed Conference DVD Set. Is this a good deal or what? DevTeach and SQLTeach are really the training you can’t get any other way.

Not bad. Not bad at all.

DeVoxx 2008: 8 - 12 December, Antwerp, Belgium

DeVoxx, the recently-renamed-formerly-named-JavaPolis conference, has brought me back to team up with Bill Venners to do a University session on Scala, and to record a few more of those Parlays videos that people can't seem to get enough of. Given that this show always seems to draw some of the Java world's best and brightest, I'm definitely looking forward to the chance to point the mike at somebody's grill and give 'em hell! Plus, I love Belgium, and I'm looking forward to getting back there. The fact that it's going to be the middle of winter is only a bonus, as... wait... Belgium, in the middle of winter? Whose bright idea was that?

(And finally, a show that Carl and Richard won't be at!)

Meanwhile, I promise to keep the "Upcoming Events" up to date for 2009. Seriously. I mean it.

Having created a Window7 VMWare image (which I then later cloned and installed the Windows7 SDK into, successfully, wahoo!), I turned to the Visual Studio 2010 bits they provided on the hard drive. Not surprisingly, though a bit frustratingly, they didn't give us an install image that I could put into a VMWare image of my own creation, but instead gave us a VPC with everything pre-installed in it.

I know that Microsoft prefers to promote its own products, and that it's probably a bit much to ask them to provide both a VMWare image and a VirtualPC image for these kind of pre-alpha things, but it's a bit of a pain considering that Virtual PC doesn't run anymore on the Mac, that I'm aware of. Please, Microsoft, a lot of .NET devs are carrying around MacBookPro machines these days, and if you're really focused on trying to get bits in the hands of developers, it would be quite the bold move to provide a VMWare image right next to the VPC image. Particularly since over half the drive was unused.

So... I don't want to have to carry around a PC (though I do at the moment) just to run VirtualPC just to be able to explore VS 2010, but fortunately VMWare provides a Converter application that can take a VPC image and flip it over to a VMWare image. Sounds like a plan. I fire up the Converter, point it at the VPC, and after the world's... slowest... wizard... takes... my... settings... and... begins... I discover that it will take upwards of 3 hours to convert. Dear God.

I decided to go to bed at that point.

When I woke up, the image had been converted successfully, but I wasn't quite finished yet. First of all, fire it up to make sure it runs, which it does without a problem, but at 640x480 in black-and-white mode (no, seriously, it's not much more than that). Install the VMWare Tools, reboot, and...

... the mouse cursor disappears. WTF?!?

Turns out this has been a nagging problem with several versions of VMWare over the years, and I vaguely remember running into the problem the last time I tried to create a Windows Server 2003/2008 image, too. Ugh. Hunting around the Web doesn't reveal an easy solution, but a couple of things do show up a few times: disconnect the CD-ROM, change the mouse pointer acceleration, delete the VMWare Mouse driver and let Windows rediscover the standard PS/2 mouse driver, or change the display hardware acceleration.

Not being really interested in debugging the problem (I know, my chance at making everybody's life better is now forever lost), I decided to take a bit of a shotgun approach to the problem. I explicitly deleted the VMWare Mouse driver, fiddled with the display settings (including resizing it to a more respectable 1400x1050), turned display hardware acceleration down, couldn't find mouse hardware acceleration settings, allowed it to reboot, and...

... yay. I have a mouse pointer again.

Now I have a VS2010 image on my Drive-o'-Virtual-Machines, and with it I plan on exploring the VS2010/C# 4.0/C++ 10/VB 10 bits some more. I fire up Visual Studio 2010, intending to poke around C# 4.0's new "dynamic" keyword and see if and how it builds on top of the DLR (as a few people have suggested in comments in prior posts). VS comes up pretty quickly (not bad for a pre-alpha), the new interface seems snappy, and I create the ubiquitous "ConsoleApplicationX" C# app.

Wait a minute...

Something niggled at the back of my head, and I went back to File | New Project, and ... something's missing.

There's no "Visual F#" tab. There's an item in the "Project types:" box on the left for Visual Basic, Visual C#, Visual C++, WiX, Modeling Projects, Database Projects, Other Project Types, and Test Projects, but no Visual F#. (And no, it doesn't show up under "Other Project Types" either, I checked.) Considering that my understanding was that F# was going to ship with VS 2010, I'm a little puzzled as to its absence. Hopefully this is just a temporary oversight.

In the meantime, I'm off to play with "dynamic" a bit more and see what comes out of it. But guys, please, let's see some F# love out of the box? Surely, if you can ship WiX with it, shipping F# can't be hard?

The organizers of TSSJS 2009 have asked me to serve as the Languages track chair, and as long-time readers of this blog will already have guessed, I've accepted quite happily. This means that if you're interested in presenting at TSSJS on a language-on-the-JVM, you now know where to send the bottle of Macallan 18.

Having said that (in jest, of course--bribes have to be at least a Macallan 25 or Macallan Cask Strength to have any real effect), I'm curious to get a sense of what languages--and what depth in each--people are interested in seeing presented there. Groovy, JRuby and Scala are obvious suggestions, but how deep would people be interested in seeing these? Would you prefer to see more languages at a shallower depth, or going really deep on a few?

(Disclaimer: emails sent to me directly or comments on this blog will weigh in on my decision-making process, but don't necessarily count as submitted abstracts; make sure you send them via the "official" channels to ensure they get considered, particularly since some proposals will be "borderline" on several different tracks, and thus could conceivably make it in via a different track than mine.)

Y'all know how to reach me....

Update: The deadline for abstracts is November 19th, so make sure to check out the website when it goes live (Nov 11th), and if you can't figure out how to submit an abstract, send it to me directly....