Interoperability Happens - Saturday, August 15, 2009

More on journalistic integrity: Sys-Con, Ulitzer, theft and libel

Recently, an email crossed my Inbox from a friend who was concerned about some questionable practices involving my content (as well as a few others'); apparently, I have been listed as an "author" for SysCon, I have a "domain" with them, and that I've been writing for them since 10 January, 2003, including two articles, "Effective Enterprise Java" and "Java/.NET Interoperability".

Given that both of those "articles" are summaries from presentations I've done at conferences past, I'm a touch skeptical. In fact, it feels like those summaries were scraped from conferences I've done in the past, and I certainly don't remember ever giving Sys-Con (or any other conference) the right to reprint my presentation as an article.

Then it turns out that apparently I'm not the only one suffering this problem. Go. Read that article, then come back. I promise, I'll wait.

(Seriously, go read it.)

Wow. Just... wow. If even half of what Aral's story is true (and I'm inclined to believe at least part of it, given that he's done some pretty meticulous documentation of at least his side of the story), then this is beyond outrageous, and squarely into "completely unethical".

Now, I'll be the first to admit, I've not heard back from Sys-Con about any of this, so if I get any sort of response I'll be sure to update this blog post. But...

Calling anyone a "homosexual son of a bitch", "terrorist" or "fag" is so unbelievably offensive it staggers the mind. Normally, I'd be a bit hesitant to just give either party the benefit of the doubt on that one, given just how ludicrous the accusation sounds, but Aral includes screen shots of the articles, which in of itself lends an air of credibility to the accusation—either Aral is the world's worst Turkish translator, or Sys-Con's translation into Turkish is a bit on the "edgy" side, or Sys-Con really did call him that. Which implies that whichever way this goes, doesn't look good for one of the two parties. But even if we leave that to one side....

Sys-Con is playing with fire by collecting my content and claiming me as an author. Sys-Con never contacted me about becoming a part of their "Ulitzer" website. They never asked me for permission to reprint my articles, though, I'll admit, I can't find where the articles actually exist, nor links to the articles, so maybe they didn't, actually, reprint the article, but just link to them... except I can't find the links to the articles or the presentations, either. They never asked me for an updated bio or photo, and in fact, they pretty clearly grabbed both bio, photo and "summaries" from an old location, because that bio lists me as a DevelopMentor instructor (which I haven't been for two years or so), and as living in Sacramento, CA (which I haven't been for about three years or so). Let me be very clear about this: I do not write for Sys-Con Media. I never have. They have never asked permission to reuse any of the content I have produced. I am appalled at being included in such a fashion.

Note that I'm not opposed to being linked to, mind you—if I put material on my blog, I generally expect (and hope) that people will link to it, and I don't demand permission or even notification when it happens. But to claim that I've written material for an entity does mean I expect to at least be asked if it's OK to use my likeness, name, or material. No such request was ever made of me, so far as I can remember or find (through my own email archives, which stretch back to 2001).

And I can say that I've thought about this issue before, from the other side of the story—back when I was editor at TheServerSide.NET, we began a "blogger's program" that would take interesting blog posts from around the Internet and "collect" them in some fashion for TSS.NET readers. Originally, the thought was to simply reproduce the content directly on our site, and I hated that idea, for the same reasons as I dislike it when somebody does it to me. Regardless of the licensing model the blog entries are published under, to me, a publication or media firm owes the author at least the right of refusal, and a chance to be notified when their material is reused. (In the end, we chose to ask authors if we could reproduce their material in the program, and we never (to my knowledge) had an author refuse.) It doesn't take a real rocket scientist's brain to figure out that asking permission is never a bad thing to do if you want to maintain good will with your sources of material.

This is an open and public request to Sys-Con media: either contact me about using my name, likeness and material on your website, or remove it. (I have emailed their editorial and asked them to acknowledge receipt of my request.)

In the meantime, I will be making every effort to make sure that other content-producers I know are aware of Sys-Con's practices, so they can act as they see fit.

If you are a reader, and find this distasteful as well, then I suggest you follow some of the suggestions mentioned in Aral's blog post:

Tell everyone you know about what Sys-Con is doing (but don't link to them so as not to give them Google Juice). If tweeting, leave out the http:// bit so that your URL is not automatically made into a link.
Sys-Con feeds upon the work of authors and speakers to live. If all authors had their content removed from Sys-Con and Ulitzer, they would not have pages to put ads on. So go through their list of authors and notify the ones you know. If they are unaware that they're listed there, they will most likely want themselves removed. Update: I've created a single list of all Sys-Con's Ulitzer authors. More information and the full list are in this post. The original list of authors is at http://www.ulitzer.com/?q=authors. You can ask for your Ulitzer/Sys-Con author page to be removed by emailing editorial@sys-con.com.
Contact their advertisers and tell them what you think of their association with Sys-Con.
If you know any speakers speaking at Sys-Con events, make sure they know the kind of company they are associating themselves with. Do the same with anyone you know who is thinking of attending one of their events. Raise awareness about their events at your place of work.
Make sure Google knows that Sys-Con/Ulitzer is spamming Google with tons of duplicate content. Report them on Google's spam page for posting duplicate content. According to their terms and conditions, Google should stop indexing Sys-Con/Ulitzer. See this comment for a template you can use when reporting them.
Make sure Google News knows that they are syndicating libelous articles from Sys-Con. Use the Google News Report an Issue form to report the following articles: http://internetvideo.sys-con.com/node/1017038, http://internetvideo.sys-con.com/node/1028923, http://www.sys-con.com/node/1035252, http://air.ulitzer.com/node/1038383, http://openwebdeveloper.sys-con.com/node/1039556, and http://cloudcomputing.sys-con.com/node/1047589

Meanwhile, I'm going to be talking about this to everybody I know at Microsoft, desperately seeking to find out which department engaged the advertising with Sys-Con, and looking to convince them that they don't need this kind of press or association. Ditto for the contacts (far fewer in number) I have with IBM, and any other Sys-Con advertiser I find.

What is "news", and what is "unethical"?

This post from TechCrunch crossed my attention inbox today, and I find myself quite flummoxed on the subject of how I think I should react.

Assume you have managed, through no overt work on your part (meaning, you didn't explicitly solicit, ask, or otherwise endeavor to obtain), to get ownership of "hundreds of confidential corporate and personal documents" for a company. Assume further that these documents are genuine—there is little to no chance that they could have been forged or fabricated. The documents span a range of sensitivity, from documents that are "somewhat embarrassing to various individuals, but not otherwise interesting", to documents that "show floorplans and security passcodes to get into the Twitter offices", to documents "showing financial projections, product plans and notes from executive strategy meetings". In other words, documents that yes, could create a certain amount of havoc to the corporate entity in question, could embarrass individuals within (and not within) that company, and documents that could lead to a competitive advantage for the entity's competitors.

Now also assume, for the purpose of the discussion, that you are an entity whose business model or raison d'etre is to publish—you are a blogger, a "social networking maven", a media outlet, whatever.

Is it unethical to publish these documents? Is it simply trolling for hits? Is there a "journalistic responsibility" to publish this material?

The people from TechCrunch feel like they have a right/responsibility to publish at least some of the documents, and are unswayed by the arguments in the blog's comments about the morality of such a move, including such comments as "This is an a**hole move" and "there's still an appearance of lapse of ethics here" (and that's just within the first half-dozen comments or so". What is particularly interesting is the response from (someone I assume to be) one of the blog's owners:

lol. if we only posted things that companies gave us permission to post this would be a press release site and none of you would be here. News is stuff someone doesn’t want you to write. The rest is advertising.

This comment disturbs me on several levels—it's only news if it's "stuff someone doesn't want you to write"? That's a pretty shallow and narrowly-defined sense of the term, if you ask me, and it puts periodicals like National Enquirer and Star magazines on the same level as the New York Times and CNN. (Although, and I'll freely admit this, having just come through the Michael Jackson media blitz, sometimes it feels difficult to tell the difference between all four of those.)

At the same time, though, it's clear from our own history that journalism has served the public good by shining a bright light into shady corners that some powers-that-be would prefer left unexposed. The abuses described by Upton Sinclair in the turn-of-the-century factories, the rampant sexual harassment in the military exposed by the Tailhook scandal, and certainly the outright blatantly violent suppression of Civil Rights movement of the 60's in the South were all shining examples of journalism at its finest, showing off dark and ugly parts of the world and—either implicitly or explicitly—demanding society to acknowledge it and either openly accept it or strive to change it (with all three of my examples seeing society choosing the latter).

What is "journalistic responsibility" here?

In our chosen field—that of computer science and software—there is clearly a responsibility for those "in the know" to reveal scenarios where information is being purloined or made available that violates individuals' rights to privacy. It's one thing if I trade my personal sales habits to a grocery store chain in exchange for a percentage off the final sale. That's a choice I'm making, consciously and knowingly. (By this point, if you haven't figured that out, you're just deliberately hiding from the fact.) But for somebody else to disclose my purchasing history without my consent to another party, that's brushing a very ugly moral dark area. And if a company is choosing to take its customers' personal data and make it available for anyone else to use as they see fit—for whatever purpose that third party can imagine—then cheers and kudos to the whistle-blower who brings media attention on that behavior.

But Twitter doesn't have much of my personal data, and they certainly didn't give it away to anybody—it was stolen from them, according to what I've read so far. What's more, I don't really have that much personal data stored with them—certainly no credit cards, birthdates, financial or medical information, or even family notes. What's there is actually pretty tame, as a Twitter customer.

(Twitter employees are a totally different matter. Admittedly. But let's just stick with the Twitter customer data for now.)

So where is the "journalistic responsibility" in publishing this material?

And are bloggers journalists? Should they be held to the same standards as journalists? And if not, then with all these formerly print-only media moving to the Internet and putting more and more of their material online, where do we draw that line? What's the difference between Fareed Zakaria writing a column on Middle East affairs for Newsweek.com on a monthly basis and Joe Sixpack posting a monthly rant on the illegal and illicit activities of his hometown rival's sports team? Is it just the domain name? And if Joe Sixpack decides to say, point blank, "TechCrunch paid for that material, they hired the guy who broke into the Twitter offices and stole it" on his blog, what avenues does TechCrunch have to decry and/or reverse that trend?

For the record, I oppose what TechCrunch is doing except if there is some blatantly legal violation of consumers' privacy. Frankly, if the hacker had approached me with those documents, I'd be working with the FBI to see the guy tossed in jail, because folks, if he did it to them, he could just as easily do it to you.

But this still leaves the deeper question about where bloggers sit in the journalistic continuum, and I admit, I have a lot of mixed feelings on the subject.

Industry | Reading | Review | Social

Wednesday, July 15, 2009 1:35:50 AM (Pacific Daylight Time, UTC-07:00)
Comments [4] |

Thoughts on the Chrome OS announcement

Google made the announcement on Tuesday: Chrome OS, a "open source, lightweight operating system that will initially be targeted at netbooks."

Huh?

I'm sorry, but from a number of perspectives, this move makes no sense to me.

Don't get me wrong—on a number of levels, the operating system needs a little shaking up. Windows7 looks good, granted, Mac OS is a strong contender, and both are clearly popular with the consuming public, but innovation in the operating system seems pretty limited right now, to eye candy graphical window-opening/window-closing effects, different window decorations (title bars and minimize/maximize buttons), and areas along the edges of the screen to store icons. At no point has any of the last three or four OS releases from any of the major vendors—Microsoft, Apple, or the various Linux distros—really introduced anything novel, just infinite variations on a theme. Filesystems are still hierarchical, users still install and manage applications, and so on. In fact, arguably the most interesting development in operating systems has been the iPhone, and most of its innovations center around two things: the two-finger interface, and the complete mental reboot of what user interface looks and acts like.

Seriously, that's the best we can do?

I see a lot of room for improvements in the operating system experience; for starters, let's do away with the "browser" and just call Firefox, IE and Chrome what they're (far too slowly) evolving into: a generic application host. Get that story right—the acquisition of applications onto the device, the updating of those applications when new versions are available, the offline application experience, and so on—and the operating system and the browser will mesh into a seamless whole. But we're not there yet, not by a long ways, and the first competitor to create such an environment will have a huge advantage over its rivals. Arguably Apple got there first with the iPhone and AppStore, and yet the iPhone still needs iTunes running on a computer to make the experience seamless, and iTunes is definitely not what I call a seamless user experience.

(Besides, the iPhone is hamstrung on a number of levels—I would absolutely despise trying to write this blog post on it, for example.)

Despite the clear window of opportunity for an innovative operating system to step in and make some serious waves in the industry, Google producing an OS really doesn't make sense to me, for a number of reasons.

Challenging your opponent on your opponent's turf is never a good idea. A maxim of battle says that one should only battle on favorable terrain, yet Google's deliberately choosing to "cross the line", as it were, into territory that is clearly foreign to them. They have no expertise in marketing it, selling it, researching it, or developing it, while their competitors in this—Microsoft, Apple being the principal two—have been doing it for decades. Literally. I realize that Google has a number of smart people working for them, but it seems pretty presumptuous and arrogant to think they can get this story better, particularly in any kind of short term.
This is a difficult problem to tackle. Microsoft's known it for decades, Apple is discovering it all over again, and Linuxers have either wallowed in it as a sign of prowess or just accepted the problem as intractable—it's really hard to get an operating system to recognize the billions of different devices out there. Apple solved it by jealously and zealously chasing anyone who ever tried to run Mac OS on non-Apple hardware. Linux consumers found themselves recompiling kernels or in some cases, having to build device drivers themselves. Microsoft just suffered through it. For a new OS, the only path possible in the beginning is to support the 20% of the devices that 80% of the people use, and hope that nobody else tries a device that isn't on that list and blogs to tell about it. Unfortunately, the chosen target market (consumer netbooks) works against them here in a big way. With developers, it's pretty easy to say, "Sorry, guys, you know how it is, give us a few years, or contribute the patch yourself!"; with consumers, if their BuyMart-bargain-bin web cam doesn't work, it's Google's fault and they'll be up in the acne-spackled BuyMart counter boy's face about it. This will not persuade BuyMart to stock the Chrome-installed netbook for much longer.
Is this really the company that swore to "do no evil"? Google's announcement is vague on so many levels, it's almost a FUD play, or else they're trying to blatantly cash in on their "geek cred" to convince investors and analysts that they've finally found a new source of revenue to supplement AdWords. (Well, modulo the fact that this new OS will be open-source, which means it's not really a revenue play, but I'm sure they've got that figured out somehow, too.) Seriously, this doesn't make sense: if you're doing an open-source OS, then where is the source? Where is the transparency? Where is my ability to contribute despite my status as a non-Google developer? What part of this project is open-source in any sense of the term?
Netbooks? I realize that netbooks are the new hotness to a lot of people, a compromise between a phone/PDA and a laptop, and that the price point of the netbook means that for the first time, consumers can get into computing for under $250 (rivalling the price of game consoles) that addresses their fundamental needs—email, web surfing and maybe an application or two—but the timing here is just too late. Google's announcement says that "netbooks running Google Chrome OS will be available for consumers in the second half of 2010". Which means that the major competitors (mostly Windows) will have twelve months to convince netbook consumers that Windows (and Windows7, in particular) is the right choice to run the netbook, and Google will be starting from some distance behind the 8-ball. Chrome needs to be available now if they're going to avoid a long and entrenched battle starting from a position of weakness.
It's a distraction from their strength. Abraham Lincoln is famous for saying. "You cannot strengthen the weak by weakening the strong", but this represents Google's third or fourth effort into a space that really isn't leveraging their core strength (their ability to scale). Even if the money and resources spent on Chrome (and Android, for that matter) have zero effect on the budgeting and resourcing for Google App Engine and other server plays, the message and story that Google presents to the world is now as disjoint and multifaceted (and therefore harder to grasp) as Microsoft's.
Haven't we seen this before? Wasn't it almost a decade ago when another company announced a plan to unify the browser and the desktop? In that case, the world either yawned, rejected it outright ("I don't want to browse my desktop, damnit" was how one friend of mine put it), or sued them over it. Even if Google doesn't run afoul of the DOJ directly, Microsoft is going to love pointing to Chrome OS as clear indication of non-monopoly status the next time DOJ comes calling. If Google does manage somehow to annoy the DOJ antitrust personalities, well... let IBM and Microsoft tell you all about how much fun it is to try to innovate and bring products to market with lawyers looking over your shoulders.
Haven't we seen this before? Not too long ago, another vendor tried to go after the "you don't need an operating system" story... except they called it "The Network Is the Computer". All you Java developers, raise your hand. Anybody who doesn't have their hand raised, ask what happened to that vendor from any of the people with their hand in the air. Or ask an Oracle DBA.
Haven't we seen this before? Even more recently, another vendor made a play for the netbook+cloud story. All those who've heard of Cloud OS, raise your hand. Anybody who doesn't have your hand raised.... well, I wish I could tell you to go talk to the people with their hand raised, except I don't think anybody does.

This whole idea just feels badly-planned and not well thought-out. Let's see how it executes, so let's meet back here in a year and compare notes, but in the meantime, I'm not hanging up my Java or .NET tools any time soon.

Review: "Iron Python in Action" by Michael Foord and Christian Muirhead

OK, OK, I admit it. Maybe significant whitespace isn't all bad. (But don't let me ever catch you quoting me say that.)

The reason for my (maybe) shift in thinking? Manning Publications sent me a copy of Iron Python in Action, and I have to say, I like the book and its approach. Getting me to like Python as a primary language for development will probably take more than just one book can give, but... *shrug* Who knows?

Bear in mind, I have plenty of reasons to like IronPython (Microsoft's Python implementation for the .NET environment):

A good friend of mine, Harry Pierson (aka @DevHawk), is the PM on the IPy project, and I'm generally prejudiced in favor of those things that people I know and respect.
I'm generally a fan of dynamic languages, particularly those that let you do strange and twisted things to the type system and its instances at runtime. (Yes, I'm looking at you, ECMAScript...)
I spent some quality time with IronPython Studio last year while researching a Visual Studio Extensibility "Deep Dive" paper.
I've known Jim Hugunin (the creator of IronPython, and Jython before that) for some years, ever since his days working on AspectJ, and he's one of those scary-smart guys that, despite knowing they're scary-smart, still render me stunned when I listen to them.
I'm a huge fan of the DLR. It's like having Parrot, but without having to wait a decade (give or take).

But, just to counterbalance the scales, I have plenty of good reasons to dislike IronPython, too:

Significant whitespace.
The "There's only one way to do it" oath that Pythonistas seem to hold as religion. (Somebody told me that building C-Python—the original implementation—only works for you if you swear a holy oath to The One True Way on the One True Way Bible. Needless to say, I believe them, and have never tried to build C-Python from sources as a result.)
Significant whitespace.
Uh.... did I mention significant whitespace yet?

I admit, it was with some hesitation that I cracked open the book. Actually, to be honest, I was really ready to just take out all my dislike of significant whitespace and pour it into a heated, vitriolic diatribe on everything that was just wrong with Python.

And...?

Well, OK, I admit it. Maybe significant whitespace isn't all bad.

But this is a review of the book, not the technology. So, on we go.

What I liked about the book

The focus is on both .NET and Python, and doesn't try to short-change either the "Python"-ness or the ".NET'-ness by trying to be a "Python book (that happens to run on .NET)" or a ".NET book (that happens to use Python for code samples)". The authors, I think, did a very good job of balancing the two, making this the book to get if you're in that area on the Venn diagram where "Python" overlaps with ".NET".
Part 2, "Core development techniques", starts down the "feed you the Python Kool-Ade" pretty quickly, heading straight into Chapter 4 ("Writing an application and design patterns with IronPython") without much of a pause for breath. The authors get into duck typing, protocols, and Model-View-Controller within the first four pages, and begin working on a running example to highlight some of the ideas. (Interestingly enough, they also take a few moments to point out that IronPython on Mono works, and include a couple of screen shots to that effect as we go, though I personally wonder just how many people are really going down this path.) I like the no-holds-barred, show-you-the-code style, but only because they also take time throughout the prose to talk about some of the concepts at work underneath and laced throughout the code. "Show me then tell me" is a time-honored tradition, but too many authors forget the "tell me" part and stop with code. These guys do a good job of following through.
The chapters in Part 3, "IronPython and advanced .NET", form an interesting collection of how IronPython can fit into the rest of the .NET stack, demonstrating how to use IronPython with WPF, ASP.NET, and IronPython's crowning glory, Silverlight. If you're into front-end stuff, this is the section where I think you're going to have the most fun.
The chapters in Part 4, "Reaching out with IronPython", is I think the most important part of the book, showing how to extend IronPython (chapter 14) with C#/VB extensions (similar to how a C-Python developer would extend Python by writing C code, but much much simpler) and the opposite—how to embed IronPython inside of existing C#/VB applications (chapter 15), which is really an exercise in using the DLR Hosting APIs. While the discussion in chapter 15 is good, I wish it'd had a bit more thorough discussion of how the DLR could be hosted regardless of the scripting language, though I admit that's pretty beyond the scope of this book (which is focused, after all, entirely on IronPython, and as a result should stay focused on how to host IPy).

What I found "Meh" about the book

Part 1 ("A new language for .NET", "Introduction to Python", and ".NET objects and IronPythong") does a good job of bringing the rank beginner up to speed, getting some basic Python ideas across in the same breath that they bring .NET home. The only problem is, it only works well if you're neither a Python programmer nor a .NET programmer. Chapter 1, for example, does a sort of Cannonball-into-the-pool kind of dive into Python, but dives equally into the "Iron" parts as it does the "Python" parts. If you're either a Pythonista or a .NETter, I suspect you're going to be tempted to flip pages pretty quickly, and (I suspect) miss a few things. Chapter 2 is all about Python (meaning .NETters will probably spend some time here), but it certainly doesn't feel like an exhaustive reference, nor does Chapter 3 stand as an exhaustive discussion about all things .NET, either. I almost wish all three chapters had been collapsed into one—suffice it to say, I don't feel like I know the Python language, and don't feel like this book could be my Python reference next to me as I learn it, and I know that it's not a great .NET reference, either. Fortunately, the goal of these three chapters feels pretty clearly to be "Teach you just enough to make you dangerous (and able to understand the rest of the book)", and once we hit Part 2, rubber meets road pretty quickly.
By the time you hit Chapter 7, less than halfway through the book, the authors have created a fairly nice, if simplistic, application for later dissection, but it's not until you hit Chapter 7 that they begin to start unit-testing, even though they insist (on page 17) that "Dynamic language programmers are often proponents of strong testing rather than strong typing" (a quote they attribute to Bruce Eckel, though I'm relatively certain I heard Dave Thomas and Neal Ford say it with respect to Ruby, long before Eckel started "Thinking in Python... or Flex... or whatever"). If unit-testing is that important, why wait three chapters into the application's development before writing a single unit-test? This doesn't jibe with me, somehow.
If you're into back-end stuff, chapter 12 on "Databases and web services" is pretty bland. The fact that the two are combined into a single chapter is indicative, all by itself, of how deep or intensive the coverage goes, and there's zero mention of anything beyond basic ADO.NET. The coverage on web services covers REST relatively well, but there's zero coverage of WCF, and the whole of SOAP-based services is all of four or five pages. And Workflow? Doesn't exist, isn't even mentioned (except for an appearance in a table, "The major new APIs of .NET 3.0"). Yikes.

What I actively disliked about the book

Actually, not much. Manning did their usual superb job of arrowed callouts to point out particular concepts in the code listings, the copyediting is professional (meaning there's no obvious typos or misspellings that just break up the flow of prose, something that not all publishers seem to take seriously), and the graphics flow nicely alongside the prose, not dominating the page but accentuating it.

In fact, about the only thing I'd care to criticize is the huge number of footnotes, particularly in the first chapter. (By page 20 in the book, there have already been 30 footnotes.) When you have three footnotes per page, on average (and sometimes more), it does tend to distract, at least to me it does. It feels like there were ways, for most of them, to inject the idea or concept into the main prose, or leave it out entirely, but that could just be a difference of writing style, too.

Summation

If you're a .NET developer interested in learning/using IronPython on your next project, this is a definite winner. If you're a Python developer looking to see how to break into .NET, I'm not so sure this is your book, but I say that mostly because I'm not a Pythonista and can't really speak to how that mindset will find this as an introduction to the .NET space. My intuition tells me that this would be a good springboard into another book on .NET for the Python programmer, but I'll have to leave that to Pythonistas who've read this book to comment one way or another.

Review: "Programming Clojure", by Stu Halloway

(Disclaimer: In the spirit of full disclosure, Stu is a friend, fellow NFJS speaker, and former co-worker of mine from DevelopMentor.)

I present this review to you in two parts.

Short version: If you want to learn Clojure, and you're familiar with at least one programming language, you'll find this a great resource. If you don't already know a programming language, or if you already know Clojure, or if you're looking for "best practices" to cut-and-paste, you're going to be disappointed.

Long version: Recently, fellow NFJS speaker Stu Halloway decided to take up a new language, and came to Clojure. He found the language interesting enough to write a book on it, something he hasn't done since his Java days, and the result is a nice walk through the language and its environment for experienced Java developers who want to understand Clojure's language, concurrency concepts, and programming model.

Now, let's be 100% honest about this: if you're coming at this book expecting it to be a language reference, you will probably be disappointed (as this guy obviously is). Stu's not like that—he's not going to re-create material that's available elsewhere, or that can be found with an easy Google search. Stu will not waste your time that way—he wants to tell you a story, one that takes you from "I'm a Java guy, but clueless about Lisp, dynamic languages, functional programming, concurrency, or macros" to "Wow. I know kung-fu." in the shortest path possible, but without trying to lobotomize you. He wants—no, expects—the readers of his book to be propping the text open with a cell phone on one side and the dinner plate on the other, craning your neck over to scan the pages and type in the examples into the REPL shell to try them out, see them work, then spend a few minutes experimenting with them before moving on to the next paragraph or page.

(Oh, I suppose you could just cut and paste them from the PDF version of the book, but where's the fun in that?)

The fact is, the concepts behind Clojure make up what's important to learn here, and readers of this book will come away like the panda from the movie, realizing that "There is no Secret Ingredient", that the power of Clojure comes not from its super-secret language sauce or special libraries, but in the way Clojure programmers approach problems and think about programming. And for that reason, if you're a programmer—even if you don't program on the JVM—you really want to take a look at what Stu's talking about (and Rich Hickey is creating).

Just remember, cellphone and dinner plate. Otherwise you'll be missing out on so much.

Interview with Scott Bellware and Scott Hanselman on the Death of the Professional Speaker

Well, OK, the title is trolling ever so slightly, but there is an interesting trend at work, and I'm genuinely concerned about its ultimate expression if the trend continues to its logical conclusion. Have a look and tell me if you agree or disagree.

The "controversy" continues

Apparently the Rails community isn't the only one pursuing that ephemeral goal of "edginess"—another blatantly sexist presentation came off without a hitch, this time at a Flash conference, and if anything, it was worse than the Rails/CouchDB presentation. I excerpt a few choice tidbits from an eyewitness here, but be warned—if you're not comfortable with language, skip the next block paragraph.

Yesterday's afternoon keynote is this guy named Hoss Gifford — I believe his major claim to fame is that viral "spank the monkey" thing that went around a few years back. Highlights of his talk:

He opens his keynote with one of those "Ignite"-esque presentations — where you have 5-minutes and 20 slides to tell a story — and the first and last are a close-up of a woman's lower half, her legs spread (wearing stilettos, of course) and her shaved vagina visible through some see-thru panties that say "drink me," with Hoss's Photoshopped, upward-looking face placed below it.

He later demos a drawing tool he has created (admittedly with someone else's code) and invites a woman to come up to try it. After she sits back down, he points out that in her doodles she's drawn a "cock."

Then he decides he wants to give a try at using the tool to draw a "cock" (he loves this word) — and draws a face, then a giant dick (he redraws it three times) that ultimately cums all over the face.

A multitude of references to penises and lots of swearing — and also "If you are easily offended, fuck you!"

And then, to top it off, a self-made flash movie of an animated woman's face, positioned as if she's having sex with you, who gradually orgasms based on the speed of your mouse movement on the page.

Wow. Just... wow. To call this unprofessional smacks of calling Hitler a "socially awkward individual"... or using a euphemism like "mild medical condition" to refer to death. This is so far "over the line" that it's unbelievable. Even Mr. Aimonetti's "CouchDB" presentation, as bad as it was, at least tried to tie the analogy together in a meaningful, if offensive, way. This is just male posturing at its worst. (I'm shocked Hoss didn't whip off his pants and demand the women in the room bow down in worship to his obviously superior manhood.)

Fortunately, according to the source, the conference organizer seems to be pretty responsive, so kudos to the one adult in the room, but....

What's worse, apparently the presenter and more than a few of his pals are (in the best traditions of assholery) blatantly unrepentant about the whole thing, claiming the moral high ground in much the same way that the Rails idiots did—it's all in good fun, if you don't find it funny you're a prude, and so on:

I checked Twitter (hashtag #flashbelt) to see what the responses were. Here are some notable remarks:

Fonx is reading the #flashbelt rants on Hoss offending the ladies w/ a few swear words & a penis drawing - r u really that prudish & sexist?

nthitz lol @hoss69 "If you are easily offended, fuck you" #flashbelt

livenootrac Ladies of #flashbelt , I am sorry for the Hoss preso, but in the flash community he gets a pass, kinda like Don Rickles - that's just Hoss.

CujoJpn @livenootrac And there were many ladies at #flashbelt who were offended by Hoss' Preso some were thick skinned and took it as is.

So, if you didn't like it then
a) you are a prude - and sexist (?)
b) fuck you
c) suck it because Hoss gets a pass here in the boy's club known as "the flash community" and
d) you are a wimpy girl who isn't strong enough / man enough / "thick-skinned" enough to deal with it.

Even more... wow. Talk about justification and marginalization. Amazing.

Before I figuratively smack this Hoss guy around the blog for a while, let's take a brief moment for reflection—what's going on here? Why all the misogynistic presentations recently? Is this reflective of a general trend in the programming industry? Of society in general? Is the world coming to an end?

A few possibilities present themselves:

The lack of women in the IT industry means there's nobody around to act as a "gender filter" to keep things on an even keel. In other words, the genders constantly filter themselves based on the company they keep, and because the boys who put these presentations together don't have female input, they simply don't know where to draw the line for mixed company. This theory also presumes that an industry that's made up primarily of women will also lack such a filter and "girls will be girls" as a result. Unfortunately I have no good counterexamples at hand to examine—anybody know of an industry populated primarily by women, and can weigh in with experience there? The closest I get is my brief experience working in a restaurant with an almost-all-woman serving staff, and from that brief experience, yep, the theory holds. Solution? Easy: get more women in IT, and things will re-balance themselves naturally.
Programmers are principally males who have no redeeming social skills. In other words, the industry gathers up exactly the kind of men who find objectifying women and reveling in late-acquired testosterone overdoses to be gratifying, and this kind of behavior is the result. If true, it leads to the conclusion that programmers are no more evolved than the Navy sailors involved in the Tailhook scandal of a few years ago. So go ahead, smack your wives and girlfriends around a little if they get a little "uppity", it's OK, 'cuz u r a l33t d00d. Personally? I find the idea ludicrous—there is definitely a strong antisocial streak that runs through the IT ecosystem (how many of you met your friends via World of Warcraft again?), but like all stereotypes, there's some elements of truth to it, and a lot of exaggeration. And frankly, anybody who believes in this theory is welcome to come with me to dinner at a No Fluff Just Stuff show and meet the other speakers, and listen in on our "boys club" conversations, including questions like, "Which movie best represents the book it was made after?" and "If given a mandate to create a programming language, what language would your language most resemble?". Oh, and the odd fart joke. We are boys, after all.
We're hypersensitive to the subject right now. In other words, these kind of presentations have always been going on, and it's just that we notice them now, in the same way that you notice a particular brand of car on the road a lot more when you're thinking about buying that brand and model of car. Frankly, I don't buy this argument—I've been to a lot of presentations over the past decade, and I've never seen any that were anything like this.
This is the YouTube generation, with access to everything the Internet has to offer, and this is "just how they do things". After all, how much maturity, sexual discretion and adult behavior can we expect of the generation that gave us "Girls Gone Wild" and its ilk? It's just a "generation gap" thing, and we old fogies who didn't grow up with Internet porn just a browser-click away just don't "get it". Hmm.... somehow, I just don't buy it. Sure, there may be some elements of this involved here (I'm really curious to see what all these "Girls Gone Wild" girls are going to say to their own daughters in a decade or so...), but I think that's too easy an answer, and an eminently unhelpful one.
We have copycatters out there trying to follow the path of people they respect. If you're looking up at this Hoss character and thinking, "I want to be just like him!", you really should see a therapist and develop a sense of self, before you find yourself without friends. Hoss gets a pass because of your misguided fan-boi hero-worship. So does Paris Hilton. You want to be the Paris Hilton of your social circle? Go for it. After all, she's highly respected and loved, right? Take a clue from the next car wreck you drive past—everybody's slowing to look not because they wish they were in the body bag, folks, but because we have a ghoulish fascination with it. In the case of Ms. Hilton, that ghoulish fascination is with those who self-destruct in spectacular fashion. (Me, I'd love to be the fly on the wall at the Hoss residence when he tries to explain this whole thing to his daughter or his date/girlfriend/wife, if he ever finds one.)
The presenters taking this tack are looking for an easy path to fame. In the grand traditions of Andrew Dice Clay ("Oh!"), the easiest way for a presenter to "stand out" from the rest of the crowd of presenters is to do something outrageous and call it "edgy", and stake out a claim on the edge of the civilization, rather than try to integrate with the rest of the crowd and build something up slowly. Don Box has already claimed "HTTP is dead", I made the analogy between a technology and a military conflict, and Matt Aimonetti claimed a data storage framework "performs like a pr0n star", so what's left but to stake out ground even further out on the fringe and just be misogynistic? Fortunately, history suggests that people with content-free/shock-heavy presentations (or even content-heavy/shock-heavy ones) don't go the distance, so to speak, and that once there's nowhere more shocking left to go, the audience comes back to the content-heavy/shock-light discussions and stays there for a while. Unfortunately, this means we're going to have to suffer through somebody's "Live YouPorn filming" talk first, which I'm not looking forward to.

And now for the smacking around... but you know, I suddenly realize that the volume of comments on the original post leave with nothing to do or say that's not already being said, so to just "pile on" would only serve to let me vent, and I have other outlets for that. But it would be inappropriate to just "walk away", so to speak, so with that in mind....

Hoss, you're an idiot. Like any sprinter, you're going to head up the pack for a bit, but soon enough, your "shtick" is going to flame out and you'll be left behind with all the other "shock jocks" of the 80's who found their material unwelcome after a while. So enjoy the spotlight (such as it is) while you can. In the meantime, I'm off to revise a few presentations, and stick with solid ideas and analogies, and maybe dropping the odd F-bomb when I want to make a point, just for emphasis, because I know something you apparently don't:

Shock makes a point because of the contrast to the rest of the talk, not because of its inherent "edginess".

Meanwhile, by all means, continue to be an idiot. You just make me look better by comparison, for which I thank you.

A eulogy: DevelopMentor, RIP

Update: See below, but I wanted to include the text Mike Abercrombie (DM's owner) posted as a comment to this post, in the body of the blog post itself. "Ted - All of us at DevelopMentor greatly appreciate your admiration. We're also grateful for your contributions to DevelopMentor when you were part of our staff. However, all of us that work here, especially our technical staff that write and delivery our courses today, would appreciate it if you would check your sources before writing our eulogy. DevelopMentor is open for business and delivering courses this week and we intend to remain doing so." Duly noted, Mike. Apology offered (and hopefully accepted).

An email crossed my desk today, announcing that DevelopMentor, home to so many good people and fond memories, has (at least temporarily) closed its doors.

I admit to a small, carefully-cushioned place in my heart where I mourn over this.

DevelopMentor was such a transcendent place for me. Much, if not most or all, of the acceleration that came in my career came not only while I was there, but because I was there.

So much of my speaking persona and skill I owe to Ron Sumida, who took a half-baked neophyte of intermediate speaking skill, and in an eight-hour marathon session still referred to in my mental memoirs as my "Night with Scary Ron", shaped me and taught me tricks about speaking that I continue to use to this day. That I got to know him as a friend and confidant later still to this day ranks as one of my greatest blessings.

I remember my first DM Instructor Retreat, where I met so many of the names I'd read about or heard about, and feeling "Oh, my God" fanboy-ish. I remember Tim Ewald giving a talk on transactions at that retreat that left me agape—I seriously didn't understand half of what he was saying, and rather than feeling overwhelmed or ashamed, I remember distinctly thinking, "Wow—I have found a home where I can learn SO much more." It was like waking up one morning to find that your writing workshop group suddenly included Neal Stephenson, Stephen Pinker, C.S. Lewis and Ernest Hemingway. (Yes, I know those last two are dead. Work with me here.)

I remember the day that Lorie (the ops manager at the time) called me to say that Don Box wanted me to work with him on the C# course. I was convinced that she'd called the wrong Ted, meaning instead to reach for Ted Pattison in her Rolodex and coming up a few letters shy. She tartly informed me, "No, I know exactly who I'm talking to, and are you interested or not?" How could I refuse? Help the Diety of COM write DM's flagship course on Microsoft's flagship technology for the next decade? "Hmm...", I say out loud, not because I needed time to think about it, but because a thread in the back of my head says, "Is there any scenario here where I say no?"

I still fondly recall doing a Guerilla .NET at the Torrance Hilton shortly after the .NET 1.0 release, and having a conversation with Don in my hotel room later that night; that was when he told me "Microsoft is working on an open-source version of the CLR". I was stunned—I had no idea that said version would factor pretty largely in my life later. But it opened my eyes, in a very practical way, to how deeply-connected DevelopMentor was to Microsoft, and how that could play out in a direct fashion.

When Peter Drayton joined, he asked me to do a quick review pass on the reference section of his C# in a Nutshell, and I agreed because Peter was a good guy (and somebody I'd hoped would become a friend), and wanted to see the book do well. That went from informal review to formal review to "well, could you maybe make it an editing pass?" to "Would you like to write a few chapters?" to "Well, let's sign you up as a co-author...". That project is what introduced me to John Osborn, which in turn led him to call me one day and say, "Some guys at Microsoft are working on an open-source version of the CLR, and would like to have a 'professional writer' help them write a book on it. Interested?" That led to SSCLI Internals, working with David Stutz, and wow, did I learn a helluvalot from that project, too.

Effective Enterprise Java came through DevelopMentor, thanks again to Don Box, who introduced me to the folks at Addison-Wesley that put the contract (and Scott Meyers, another blessing) in front of me.

DM got me my start in the conference circuit, as well. In 2002, John Lam pinged me over email—he'd recently become track chair for Connections down in Orlando, and was I interested in speaking there? I was such a newbie to the whole idea, but having taught classes roughly twice every month, I wasn't worried about the speaking part, but the rest of the process. John walked me through the process, and in doing so, set me down a path that would almost completely redefine my career within a year or so.

Even my Java chops got built up—the head of our Java curriculum was Stu Halloway (recently of Clojure fame), and between him, Kevin Jones, Si Horrell, Brian Maso and Owen Tallman, man, did I feel simultaneously like a small child among giants and like a kid in a candy store. Every time I turned around, they'd discovered something new about the Java platform that floored me. Bob Beauchemin has forgotten more about databases in general than I will ever learn, and he had some insights on the intersection of Java + databases that still hang with me today.

And my start with No Fluff Just Stuff came through DevelopMentor, too. Jason Whittington heard through a mutual friend (Erik Hatcher, of Ant fame) about this cool little conference being held in Denver, and maybe I should look into it. That led to an email intro to Jay Zimmerman, a dinner together while I was teaching in Denver a few weeks later, and before I knew it, I was on the Denver NFJS schedule, including the speaker panel, where I uttered the then-infamous line, "Swing sucks. Get over it."

DevelopMentor, you shaped my career—and my life—in so many ways, you will always be a source of pleasant memories and a group of friends and acquaintances that I would never have had otherwise. Thank you so much.

Rest in peace.

Update: Well, as it turns out, I have to rescind at least part of my eulogy, as the post itself generated quite a stir—the folks at DevelopMentor were pretty quick to email me, pointing out that they're still alive and well. In fact, as one of them (a friend of mine still working there) put it, "We were all kinda surprised when we came to work this morning and discovered that we could go home." Fortunately, the DevelopMentor folks were pretty gracious about what could've been a very ugly situation, and I apologize for to them for the misunderstanding—all I can say is that my "source" must've also been mistaken, and I'm glad that we're all still good. And lest it need to be said out loud, I heartily want nothing but the best for DM, and hope that I never have to write this message again.

SSCLI 2.0 Internals

Joel's weblog appears to be down, so in response to some emails I've posted my draft copy of SSCLI 2.0 Internals here. I think it's the same PDF that Joel had on his weblog, but I haven't made absolutely certain of the fact. :-/

If you've not checked out the first version of SSCLI Internals, it's cool—the second edition is basically everything that the first edition is, plus a new chapter on Generics (and how they changed the internals of the CLR to reflect generics all the way through the system), so you're good. And if you're not sure where to get the codebase for Rotor 2.0 (the SSCLI), well, here, I'll make it easy for you. ;-)

Gotta say, this is almost without question my favorite book to have written. Just wish Microsoft would've kept Rotor up with the successive CLR releases (3.5 SP 1 and now the forthcoming 4.0). Maybe, if I can find that wishing ring....

.NET | C# | C++ | F# | Languages | Reading | Visual Basic | Windows

Tuesday, May 26, 2009 6:42:49 PM (Pacific Daylight Time, UTC-07:00)
Comments [0] |

He was Aaron Erickson... Now he's Aaron Erickson, ThoughtWorker

Yep, you heard that right—Aaron Erickson, author of The Nomadic Developer, is now a ThoughtWorker.

For those of who you don't know Aaron, he's been a consultant at another consulting company for a while, and has been exploring a number of different topics in the .NET space for a few years now, not least of which is one of my favorites (F#) and one of THoughtWorks' favorites (agile). He's been speaking at a number of events, including the Connections conferences, and he's going to bring some serious market-development potential to our Chicago office, something that's obviously of concern right now in these current economic conditions.

He also cooks a mean bacon-wrapped scallop, but that's another story for another day.

I'm looking forward to having him be a part of the growing collection of .NET rock stars at ThoughtWorks. Wanna come join us? Always room for a few more....

.NET | C++ | Conferences | F# | Languages | Visual Basic | WCF | Windows | XML Services

Saturday, May 23, 2009 7:05:09 PM (Pacific Daylight Time, UTC-07:00)
Comments [3] |

Of Tomcat 6, native services, Windows 2008R2, and pain...

So I'm putting together a Windows 2008 R2 x64 RC Java image for a client (more on that later), and everything's breezing along fine. Install the OS, check. Install JDK 1.6 (u13) into the machine, check. Install Tomcat 6 into the machine, running as a native Windows service, check. Open localhost on port 8080, and... not check. Times out, no response, not good.

Naturally, the first thing to check is the logs, and I get the strangest error I've seen in a while. "Cannot create Java". This is odd—what's happening, in the aggregate, is easy enough to understand, in that the native Windows .exe launcher (ProcRun, a generic service launcher from Apache) is using JNI to create the JVM inside the launched service process and, for some reason, failing; what's not clear is why. Unfortunately, the error codes offered up by the two players involved (Tomcat/ProcRun and the Windows OS) are not helpful—the Windows Event Log basically says "Service failed to start. Check the error code", which reports 0 (not helpful, thanks), and the Tomcat "jakarta_service_date.log" file reports something along the lines of...

[2009-05-23 17:33:41] [1343 prunsrv.c] [debug] Procrun log initialized
[2009-05-23 17:33:41] [info] Procrun (2.0.4.0) started
[2009-05-23 17:33:41] [1166 prunsrv.c] [debug] Inside ServiceMain...
[2009-05-23 17:33:41] [info] Starting service...
[2009-05-23 17:33:41] [174 javajni.c] [error] The specified module could not be found.
[2009-05-23 17:33:41] [994  prunsrv.c] [error] Failed creating java C:\Java\Tomcat6.0\jre64\bin\server\jvm.dll
[2009-05-23 17:33:41] [1269 prunsrv.c] [error] ServiceStart returned 1
[2009-05-23 17:33:41] [info] Procrun finished.

... which is not really all that helpful, either.

Hmm.

The fact that it can't create Java is not a really strong clue, so I start searching the Web for some solutions. Several people report running into this same problem, but solutions are not easily found—one web page reports that there's a missing "msvcr71.dll" file from the Windows installer installation script, and that copying the file into C:\WINDOWS\System32 fixes it, but when I go look in that directory, no dice—the DLL's there, and a quick "DUMPBIN" on the file reveals it looks good, no accidental file corruption or anything. Rats.

Maybe the problem's somewhere in the service configuration—it's possible that the Tomcat installer put the wrong configuration in or something. So I fire up the Tomcat configuration (tomcat6w.exe) from the "bin" directory, and just to be sure, I go hunting up the Service entry in the Registry (on the off-chance that the configuration utility is the source of the bug). Granted, this is kind of a stretch, but unfortunately, like I said, there's not much to go on. Sure enough, make a few changes (one of which is to tell the Tomcat native launcher to use the "server" VM, instead of the "client" VM, by default—why, oh why, hasn't Apache changed that yet?!?), verify that the changes are percolating all the way through into the Registry, and try kicking off the service. Still no luck. Still the same error.

While I'm rooting around in the Registry, I notice that there's another node in there that I'm not familiar with—the Wow6432Node. And buried underneath it (thank you, Registry Search, for finding this!) is a node for Apache Software Foundation/ProcRun2.0/Tomcat6, and a whole slew of configuration options under there, as well. Hmm. Errors in the ProcRun configuration perhaps? Sure enough... no, everything's working fine.

But now the synapses are firing in a different direction—the ProcRun bits are underneath the "Wow6432Node", and the "Wow" part of that name has me wondering—in the old 16-bit-to-32-bit transition Windows went through once before, "Wow" was an acronym for "Windows-on-Windows", meaning that the 32-bit version of Windows was opening up an emulation layer to run 16-bit programs. Given that this is an x64 image that I'm working with... is it that the service wants to be using the x64 version of Java rather than the 32-bit version I downloaded out of pure habit? Hmm. Go grab the x64 image, install it, and... still no love.

The WoW64 thing is still tickling at the back of my brain, though, and suddenly a new synapse fires off. If this is a 64-bit version of Windows, then there has to be.... Yep, sure enough, underneath the C:\WINDOWS directory there are not two, but three, "system" directories—the "C:\WINDOWS\System" directory that used to be the hangout place for 16-bit DLLs, the "C:\WINDOWS\System32" directory where 32-bit DLLs were encouraged to reside, and, just as pretty as you please, there it is, a "C:\WINDOWS\SysWOW64" directory, and inside there... no "msvcr71.dll". Copy the "msvcr71.dll" over from System32 into SysWOW64, and.... Voila. Service starts, log file looks good, and "localhost:8080" comes back with the Tomcat home page.

What have we learned from this little experience? A couple of things, some personal, some observational about the state of the universe and the industry:

Tomcat still installs itself to depend on a JRE found elsewhere on the system. This isn't a problem, per se, but the Windows installer for Tomcat tries to discover the JRE to use to run the Tomcat bits, and usually comes up with the "public" JRE installed underneath C:\Windows\Java\... . Fact is, I would really prefer if Tomcat made use of a private JRE (one inside the Tomcat directory) rather than the "public" one—too many times an installer will take liberties with the public JRE, and as a general rule, I really don't want installers messing around with those settings or deployment picture (contents of jre/lib/ext, for example).
I feel a little out-of-touch with x64 operating systems. Fact is, I have gotten a bit rusty on my operating system operation with respect to the 64-bit operating systems (Windows in particular), as highlighted by the fact that I really don't know what, if any, differences there are between the 64-bit version of a native executable and it's 32-bit cousin, or what the 32/64-bit transition story is. Anybody got any good book recommendations on the 64-bit Windows story?
I feel a little out-of-touch with the Java 64-bit story. Same thing—anybody have a good overview of what's different between 32-bit and 64-bit Java on Windows, and more importantly, why, even now, when I switch back and try to run the 64-bit version of Java via the service, it fails (this time with a "not a valid Win32 image" error in the log file)? Is it worth it enough to try and diagnose/debug/develop a solution to let Tomcat run with the 64-bit version of Java instead of the 32-bit it's now using?
The fact that this was harder to unearth via Google than usual bothers me a bit. Google usually helps with troubleshooting a lot more than it did, usually because commonly-hit errors and their fixes are reported all over the place, in blogs and forums and so on. The fact that there was relatively few hits (with potential solutions, anyway) makes me wonder: Are people not running Tomcat on Windows, not running Tomcat as a service on Windows, not running Tomcat on 64-bit Windows, or just not generally having problems? If you're running Tomcat on Windows, I'd love to hear your story.
Diagnosing Windows services is still a pain. I was a heartbeat away from trying to debug the native parts of the Tomcat service, using either SysInternals' Process Explorer or Visual Studio itself, and really wished there was some better error-logging to indicate what the problem was so I didn't have to. Granted, from my time writing Windows services way back when, I remember there not being a lot that a service author can do to make that a more transparent experience, so I can't necessarily fault the authors of ProcRun, since they're (probably) faithfully reporting the return value of CreateProcess or LoadLibrary, but it's still frustrating and I think more information (maybe the return value of GetLastError?) might have helped out here a bit.

Meanwhile, my installations continue....

C++ | Java/J2EE | Windows

Saturday, May 23, 2009 6:37:23 PM (Pacific Daylight Time, UTC-07:00)
Comments [3] |

TechEd 2009 Thoughts

These are the things I think as I wing my way out of LA fresh from this year's TechEd 2009 conference:

I think I owe the attendees at DTL309 ("Busy .NET Developer's Guide to F#") an explanation. It's always embarrassing when your brain freezes during a presentation, and that's precisely what happened during the F# talk—I completely spaced on the syntax for implementing an interface on a class in F#. (To the attendees who commented "consider preparing a bit better so you dont forget the sintax :)" and "Not remembering the language syntax sorta comes across bad doesn't it?", you're absolutely right, which prompts this next sentence.) I apologize profusely to those who were there—I just blew it. For the record, the missing syntax looks like this:
```
#light

type IStudy =
    abstract Study: string -> unit

type Person(firstName : string, lastName : string, age : int) =
    member p.FirstName = firstName
    member p.LastName = lastName
    member p.Age = age
    override p.ToString() = 
        System.String.Format("[Person: firstName={0}, lastName={1}, age={2}]",
            p.FirstName, p.LastName, p.Age);
            
type Student(firstName : string, lastName : string, age : int, subject : string) =
    inherit Person(firstName, lastName, age)
    interface IStudy with
        member s.Study(sub : string) =
            System.Console.WriteLine("Hey, Ma, I'm studying {0}!", sub)
    member s.Subject = subject
    override s.ToString() =
        System.String.Format("[Student: " + base.ToString() + " subject={0}]", s.Subject);
```
Truth is, though, right now not a lot of people (myself included) are writing types that formally implement a given interface—the current common practice appears to be an object expression instead, something along these lines:
```
let monkey =
    { new IStudy with
        member p.Study(subject : string) =
            System.Console.WriteLine("Oook eeek aah aah {0}!", subject) }
monkey.Study("Visual Basic")
```
In this way, the object handed back still implements the interface type that the client wants to call through, but the defined type remains anonymous (and thus provides an extra layer of encapsulation against implementation details leaking out). The most frustrating part about that particular snafu? I had a Notepad window open with some prepared code snippets waiting for me (a fully-defined Person type, a fully-defined Student type inheriting from Person, and so on) if I needed to grab that code because typing it out was taking too long. Why didn't I use it? I just forgot. Oy.....
Clearly Microsoft is thinking big things about Azure. There were a lot of sessions around Azure and cloud computing, far more than I'd honestly expected, given how new (and unreleased) the Azure bits are. This is a subject I would have expected to see covered this deeply at PDC, not TechEd.
TechEd Speaker Idol is a definite win, to me. I watched the final round of Speaker Idol on Thursday night (before catching the redeye out to Atlanta for the NFJS show there this weekend), and quite honestly, I was blown away by the quality of the presentations—they were all of them better than some of the TechEd speakers I'd seen, and it was great to hear that not only will the winner, who did a great presentation on legacy application support in Windows7 (and whose name I didn't catch, sorry) be guaranteed a slot at TechEd, but I overheard that the runner-up, a Polish security expert who demoed how to break Process Explorer (in front of Mark Russinovich, no less!), will also be speaking at TechEd Berlin this year.
As always, the parties at TechEd were where the real value lay. This may seem like an odd statement to those whose heads are a bit full right now from five days' worth of material (six, if you attended a pre-con), but remember that I'm a speaker, so the sessions aren't always as useful as they are to people who've not seen this content before (or have the kind of easy access to the people building it and/or presenting it that I'm fortunate and privileged to have). Any future attendees should take serious note, though: networking is a serious part of this business, and if you're not going out to the parties (or creating a few of your own while you're there) and handing out business cards left and right, you're missing a valuable opportunity.
I'm looking forward to TechEd 2010. Particularly because, thanks to a few technical snafus, I had the chance to sit down with the folks who organize and run TechEd and vent for a little bit about everything I found annoying (as a speaker). Not only were my comments not blown off, but it started a really productive discussion about how to make the behind-the-scenes experience for the TechEd speakers a more pleasant and streamlined one. What's more, we're planning to revisit some of these discussions in the months to come as they start their preparations for TechEd 2010 (in New Orleans). I'm looking forward to those conversations and (hopefully) helping them eliminate some of the awkwardness that I've seethed over in the past.

New Orleans in the summer will not be an entirely wonderful experience (I'm told it gets monstrously humid there in the summers, but it can't be any worse than Orlando is/was), but I'm honestly very curious to get back there to see what post-Katrina New Orleans looks and feels like, and to maybe do my (very little) part to help the area claw its way back by maybe staying an extra day or two and taking in some of the sights. (I'm hoping that Sara Ford will be willing to act as tour guide.....)

In the meantime, thanks to all of you who came, and remember—if you attended a talk and you want to say "thanks" to the speaker who gave it, the best way is to take the five minutes to fill out the evals for that talk. (Speaking personally, I don't even care so much about the scores you give me, but the comments are absolutely invaluable.)

See y'all next year!

Windows 7 RC install experience

Since a number of people have been connecting to my blog via my last post on installing Windows 7 into a VMWare image, I thought since the Windows7 RC is now available, I'd update my experiences with installing it.

I downloaded the Windows7 RC ISO image (a freakishly hideous name containing every character on my US keyboard, plus a few in Klingon, I think.... if you can stand it, the full name of the ISO is 7100.0.090421-1700_x86fre_client_en-us_Retail_Ultimate-GRC1CULFRER_EN_DVD) from the Microsoft CONNECT website, not bothering with any of the other images (x64, ia64, and a "server" image I've not explored yet), using Microsoft's File Transfer Manager. (I know, I know, somebody's going to complain again about the ISOs not being available via a straight HTTP download or Torrent, but this is just an RC release, folks, and this is ostensibly to Microsoft-friendly customers who already have the FTM utility installed.) Took about 3+ hours to download on my home connection... or so it claimed. I went to bed after starting it last night. It was done when I woke up. What more do you want from me?

I created a new VMWare image, as a "Windows Vista" VM with 1GB RAM, a 60GB IDE hard disk (by default Fusion wants to create a 40 GB SCSI disk, but IDE seems to play nicer with the early betas of Microsoft OS'es, and I made it all one file rather than Fusion's default "Split into 2GB files" option), with the experimental 3D graphics turned on, battery status turned off, and (this is HUGE) the "Allow your Mac to open applications in the virtual machine" option turned OFF. Can't repeat this enough, for ANY VMWare VM containing Windows inside of it, turn off that option—leaving it on sucks up HUGE amounts of CPU time. (It's barely documented, and only determined Googling found that this was what was rendering my VMWare Fusion 2 images all but unusable.)

I attached the ISO to the VMWare CD and turned the thing loose. It takes a while, but so long as the ISO file and the VMWare VMDK disk file are on separate drives, the perf isn't too bad—roughly twenty minutes later (or, as I measure things, one randomly-generated map game of Pax Galaxia later), the image had installed all the core files on the VM disk, restarted itself, finished the installation, and restarted itself again. (I have no idea why Win7 wants to reboot itself twice during the install—if I remember the Vista installs correctly, it only restarts once). As I write this, I'm starting at the "Setup is preparing your computer for first use" screen with the funky Cylon-like flashing bar underneath the text (I'm serious, it really looks like the graphic artists at Microsoft are paying homage to BSG during that Setup screen). Whoops, I take it back—got through that screen rather quickly, and now we're into the username/password/product key stage. Plug that in, set the Update policy, the date and time, the network defaults (Public Location for all my VMs, just because), and.... "Welcome".

There's no Step Four. Although, according to Windows Update, there's already an update for Windows7 that should be downloaded and installed. *grin* Actually, it seems like the driver it installed was for the VMWare virtual sound device, which normally doesn't kick in until I install the VMWare Tools. It tells me that this is an "Unsupported Creative Sound Device", however, so maybe it's an older driver. *shrug* Not sure, don't care, because my next step is....

Install the VMWare Tools. I install VMWare Tools in the image, after the Update is complete. (No restart was required, so why not?) Actually, let me rephrase that—I tried to install the VMWare tools, but when I selected it from the Fusion menu bar... nothing happened. Hmm. OK, let's do the restart and see what happens. VM shuts down quickly enough (no having to wait for updates to finish, which was somewhat annoying with Vista), and when I restart, it seems to restart quickly enough (again, no obvious updates to be installed), so I get to a working desktop (640x480, how did we ever think this was reasonable?!?), and try the Install VMWare Tools option from the Fusion menubar again. It thinks for a bit, and the cursor flashes to the "pointer-with-CD" icon for a second before flashing back, but after a few seconds, the "What do you want to do?" (Autoplay) menu pops up as if I'd slipped the CD into the drive, so all looks good. Go through the UAC "Continue/Cancel" dialog (see below), choose "Complete" for the VMWare Tools install options, and let 'er rip. Disks spin, lights flicker, and a "VMWare Shared Tools" network folder shows up on the Win7 desktop, indicating that it's suddenly discovered the Shared Folder (to my MacOS user account) is there. But now we're back to the Windows-display-exercise program, which leads me to believe that it's the VMWare driver that's doing the exercise, not Windows itself. (VMWare? Anybody listening and care to comment?)

And now I'm into Win7 desktop customization steps, things like display sizing and desktop icon selection, background image, and all that other jazz that you probably don't care about. (If you do, then I'm a bit worried about you—be an individual! Choose your own settings!) All in all, pretty flawless and smooth.

Thoughts on the process:

It feels like we're getting away from the "minimal install" process that Vista tried to create. For a while, there was a meme that said that installing Windows was too hard for the average person, and Microsoft promised to reduce the number of steps it had to go through to install the OS. Take the date/time screen, for example: it picked up the defaults from the underlying (virtual) hardware, why not just assume those and skip that step? Users can always change it later.
I still have to set a Administrator password. I know that Microsoft is trying to find that sweet-spot balance between "too secure" and "unsecure" for desktop operating systems, but I have to hand it to the Ubuntu folks here—the "passwordless root" idea that they use is pretty slick. MacOS uses it (for the most part) in places, as well. I like the balance that approach achieves: it forces the user to enter "superuser" mode to do something sensitive, but it isn't challenging for a password (unless the superuser installs one) every time.
It's not going through display-screen calesthenics on each startup with this build. My previous Win7 image, every time I restart the VM, goes through every possible video/monitor size combination before settling in on the resolution I established in the last session. That was a bit disconcerting, until I realized that it's Windows trying to get some exercise in to be less overweight. *grin*
What, no PowerShell installed by default? Either it's not there, or it's buried pretty deeply. Command Prompt (cmd.exe) is right where it's always been, under Accessories, but no PowerShell.... Whoops, no, I take it back, it's in a folder underneath Accessories, forcing one more click to get to it. Hey Microsoft: do me a favor and pin that guy to the Start Menu. Make it easy for me to use, if you really want me to believe that this is supposed to replace Command Prompt someday.
On that note, though, the PowerShell "ISE" (Interactive Scripting Environment) is an interesting and new toy to play with.
"Pin to Taskbar" is an interesting option that I'm going to have to play around with. Not being a huge MacOS Dock fan (which is pretty clearly the inspiration for the new Taskbar), I'm not sure how well I'll like the new "it's the QuickLaunch and the Taskbar combined" idea.

Overall, I'm looking forward to putting a few things into this image (VS 2008, VS 2010, Office, and so on) and seeing how it reacts. As always, your mileage may vary, no implied warranties with this blog post, blah blah blah, but if you do anything with the Windows OS, you really should get hold of the RC (build 7100) and put it into a Virtual PC, VMWare, VirtualBox, Xen or some other virtualized box to play with. Like it or not, it's entirely reasonable to believe that Windows7 is going to win a few folks back from the Vista "less-than-I-expected" crowd.

As always, caveat emptor, and feel free to comment....

.NET | C# | F# | Industry | Review | Windows

Saturday, May 02, 2009 12:18:20 AM (Pacific Daylight Time, UTC-07:00)
Comments [2] |

On speaking, trolling, inciting and growing

It's been going around in developer circles now for a few days, this whole controversy about the "Perform like a pr0n star" presentation from the Golden Gate Ruby Conference and the related accusations of misogyny and sexism and overblown accusations and double-standardisms and what-all else, and I've deliberately waited to let opinions in my head settle out before blogging on the whole thing. Sara J Chipps reacts on her blog, and the comments to her comments are also somewhat... interesting... to note.

Without any particular implied importance or order:

Matt Aimonetti, you are an idiot. You had to know that this was going to generate more than a few strong reactions. I'll admit, it's a funny title, and it definitely generated a ton of buzz around your name, but for the rest of your life, you're going to be "the porno Rails guy", and in about a year or so, it's not going to be funny anymore. You've touched off a firestorm, and you can't very well hide from it, and frankly, I think the short-term boost to your public recognizance is going to be more than outweighed by the long-term judgments that will be levied against you. "Wait, this is the guy who did that talk? Wow. I bet he's a good developer, but can I risk him pulling the same kind of stunt at a meeting with our VP or clients? Nah, I'll go for this other guy...."
Clearly we have a lot of issues to work out in the programming industry. I'm not going to go into the rights or wrongs of putting those images into his talk. I'm talking about the discussion that followed (one comment here says, "Matt Aimonetti is obviously an antisocial twerp still living in his mothers basement at the age of 35 who has never even been able to muster up the courage to actually talk to a real-life woman, let alone respect one.", and a follow-up comment says, "Great presentation, nevermind the jackasses, keep up the good work!"), and the fact that at no point in the time leading up to this presentation did anybody pull Mr. Aimonetti off to one side and say, "Dude, it was funny when we thought of it, sure, but it's time to stop." If ever we wanted to convince the rest of the world that the programming industry wasn't populated by a bunch of 13-year-olds giggling over the fact that somebody said, "Boobies".... well, maybe next year.
Ruby community, you have a long way to go if you want to convince people to spend money on you. Maybe you don't mind that corporations think that you guys are clearly unstable and immature. If/when you want to gain some degree of corporate acceptance, and maybe make it out of your parents' basement someday, you're going to have to learn that how you handle yourself in public goes a long way towards establishing peoples' attitudes towards you as professionals, and right now, you all collectively look like a bunch of 13-year-olds, what between this and DHH's famous "FUCK YOU" presentation of a few years ago. If you're OK with not being taken seriously, then cool, more power to you. But personally, I like the idea of making money at things I like to do and have fun doing, and you're not helping yourselves.
Why are we such prudes? Whether you agree or not with the rightness of the "porn" metaphor, you have to admit that there is factual basis in the bones of this particular comment: "This is probably the least offensive thing I've seen in 3 weeks." Glance at the billboards in the airport next time you're walking to the gate. Glance at the racks of magazines in the grocery store as you prepare to check out. Glance at the beer commercials on TV during prime-time. In every case, sexy, young, attractive, scantily-clad men and women seek to create an instinctive emotional reaction inside your head to subconsciously create a feel-good link between whatever product is being hawked and your id. Honestly, the photos in the presentation are hardly all that titillating—and a very long ways from the kind of commercials you can see on TV in Europe—so why are we getting up in arms over this?
Matt Aimonetti, you are an idiot. Notice how nobody's talking about the actual subject of your presentation? A good presenter knows that the message should never outstrip the delivery mechanism, just like a sauce should never overpower the flavor of the dish it accompanies. For all that the content of your presentation might have been spot-on, the lessons that might have been learned from the presentation have drowned in the "He's a pig!" "No he's not!" that has followed. Great job there, mate. Way to get your message across.
To the commenter on the presentation page who said, "ps [sic] feminism is dead", get a clue. Women still, on average, get paid less than men do for an equivalently-skilled employee in the same job. Maybe it's not $.50 to every $1 as it used to be, but so long as it's even measurable, there's work to be done. This industry in particular has absolutely no reason for gender discrimination in any form, since there's absolutely nothing "physical" about what we do. (Ditto for medicine and law, for that matter.)
Presentations reach far beyond just the attendees. One commenter on Sara's blog notes, "What an over reaction, there was nothing wrong with that presentation, i wouldn't show it to a board room but as far as showing it to a ruby developers conference then no probs." Frankly, that's a short-sighted attitude, making the presumption that someone of the suit-and-tie set (those supposedly inhabiting the "board room" where this kind of presentation isn't appropriate) wouldn't actually be in the audience at a ruby developers conference. Oh, granted, when in Rome, one has to expect Romans to act like Romans, but that just means that the Ruby community isn't welcome inside the board room, right? (Somehow I doubt this is what the numerous people who are trying to make money off of Ruby really want.) Fact is, that presentation is now captured by the Internet for all time, and it will forever be known as "The Ruby Porno Presentation", and it's an even money bet that somebody in that board room has seen the presentation (and the video, and the play-by-play from the people who had friends who had friends that were there....).
To the commenters who say, "You asked for it", get a clue. Commenters have suggested that the title should have clued people into what was coming: "I'm totally flabbergasted no one has stated the obvious here: if you see a presentation labeled "CouchDB: Perform like a pr0n star" and you choose to go to it, don't act all surprised when R-rated images are used as props." Sorry, no biscuit. Presenters use analogies and imagery all the time in their titles in order to "sell" their talks. Recently I was part of a talk that was labeled as a "smackdown"—did that mean the audience should have expected to see images of physical violence? If I title my next talk as something that's "hard-core", should you expect to see images of ball gags and snuff film clips? This is what happens when we co-opt terms like "smackdown" and "hard-core"—you can't fall back to the original meanings and then claim ignorance when people misunderstand how you're going to use them. (God only knows what Mr. Aimonetti would have done for a presentation on "Naked Objects". *shudder*)
Matt Aimonetti, you are an idiot. You could have had your joke and keep it tasteful too. You do, in fact, from time to time in the early part of the presentation: the photo of the "little blue pills" was perfect, offering a hint as to what you meant while keeping the double-entendre alive. Every single "objectionable" photo in that presentation could have been replaced by a more subtle one that kept everybody's mind on the subject and still got the point across. The fact that you resorted to the heavy-handed imagery only proves that you wanted to beat the audience's head with it.
Please, let the one-ups-manship stop. Can we please agree that moving and powerful presentations can be done without having to resort to cheap tricks? They almost always come off badly, particularly when you have to keep the gag running for a full hour or so. Anybody remember Marc Fleury's "Joker" retinue at TheServerSide a half-decade ago? Can you tell me what his presentation was about? Now, consider Dave Thomas' "Cargo Cults" talk from NFJS around the same time—what was he covering? If you were there for both talks, chances are you remember Dave's talk far better than you remember the Fleury keynote beyond the fact that he wore Joker face paint the entire time. Good presentations are about using humor to underscore and support the message, and not making humor the central point of the message. Think about that before you start reaching for the bad innuendo.
Is this really the kind of industry we want? Granted, it may seem like all of this is way overblown if you're a 25-year-old guy recently graduated from college and hacking on your first or second Rails project. "What do these grumpy idiots not understand about 'it's a joke'? My God, is everybody nuts? Are they trying to say that we can't have fun at work or with what we do?" To which all I can say is two things: one, check in with yourself five or ten years from now, when your daughters are learning about body images by staring at pictures of women who are entirely artificial (and yes, guys, those pictures you see are entirely artificial, having been touched up and enhanced in many ways), and two, you're more than welcome to have whatever jokes you like at whomever's expense you like, in private. This wasn't in private. A developer conference is not a private locale. More importantly, though, think about it—when you bring your girlfriend to work, do you want her hearing those same jokes that buddies toss off back and forth? What seems like "harmless fun" now, may have a very different feel to it for you a few years from now.

I'll freely admit, I drop profanity from time to time in my presentations. And to everyone who comes up afterwords (figuratively and literally) saying I shouldn't use such offensive language, I apologize, and point out that I did so in order to underscore the point, knowing that I'm taking that risk, and knowing that I may be required to offer up apologies after the fact for having offended them. (To date, those apologies still number in the single digits.) So perhaps I am no better than Mr. Aimonetti in the final accounting of things.

But all of this loses sight of a core principle. Regardless of the efficacy of his presentation, regardless of your feelings about the subject matter, regardless of your thoughts around the overblown-or-not nature of this discussion, a deeper principle is at stake here, that of professional presentation etiquette: Mr. Aimonetti, you owe an apology to anyone and everyone that was offended by your presentation (for whatever reason). Failure to deliver that, in my mind, equates to a personal and professional FAIL on your part.

When you stand up on stage, and you say something that somebody finds offensive, you owe that person an apology, even if you think their reasoning or rationale is bogus.

It's simple common courtesy.

Conferences | Industry | Ruby | Social

Thursday, April 30, 2009 11:03:32 PM (Pacific Daylight Time, UTC-07:00)
Comments [3] |

"From each, according to its abilities...."

Recently, NFJS alum and buddy Dion Almaer questioned the widespread, almost default, usage of a relational database for all things storage related:

Ian Hickson: “I expect I’ll be reverse-engineering SQLite and speccing that, if nothing better is picked first. As it is, people are starting to use the database feature in actual Web apps (e.g. mobile GMail, iirc).”

When I read that comment to Vlad’s post on HTML 5 Web Storage I gulped. This would basically make SQLite the HTML 5 for storage in the browser. You would have to be a little crazy to re-write the exact semantics (including bugs) of SQLite and its dialect. What if you couldn’t use the public domain code?

Gears lead out strong with making a relational database part of the toolbox for developers. It embedded its own SQLite, in fact one that was customized to have the very cool full text search ability. However, this brings up the point of “which SQLite do you standardize on?”

The beauty of using SQL and SQLite is that many developers already know it. RDBMS has been mainstream for donkey’s years; we have tools to manage SQL, to view the model, and to tweak for performance. It has gone through the test of time.

However, SQL has always been at odds with many developers. Ted Neward brought up ORM as the vietnam of computer science (which is going a touch far ;). I was just lamenting with a friend at Microsoft on how developers spend 90% of their time munging data. Our life is one of transformations, and that is why I am interested in a world of JavaScript on client and server AND database. We aren’t there yet, but hopefully we can make progress.

One of Vlad’s main questions is “Is SQL the right API for Web developers?” and it is a valid one. I quickly found that for most of my tasks with the DB I just wanted to deal with JSON and hence created a wrapper GearsDB to let me insert/update/select/delete the database with a JSON view of the world. You probably wouldn’t want to do this on large production applications for performance reasons, but it works well for me.

Now a days, we have interesting APIs such as JSONQuery which Persevere (and other databases) use. I would love to see Firefox and other browsers support something like this and let us live in JSON throughout the stack. It feels so much more Webby, and also, some of the reasons that made us stay with SQL don’t matter as much in the client side world. For example, when OODBMS took off in some Enterprises, I remember having all of these Versant to Oracle exports just so people could report on the darn data. On the client the database is used for a very different reason (local storage) so lets use JSON!

That being said, at this point there are applications such as Gmail, MySpace search, Zoho, and many iPhone Web applications that use the SQL storage in browsers. In fact, if we had the API in Firefox I would have Bespin using it right now! We had a version of this that abstracted on top of stores, but it was a pain. I would love to just use HTML 5 storage and be done.

So, I think that Firefox should actually support this for practical reasons (and we have SQLite right there!) but should push JSON APIs and let developers decide. I hope that JSON wins, you? I also hope that Hixie doesn’t have to spec SQLite :/

Dion's right when he says "developers spend 90% of their time munging data" and that "Our life is one of transformations", but I think he's being short-sighted and entirely narrow-minded when he says, "I am interested in a world of JavaScript on client and server AND database." Dion, I love you, man, but you're falling prey to the Fallacy of the One True Language. JavaScript (or ECMAScript, as its official name is given) is an interesting and powerful language, but why do you want to force your biases and perceptions on the rest of the world, man? You're being just as bad as the C++ or Java guys were in their heyday—remember when Java stored procedures were all the rage because "everybody knows that Java is the wave of the future"?

The fact is, from where I stand, there is no one storage solution or language solution or user-interface solution that is the Right Thing To Do in all situations. Not even inside the browser. There will be situations where a SQLite is the Right Thing, and other situations where a document-oriented JSON-like or CouchDB-like thing will be the Right Thing, and trying to force-feed one into a situation that's best solved by the other is a bad idea.

Dion alludes to my article about the Vietnam of Computer Science, but in fact, his suggestion charges right into another quagmire—how long before somebody starts trying to create a JSON-to-RDBMS adaption layer? Or JSON-to-CouchDB? Or things equally ridiculous? The fact is, data has three fundamentally different "shapes" to it, and trying to pound data from one shape into the other has all the efficacy and elegance to it just as much as pounding round pegs into square holes does. Dion even alludes to this with this paragraph:

One of Vlad’s main questions is “Is SQL the right API for Web developers?” and it is a valid one. I quickly found that for most of my tasks with the DB I just wanted to deal with JSON and hence created a wrapper GearsDB to let me insert/update/select/delete the database with a JSON view of the world. You probably wouldn’t want to do this on large production applications for performance reasons, but it works well for me.

JSON is certainly an attractive representation format for ECMAScript objects, thanks to its fundamental roots in ECMAScript's object literal syntax, and the powerful/dangerous eval() functionality offered by ECMAScript environments, but JSON also lacks a number of things a SQL-based dialect has, including a powerful query syntax for selecting individual and subsets of entities from the whole, which only becomes more and more necessary as the data base itself gets larger and larger. (Anybody who suggests that a local browser store would only remain within a certain size is clearly not thinking further ahead than the current day. Look at how cookies are outrageously abused as local storage for a lot of sites, or how Viewstate was abused in early ASP.NET apps—if you give the HTML/front-end developer a local storage mechanism, they will use it, and use it as far and as long and as hard as they can.) On top of which, JSON simply doesn't have the years of solid backing behind it than a SQL-based storage format does. And so on, and so on, and so on.

Ironically, just as JSON is a scheme for representing native objects in some kind of data format (in this case, a plain-text one), developers casually ignore the idea of storing objects in a native data format with all of the other bells-and-whistles that a database provides. Naturally, I'm referring to the idea of an object database—if JSON is appropriate for storing certain kinds of data in certain scenarios, then why isn't it appropriate to consider a native object database for some of those same certain kinds of scenarios? Not that I have anything against a JSON-based database scenario—in fact, I can easily imagine a JSON database that indexes the properties of the stored objects and takes ECMAScript functions as "native queries" in the same way that db4o doe. But let's stop with the repeated attempts at "one size fits all", and just accept that the world is a polyglot world, and that no one language—or data storage format, or data access API—will be the Right Thing To Do for all scenarios. Each language, format, API or tool has a reason to exist, a particular way it looks at the world, and optimizes itself to work best when used in that particular style. Trying to force one into the terms of the other is the road to another Computer Science quagmire.

Viva la Polyglot!

Out, out, you damn foreigners!

A friend of mine, from Canada, recently decided not to come to the US anymore.

Today was my final time trying to enter the US to do what many other people have done in my industry before: go and speak at a conference.

The reason I was given this time was that although I had forfeit the speaking fee they were going to pay me, I was still going to be speaking at a conference where other speakers were getting paid, and that there was no reason an American couldn’t fill that spot. When I asked if there would have been any issue if the conference was a free one and nobody was getting paid, I didn’t get an answer.

D'Arcy's experience at the border control reflects a growing dilemma that other speakers in my industry have also been facing: when you travel overseas to speak at a conference, and you get the dreaded "What are you here for?" question, should you tell them the truth and face the battery of questions that boil down to "Are you taking any money out of the country?", or should you lie, claim you're on vacation, and point out how you're putting money into the country in question?

Particularly when the organizers of the conference have every reason to prefer people at home—financial, lack of cultural barriers, reduced language barriers, and more—and invite me to come speak, anyway?

Note that because the US Border Patrol apparently Googles people when they stop at the border,

This all started of course when I was up-front and honest about the speaking engagement the first time I went through, which flagged me in their system. This became very obvious this past weekend when I attended the Twin Cities Code Camp and was at the border for an hour. On that entry I specified that I was going for a shopping weekend, which I was; I was also planning on going to the Twin Cities Code Camp, a free event and one that I was volunteering at. I didn’t mention that because why confuse the issue trying to explain what a code camp was, that it was free, and why I would consider speaking for free. This was a mistake for two reasons…

For one, they do have internet at CBP offices. So if you’re flagged, and you have to go for secondary interviewing, realize that you may be Googled. And as such, blog posts talking about said code camp or eating a Chipotle Burrito may appear as well (“So how was the burrito?” was a question I was asked).

... and because there's no reason to assume other nations' border patrol won't do the same thing, I'm not going to answer that question. I don't want my views aired on a public forum and in the context of a particular discussion acting as a convenient reason for a bureaucrat to create heartache for the citizens of his country that are expecting me to come and help them be more useful and productive and competitive.

D'Arcy's spot-on right on one point, and I applaud him for saying it:

Canadians have long taken for granted our border with the USA. If there’s one thing this experience has taught me, its that there is an air of entitlement that we’ve had in regards to being able to cross over and do whatever we want in the US. We assume that we’ll be as welcome as we were in the past, and that there really isn’t that much difference between us: we drive the same cars, watch the same television and movies, listen to the same music, read the same books.

That "entitlement" isn't limited to just Canadians—other citizens of other Western nations, including my own, feel that same sense of entitlement. Border control is just a hassle, just another annoying obstacle keeping me from my travel destination, just like airport security and agricultural inspections. (Having lived in Stamford, CT in the 70s when entire forests were being depopulated by some sort of caterpiller/moth infestation, and in LA in the 80s when we had to stay indoors at night as the authorities did overhead spraying of Malathion over our house at night to kill off the fruit fly infestation, I'm really kinda sensitive to the need for those ag inspections.)

But the fact is, you are leaving your country, and the laws you grew up with, and entering a new country, one which owes you nothing.

But we are different. We are separate, independent entities with different history, values, and morals. So to the second reason why that was a mistake: I, as a Canadian, have no right to make a call as to whether I’m of a benefit to a neighbouring country. I can rationalize all I want that the event is free, and that I’m actually trying to help other Americans by sharing my knowledge, but that’s not my call to make.

The US is in a state of protectionism right now whether they admit it or not. When you continue to hear about the vast number of jobs being lost, it makes sense that they want to ensure their people are being protected first and foremost. Many of those people include friends of mine whose companies are laying off people.

(By the way, D'Arcy, you misspelled "neighboring".)

As much as D'Arcy has the right attitude about the ways in which nations get to make decisions for their little plots of land upon the earth, and our ability to argue with them, I still want to point out that the whole economic protectionist argument has been used before, and it's pretty much been debunked at a number of levels. (I'm not going down the path of talking about border security, which is a different issue entirely and not what stops D'Arcy from coming to the US.)

The debate around protectionism has been around as long as people have studied economics as a formal "science", and the end results are pretty clear: everybody benefits when the borders are open and unrestricted. The "multiplier effect" that macroeconomists talk about more than makes up for whatever "drain" a foreigner imposes on the local economy.

Note: For those of you who haven't heard of the multiplier effect, it works like this: while in the US to speak at whatever conference he wants to speak at, D'Arcy spends a dollar at a hotel gift shop, of which the hotel uses $.95 to pay its local worker's hourly wage, of which the worker spends $.90 on a hot dog for lunch, of which the hot dog stand operator uses $.85 to buy buns for tomorrow's customers.... And so on. Why aren't we spending the full dollar each time? Mostly because people will often save some portion of that dollar (unless you're American, because we don't save anything, it seems), and because the government will take some portion of that dollar each time in taxes. What this means, though, is that the US$1 that D'Arcy spent turned into US$4 or US$5 or more towards the total GDP of the country. Econ is a fascinating subject sometimes.

And, of course, ask any three economists a question, and you'll get five different answers. This subject is no different: protectionism has its proponents, too, usually when the local economy is taking a hit... like now. It feels right, protecting those who are "close to home" (and believe me, I'm sympathetic, I've had friends who've pinged me about finding a new job within the last six months), but in the end, everything it does is artificial—in much the same way that unions artificially keep wages high for union workers, and impose some serious constraints on the companies that employ them. (I don't think it's an accident that industries being hammered mercilessly by the soft economy—the auto manufacturers and the airlines—are also ones with large union populations.) Protectionism is almost always a short-term gain, long-term loss kind of operation. The "perennial gale of creative destruction" (from Alan Greenspan's Age of Turbulence) isn't always gentle, but it is necessary.

D'Arcy, in the end, closes his piece with a fond wish:

My hope is that at some point the US and Canada will be able to get back to where our countries were before 9/11. At the same time though, I hope that Canada realizes during this time that it has its own identity; that we are more than just who we border against. Maybe locking down the border will become a good thing after all.

Frankly, my wish would be for Canadians to realize their own identity (and I think Canadians are pretty aware of this in the same way that Americans don't even realize that it's a problem), as well, assuming that's even a problem. What's more, I think that Canadians will find that they don't need the US nearly as much as Americans like to think they do.

But locking down the border is going to affect more than just Canadians—my fear is that this protectionist attitude will in fact deter other really bright people from coming to the US and sharing their knowledge and wisdom, or even just participating in our economy for a while. Assume for just a moment that the million or so H-1B visas currently allocated are suddenly all revoked and their holders must return to their countries of origin—how many rent checks, car payments, utility bills, movie nights, dinners at local restaurants and bank accounts are going to be exiled with them? And this doesn't even begin to touch the potentials for racism that lurk hidden within the system—granting visas and citizenship more easily to "Westerners" (Brits, Germans, Australians, whatever) than "foreigners" (Hispanics, Indians, Chinese).

The fact is, this "locking down the border" won't help us, in the long-term. Whatever benefits we as Americans accrue from keeping our jobs intact will be lost when those barriers finally come down and we find we can't compete on the global scale. The "perennial gale of creative destruction" can't be bought off, it can only be delayed. (Ask the ~~Soviets~~ Russians about their success with the high-protectionist tactic the next time you're in Moscow or St. Petersburg.)

At some point, the borderless Internet is going to come crashing against the bordered "real world", and it's not going to be a pretty fight. And we, those of us who define and shape and act as the primary consumer and producer of the Internet's benefits, are going to find ourselves facing some uncomfortable choices.

In the meantime, however this story ends, I want to be able to say that my country acted in its own defense, but without prejudice, malice, or ignorance. But I'm very worried that I won't be able to say that... and I'm worried what damage we will do to ourselves in the interim.

(Editor's note: It will be fascinating to see how many people call me an ignorant racist based on nothing more than the blog title. You want to disagree with me, that's fine, just do so on a material basis from the body of the post, not just the title.)

Conferences | Industry | Social

Wednesday, April 08, 2009 2:47:21 PM (Pacific Daylight Time, UTC-07:00)
Comments [12] |

"Multi-core Mania": A Rebuttal

The Simple-Talk newsletter is a monthly e-zine that the folks over at Red Gate Software (makers of some pretty cool toys, including their ANTS Profiler, and recent inheritors of the Reflector utility legacy) produce, usually to good effect.

But this month carried with it an interesting editorial piece, which I reproduce in its entirety here:

When the market is slack, nothing succeeds better at tightening it up than promoting serial group-panic within the community. As an example of this, a wave of multi-core panic spread across the Internet about 18 months ago. IT organizations, it was said, urgently had to improve application performance by an order of magnitude in order to cope with rising demand. We wouldn't be able to meet that need because we were at the "end of the road" with regard to step changes in processor power and clock speed. Multi-core technology was the only sure route to improving the speed of applications but, unfortunately, our current "serial" programming techniques, and the limited multithreading capabilities of our programming languages and programmers, left us ill-equipped to exploit it. Multi-core mania gripped the industry.

However, the fever was surprisingly short-lived. Intel's "largest open-source effort ever" to provide a standard tool for writing multi-threaded code, caused little more than a ripple of interest. Various books, rushed out while the temperature soared, advocated the urgent need for new "multi-core-friendly" programming models, involving such things as "software pipelines". Interesting as they undoubtedly are, they sit stolidly on bookshelves, unread.

The truth is that it's simply not a big issue for the majority of people. Writing truly "concurrent" applications in languages such as C# is difficult, as you get very little help from the language. It means getting involved with low-level concurrency primitives, such as lock statements and so on.

Many programmers lack the skills to do this, but more pertinently lack the need. Increasingly, programmers work in a web environment. As long as these web applications are deployed to a load-balanced web farm, then page requests can be handled in parallel so all available cores will be used efficiently without the need for the programmer to be concerned with fine-grained parallelism.

Furthermore, the SQL Server engine behind these web applications is intrinsically "parallel", and can handle and use effectively about as many cores as you care to throw at it. SQL itself is a declarative rather than procedural language, so it is fundamentally concurrent.

A minority of programmers, for example games programmers or those who deal with "embarrassingly parallel" desktop applications such as Photoshop, do need to start working with the current tools and 'low-level' coding techniques that will allow them to exploit multi-core technology. Although currently perceived to be more of "academic" interest, concurrent languages such as Erlang, and concurrency techniques such as "software transactional memory", may yet prove to be significant.

For most programmers and for most web applications, however, the multi-core furore is a storm in a teacup; it's just not relevant. The web and database platforms already cope with concurrency requirements. We are already doing it.

My hope is that this newsletter, sent on April 1st, was intended to be a joke. Having said that, I can’t find any verbage in the email that suggests that it is, in which case, I have to treat it as a legitimate editorial.

And frankly, I think it’s all crap.

It's dangerously ostrichian in nature—it encourages developers to simply bury their heads in the sand and ignore the freight train that's coming their way. Permit me, if you will, a few minutes of your time, that I may be allowed to go through and demonstrate the reasons why I say this.

To begin ...

When the market is slack, nothing succeeds better at tightening it up than promoting serial group-panic within the community. As an example of this, a wave of multi-core panic spread across the Internet about 18 months ago. IT organizations, it was said, urgently had to improve application performance by an order of magnitude in order to cope with rising demand. [...] Multi-core mania gripped the industry.

Point of fact: The “panic” cited here didn’t start about 18 months ago, it started with Herb Sutter’s most excellent (and not only highly recommended but highly required) article, “The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software”, appeared in the pages of Dr. Dobb’s Journal in March of 2005. (Herb’s website notes that “a much briefer version under the title “The Concurrency Revolution” appeared in C/C++ User’s Journal” the previous month.) And the panic itself wasn’t rooted in the idea that we weren’t going to be able to cope with rising demand, but that multi-core CPUs, back then a rarity and reserved only for hardware systems in highly-specialized roles, were in fact becoming commonplace in servers, and worse, as they migrated into desktops, they would quickly a fact of life that every developer would need to face. Herb demonstrated this by pointing out that CPU speeds had taken an interesting change of pace in early 2003:

Around the beginning of 2003, [looking at the website Figure 1 graph] you’ll note a disturbing sharp turn in the previous trend toward ever-faster CPU clock speeds. I’ve added lines to show the limit trends in maximum clock speed; instead of continuing on the previous path, as indicated by the thin dotted line, there is a sharp flattening. It has become harder and harder to exploit higher clock speeds due to not just one but several physical issues, notably heat (too much of it and too hard to dissipate), power consumption (too high), and current leakage problems.

Joe Armstrong, creator of Erlang, noted in a presentation at QCon London 2007 that another of those physical limitations was the speed of light—that for the first time, CPU signal couldn't get from one end of the chip to the other in a single clock cycle.

Quick: What’s the clock speed on the CPU(s) in your current workstation? Are you running at 10GHz? On Intel chips, we reached 2GHz a long time ago (August 2001), and according to CPU trends before 2003, now in early 2005 we should have the first 10GHz Pentium-family chips.

Just to (re-)emphasize the point, here, now, in early 2009, we should be seeing the first 20 or 40 GHz processors, and clearly we’re still plodding along in the 2 – 3 GHz range. The "Quake Rule" (when asked about perf problems, tell your boss you'll need eighteen months to get a 2X improvement, then bury yourselves in a closet for 18 months playing Quake until the next gen of Intel hardware comes out) no longer works.

For the near-term future, meaning for the next few years, the performance gains in new chips will be fueled by three main approaches, only one of which is the same as in the past. The near-term future performance growth drivers are:

hyperthreading

multicore

cache

Hyperthreading is about running two or more threads in parallel inside a single CPU. Hyperthreaded CPUs are already available today, and they do allow some instructions to run in parallel. A limiting factor, however, is that although a hyper-threaded CPU has some extra hardware including extra registers, it still has just one cache, one integer math unit, one FPU, and in general just one each of most basic CPU features. Hyperthreading is sometimes cited as offering a 5% to 15% performance boost for reasonably well-written multi-threaded applications, or even as much as 40% under ideal conditions for carefully written multi-threaded applications. That’s good, but it’s hardly double, and it doesn’t help single-threaded applications.

Multicore is about running two or more actual CPUs on one chip. Some chips, including Sparc and PowerPC, have multicore versions available already. The initial Intel and AMD designs, both due in 2005, vary in their level of integration but are functionally similar. AMD’s seems to have some initial performance design advantages, such as better integration of support functions on the same die, whereas Intel’s initial entry basically just glues together two Xeons on a single die. The performance gains should initially be about the same as having a true dual-CPU system (only the system will be cheaper because the motherboard doesn’t have to have two sockets and associated “glue” chippery), which means something less than double the speed even in the ideal case, and just like today it will boost reasonably well-written multi-threaded applications. Not single-threaded ones.

Finally, on-die cache sizes can be expected to continue to grow, at least in the near term. Of these three areas, only this one will broadly benefit most existing applications. The continuing growth in on-die cache sizes is an incredibly important and highly applicable benefit for many applications, simply because space is speed. Accessing main memory is expensive, and you really don’t want to touch RAM if you can help it. On today’s systems, a cache miss that goes out to main memory often costs 10 to 50 times as much getting the information from the cache; this, incidentally, continues to surprise people because we all think of memory as fast, and it is fast compared to disks and networks, but not compared to on-board cache which runs at faster speeds. If an application’s working set fits into cache, we’re golden, and if it doesn’t, we’re not. That is why increased cache sizes will save some existing applications and breathe life into them for a few more years without requiring significant redesign: As existing applications manipulate more and more data, and as they are incrementally updated to include more code for new features, performance-sensitive operations need to continue to fit into cache. As the Depression-era old-timers will be quick to remind you, “Cache is king.”

Herb’s article was a pretty serious wake-up call to programmers who hadn’t noticed the trend themselves. (Being one of those who hadn’t noticed, I remember reading his piece, looking at that graph, glancing at the open ad from Fry’s Electronics sitting on the dining room table next to me, and saying to myself, “Holy sh*t, he’s right!”.) Does that qualify it as a “mania”? Perhaps if you’re trying to pooh-pooh the concern, sure. But if you’re a developer who’s wondering where you’re going to get the processing power to address the ever-expanding list of features your users want, something Herb points out as a basic fact of life in the software development world ...

There’s an interesting phenomenon that’s known as “Andy giveth, and Bill taketh away.” No matter how fast processors get, software consistently finds new ways to eat up the extra speed. Make a CPU ten times as fast, and software will usually find ten times as much to do (or, in some cases, will feel at liberty to do it ten times less efficiently).

... then eking out the best performance from an application is going to remain at the top of the priority list. Users are classic consumers: they will always want more and more for the same money as before. Ignore this truth of software (actually, of basic microeconomics) at your peril.

To get back to the editorial, we next come to ...

However, the fever was surprisingly short-lived. Intel's "largest open-source effort ever" to provide a standard tool for writing multi-threaded code, caused little more than a ripple of interest. Various books, rushed out while the temperature soared, advocated the urgent need for new "multi-core-friendly" programming models, involving such things as "software pipelines". Interesting as they undoubtedly are, they sit stolidly on bookshelves, unread.

Wow. Talk about your pretty aggressive accusation without any supporting evidence or citation whatsoever.

Intel's not big into the open-source space, so it doesn't take much for an open-source project from them to be their "largest open-source effort ever". (What, they're going to open-source the schematics for the Intel chipline? Who could read them even if they did? Who would offer up a patch? What good would it do?) The fact that Intel made the software available in the first place meant that they knew the hurdle that had yet to be overcome, and wanted to aid developers in overcoming it. They're members of the OpenMP group for the same reason.

Rogue Wave's software pipelines programming model is another case where real benefits have accrued, backed by case studies. (Disclaimer: I know this because I ghost-wrote an article for them on their Software Pipelines implementation.) Let's not knock something that's actually delivered value. Pipelines aren't going to be the solution to every problem, granted, but they're a useful way of structuring a design, one that's curiously similar to what I see in functional programming languages.

But simply defending Intel's generosity or the validity of an alternative programming model doesn't support the idea that concurrency is still a hot topic. No, for that, I need real evidence, something with actual concrete numbers and verifiable fact to it.

Thus, I point to Brian Goetz’s Java Concurrency in Practice, one of those “books, rushed out while the temperature soared”, which also turned out to be the best-selling book at Java One 2007, and the second-best-selling book (behind only Joshua Bloch’s unbelievably good Effective Java (2nd Ed) ) at Java One 2008. Clearly, yes, bestselling concurrency books are just a myth, alongside the magical device that will receive messages from all over the world and play them into your brain (by way of your ears) on demand, or the magical silver bird that can wing its way through the air with no visible means of support as it does so. Myths, clearly, all of them.

To continue...

The truth is that it's simply not a big issue for the majority of people. Writing truly "concurrent" applications in languages such as C# is difficult, as you get very little help from the language. It means getting involved with low-level concurrency primitives, such as lock statements and so on.

Many programmers lack the skills to do this, but more pertinently lack the need. Increasingly, programmers work in a web environment. As long as these web applications are deployed to a load-balanced web farm, then page requests can be handled in parallel so all available cores will be used efficiently without the need for the programmer to be concerned with fine-grained parallelism.

He’s right when he says you get very little help from the language, be it C# or Java or C++. And getting involved with low-level concurrency primitives is clearly not in anybody’s best interests, particularly if you’re not a concurrency guru like Brian. (And let’s be honest, even low-level concurrency gurus like Brian, or Joe Duffy, who wrote Concurrent Programming on Windows, or Mike Woodring, who co-authored Win32 Multithreaded Programming, have better things to do.) But to say that they “pertinently lack the need” is a rather impertinent statement. “As long as these web applications are deployed to a load-balanced web farm", which is very likely to continue to happen, “then page requests can be handled in parallel so all available cores will be used …”

Um... excuse me?

Didn’t you just say that programmers didn’t need to learn concurrency constructs? It would strike me that if their page requests are being handled in parallel that they have to learn how to write code that won’t break when it’s accessed in parallel or lead to data-corruption problems or race conditions when their pages are accessed in parallel. If parallelism is a fundamental part of the Web, don’t you think it’s important for them to learn how to write programs that can behave correctly in parallel?

Look for just a moment at the average web application: if data is stored in a per-user collection, and two simultaneous requests come in from a given user (perhaps because the page has AJAX requests being generated by the user on the page, or perhaps because there’s a frameset that’s generating requests for each sub-frame, or ...), what happens if the code is written to read a value from the session, increment it, and store it back? ASP.NET can save you here, a little, in that it used to establish a per-user lock on the entirety of the page request (I don’t know if it still does this—I really have lost any desire to build web apps ever again), but that essentially puts an artificial throttle on the scalability of your system, and makes the end-users’ experience that much slower. Load-balancer going to spray the request all over the farm? So long as the user session state is stored on every machine in the farm, that’ll work... But of course if you store the user’s state in the SQL instance behind each of those machines on the farm, then you take the performance hit of an extra network round-trip (at which point we’re back to concurrency in the database) ...

... all because the programmer couldn’t figure out how to make “lock” work? This is progress?

The Java Servlet specification specifically backed away from this "lock on every request" approach because of the performance implications. I heard a fair amount of wailing and gnashing during the early ASP.NET days over this. I heard the ASP.NET dev team say they made their decision because the average developer can't figure out concurrency correctly anyway.

And, by the way folks, this editorial completely ignores XML services. I guess "real" applications don't write services much, either.

The next part is even better:

Furthermore, the SQL Server engine behind these web applications is intrinsically "parallel", and can handle and use effectively about as many cores as you care to throw at it. SQL itself is a declarative rather than procedural language, so it is fundamentally concurrent.

True… and false. SQL is fundamentally “parallel” (largely because SQL is a non-strict functional language, not just a “declarative” one), but T-SQL isn’t. And how many developers actually know where the line is drawn between SQL and T-SQL? More importantly, though, how many effective applications can be written with a complete ignorance of the underlying locking model? Why do DBAs spend hours tuning the database’s physical constructs, establishing where isolation levels can be turned down, establishing where the scope of a transaction is too large, putting in indexed columns where necessary, and figuring out where page, row, or table locking will be most efficient? Because despite the view that a relational database presents, these queries are being executed in parallel, and if a developer wants to avoid writing an application that requires a new server for each and every new user added to the system, they need to learn how to maximize their use of the database’s parallelism. So even if the language is "fundamentally concurrent" and can thus be relied upon to do the right thing on behalf of the developer, the implementation isn't, and needs to be understood in order to be implemented efficiently.

He finishes:

For most programmers and for most web applications, however, the multi-core furore is a storm in a teacup; it's just not relevant. The web and database platforms already cope with concurrency requirements. We are already doing it.

This is one of those times I wish I had a time machine handy—I'd love to step forward five years, have a look around, then come back and report the findings. I'm tempted to close with the challenge to just let’s come back in five years and see what the programming language landscape and hardware landscape looks like. But that's too easy an "out", and frankly, doesn't do much to really instill confidence, in my opinion.

To ignore the developers building "rich" applications (be they being done in Flex/Flash, Cocoa/iPhone, WinForms, Swing, WPF, or what-have-you) is to also ignore a relatively large segment of the market. Not every application is being built on the web and is backed by a relational database—to simply brush those off and not even consider them as part of the editorial reveals a dangerous bias on the editor's part. And those applications aren't hosted in an "intrinsically 'parallel'" container that developers can just bury their head inside.

Like it or not, folks, the path forward isn't one that you get to choose. Intel, AMD, and other chip manufacturers have already made that clear. They're not going to abandon the multicore approach now, not when doing so would mean trying to wrestle with so many problems (including trying to change the speed of light) that simply aren't there when using a multicore foundation. That isn't up for debate anymore. Multicore has won for the forseeable future. And, as a result, multicore is going to be a fact of the developer's life for the forseeable future. Concurrency is thus also a fact of the developer's life for the forseeable future.

The web and database platforms “cope” with concurrency requirements by either making "one-size-fits-all" decisions that almost always end up being the wrong decision for high-scale systems (but I'm sure your new startup-based idea, like a system that allows people to push "micro-entries" of no more than 140 characters in length to a publicly-trackable feed would never actually take off and start carrying millions and millions of messages every day, right?), or by punting entirely and forcing developers to dig deeper beneath the covers to see the concurrency there. So if you're happy with your applications running no faster than 2GHz for the rest of the forseeable future, then sure, you don't need to worry about learning concurrency-friendly kinds of programming techniques. Bear in mind, by the way, that this essentially locks you in to small-scale, web-plus-database systems for the forseeable future, and clearly nothing with any sort of CPU intensiveness to it whatsoever. Be happy in your niche, and wave to the other COBOL programmers who made the same decision.

This is a leaky abstraction, full stop, end of story. Anyone who tells you otherwise is either trolling for hits, trying to sell you something, or striving to persuade developers that ignorance isn't such a bad place to be.

All you ignorant developers, this is the phrase you will be forced to learn before you start your next job: "Would you like fries with that?"

Laziness in Scala

While playing around with a recent research-oriented project for myself (more on that later), I discovered something that I haven't seen mentioned anywhere in the Scala universe before. (OK, not really--as you'll see towards the end of this piece, it really is documented, but allow me my brief delusions of grandeur as I write this. They'll get deflated quickly enough.)

So the core of the thing was a stack-oriented execution engine; essentially I'm processing commands delivered in a postfix manner. Since some of these commands are relational operators, it's important that there be two things to relationally operate on the execution stack, after which I want to evaluate the relational operation and push its result (1 if true, 0 if false) back on the stack; this is pretty easily done via the following:

def compareOp(op : (Int, Int) => Boolean) =
{
  checkStack(2)
  val v1 = (execStack.pop()).asInstanceOf[Int]
  val v2 = (execStack.pop()).asInstanceOf[Int]
  val vr = op(v1, v2)
  execStack.push(if (vr) 1 else 0)
}

where "execStack" is a mutable.Stack[Any] held in an enclosing function.

Interestingly enough, however, when I wrote this the first time, I wrote it like this, which is a very different sequence of operations:

def compareOp(op : (Int, Int) => Boolean) =
{
    checkStack(2)
    def v1 = (execStack.pop()).asInstanceOf[Int]
    def v2 = (execStack.pop()).asInstanceOf[Int]
    def vr = op(v1, v2)
    execStack.push(if (vr) 1 else 0)
}

See the difference? Subtle, is it not? But the actual code is significantly different, something that's more easily seen with a much simpler (and standalone) example:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        def v1 = (stack.pop()).asInstanceOf[Int]
        def v2 = (stack.pop()).asInstanceOf[Int]
        def vr = v1 + v2
        System.out.println(vr)
    }
}

When run, the console prints out "36", as we'd well expect.

But suppose we want to look at those values of v1 and v2 along the way, perhaps as part of a logging operation, or perhaps because you're just screwing around with some ideas in your head and you don't want to bother to fire up an IDE with Scala support in it. So you decide to spit those values to a console:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        def v1 = (stack.pop()).asInstanceOf[Int]
        def v2 = (stack.pop()).asInstanceOf[Int]
        System.out.println(v1)
        System.out.println(v2)
        def vr = v1 + v2
        System.out.println(vr)
    }
}

And then something *very* different happens; you get "24", "12", and then a NoSuchElementException.

If you're like me the first time I ran into this, your first reaction is, "Eh?". Actually, if you're like me, when you're programming, your profanity filters are probaby at an ebb, so your first reaction is "WTF?!?", said with great gusto and emphasis. Which has a tendency to get some strange looks when you're at a Denny's doing your research, I will admit. Particularly when it's at 3 AM in the morning. And the bar crowd is in full alcoholic haze and slightly nervous about the long-haired, goatee-sporting guy in his headphones, wearing his black leather jacket and swearing like a drunken sailor at his laptop. But I digress.

What is Scala doing here?

Turns out this is exactly as the language designers intended, but it's subtle. (Or maybe it's just subtle to me at 3AM when I'm pumped full of caffeine.)

Let's take this a different way:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        def v1 = (stack.pop()).asInstanceOf[Int]
        def v2 = (stack.pop()).asInstanceOf[Int]
        System.out.println(stack)
    }
}

When run, the console prints "Stack(12, 24)", which *really* starts to play with your mind when you're a little short on sleep and a little high on Diet Coke. At first glance, it looks like Scala is broken somehow--after all, those "pop" operations are supposed to modify the Stack against which they're operating, just as the push()es do. So why is the stack convinced that it still holds the values of 12 and 24?

Because Scala hasn't actually executed those pop()s yet.

The "def" keyword, it turns out, isn't what I wanted here--what I wanted (and in retrospect it’s painfully obvious) was a "val", instead, in order to force the execution of those statements and capture the value into a local value (an immutable local variable). The "def" keyword, instead, creates a function binding that waits for formal execution before evaluating. So that when I previously said

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        def v1 = (stack.pop()).asInstanceOf[Int]
        def v2 = (stack.pop()).asInstanceOf[Int]
        def vr = v1 + v2
        System.out.println(vr)
    }
}

… what in fact I was saying was this:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        def v1 = (stack.pop()).asInstanceOf[Int]
        def v2 = (stack.pop()).asInstanceOf[Int]
        System.out.println(v1 + v2)
    }
}

… which is the same as:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        System.out.println((stack.pop()).asInstanceOf[Int] + (stack.pop()).asInstanceOf[Int])
    }
}

… which, when we look back at my most recent "debugging" version of the code, substituting the "def"ed versions of v1 and v2 (and vr) where they're used, makes the reason for the NoSuchElementException become entirely more clear:

object App
{
    def main(args : Array[String]) =
    {
        import scala.collection.mutable.Stack
        var stack : Stack[Any] = new Stack()
        stack.push(12)
        stack.push(24)
        System.out.println((stack.pop()).asInstanceOf[Int])
        System.out.println((stack.pop()).asInstanceOf[Int])
        System.out.println((stack.pop()).asInstanceOf[Int] + (stack.pop()).asInstanceOf[Int])
    }
}

Now, normally, this would probably set off all kinds of alarm bells in your head, but the reaction that went off in mine was "COOL!", the reasons for which revolve around the concept of "laziness"; in a functional language, we frequently don't want to evaluate the results right away, instead preferring to defer their execution until actually requiring it. In fact, many functional languages—such as Haskell—take laziness to new heights, baking it directly into the language definition and assuming laziness everywhere, so much so that you have to take special steps to avoid it. There’s a variety of reasons why this is advantageous, but I’ll leave those discussions to the Haskellians of the world, like Matt Podwysocki and Simon Peyton-Jones.

From a Scalist’s perspective, laziness is still a useful tool to have in your toolbox. Suppose you have a really powerful function that calculates PI to a ridiculous number of decimal places. In Java, you might be tempted to do something like this:

class MyMath
{
  public static final double PI = calculatePiToARidiculousNumberOfPlaces();
  private static double calculatePiToARidiculousNumberOfPlaces()
  {
    // implementation left to the reader's imagination
    // imagine it being "really cool"
  }
}

The problem with this is that if that method takes any length of time to execute, it's being done during class initialization during its ClassLoading phase, and aside from introducing a window of time where the class *could* be used before that initialization is finished (it's subtle, it's not going to happen very often, but it can, according to older versions of the JVM Spec), the problem is that the time required to do that initialization is paid for *regardless of whether you use PI*. In other words, the classic Stroustrup-ian "Don't pay for it if you don't use it" principle is being completely tossed aside.

In Scala, using the "def" keyword here, aside from avoiding the need for the additional decorators, completely eliminates this cost--people won't need the value of PI until it becomes used:

object App
{
    def PI = calculatePiToARidiculousNumberOfPlaces()
    def calculatePiToARidiculousNumberOfPlaces() =
    {
        System.out.println("Calculating PI")
        3 + 0.14
    }
    def main(args : Array[String]) =
    {
        System.out.println("Entering main")
        System.out.println("PI = " + PI)
    }
}

(In fact, you'd probably just write it without the calculating method definition, since it's easier that way, but bear with me.)

When you run this, of course, we see PI being calculated after main()'s been entered, thus proving that PI is being calculated only on demand, not ahead of time, as a public-static-final-constant would be.

The problem with this approach is, you end up calculating PI on each access:

object App
{
    def PI = calculatePiToARidiculousNumberOfPlaces()
    def calculatePiToARidiculousNumberOfPlaces() =
    {
        System.out.println("Calculating PI")
        3 + 0.14
    }
    def main(args : Array[String]) =
    {
        System.out.println("Entering main")
        System.out.println("PI = " + PI)
        System.out.println("PI = " + PI)
            // prints twice! Not good!
    }
}

Which sort of defeats the advantage of lazy evaluation.

This got me wondering--in F#, we have lazy as a baked-in concept (sort of), such that when I write

#light
let sixty = lazy (30 + 30)
System.Console.WriteLine(sixty)

What I see on the console is not 60, but a Lazy<T> type instance, which effectively defers execution until it's Force() method is invoked (among other scenarios). This means I can write things like

let reallyBigList = lazy ([1..1000000000000] |> complexCalculation |> anotherComplexCalcuation)

without fear of blowing the stack or heap apart, since laziness means the list won't actually be calculated until it's forced; we can see this from the following (from the F# interactive console):

> let sixtyWithSideEffect = lazy (printfn "Hello world"; 30+30);;
val sixtyWithSideEffect: Lazy<int>
> sixtyWithSideEffect.Force();;
Hello world
val it : int = 60
> sixtyWithSideEffect.Force();;
val it : int = 60

(Examples taken from the excellent Expert F# by Syme/Granicz/Cisternino; highly recommended, if a touch out-of-date to the current language definition. I expect Chris Smith’s Programming F#, from O’Reilly, to correct that before too long.)

It would be nice if something similar were doable in Scala. Of course, once I start looking for it, it makes itself visible, in the wonderful Venners/Odersky/Spoon book, Programming In Scala, p. 444:

You can use pre-initialized fields to simulate precisely the initialization behavior
of class constructor arguments. Sometimes, however, you might prefer
to let the system itself sort out how things should be initialized. This can
be achieved by making your val definitions lazy. If you prefix a val definition
with a lazy modifier, the initializing expression on the right-hand side
will only be evaluated the first time the val is used.

[...]

This is similar to the situation where x is defined as a parameterless
method, using a def. However, unlike a def a lazy val is never evaluated
more than once. In fact, after the first evaluation of a lazy val the result of the
evaluation is stored, to be reused when the same val is used subsequently.

Perfect! The key, then, is to define PI like so:

object App
{
    lazy val PI = calculatePiToARidiculousNumberOfPlaces()
    def calculatePiToARidiculousNumberOfPlaces() =
    {
        System.out.println("Calculating PI")
        3 + 0.14
    }
    def main(args : Array[String]) =
    {
        System.out.println("Entering main")
        System.out.println("PI = " + PI)
        System.out.println("PI = " + PI)
            // prints once! Awesome!
    }
}

That means, if I apply it to my Stack example from before, I should get the same deferred-execution properties of the "def"-based version ...

def main(args : Array[String]) =
{
    import scala.collection.mutable.Stack
    var stack : Stack[Any] = new Stack()
    stack.push(12)
    stack.push(24)
    lazy val v1 = (stack.pop()).asInstanceOf[Int]
    lazy val v2 = (stack.pop()).asInstanceOf[Int]
    System.out.println(stack)
        // prints out "Stack(12,24)
}

... but if I go back to the version that blows up because the stack is empty, using lazy val works exactly the way I would want it to:

def main(args : Array[String]) =
{
    import scala.collection.mutable.Stack
    var stack : Stack[Any] = new Stack()
    stack.push(12)
    stack.push(24)
    lazy val v1 = (stack.pop()).asInstanceOf[Int]
    lazy val v2 = (stack.pop()).asInstanceOf[Int]
    System.out.println(v1)
    System.out.println(v2)
    lazy val vr = v1 + v2
    System.out.println(vr)
        // prints 12, 24, then 36
        // and no exception!
}

Nice.

So, it turns out that my accidental use of "def" inside the compareOp function behaves exactly the way the language designers wanted it to, which is not surprising, and that Scala provides nifty abilities to defer processing or extraction of values until called for.

Curiously, the two languages differ in how laziness is implemented; in F#, the lazy modifier defines the type to be a Lazy<T> instance, an ordinary type that we can pass around from F# to C# and back again as necessary (in much the same way that C# defined nullable types to be instances of Nullable<T> under the hood). We can see that from the interactive console output above, and from the fact that we call Force() on the instance to evaluate its value.

In Scala, however, there is no corresponding Lazy[T] type; instead, the PI() method is defined to determine whether or not the value has already been evaluated:

public double PI();
  Code:
   0:   aload_0
   1:   getfield        #135; //Field bitmap$0:I
   4:   iconst_1
   5:   iand
   6:   iconst_0
   7:   if_icmpne       48
   10:  aload_0
   11:  dup
   12:  astore_1
   13:  monitorenter
   14:  aload_0
   15:  getfield        #135; //Field bitmap$0:I
   18:  iconst_1
   19:  iand
   20:  iconst_0
   21:  if_icmpne       42
   24:  aload_0
   25:  aload_0
   26:  invokevirtual   #137; //Method calculatePiToARidiculousNumberOfPlaces:()D
   29:  putfield        #139; //Field PI:D
   32:  aload_0
   33:  aload_0
   34:  getfield        #135; //Field bitmap$0:I
   37:  iconst_1
   38:  ior
   39:  putfield        #135; //Field bitmap$0:I
   42:  getstatic       #145; //Field scala/runtime/BoxedUnit.UNIT:Lscala/runtime/BoxedUnit;
   45:  pop
   46:  aload_1
   47:  monitorexit
   48:  aload_0
   49:  getfield        #139; //Field PI:D
   52:  dreturn
   53:  aload_1
   54:  monitorexit
   55:  athrow
  Exception table:
   from   to  target type
    14    48    53   any

If you look carefully at the bytecode, the implementation of PI is checking a bitmask field (!) to determine if the first bit is flipped (!) to know whether or not the value is held in the local field PI, and if not, calculate it and store it there. This means that Java developers will just need to call PI() over and over again, rather than have to know that the instance is actually a Lazy[T] on which they need to call Value or Force (such as one would from C# in the F# case). Frankly, I don’t know at this point which approach I prefer, but I’m slightly leaning towards the Scala version for now. (If only Java supported properties, then the syntax “MyMath.PI” would look like a constant, act lazily, and everything would be great.)

(It strikes me that the F# developer looking to write something C#-accessible need only tuck the Lazy<T> instance behind a property accessor and the problem goes away, by the way; it would just be nicer to not have to do anything special on either side, to have my laziness and Force() it, too. Pipe dream, perhaps.)

In retrospect, I could wish that Scala weren't *quite* so subtle in its treatment of "def" vs "val", but now that I'm aware of it, it'll (hopefully) not bite me quite so subtly in the sensitive spots of my anatomy again.

And any experience in which you learn something is a good one, right?

.NET | C# | F# | Java/J2EE | Languages | Scala | Visual Basic

Sunday, March 29, 2009 5:18:12 AM (Pacific Daylight Time, UTC-07:00)
Comments [4] |

A new stack: JOSH

An interesting blog post was forwarded to me by another of my fellow ThoughtWorkers, which suggests a new software stack for building an enterprise system, acronymized as “JOSH”:

The Book Of JOSH

Through a marvelous, even devious, set of circumstances, I'm presented with the opportunity to address my little problem without proscribed constraints, a true green field opportunity.

Json OSGi Scala HTTP

Json delivers on what XML promised. Simple to understand, effective data markup accessible and usable by human and computer alike. Serialization/Deserialization is on par with or faster then XML, Thrift and Protocol Buffers. Sure I'm losing XSD Schema type checking, SOAP and WS-* standardization. I'm taking that trade.

OSGi a standardized dynamic, modular framework for versioned components and services. Pick a logger component, a HTTP server component, a ??? component, add your own internal components and you have a dedicated application solution. Micro deployment with true replacement. What am I giving up? The monolithic J2EE application servlet loaded with 25 frameworks, SCA and XML configuration hell. Taking the trade.

HTTP is simple, effective, fast enough, and widely supported. I'm tired of needlessly complex and endless proprietary protocols to move simple data from A to B with all the accompanying firewall port insanity. Yes, HTTP is not perfect. But I'm taking this trade where I can as well.

All interfaces will be simple REST inspired APIs based on HTTP+JSON. This is an immediate consequence of the JOSH stack.

Scala is by far the toughest, yet the easiest selection in the JOSH stack. I wrestled far more with the JSON or XML or Thrift or Protocol Buffers decision.

And, let’s be honest, the stack sounds a lot better than what he was working with before....

[...] Yes, you see, I have a small problem.

So whats the issue, you say? I write a whole blog about nothing, you say? We all know the right answer, you're pointing out? Yea, I know, its intuitively obvious to the casual observer.

We'll rewrite it from scratch.

Course we'll need a cluster of WebSphere Application Servers, and an Oracle RAC cluster for all that data. Don't forget the middleware needed to transition over from the legacy systems, so toss in an ESB cluster, and what heck a couple of BPEL servers too.

Need a SOA Center of Excellence of course too. Can't integrate without some common XML Business Object Schemas. Also need to roll the Rational RUP suite and some beefy IDE environments and for that shiny look, sprinkle the works with lots of WS-* sparkly dust. Bake 3-5 years or until done, whenever.

My presentation slides for all this will be killer. I can sell this stuff. I'm good at it. I'll look like a bloody genius. I'll have Vendors fawning all over me. And the best part is the bubble on this mess won't pop for YEARS, when I'll have plenty of plausible deniability. "Hey the plan was perfect, the business, IT managers and their people were incapable of executing it."

I feel like the enterprise IT equivalent of an AIG trader pocketing ill gotten gains from writing Credit Default Swaps that we can't pay off.

Ewww... even thinking about all that makes me want to go upstairs, step into the shower, turn the water as hot as it will go, and wash. Scrub my skin raw with soap and sponge until the top five layers of epidermis are gone, and still not feel clean.

On the surface of things, the stack sounds pretty good. OSGi is a pretty solid spec for managing versioning and modularity within a running Java system, and more importantly, it’s well-known, relatively reliable, and pretty well-proven to handle the classic problems well. And of course, anybody who knows me knows that I’m a fan of the Scala language as a potential complement or supplement to the Java programming language, so that’s hardly in debate.

But there are a few concerns. JSON is a simple wire protocol, granted, but that is both a good thing and a bad thing (it’s object-centric, for one, and will run into some of the same issues as objects do with certain relationships), and it lacks the ubiquity that XML provides. Granted, XML clearly suffered from an overabundance of adoption, but it still doesn’t take away the fact that ubiquity is really necessary if you’re building a baseline for something that will talk to a variety of different systems. Which, I admit, may not be in his list of requirements, I don’t know. And HTTP is great for long-haul, client-initiated communication, but it definitely has its limitations (which he acknowledges, openly, to his credit), at least to internal-facing consumers. There is no peer for external-facing consumers, that’s a given.

And the stack is clearly also missing something else...

The JOSH stack is lacking a letter, because a solution for persisted data is missing in the stack.

A great deal of what needs to be done does not require a ACID RDB cluster. Some of it does and I'm kicking that can down the road.

For the rest, either the data is ReadOnly and loaded a 1-3 times a day or is best persisted by a distributed Key-Value storage system. A number of these are now available as open source solutions and at the right moment I'll need to pick one and add that letter to the JOSH stack.

As a commenter suggested, CouchDB might be a solution here, or I’ll even throw db4o into the ring for discussion as an option. Again, it’ll depend on how far-and-wide the data will be seen by other systems—the more other systems need to see it, the less further away from a “regular” RDBMS we can go.

Certainly, it’s a great start for discussion, even if the acronym is likely to give those named Joshua an unhealthy ego boost. :-)

Part of me wonders, though... what would the equivalent on .NET look like? JSON + Assemblies + F# + HTTP = JAFH?

Java/J2EE | Languages | Scala | XML Services

Tuesday, March 24, 2009 2:25:43 AM (Pacific Daylight Time, UTC-07:00)
Comments [3] |

From the Mailbag: Polyglot Programmer vs. Polyactivist Language

This crossed my Inbox:

I read your article entitled: The Polyglot Programmer. How about the thought that rather than becoming a polyglot-software engineer; pick a polyglot-language. For example, C# is borrowing techniques from functional and dynamic languages. Let the compiler designer worry about mixing features and software engineers worry about keep up with the mixture. Is this a good approach? [From Phil, at http://greensoftwareengineer.spaces.live.com/]

Phil, it’s an interesting thought you’ve raised—which is the better/easier approach to take, that of incorporating the language features we want into a single language, rather than needing to learn all those different languages (and their own unique syntaxes) in order to take advantage of those features we want?

After all, we’re starting to see this taking place within a certain number of languages already, particularly C#; first, in 3.0, they introduced a number of features in support of LINQ that make C# a useful starting point for working with a functional language. Extension methods, for example, allow us to add a number of different methods to the collection classes that provide some functional capabilities (Select<>, GroupBy<>, and so on), as Matt Podwysocki demonstrates, generics contribute the type-safety that most functional languages embrace, anonymous methods and delegates provide better functions-as-first-class-constructs (including lambdas), and anonymous types make it vastly easier to return and pass tuples. And now, in 4.0, we’re getting the “dynamic” keyword, which will add support for invoking methods and properties dynamically, in the grand tradition of most dynamic languages (like Python and Ruby), and 3.0’s local variable type inference allows us to write “var x = ...”, which feels pretty dynamic (even if it’s not, under the hood).

Unfortunately, I think for the most part, the answer’s going to be, “Yes, it would be nice, if it weren’t for the fact that there are very few languages that won’t collapse underneath their own weight if they did so.”

Consider, for example, the C# language. Already, with the C# 3.0 definition, the language specification weighs in at close to a thousand pages. The additional features in 4.0 could easily push it over a thousand and possibly, with all the places where “dynamic” behavior will need to be factored into the existing specification, could push that well into the 1200 to 1300 page range. What’s the upper limit on a language’s complexity to maintain and enhance, much less for its programmers to comprehend?

(By comparison, the C++ specification, as I can best remember, didn’t weigh in at more than a thousand pages, but given that the current working draft is under password protection, and I can’t find the prior spec as a freely-available download, I can’t see if memory is correct or not.)

Or, consider the various edge cases that came up around the introduction of nullable types in C# 2.0. What started out as a fairly simple suggestion—“let’s let T? represent the idea that this instance of T could be nullable, and at runtime it’ll be a Nullable<T> instance behind the scenes”—turned into a pretty ugly morass of edge cases at the language level that resulted in some serious bug-fixing right up until the final ship date.

Thing is, languages that aren’t written deliberately to allow their own modification and evolution tend to fail over time. C++ was one such example, and I think both Java and C# will stand as successor examples before long.

Right now, in C# 3.0, type inference is limited entirely to local variables because the language isn’t syntactically set up to leave out type names wherever possible—the “var” token is a type placeholder, largely because the parser has to have a type first. (This is the same purpose the “dynamic” keyword seems to be playing for 4.0, though I can’t say so for certain.) In F# and Scala, this syntax is deliberately written Pascal-style, with the name first, optionally followed by a colon and the type, because the parser can see the colon and realize the type is already specified, or see no colon and realize the type should be inferred. That syntax is used consistently throughout the F# and Scala languages, and that means it’s pretty easy, lexically speaking, for the languages to recognize when type inference should kick in.

What’s more, both F# and Scala don’t really support the O-O notion of method overloading, because again, it gets confusing when trying to kick in type inference—something about too many possibilities confusing the type-inferencer. (I’m not entirely positive of this point, by the way, it’s based on some conversations I’ve had with language designers over the last few years. I could be wrong, and would love to see a language that supports both.) Instead, they force developers to be more explicit about parameters being passed—F# won’t even do implicit widening conversions, in fact, such as automatically widening ints to longs.

But both F# and Scala have a very interesting facility to allow definitions of methods/functions using very flexible syntactic rules, such that they look like operators or keywords built into the language; F# defines its pipeline operator ( |> ) in its library definitions, for example. Scala defines numerous “keywords”, like synchronized or transient, as classes in the Scala package extending “StaticAnnotation”—in other words, their syntax and behavior is defined as an annotation, rather than as a built-in part of the language. Ditto for Scala’s XML support.

Lisp, of course, was one of the first (if not the first) language to do this, and it’s my understanding that this has been one of the principal reasons it has survived all these years as a language—because it’s an abstraction built on top of an abstraction built on top of an abstraction, et al, it makes it easier to change those underlying abstractions when the context changes.

This doesn’t mean those “polyactivist” languages like C# are bad things, it just means that there’s a danger that they’ll eventually collapse from too many moving parts all trying to talk to each other at the same time. As an exercise, open the C# 3.0 spec, and start checking off all the sections that will need to be touched by the introduction of the “dynamic” keyword as a new type.

Or, to put it analagously, yes, for a lot of work, a single multifunction tool can be useful, but for a lot of other work, you want tools that are specialized to the task at hand. Let’s not minimize the usefulness of that multifunction tool, but let’s not try to use a Swiss Army knife where a jeweler’s screwdriver is really needed.

SDWest, SDBestPractices, SDArch&Design: RIP, 1975 - 2009

This email crossed my Inbox last week while I was on the road:

Due to the current economic situation, TechWeb has made the difficult decision to discontinue the Software Development events, including SD West, SD Best Practices and Architecture & Design World. We are grateful for your support during SD's twenty-four year history and are disappointed to see the events end.

This really bums me out, because the SD shows were some of the best shows I’ve been to, particularly SD West, which always had a great cross-cutting collection of experts from all across the industry’s big technical areas: C++, Java, .NET, security, agile, and more. It was also where I got to meet and interview Bjarne Stroustrup, a personal hero of mine from back in my days as a C++ developer, where I got to hang out each year with Scott Meyers, another personal hero (and now a good friend) as well as editor on Effective Enterprise Java, and Mike Cohn, another good friend as well as a great guy to work for. It was where I first met Gary McGraw, in a rather embarrassing fashion—in the middle of his presentation on security, my cell phone went off with a klaxon alarm ring tone loud enough to be heard throughout the entire room, and as every head turned to look at me, he commented dryly, “That’s the buffer overrun alarm—somewhere in the world, a buffer overrun attack is taking place.”

On a positive note, however, the email goes on to say that “Cloud Connect [will] take over SD West's dates in March 2010 at the Santa Clara Convention Center”, which is good news, since it means (hopefully) that I’ll still get a chance to make my yearly pilgrimage to In-N-Out....

Rest in peace, SD. You will be missed.

As for Peer Review, Code Review?

Interesting little tidbit crossed my Inbox today...

Only 8% members of the Scientific Research Society agreed that "peer review works well as it is". (Chubin and Hackett, 1990; p.192).
"A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research." (Horrobin, 2001)
Horrobin concludes that peer review "is a non-validated charade whose processes generate results little better than does chance." (Horrobin, 2001). This has been statistically proven and reported by an increasing number of journal editors.
But, "Peer Review is one of the sacred pillars of the scientific edifice" (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and "Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process" (Horrobin, 2001).
...
Chubin, D. R. and Hackett E. J., 1990, Peerless Science, Peer Review and U.S. Science Policy; New York, State University of New York Press.
Horrobin, D., 2001, "Something Rotten at the Core of Science?" Trends in Pharmacological Sciences, Vol. 22, No. 2, February 2001. Also at http://www.whale.to/vaccine/sci.html and http://post.queensu.ca/~forsdyke/peerrev4.htm (both pages were accessed on February 1, 2009)
Goodstein, D., 2000, "How Science Works", U.S. Federal Judiciary Reference Manual on Evidence, pp. 66-72 (referenced in Hoorobin, 2000)

I know that we don't generally cite the scientific process as part of the rationale for justifying code reviews, but it seems to have a distinct relationship. If the peer review process is similar in concept to the code review process, and the scientific types are starting to doubt the efficacy of peer reviews, what does that say about the code review?

(Note: I'm not a scientist, so my familiarity with peer review is third-hand at best; I'm wide open to education here. How are the code review and peer review processes different, if in fact, they are different?)

The Horrobin "sacred pillars" quote, in particular, makes me curious: Don't we already apply "scientific [and engineering] methods" to the peer review process? And can we honestly say that we in the software industry apply "scientific [and engineering]" methods to the code review process? Can we iterate the list? Or do we just trust that intuition and "more eyeballs" will help spot any obvious defects?

The implications here, when tied up next to the open source fundamental principle that states that "more eyeballs is better", are interesting to consider. If review is not a scientifically-proven or "engineeringly-sound" principle, then the open source folks are kidding themselves in thinking they're more secure or better-engineered. If we conduct a scientific measurement of code-reviewed code and find that it is "a non-validated charade whose processes generate results little better than does chance", we've at least conducted the study, and can start thinking about ways to make it better. (I do wish the email author had cited sources that provide the background to the statement, "This has been statistically proven", though.)

I know this is going to seem like a trolling post, but I'm genuinely curious--do we, in the software industry, have any scientifically-conducted studies with quantifiable metrics that imply that code-reviewed code is better than non-reviewed code? Or are we just taking it as another article of faith?

(For those who are curious, the email that triggered all this was an invitation to a conference on peer review.

This is the purpose of the International Symposium on Peer Reviewing: ISPR (http://www.ICTconfer.org/ispr) being organized in the context of The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009 (http://www.ICTconfer.org/kgcm), which will be held on July 10-13, 2009, in Orlando, Florida, USA.

I doubt it has any direct relevance to software, but I could be wrong. If you go, let me know of your adventures and conclusions. ;-) )

Conferences | Development Processes | Security | Windows

Sunday, February 22, 2009 10:36:43 AM (Pacific Standard Time, UTC-08:00)
Comments [9] |

Woo-hoo! Speaking at DSL DevCon 2009!

Just got this email from Chris Sells:

For twelve 45-minute slots at this year’s DSL DevCon (April 16-17 in Redmond, WA), we had 49 proposals. You have been selected as speakers for the following talks. Please confirm that you’ll be there for both days so that I can put together the schedule and post it on the conference site. This DevCon should rock. Thanks!
Martin Fowler - Keynote
Paul Vick + Gio - Mgrammar Deep Dive
Tom Rodgers - Domain Specific Languages for automated testing of equity order management systems and trading machines
Paul Cowan - DSLs in the Horn Package Manager
Guillaume Laforge - How to implement DSLs with Groovy
Markus Voelter - Eclipse tooling for Model-Driven stuff
Dionysios G. Synodinos - JavaScript DSLs for the Client Side
Ted Neward, Bradford Cross - Functional vs. Dynamic DSLs: The Smackdown
Gilad Bracha - embedding EBNF in a general purpose language
Umit Yalcinalp, Tilman Giese - RUMBA: RUby Managed Business data for Applications
Bob Archer - A DSL for Cool Effects in Adobe Pixel Blender
Chance Coble - Language Oriented Programming in F#

As my 15-year-old son Michael has grown fond of saying... w00t! The list of topics is fascinating, and I'm really looking forward to most, if not all, of them. Chance's talk on LOP in F# should be good, I'm really curious to see Gilad's discussion of EBNF (and wondering if this is Newspeak we'll be seeing), and Guillaume is always fun to watch when he's going on about Groovy. Of course, I'm also excited to be paired up with Brad, who's an insanely smart guy--I have a feeling I'll learn a lot just by standing next to him. (Sort of a speakers' osmosis.)

If you're not planning to be here for this (and the Lang.NET Symposium), either you have life-saving surgery scheduled that can't be pushed back, or you're clearly not interested in DSLs. For your own sake, I hope it's the latter. ;-)

Seriously, come for the full week. The Lang.NET Symposium last year was an amazing event, for a number of reasons, not the least of which is that it saw Sun celebrities John Rose, Charlie Nutter and Brian Goetz step on to the Microsoft campus, deliver a great presentation on the JVM, MLVM/invokedynamic, and JRuby, and get good feedback and discussion from Microsoft engineers and other notables. You don't get to see that every day. :-)

What do beer, London, Alt.NET and ThoughtWorks have in common?

Answer: "I don't know, but I'm damn well going to find out!"

(Now I really wish I were in London. Ah, well, will just have to go see Ward Cunningham speak at Alt.NET Seattle, instead.)

.NET | C# | Conferences | F# | Languages | Social | WCF | Windows | XML Services

Tuesday, February 17, 2009 10:26:00 AM (Pacific Standard Time, UTC-08:00)
Comments [0] |

NOW you know why you want to learn Haskell

Matt Podwysocki makes it all clear:

Hey, I'd have learned Haskell a LONG time ago if I'd known it could yield up a beer!

Nice little montage from JDD08

Last year I had the opportunity to return to the land of my roots, Poland, and speak at Java Developer Days (JDD). Just today, the organizers from JDD sent me a link with a nice little photo montage from the conference. (I did notice a few photos from the after-party were selectively left out of the montage, however, which is probably a good thing because that was the first time I'd ever met a Polish Mad Dog, and boy did they all go down easy...)

If you're anywhere in the area around Krakow in March, you definitely should swing by for their follow-up conference, 4Developers--it sounds like it's going to be another fun event, and this time it's going to reach out to more than just the Java folks, but also the .NET crowd (and a few others), as well.

(I don't really expect any of the readers of this blog living outside Poland to really pack up and head over to Krakow for a weekend, mind you, but if you're a technology speaker and you're interested in hanging with an extremely good group of people, the people who put these shows on--ProIdea--are top-notch, take great care of the speakers, and overall make the entire experience well worth the trip.)

Building WCF services with F#, Interlude

Because I’m about to start my third part in the WCF/F# series, I realized that I’ve now hit the “rule of three” mark: in this particular case, this will mark the third project I’m creating that unifies WCF and F#, and frankly, it’s a pain in the *ss to do it all by hand each time: create an F# Library, add the System.ServiceModel and System.Runtime.Serialization assemblies, go create an App.config file and add it to the project as an Existing Item…. Painful.

So… as a brief interlude, I decided to go re-acquaint myself with the Visual Studio project template system, and sure enough, it’s basically what I remember: a collection of files with some template-style functionality, bundled into a .zip file and stored in the Visual Studio directory, under <VSDir>\Common7\IDE\ProjectTemplates. What was new to me, however, was the highly useful “File | Export Template…” menu option, allowing me to take an existing F#/WCF project and use it as a template to create the .zip bundle. (Naturally, I didn’t discover this until I’d built the silly thing by hand.)

Sara Ford has more on creating a VS template on her Visual Studio Tools blog/column, number 336 to be precise. (You should read all of them, by the way—start with #1 and work your way there. When you’re done, you’ll have a much better appreciation of everything Visual Studio can do, and you’ll be able to find a ton of ways to save yourself and your team some time and effort.)

You can always take a .zip bundle like this and drop it into the Visual Studio 2008 “My Exported Templates” directory, but quite frankly, I didn’t want that. I wanted my template to appear in a subcategory of Visual F# in the New Project dialog box, under “WCF”, just as the C# versions do. The easiest way to do this is to manually create the “WCF” directory (full path thus being <VSDir>\Common7\IDE\ProjectTemplates\FSharp\WCF), and drop the .zip file there. Note that if you restart Visual Studio at this point, you won’t see the new template; it builds a cache of the .zip templates in a sister directory (ProjectTemplatesCache), so instead, you have to tell Visual Studio to reset that cache by firing “devenv /setup” from the command-line. (This will require admin privileges, by the way.)

After that, you have an F#/WCF project template, and you’re good to go.

.NET | C# | F# | Languages | Reading | WCF | Windows | XML Services

Saturday, January 24, 2009 12:15:53 AM (Pacific Standard Time, UTC-08:00)
Comments [3] |

Building WCF services with F#, Part 2

If you’ve not read the first part in the series, take a look there first.

While it’s always easier to build WCF services with nothing but primitive types understood by all the platforms to which you’re communicating (be it Java through XML services or other .NET systems via WCF’s more efficient binding types), this gets old and limiting very quickly. The WCF service author will want to develop whole composite types that can be exchanged across the wire, and this is most often done via the DataContract attribute applied to the types that will be exchanged.

In Michele Leroux Bustamente’s Learning WCF examples, this is covered in Chapter #2, and the corresponding code I’m using as a basis for conversion to F# is in Labs\Chapter2\DataContracts_Part1.

One notable difference between this example and the previous one is that the type definitions are stored in a separate assembly, ContentTypes.dll. There’s two basic choices to use here: one, to use the C# types as defined, from a service written in F#, or two, to define the types in F# and use them from the service. A third choice, defining the types in F# and using them from C#, also presents itself, but is uninteresting to us from a purely instructional standpoint—if you know how to write C#, then you can take the types defined in F# and use them just as you would have the C# types.

For instructional purposes, I’m going to take the second approach: I’m first going to convert the ContentTypes.dll assembly over to F#, again to show how to create types in F# that are structurally equivalent to the types defined in C#, since that’s something that has changed since Nick Holmes blogged about this last year), then I’m going to flip the service over to F# as well.

Defining the Data Types

The first step, for many service authors, is to define the interfaces for the service(s) and the types that will be exchanged; in this case, since I’m building from Michele’s example, these have already been defined as:

        1: using System;

       2: using System.ServiceModel;

       3: using System.Runtime.Serialization;

       4:  

       5: namespace ContentTypes

       6: {

       7:     

       8:    [DataContract(Namespace="http://schemas.thatindigogirl.com/samples/2006/06")]

       9:     public class LinkItem

    {

  

        [DataMember(Name = "Id", IsRequired = false, Order = 0)]

        private long m_id;

        [DataMember(Name = "Title", IsRequired = true, Order = 1)]

        private string m_title;

        [DataMember(Name = "Description", IsRequired = true, Order = 2)]

        private string m_description;

        [DataMember(Name = "DateStart", IsRequired = true, Order = 3)]

        private DateTime m_dateStart;

        [DataMember(Name = "DateEnd", IsRequired = false, Order = 4)]

        private DateTime m_dateEnd;

        [DataMember(Name = "Url", IsRequired = false, Order = 5)]

        private string m_url;

  

        public DateTime DateStart

        {

            get { return m_dateStart; }

            set { m_dateStart = value; }

        } 

  

        public DateTime DateEnd

        {

            get { return m_dateEnd; }

            set { m_dateEnd = value; }

        }

       

        public string Url

        {

            get { return m_url; }

            set { m_url = value; }

        }

        

        public long Id

        {

            get { return m_id; }

            set { m_id = value; }

        }

  

        public string Title

        {

            get { return m_title; }

            set { m_title = value; }

        }

  

        public string Description

        {

            get { return m_description; }

            set { m_description = value; }

        }

    }

}

Note that now, in a C#3-friendly world, we can slim the definition of the LinkItem down to a much smaller level thanks to the power of automatic properties:

       1: using System;

       2: using System.ServiceModel;

       3: using System.Runtime.Serialization;

       4:  

       5: namespace ContentTypes

       6: {    

       7:     [DataContract(Namespace="http://schemas.thatindigogirl.com/samples/2006/06")]

       8:     public class LinkItem

       9:     {

      10:         [DataMember(Name = "Id", IsRequired = false, Order = 0)]

      11:         public long Id { get; set; }

      12:         [DataMember(Name = "Title", IsRequired = true, Order = 1)]

      13:         public string Title { get; set; }

      14:         [DataMember(Name = "Description", IsRequired = true, Order = 2)]

      15:         public string Description { get; set; }

      16:         [DataMember(Name = "DateStart", IsRequired = true, Order = 3)]

      17:         public DateTime DateStart { get; set; }

      18:         [DataMember(Name = "DateEnd", IsRequired = false, Order = 4)]

      19:         public DateTime DateEnd { get; set; }

      20:         [DataMember(Name = "Url", IsRequired = false, Order = 5)]

      21:         public string Url { get; set; }

      22:     }

      23: }

… but either way, the type ends up looking the same. Converting this over to F# is relatively easy, if not any shorter or more convenient than the C# 3.0 version, owing to the fact that, by default, F# will not generate mutable properties by default:

       1: #light

       2:  

       3: namespace ContentTypes

       4:     

       5: open System

       6: open System.Runtime.Serialization

       7: open System.ServiceModel

       8:  

       9: [<DataContract(Namespace="http://schemas.thatindigogirl.com/samples/2006/06")>]

type LinkItem() =

    let mutable id : int64 = 0L

    let mutable title : string = String.Empty

    let mutable description : string = String.Empty

    let mutable dateStart : DateTime = DateTime.Now

    let mutable dateEnd : DateTime = DateTime.Now

    let mutable url : string = String.Empty

  

    [<DataMember(Name = "Id", IsRequired = false, Order = 0)>]

    member public l.Id

        with get() = id

        and set(value) = id <- value

    [<DataMember(Name = "Title", IsRequired = true, Order = 1)>]

    member public l.Title

        with get() = title

        and set(value) = title <- value

    [<DataMember(Name = "Description", IsRequired = true, Order = 2)>]

    member public l.Description

        with get() = description

        and set(value) = description <- value

    [<DataMember(Name = "DateStart", IsRequired = true, Order = 3)>]

    member public l.DateStart

        with get() = dateStart

        and set(value) = dateStart <- value

    [<DataMember(Name = "DateEnd", IsRequired = false, Order = 4)>]

    member public l.DateEnd

        with get() = dateEnd

        and set(value) = dateEnd <- value

    [<DataMember(Name = "Url", IsRequired = false, Order = 5)>]

    member public l.Url

        with get() = url

        and set(value) = url <- value

Notice that I have to create a mutable backing field, and define the properties in the F# LinkItem type to explicitly access and mutate those values. This is a bit frustrating, because it seems like F# should be able to infer what I want from a simple property declaration, in the same way that C# can, but perhaps that’s asking too much from the language right now, considering the silly thing hasn’t even shipped yet.

(Psssst, Luke, Don, if you’re listening, automatic property generation in F# would be a nifty feature to add between now and then, if you guys can ninja it in there before the next CTP…)

Notice, by the way, the namespace directive at the top of the F# code; this is necessary to set the prefix around the LinkItem type. Without it, remember, the F# code is going to be slipped inside an outer class declaration matching the filename, effectively naming the class Module1+LinkItem, which would not be structurally equivalent to the C# type.

Lesson #4: Always put a namespace or module declaration around the types exported from a service.

Notice that LinkItem also has a default constructor, as per Lesson #2; this is necessary because the DataContract-related code inside of WCF is going to need to be able to construct one of these and set its properties. If we want to set any reasonable defaults, that’s easily done in the mutable member definitions.

One principal difference between the F# version and the C# version is that the DataMember attributes are applied to the properties, instead of the fields, largely because the F# language wants to keep a layer of encapsulation between the code you write as an F# programmer, and the actual code generated. So, for example, the “field” id, above, doesn’t actually get generated exactly as described—in truth, it turns into a field called “id@11”. This is a marked difference from C# (or even VB), which deliberately gives us more control over how the physical structure of classes looks. This is even more obvious in a basic F# program where a top-level declaration reads, “let x = 12”; where it might be tempting to assume that x will be a static field on the class surrounding the declaration, the F# compiler actually generates a property.

In this particular case, whether the attribute applies to the fields or the property declarations isn’t going to make a large difference, but in more sophisticated classes, it might, so it’s better to apply the attribute to the property and not the field, at least, from what I’ve found so far.

Lesson #5: Put DataMember attributes on the properties of the DataContract, not the fields.

Defining the Service

The definition of the service is actually pretty straightforward. Add either the C# ContentTypes.dll or the F# ContentTypes.dll as an assembly reference, and where the C# code (GigManagerService.cs) reads:

       1: using System;

       2: using System.Collections.Generic;

       3: using System.Text;

       4: using System.ServiceModel;

       5: using ContentTypes;

       6:  

       7: namespace GigManager

       8: {

       9:     [ServiceContract(Name = "GigManagerServiceContract", Namespace = "http://www.thatindigogirl.com/samples/2006/06", SessionMode = SessionMode.Required)]

    public interface IGigManagerService

    {

        [OperationContract]

        void SaveGig(LinkItem item);

  

        [OperationContract]

        LinkItem GetGig();

    }

  

    [ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession)]

    public class GigManagerService : IGigManagerService

    {

  

        private LinkItem m_linkItem;

  

        public void SaveGig(LinkItem item)

        {

            m_linkItem = item;

        }

  

        public LinkItem GetGig()

        {

            return m_linkItem;

        }

    }

}

… the corresponding F# code (Program.fs) reads like so:

       1: #light

       2:  

       3: module GigManager =

       4:     open System

       5:     open System.Runtime.Serialization

       6:     open System.ServiceModel

       7:     

       8:     open ContentTypes

       9:     

      10:     [<ServiceContract(Name = "GigManagerServiceContract", 

      11:         ConfigurationName = "IGigManagerService",

      12:         Namespace = "http://www.thatindigogirl.com/samples/2006/06", 

      13:         SessionMode = SessionMode.Required)>]

      14:     type IGigManagerService =

      15:         [<OperationContract>]

      16:         abstract SaveGig: item : LinkItem -> unit

      17:         [<OperationContract>]

      18:         abstract GetGig: unit -> LinkItem

      19:         

      20:     [<ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession)>]

      21:     type GigManagerService() =

      22:         let mutable li : LinkItem = LinkItem()

      23:         interface IGigManagerService with

      24:             member gms.SaveGig(item) = li <- item                

      25:             member gms.GetGig() = li

Careful readers will notice that there’s one additional element in the F# version that isn’t in the C# version; specifically, on line 11, I’ve added a “ConfigurationName” element to the IGigManagerService’s ServiceContract attribute. I do this because, again, the F# compiler is doing some interesting things to the code under the hood. In particular, the interface IGigManagerService is actually exposed under a slightly different name—remember, F# likes to use nested classes, not namespaces, so where the C# version of IGigManagerService is formally known as “GigManager::IGigManagerService”, the F# version is “Program/GigManager/GigManagerService”, where Program is the name of the .fs file. This seems to cause WCF some heartache when it starts looking through the App.config file and matching it up against the names exported from the actual class—it won’t match up correctly. So, by giving it a ConfigurationName that matches the human-readable interface name, WCF is happy again.

Lesson #5: Use ConfigurationName on ServiceContract to avoid having to learn F#’s naming bindings to the CLR.

The rest of the code in Program.fs is the hosting code, which structurally is no different than that of the previous post.

One key thing to remember, however, is that the host “service” element will also be looking at type names, so if you forget to set the name of the service, you’ll need to use a type-investigation tool (ILDasm or Reflector) to figure out what the host class name is; in the case above, it would be “Program+GigManager+GigManagerService”, forcing the App.config file to read as follows:

       1: <?xml version="1.0" encoding="utf-8" ?>

       2: <configuration>

       3:   <system.serviceModel>

       4:     <services>

       5:       <service name="Program+GigManager+GigManagerService" 

       6:                behaviorConfiguration="serviceBehavior">

       7:         <host>

       8:           <baseAddresses>

       9:             <add baseAddress="http://localhost:8000"/>

            <add baseAddress="net.tcp://localhost:9000"/>

          </baseAddresses>

        </host>

        <endpoint address="GigManagerService"

                  binding="netTcpBinding"

                  contract="IGigManagerService" />

        <endpoint address="mex"

                  binding="mexHttpBinding"

                  contract="IMetadataExchange" />

      </service>

    </services>

      <behaviors>

          <serviceBehaviors>

              <behavior name="serviceBehavior">

                  <serviceMetadata httpGetEnabled="true"/>

              </behavior>

          </serviceBehaviors>

      </behaviors>

    <!-- This <diagnostics> section should be placed inside the <system.serviceModel> section. In addition, you'll need to add the <system.diagnostics> snippet to specify service model trace listeners and a file for output. -->

    <diagnostics performanceCounters="All" wmiProviderEnabled="true" >

      <messageLogging logEntireMessage="true" logMalformedMessages="true" logMessagesAtServiceLevel="true" logMessagesAtTransportLevel="true" maxMessagesToLog="100000"  />

    </diagnostics>

  </system.serviceModel>

  <!-- This <system.diagnostics> section illustrates the use of a shared listener for service model output. It requires you to also add the <diagnostics> snippet for the <system.serviceModel> section. -->

  <system.diagnostics >

    <sharedListeners>

      <add name="sharedListener" 

                 type="System.Diagnostics.XmlWriterTraceListener"

                 initializeData="c:\logs\servicetrace.svclog" />

    </sharedListeners>

    <sources>

      <source name="System.ServiceModel" switchValue="Verbose, ActivityTracing" >

        <listeners>

          <add name="sharedListener" />

        </listeners>

      </source>

      <source name="System.ServiceModel.MessageLogging" switchValue="Verbose">

        <listeners>

          <add name="sharedListener" />

        </listeners>

      </source>

    </sources>

  </system.diagnostics>

</configuration>

Caveat emptor. In all honesty, despite the motivation of Lesson #5, I don’t think there’s any way around learning at least a little bit of F#’s name-mapping scheme, but at least we can be selective about where and when we apply it.

.NET | C# | F# | Languages | Reading | WCF | Windows | XML Services

Friday, January 23, 2009 7:11:15 PM (Pacific Standard Time, UTC-08:00)
Comments [2] |

Seattle/Redmond/Bellevue Nerd Dinner

From Scott Hanselman's blog:

Are you in King County/Seattle/Redmond/Bellevue Washington and surrounding areas? Are you a huge nerd? Perhaps a geek? No? Maybe a dork, dweeb or wonk. Maybe you're in town for an SDR (Software Design Review) visiting BillG. Quite possibly you're just a normal person.
Regardless, why not join us for some Mall Food at the Crossroads Bellevue Mall Food Court on Monday, January 19th around 6:30pm?
...
NOTE: RSVP by leaving a comment here and show up on January 19th at 6:30pm! Feel free to bring friends, kids or family. Bring a Ruby or Java person!

Any of the SeaJUG want to attend? (Anybody know of a Ruby JUG in the Eastside area, by the way?) I'm game....

Building WCF services with F#, Part 1

For a while now, I’ve held the opinion that the “sweet spot” for functional languages on the JVM and CLR will be in the services space, since services and functions seem pretty similar to one another in spirit—a given input produces a given output, with (ideally) no shared state, high concurrency expectations, idempotent processing, and so on. This isn’t to say that a functional language is going to make a non-trivial service trivial, but I think it will make it simpler and more likely to scale better over time, particularly as the service gets more complicated.

As part those explorations into the union of services and functional languages, I’ve been taking some of Michele Leroux Bustamente’s excellent labs from Learning WCF and flipping the services over to F#. Along the way, I’ve discovered a few “quirks” of F# that make building a WCF service a tad more complicated than it needs to be, so I’ve decided to blog what’s going on so others can find it easier.

(Many thanks to Nick Holmes’ blog, which helped identify one of the first problems I ran into, though a few things have changed since he blogged back in February, so I thought I’d catch everything up to the Sep 08 CTP of F#.)

This isn’t intended to be a tutorial on WCF, so if you’re not familiar with WCF, I strongly suggest you go get Michele’s book. I’m assuming you’ll know the WCF basics (address, binding, contract, config files, behaviors, etc), and I just want to show the deltas necessary to make F# work. Note that I’m just doing the service side of things—I believe clients will probably continue to be written in C# or VB or some other OO language, in keeping with the theory that OO will remain the predominant way of developing client-facing stuff. (Note that this also neatly avoids the basic problem that svcutil.exe only generates either C# or VB proxy code, and that “Add Service Reference” isn’t available inside an F# project, as of this writing.)

Defining Contracts in F#

The first step in any straight-up WCF service is, of course, to define the contract that both sides will agree to. (Yes, I know, we could do everything in terms of picking Message types apart; I’ll get to that in a later piece.) First things first: taking Michele’s HelloIndigo_Part1 solution, I add a new project to it, “FHost”, an F# application. Add the System.ServiceModel and System.Runtime.Serialization assemblies, and we’re good to get going.

Michele’s “HelloIndigo_Part1” solution defines the contract between client and service this way:

        1: namespace Host

       2: {

       3:     [ServiceContract(Namespace = "http://www.thatindigogirl.com/samples/2006/06")]

       4:     public interface IHelloIndigoService

       5:     {

       6:         [OperationContract]

       7:         string HelloIndigo();

       8:     }

       9:     // ...

      10: }

This contract can be consumed in two ways; one is to build this interface into its own assembly that’s linked to both the WCF service host and to the WCF client, but in her example (as is perfectly reasonable in a WCF project), she repeats the interface in both the client and the service, so to be faithful to that, let’s define the interface in the F# code:

       1: #light

       2:  

       3: open System

       4: open System.ServiceModel

       5:  

       6: [< ServiceContract(Namespace = "http://www.thatindigogirl.com/samples/2006/06") >]

       7: type IHelloIndigoService =

       8:     [< OperationContract >]

       9:     abstract HelloIndigo: unit -> string

(The color syntax highlighting is off because I’m using the C# mode of the “Code Snippet” plugin in Windows Live Writer to post the code, and it doesn’t have an F# mode. Yet.)

Pay very close attention to the interface definition in F#, because there is a subtle WCF “bug” that F# exposes by accident. When F# compiles an interface, if a method in the interface has parameters, if no name is specified for that parameter, then WCF will throw an ArgumentNullException when you try to run svcutil.exe over the compiled assembly, or when you pass the type in to the ServiceHost constructor, claiming “Value cannot be null. Parameter name: name”. The problem is that F#, unlike C# or VB, allows methods to have parameters without names, and WCF can’t handle this. Verifying this is a b*tch; if you use ILDasm to view the F#-compiled assembly, it looks like there are parameter names there, because ILDasm generates them as placeholders for display. (Reflector is your friend here.)

The WCF team has basically said that this behavior is by design—SOAP, which is a key concept to the WCF stack, doesn’t really have great support for unnamed parameters (and yes, I know, this is not exactly true, but I’m not going to get into that debate here), so the WCF team has basically said there’s really nothing they can do but maybe issue a better error message than ArgumentNullException.

Lesson #1: Always name your WCF contract interface params.

Caveat emptor.

Defining the Service Implementation

Next step is to define the service implementation. Again, Michele’s code looks like so:

       1: public class HelloIndigoService : IHelloIndigoService

       2: {

       3:     public string HelloIndigo()

       4:     {

       5:         return "Hello Indigo";

       6:     }

       7: }

Not a really difficult operation, so converting that to F# is pretty straightforward:

       1: [< ServiceBehavior(ConfigurationName="HIS") >]    

       2: type HelloIndigoService() =

       3:     interface IHelloIndigoService with

       4:         member s.HelloIndigo() : string =

       5:             "Hello Indigo"

There are two things important to this definition. First, the parentheses at the end of the “type” declaration line create a default no-argument constructor for the HelloIndigoService, which is required—without it, WCF is going to complain about being unable to construct an instance of this type.

Lesson #2: Always provide the default type constructor in the service implementation.

Second, the ServiceBehavior attribute is one I’ve added, because F# does some funky things with the type names during compilation; for example, since my F# code is in a file called “Host.fs”, the F# compiler synthesizes a class called “Host” which acts as a nesting wrapper around everything else in the file, so technically the typename of HelloIndigoService is now “Host+HelloIndigoService”, which will cause some chaos when WCF tries to match up the service name with the appropriate entry in the App.config file. You can either make sure the App.config matches the CLR-level type names generated by the F# compiler, or you can explicitly specify the configuration names; I choose the latter, so that it’s a bit more clear what’s going on.

Lesson #3: Always specify the configuration name on the service implementation.

The App.config file, by the way, now looks like this, the only change from Michele’s labs being the changes to the configuration name of the service behavior (line 13):

       1: <?xml version="1.0" encoding="utf-8" ?>

       2: <configuration>

       3:     <system.serviceModel>

       4:         <behaviors>

       5:             <serviceBehaviors>

       6:                 <behavior name="serviceBehavior">

       7:                     <serviceMetadata httpGetEnabled="false" />

       8:                 </behavior>

       9:             </serviceBehaviors>

        </behaviors>

        <services>

            <service behaviorConfiguration="serviceBehavior" 

                     name="HIS">

                <clear />

                <endpoint address="HelloIndigoService" 

                          binding="basicHttpBinding"

                          name="basicHttp" 

                          contract="Host+IHelloIndigoService" />

                <endpoint binding="mexHttpBinding" 

                          name="mex" 

                          contract="IMetadataExchange" />

                <host>

                    <baseAddresses>

                        <add baseAddress="http://localhost:8000/HelloIndigo" />

                    </baseAddresses>

                </host>

            </service>

        </services>

    </system.serviceModel>

</configuration>

Still with me? One last part to go, defining the (self-hosting) host.

Defining the Self-Hosting Host

In simple examples, frequently the service code self-hosts, meaning it doesn’ t need to be deployed into IIS. Michele uses a wrapper class to defer some of the hosting details, a la:

       1: internal class MyServiceHost

       2: {

       3:     internal static ServiceHost myServiceHost = null;

       4:  

       5:     internal static void StartService()

       6:     {

       7:         // Instantiate new ServiceHost 

       8:         myServiceHost = new ServiceHost(typeof(HelloIndigoService));

       9:         myServiceHost.Open();

    }

  

    internal static void StopService()

    {

        // Call StopService from your shutdown logic (i.e. dispose method)

        if (myServiceHost.State != CommunicationState.Closed)

            myServiceHost.Close();

    }

}

  

class Program

{

    static void Main(string[] args)

    {

        try

        {

            MyServiceHost.StartService();

            Console.WriteLine("Press <ENTER> to terminate the host application");

            Console.ReadLine();

        }

        finally

        {

            MyServiceHost.StopService();

        }

    }

}

I don’t quite think the wrapper is necessary, so I simplified it down to:

       1: let main() =

       2:     Console.WriteLine("IHelloIndigoService = " + typeof<IHelloIndigoService>.ToString() )

       3:  

       4:     let hisType = typeof<HelloIndigoService>

       5:     let host = new ServiceHost(hisType, ([| |] : Uri[] ) )

       6:     host.Open()

       7:     Console.WriteLine("Press <ENTER> to terminate the host application")

       8:     Console.ReadLine() |> ignore

       9:     host.Close()

      10:  

      11: main()

One quirk of the current (Sept 08) F# CTP is that when working with variable-argument parameters (like the second argument of the ServiceHost constructor), F# doesn’t have a great syntax. We have to explicitly specify, in this case, an empty array of Uri objects, but simply specifying an empty array (“[| |]”) will be interpreted as an empty array of objects, and thus generate a compile error. We have to explicitly set the type of the array to be an array of Uri, hence the type specifier.

Oh, and don’t forget, if you’re running as a non-Administrator on Vista or XP, you’ll need to create a URL ACL to allow a non-Administrator user to create an HTTP endpoint; the relevant command for the example above is this:

netsh http add urlacl url=http://+:8000/HelloIndigo user=devtop-t42p\ted

(Obviously, you substitute in your own domain and username for mine.) Make sure to do this from an Administrator-enabled command prompt, or you’ll just get another security error. :-)

The beautiful thing about this example is that if it works, you can use the Client written in C# without a hitch of a problem, thus demonstrating quite clearly that WCF isn’t sharing assemblies between client and service. Given that this service also sets up the MEX endpoint, you should also be able to run svcutil against the running service and generate proxy code if you want to prove that it’s doable; I didn’t do it for this example, since I trust that the App.config-specified MEX endpoint will still be there, and because I was more interested in taking the existing Client and making it work as-is.

More to come, but this should get you started, anyway. Thanks again to Michele for letting me scaffold off of her!

.NET | C# | F# | Languages | WCF | Windows | XML Services

Saturday, January 17, 2009 5:56:06 PM (Pacific Standard Time, UTC-08:00)
Comments [0] |