Hey, anybody who’s got significant VMWare mojo, help out a bro?

I’ve got a Win7 VM (one of many) that appears to be exhibiting weird disk behavior—the vmdk, a growable single-file VMDK, is almost precisely twice the used space. It’s a 120GB growable disk, and the Win7 guest reports about 35GB used, but the VMDK takes about 70GB on host disk. CHKDSK inside Windows says everything’s good, and the VMWare “Disk Cleanup” doesn’t change anything, either. It doesn’t seem to be a Windows7 thing, because I’ve got a half-dozen other Win7 VMs that operate… well, normally (by which I mean, 30GB used in the VMDK means 30GB used on disk). It’s a VMWare Fusion host, if that makes any difference. Any other details that might be relevant, let me know and I’ll post.

Anybody got any ideas what the heck is going on inside this disk?

The JavaZone conference has just become one of my favorite conferences, EVAH. Check out this trailer they put together, entitled "Java 4-Ever". Yes, Microsofties, you should watch, too. Just leave off the evangelism for a moment and enjoy the humor of it. You've had your own fun over the years, too, or need I remind you of the Matrix video with Gates and Ballmer and the blue pill/red pill? ;-)

This video brings several things to mind:

• Wow, that's well done. And take heed, the "R" rating at the front of the trailer is actually pretty serious. NSFW.
• I remember speaking at JavaZone a half-dozen years ago, and remember it fondly. Which reminds me, I need to get back there before long. I missed NDC this year, and I need my Oslo on before long.
• Whatever happened to Microsoft marketing? They used to do things like this on a more regular basis, but it seems they've been silent over the past few years. C'mon back, guys! The water's fine!

Oh, and by the way, pay absolutely no attention to most of the comments that appeared on the trailer page—most of them are ridiculous and stupid. (To the .NET advocate who said that ".NET doesn't use a virtual machine", you're the biggest idiot of the lot.)

By now, the Twitter messages have spread, and the word is out: at Uberconf this year, I did a session ("Pragmatic Architecture"), which I've done at other venues before, but this time we made it into a 180-minute workshop instead of a 90-minute session, and the workshop included breaking the room up into small (10-ish, which was still a teensy bit too big) groups and giving each one an "architectural kata" to work on.

The architectural kata is a take on PragDave's coding kata, except taken to a higher level: the architectural kata is an exercise in which the group seeks to create an architecture to solve the problem presented. The inspiration for this came from Frederick Brooks' latest book, The Design of Design, in which he points out that the only way to get great designers is to get them to design. The corollary, of course, is that in order to create great architects, we have to get them to architect. But few architects get a chance to architect a system more than a half-dozen times or so over the lifetime of a career, and that's only for those who are fortunate to be given the opportunity to architect in the first place. Of course, the problem here is, you have to be an architect in order to get hired as an architect, but if you're not an architect, then how can you architect in order to become an architect?

Um... hang on, let me make sure I wrote that right.

Anyway, the "rules" around the kata (which makes it more difficult to consume the kata but makes the scenario more realistic, IMHO):

• you may ask the instructor questions about the project
• you must be prepared to present a rough architectural vision of the project and defend questions about it
• you must be prepared to ask questions of other participants' presentations
• you may safely make assumptions about technologies you don't know well as long as those assumptions are clearly defined and spelled out
• you may not assume you have hiring/firing authority over the development team
• any technology is fair game (but you must justify its use)
• any other rules, you may ask about

The groups were given 30 minutes in which to formulate some ideas, and then three of them were given a few minutes to present their ideas and defend it against some questions from the crowd.

An example kata is below:

Architectural Kata #5: I'll have the BLT

a national sandwich shop wants to enable "fax in your order" but over the Internet instead

users: millions+

requirements: users will place their order, then be given a time to pick up their sandwich and directions to the shop (which must integrate with Google Maps); if the shop offers a delivery service, dispatch the driver with the sandwich to the user; mobile-device accessibility; offer national daily promotionals/specials; offer local daily promotionals/specials; accept payment online or in person/on delivery

As you can tell, it's vague in some ways, and this is somewhat deliberate—as one group discovered, part of the architect's job is to ask questions of the project champion (me), and they didn't, and felt like they failed pretty miserably. (In their defense, the kata they drew—randomly—was pretty much universally thought to be the hardest of the lot.) But overall, the exercise was well-received, lots of people found it a great opportunity to try being an architect, and even the team that failed felt that it was a valuable exercise.

I'm definitely going to do more of these, and refine the whole thing a little. (Thanks to everyone who participated and gave me great feedback on how to make it better.) If you're interested in having it done as a practice exercise for your development team before the start of a big project, ping me. I think this would be a *great* exercise to do during a user group meeting, too.

Code Katas are small, relatively simple exercises designed to give you a problem to try and solve. I like to use them as a way to get my feet wet and help write something more interesting than "Hello World" but less complicated than "The Internet's Next Killer App".

Rick Minerich mentioned this one on his blog already, but here is the original "problem"/challenge as it was presented to me and which I in turn shot to him over a Twitter DM:

I have a list, say something like [4, 4, 4, 4, 2, 2, 2, 3, 3, 2, 2, 2, 2, 1, 1, 1, 5, 5], which consists of varying repetitions of integers. (We can assume that it's always numbers, and the use of the term "list" here is generic—it could be a list, array, or some other collection class, your choice.) The goal is to take this list of numbers, and "compress" it down into a (theoretically smaller) list of numbers in pairs, where the first of the pair is the occurrence number of the value, which is the second number. So, since the list above has four 4's, followed by three 2's, two 3's, four 2's, three 1's and two 5's, it should compress into [4, 4, 3, 2, 2, 3, 3, 1, 2, 5].

Update: Typo! It should compress into [4, 4, 3, 2, 2, 3, 4, 2, 3, 1, 2, 5], not [4, 4, 3, 2, 2, 3, 3, 1, 2, 5]. Sorry!

Using your functional language of choice, implement a solution. (No looking at Rick's solution first, by the way—that's cheating!) Feel free to post proposed solutions here as comments, by the way.

This is a pretty easy challenge, but I wanted to try and solve it in a functional mindset, which the challenger had never seen before. I also thought it made for an interesting challenge for people who've never programming in functional languages before, because it requires a very different approach than the imperative solution.

Extensions to the kata (a.k.a. "extra credit"):

• How does the implementation change (if any) to generalize it to a list of any particular type? (Assume the list is of homogenous type—always strings, always ints, always whatever.)
• How does the implementation change (if any) to generalize it to a list of any type? (In other words, a list of strings, ints, Dates, whatever, mixed together within the list: [1, 1, "one", "one", "one", ...] .)
• How does the implementation change (if any) to generate a list of two-item tuples (the first being the occurence, the second being the value) as the result instead? Are there significant advantages to this?
• How does the implementation change (if any) to parallelize/multi-thread it? For your particular language how many elements have to be in the list before doing so yields a significant payoff?

By the way, some of the extension questions make the Kata somewhat interesting even for the imperative/O-O developer; have at, and let me know what you think.

... with my earlier speaking about F# post, which I will admit, surprises me, since I would've thought somebody interested in promoting F# would've been more supportive of the idea of putting some ideas out to help other speakers get F# more easily adopted by the community. Perhaps I misunderstood her objections, but I thought a response was required in any event.

Amanda opens with:

Let's start with the "Do" category.

OK, then, let's. :-)

First you say you want the speaker to show inheritance... in a functional-first language. This is an obvious no-no. Inheritance should be used extremely lightly in any language and it should be hidden completely in F#. You should NEVER have a student/instructor/employee inherit from a person. This language isn't used that way.

That's odd.... that's entirely contradictory to what I've heard from the F# team. I've never heard anyone on the F# team ever call it a "functional-first" language, nor that inheritance (or any other object-oriented feature) is something that should be used "extremely lightly" or "hidden completely". Quite the contrary, in fact; when I did a tag-team presentation on F# with Luke Hoban, the PM of the F# team, he gently corrected my use of the phrase describing F# as a "functional-object hybrid" language to suggest instead that it was a "fusion" of both features.

But even if that's not the case (or perhaps isn't the case anymore), I think it's critical to give audience members something concrete and familiar to hang onto as they start the roller-coaster ride of learning not only a new syntax, but new concepts. To simply say, "Everything you know from objects is wrong" is to do them a disservice, particularly when the language clearly is intended to expose object-oriented concepts as a first-class citizen.

Second you say to show interop. This will show nothing about the language. You might as well just say it is a .net language. If you spend your F# session discussing what it means to be on .net, you fail. Nobody expects that one dll will not be able to call another. If they do, I assure you that they will not be writing F# anytime soon.

Ah, but here is where my decades of experience teaching languages to audiences all over the world kicks in: they don't know that. DLLs are not all created equal, as anyone who's ever tried to get COM components to interop with native C++ DLLs that in turn want to call into managed code DLLs will tell you. It's important to stress, again, that what they know is still relevant in this new world. In fact, the goal of showing them interoperability is to reassure them that, in fact, it's not a new world at all, but simply a different spin on the world they already know and love.

Next you say give concrete examples of where F# is a win. This is a sales pitch. It's fine for some audiences but if you intend to teach F# to the audience, you likely are already there. Just make sure your examples are real world and you should be fine. I challenge you to make your next blog a "Why F#" which contains real world examples. I've not ever heard you give valuable advice about when to use F#. Also please post what your real world experience is with F#. Where did you implement a solution? What was that project like? Why was F# the best choice?

Interesting. Based on the conversations I've had with others, the main reason people come to technical talks, at least the talks I've been to (both as an audience member and as a speaker) is to know when and where and how they can use this technology (whatever it is) to solve the problems they face. That means that they need to see and hear where a technology fits well as a solution against a given problem domain or case, and the sooner they get that information, the sooner they can start to evaluate where, how and when they should use a particular technology. This has been true of almost every "new" technology I've evaluated—from the more recent presentations and articles around WCF, Workflow, MongoDB and Axum to the older talks/trainings I've given for C#, Java (including servlets, JSPs, EJBs, JMS, and so on), C++ and patterns. Case in point: does F# offer up a great experience in building UIs? Not really—Visual Studio 2010 doesn't have any of the templates or designer support that C# and Visual Basic will have, making it awkward at best to build a UI around it. On top of that, the data-binding architecture present in both WinForms and WPF rely on the idea of mutable objects, which while something F# allows, isn't something it encourages. So, it seems pretty reasonable to assume that F# is not great for UI scenarios.

Oh, and your memory is letting you down here—your comment "I've not ever heard you give valuable advice about when to use F#" is patently false. You were standing next to me at DevTeach 2008, talking about F# to an audience of about 20 or so when I said that I thought that functional-object languages were a natural fit for building services (XML or otherwise).

More importantly, these were tips to speakers interested in F#—where they think F# is strong and they think F# is weak is a personal judgment, not something that I should dictate. You used F# to implement an insurance-scoring engine, as I recall. I've used it (in conjunction with AbsIL, which used to ship with the F# bits back when they were a MSR technology) to do some IL weaving in the spirit of AOP. I've used it in a couple of other cases, but alas I cannot divulge the details due to NDA. But where I've used it and where you've used it isn't the point—it's what the speaker talking about F# has done that's important. This isn't about us—it's about the guy or gal on the stage who's giving the talk.

Then you say to inform the audience that the language is Turing complete. This seems like a huge waste as well. If the audience needs to understand that you can accomplish the same things in C#/VB/F#/Iron*/etc, you are speaking to people who are very young in the understanding of programming. They won't be using F# anytime soon.

Hmm. I think this is a reaction to the comment "DO stress that F# can do everything that C# or Visual Basic can do", which is a very different creature than simply informing the audience that the language is Turing complete. Again, based on my decade's-plus years of training experience, it's important to let the audience know that they don't have to throw away everything they already know in order to use this language. I know that it's fashionable among the functional programming community to suggest that we should just "toss away all that object stuff", but frankly I've not found that to be the attitude among the "heavyweights" in that part of the industry, nor do I find that attitude laced throughout F#. If that were the case, why would F# go to such great lengths to incorporate object-orientation as a full part of its linguistic capabilities? It would be far simpler to be a CLI Consumer (much as managed JScript is/was) and only offer up functional mechanisms, a la Yeti in the Java space.

I lived through the procedural-to-object transition back in the late 80's/early 90's, and realized that if you want to bring the previous generation of programmers along with you into a brave new world, you have to show them that a complete reboot of their mental processes is not necessary. Otherwise, you're basically calling them idiots if they can't keep up. Perhaps you're OK with that; I'm not.

Finally you say to Tease them for 20 minutes. I am not sure what this means. Can you post those 35 lines to wow us? I'd love to see your real world demo that is 35 lines. I'm curious as to why you wouldn't be able to explain the 35 lines as well. I guess there isn't time because you're busy showing interop examples that prove F# is a Turing complete, .net language.

Alas, I doubt my 35 lines would impress you. However, my 35 lines of F# service code, or Aaron's 35 lines of F# natural-language parser code might impress the crowd we're speaking to. I dunno. More importantly, again, this isn't about what *I* want to do in a talk, it's about helping other F# speakers be able to better reach their audience.

Let's get into the Don't category:

So soon? But we were just getting comfortable with all the DO's being judged completely out of order from their corresponding DON'Ts. *shrug* Ah, well.

First you say to stay away from mathematical examples because people don't write mathematical code every day. I think you already mentioned that F# is not meant to be the language you use for every scenario. Now it seems you want to say it should be the everyday tool. I'm confused. I agree that some of these simple examples aren't very useful but then again it's not because they are mathematical. It's because they are simple and ridiculous. I don't use a web crawler everyday either but I see value in the demo. I think the examples need to be more real world, period. Have you posted that blog I requested yet? :)

Ah, the black/white pedagogical argument: if it's not black, it must be white, and if it's not white, it must be black. Your confusion is clear: if it is not a language to be used for everything, it must be a niche language solely for creating high-end mathematical systems, and if it isn't just for creating high-end mathematical systems, it must be a language used for everything.

My reasoning for avoiding the exponent-hugging example is pretty easy, I think: Mathematical examples reinforce the idea that F# is solely to be used for high-end mathematical scenarios. If you're OK with the language only appealing to that crowd, please, by all means, continue to use those examples. Myself, I think functional concepts are powerful, and I try to show people the power of extracting behavior by showing them widely-disparate uses of foldLeft across lists of things to produce concrete yet widely different results. Simple examples, but without a shred of "derivatives" found anywhere.

Alas, that blog post will have to wait—I have an F# book I'm finishing up, and I'd rather put the energy there.

Next up you say to not stress FSI or the REPL. I'll start by reminding you that FSI is the REPL. There aren't two different things here. I think it's great to show a REPL! This is not just a cool F# thing. It's common to most functional languages, statically typed or not. The statically typed argument might be a better one to have than Turing completeness. I'd much rather discuss those benefits for the types of code that are written in F#.

Wow. I wouldn't have thought I would have to remind you that REPL is a generic phrase that can apply to both FSI and the Interactive Window inside Visual Studio. And while I'm certainly happy to hear that you think it's great to show a REPL, the fact remains that most .NET developers don't know what to do with it. More importantly, demonstrating a REPL reinforces the idea that this is a shell-scripting language like Python and Ruby and PowerShell, hence the questions comparing F# to Python or Perl that come up every time I've seen an F# talk show off FSI or the Interactive Window. Business developers using .NET build using Visual Studio (with the exception of that small percentage who've discovered IPy or IRb) and, again, need to be brought gently into this new approach.

(For those readers still following along, the REPL concept is hardly restricted to the functional language cadre; in fact, object-oriented developers would be well-advised to play with one of their own ancient progenitors, Smalltalk, and its environment that is essentially one giant REPL baked into a GUI image that can be frozen and re-hydrated at any time. Long-time readers of this blog will know I've talked about this before, and how incredibly powerful it would be if we could do similar kinds of things to the JVM or CLR.)

You go back into the Why F# question without giving any real reason. Can you post that blog please? I think many of your readers would appreciate that! PS: The Steelers are fantastic! :)

If I'm following your point-by-point refutation correctly, you're now saying I'm "going back" to the "Why F#" question for no real reason; I would've thought the progression of DON'T followed by DO would've been pretty obvious, but perhaps I was assuming too much on the part of at least one of the post's readership. The DO was designed to offer up prescriptive advice about how to accomplish something I'd said to DON'T previously. And thus is true here: DON'T answer the "Why F#" question with "Productivity", DO answer it with something more concrete and tangible than that, either in the form of real-world examples or concrete scenarios.

I think by this point, given all the wheedling for that blog post, the general readership would probably be very interested in your own rationale blog post, by the way.

Alas, your Steelers barely made it to .500 last year, their franchise quarterback is now the target of his second (and possibly more, if the rumors are to be believed) sexual assault charge, and their principal receiver has a reputation around the league as being a dirty player. So perhaps we will simply have to disagree on how fantastic they are. Which, you will note, proves my point—as the old saying goes, "there is no accounting for taste", because I can't understand how you think. Which then means "It's just how I think" is pretty ridiculous as a justification for using a language.

You say to stay away from the "functional jazz" or the reason why anyone should be looking at F# to start with. People don't come to these types of talks to see how F# is just like C#. They want to see what is different. Don't stress the jargon but if someone asks, let them know there is a name for what they are looking at. I remember when I was learning F# that everyone hid the meaning of let!. They would say "Something special happens here" and that would leave me thinking they were trying to hide the magic. There is no magic! I don't assume people are morons. They can handle the truth. If they want to learn more I want to give them a term to google and some potential resources. There isn't time to cover that completely in most sessions though. It's something to be careful of, not to avoid completely.

Interesting how your anecdotal evidence differs from mine—what I've seen, based on the quick poll I took of the attendees at the user group meeting last night, and based on conversations I've had with hundreds of developers from companies all over the world over the last four years, vastly more attendees come to a talk on a given subject because they have no clue what this thing is and want to see a general overview of it. Shy Cohen, one of the attendees last night, whom I first met during my days as a digerati on the WCF team back when it was still called "Indigo", admitted as much during a whispered conversation at the back of the room. If Shy, old Microsoft hand that he is/was, bright guy that he is, and close friend to Lisa Feigenbaum, who's a Program Manager for Visual Studio, has no clue what F# is and comes to a talk on it so he can get a quick overview of it, how likely is it that everybody is coming to an F# talk with a predetermined idea of what the language is and are thus ready to be given "the truth" complete with all the big dime-store words?

Yes, people want to know what is different, but to do that, they also have to see what is the same. Which takes us back to my earlier points about showing them what is the same between F# and C#.

As for people waving their hands and saying "something special happens here", well, maybe you just listened to the wrong people. *shrug* Can't help you there. For as long as I've been giving talks on F#, dating back to SDWest back in 2005 when I gave a talk on "A Tour of Microsoft Research" during which I talked about Fugue, Detours, AbsIL and F#, I've shown the language, talked about what's happening in there, and shown the IL bindings underneath to give people concrete ideas to hold on to. It's the truth, but without the pretentiousness of big words.

The last point is obvious. Nobody can learn F# in 20 (or 30 as it was) minutes.

Unfortunately, that doesn't stop people from trying to teach the entirety of the language in 20 minutes. Or even in a full day. (From having taught languages for many years, and knowing that it took most of a week to teach C# back in the 1.0/2.0 timeframe, I'm finding that it takes about 5 days of full 8-to-5 training to get them competent and confident in using the language. Less than that, by about a day or so, if they have a strong background in C#.)

Context, context, context.

Indeed. But for now, Amanda, if you take such strong issue with my suggested guidelines for F# speakers, I encourage you to create your own guidelines and post them to your blog. Let's rise the tide to raise all the ships, and encourage a broad spectrum of talk styles.

In the meantime, though, I have a lunch with Michael later this week, some OTN and developerWorks articles to write, an F# book to finish, a Scala book to start, some client code to wrap up, a slew of Scala recordings to work through, soccer practice Thursday night, and a Seattle Tech Speakers Workshop meeting next month to prep for, in addition to a class next week that requires some final polish, so you'll have to excuse me if I don't respond further down this particular path.

Cheers!

Michael Easter called me out over Twitter tonight, entirely fairly. This blog post is to attempt to make right.

Context: Tonight was a .NET Developer Association meeting in Redmond, during which we had two presentations: one on Entity Framework, and one on F#. The talk on F#, while well-meaning and delivered by somebody I've not yet met personally, suffered from several failures that I believe to be endemic to Microsoft's approach to presenting F#. I don't fault the speaker—I think Michael was set up to fail from the very beginning. Thus, I decided that it was time for me to "put up" and describe the structural failures I've seen in several talks attempting to describe F# to the general .NET computing community. (I think these could probably be generalized to presenting a new language to any general computing community, but I'll keep it focused on F# for now.)

In no particular order:

• DON'T use a demo based on a mathematical principle (like Fibonacci, factorial, or some other exponent-hugging formula). I ask you, how many developers find themselves writing that kind of code on a daily basis? If you offer up purely mathematical examples, you will create the impression that F# is only good for high-scale numerical and mathematical computing, such as what scientists use, and you will essentially convince everybody in the room that F# belongs in that class of programming language that doesn't have anything to do with them.
• DO use a demo based on real-world environments or problems. Use domain types that could have come from a regular line-of-business scenario; my favorite is "Person", since that can serve as a base type for other, more domain-specific, types (like "Student", "Instructor", "Employee", and whatever).
• DON'T stress the F# Interactive environment. Yes, it's great that F# has an interactive environment and a REPL. But accept that this is not what the general development community cares about, or even sees value in. In fact, the more you stress the REPL/interactive window in F#, the more likely you are to get a question at the end of the talk asking you to compare F# to Python or Perl. Then you end up having to argue the benefits of static typing and type inference over dynamic/duck typing, which really makes no sense in a scripting tool, which is only on the questioners' mind because you put it there by stressing the REPL.
• DO show F# code being called by other assemblies, and vice versa. At the end of the day, the watchword here should be "interoperability", because no matter how eloquent your presentation, you're not going to get the audience to suddenly abandon their C# and Visual Basic and switch over to writing everything in F#, because there's just too many scenarios where F# is not the right answer (UI "top of the stack" kinds of things being at the top of my "not great for F#" list). Stress how an F# type is just a class, with methods that can be invoked from C# and vice versa.
• DON'T answer the inevitable "why should I care?" question with the word "productivity". I hate to be the one to point this out, but every language ever introduced has held this up as a reason to switch to it, and none of them have ever really felt like they were a productivity boost, at least not in the long run. And if you answer with, "Because I just think that way", that's a FAIL on your part, because I can't see how your thinking changes mine. (You may also like the Pittsburgh Steelers, while I know they can't hold a candle to the New Orleans Saints—now where are we?)
• DO answer the inevitable "why should I care?" question with tangible real-world scenarios or examples. Give two or three cases, abstract or concrete, where F# makes the developers' life easier, and how. And frankly, I would sprinkle in a few cases where F# isn't a net win, because everybody knows, deep down, that no one language is perfect for all scenarios. (Only marketing and sales people seem to think there is.)
• DON'T jump straight into all this functional jazz. I hate to tell you this, but most of the developer community is not convinced that functional programming is "obviously" the right way to program. Attempting to take them deep into functional mojo is only going to lose them and overwhelm them and quite likely convince them that functional programming is for math majors. Use of the terms "catamorphism" or "monad" or "partial application" or "currying" in your introductory talk is an exercise in stroking your own ego, not in teaching the audience something useful.
• DO stress that F# can do everything C# or Visual Basic can do. Developers like to start with the familiar—it's why every programming language starts with the "Hello World" example, not only because it's simple and straightforward but because developers have come to expect it. F# can build types just like C# can, so do that, and use that as a framework from which to build up their understanding of the syntax and semantics.
• DON'T assume you can give an introduction to a programming language in 20 minutes. I don't care how good you are as a presenter, it can't be done. 50 minutes would be pushing it. 90 minutes is maybe just enough to get through enough syntax to get the audience to the point where they can read a commonplace F# program. Maybe.
• DO tease the hell out of them for 20 minutes. If you only have 20 minutes, then create a super-sexy demo (not a math-based or scripting-based one), show them the demo, then point out that this is written in 35 lines of F#, and if they want to understand what's going on in that 35 lines, here's some resources to go learn F#. Leave them wanting more.

Again, I'm not faulting Michael (tonight's speaker): I think he bravely attempted what was likely to be a failure regardless of who was giving the talk. My hope is that as others start to step up to talk about F# to their coworkers and fellow user group members, this will help avoid a few more "Oh, so F# is totally irrelevant to me" reactions.

A couple of days ago, a buddy of mine, Scott Hanselman, wrote a nice little intro to the "dynamic" type in C# 4.0. In particular, I like (though don't necessarily 100% agree with) his one-sentence summation of dynamic as "There's no way for you or I to know the type of this now, compiler, so let's hope that the runtime figures it out." It's an interesting characterization, but my disagreement with his characterization is not the point here, at least not of this particular blog entry.

I've been waiting for it for a while, ever since C# 4 was announced, and sure enough, here we go: Scott's blog is the victim of the Static-Typing Fundamentalist, the bearded and grizzled veteran of the Static/Dynamic Code Wars, come out to proclaim the sins of dynamic programming, the evils of those who use(d) it, and why C#/C++/Java was so much better than Visual Basic/Ruby/Python/whatever. Be careful of these creatures. They rival Al-Qaeda in their ferocity and zeal, Fox News in their attention to detail and evidence, and George Bush in their pronouncements of gloom and doom for the future if we don't act now and eliminate this evil.

Allow me to quote (liberally) from Rob's comment on Scott's blog, and comment in turn as we go:

It's such a shame that you promote this stuff. You should've seen the horrific devastation that "Variant" caused in the old VB days. Variant single-handedly create job security for so many people since the late 90's, because of the horrible, horrible, horrible things that developers did with that ridiculous, 12-byte data type!

I just love it when people make comments like "horrific devastation". Nothing like a little hyperbole to liven things up! I mean, it didn't cause exceptions, it didn't make code hard to read, it didn't make it tricky for developers to modify and refactor safely, it leveled cities! burned forests! slaughtered kittens! and even worse, it was 12 bytes in size!

Never mind the fact that Visual Basic developers frequently churned out apps twice, three, five times faster than their C++ cousins did. (I know this—I was one of those C++ developers, and routinely mocked the VB guys across the hall for their crappy language and code.... until they built an app in a few days that I tried to build at home in C++ and gave up after two weeks. And all the damn thing did was basic dialogs-and-data kinds of stuff, too.)

This weak-typing with late-binding is just such a bad idea. I know you'll say "But wait, these are powerful tools that skilled developers can leverage!" - and maybe so, but 98% of the people that truly use these sorts of techniques out in the real world, are unskilled developers making a mess of software all across this great land, because the compiler is so forgiving.

Ah, the "All Developers (Except Me) Are Idiots" argument. I love this one—the hubris involved here is just too precious for words. I have no doubt that the author of this post, being (of course) the classically-trained object-oriented developer and therefore too smart/disciplined/experienced/whatever to fall into such a ridiculous temptation as to use dynamic typing, would never use this feature except in the Most Dire of Emergencies, but his fellow programmers, all of them being much less disciplined/smart/trained/whatever than he is, will fall for the temptation and write code that levels cities! burns forests! kills kittens! and worse, uses 12 bytes! (Oh, wait, it's only 3 bytes, because dynamic is just a placeholder for an object reference, and all object references are 3 bytes in the CLR. Or at least they used to be—I admit, I haven't checked in CLR 4.) Those poor souls, they won't have any hope! There they'll be, staring at Visual Studio, wanting desperately to do the Right Thing, and that evil little programmer devil on their shoulder (probably wearing a T-shirt that says, "P3rl is l33t" or something equally blasphemous) will whisper, "You know, if you just make it a dynamic, you can get the compiler to shut up and you can go home early...."

Oh, right—sorry, I forgot. That devil will whisper, "You know, if you write this code in Visual Basic .NET, you can make the entire codebase Option Strict Off and Option Explicit Off, make the compiler shut up and you can go home early...." Hell, they've been whispering that bit of subversion since 2001. And ye Gods! The leveled cities! burned forests! cute little kitten bodies! all over the place! It's fortunate that we C# developers have kept all those Visual Basic developers on the straight-and-narrow path of true salvation static typing.

This is a huge step backwards for C#, in my opinion - and creates the same scenario VB always did - where it is so forgiving, that it allows developers to write horrible code and you won't so much as see a compiler warning!! I've always tauted that C# was better, simply because it gave the developer "tough love", and forced him/her to be better coder and to "make good choices"! :-)

Ah, yes, the C# compiler and its "tough love". The "prefer compile errors over runtime errors" argument, vis-a-vis Scott Meyers' "Effective C++" circa 1994 or so. It's vastly preferable to see errors early, before the big demo in front of the VP/President/potential customer. (Anybody who disagrees with this obviously hasn't had a demo fail in front of a VP/President/potential customer.) How fortunate that the C# compiler catches all these ugly errors at compile-time, like

               1:
              static
              void DoSomething()

               2: {

               3: List<object>
intList = new List<object>();

               4: intList.Add(5);

               5:
              string s
= (string) intList[0];

               6: Console.WriteLine(s);

               7: }

... because boy, that would be embarrassing if it didn't. I mean, can you imagine the horror other disciplined/smart/experienced developers would feel if a lenient compiler actually allowed code like this:

               1:
              class Point

               2: {

               3:
              internal
              int x;

               4:
              internal
              int y;

               5:
              public Point(int x, int y)

               6: {

               7: x
= x;

               8: y
= y;

               9: }

               10: }

or this:

               1:
              class Point

               2: {

               3:
              internal
              int x;

               4:
              internal
              int y;

               5:
              public Point(int x, int y)

               6: {

               7:
              this.x
= x;

               8:
              this.y
= y;

               9: }

               10:
              public
              override
              string ToString()

               11: {

               12:
              return String.Format("({0},{1})",
x, y);

               13: }

               14: }

               15:
              static
              void DoSomething()

               16: {

               17: Point
pt = new Point(12, 12);

               18: pt.GetType()

               19: .GetField("x",
BindingFlags.Instance | 

               20: BindingFlags.NonPublic)

               21: .SetValue(pt,
24);

               22: Console.WriteLine(pt);

               23: }

to compile? Cities! Forests! Kittens! Thank God C# isn't that kind of lustfully promiscuous... I mean, "lenient"... compiler!

(Now if only we could tout blog comment engines with spellcheck....)

Specific to this blog post, if you are doing somewhere where you can't even quantify what the data type that is coming back? Guess waht, you've got yourself a bad design.

Wow. There's just no arguing with that one. I mean, knowing the actual type on which the method is being dispatched is such a huge part of the C# development experience:

               1:
              static
              void DoSomething()

               2: {

               3: List<Point>
ptList = new List<Point>();

               4: ptList.Add(new Point(12,
12));

               5:
              object o
= ptList[0];

               6: Console.WriteLine(o.ToString());

               7: }

Gah. Just the thought of not knowing the concrete type on which the method is being dispatched gives me the heebie-jeebies.

Just because the framework allows you use weak-typing and late-binding, doesn't mean you should - nor should you endorse it's use, in my opinion.

Somebody better tell all those users of NHibernate, NUnit, Spring.NET, MEF and all those other Reflection-based tools... including WinForms, ASP.NET, WPF, Workflow and WCF, come to think about it... that they're using frameworks that clearly were designed by idiots. (The gall of those people.)

I'm just saying, it's a shame that popular "nerd celebrities" like you (and I mean zero offense by that!) - endorse all this loosey-goosey typing. I say that becuase I've never seen a single case where weak typing or late binding: A) made a design better or B) where it didn't make the component or application worse, because it was a looser design.

I'm so glad you were here to set Scott and me straight, Rob. Because otherwise, we might actually get something done. God forbid.

Little tidbits of thought for those who are still thinking about this one.

• Ola Bini describes the application of the right language at the right level of the stack as a three-layer pyramid.
• Any C# or Java developer who's not writing unit tests to test their code "because the compiler will catch all those errors" and provide "tough love" needs to be fired. Immediately. I cannot conceive of a situation where unit tests can be passed over in favor of static typing in a professionally-responsible development project. (Oh, don't mis-read that, I can see lots of situations where unit tests aren't necessary. But not on code that's going to reach Production.)
• The argument for the degree of static typing in C# or Java is completely indefensible compared to what statically-typed type-inferenced languages like Haskell, F# or Scala provide. And their syntax frequently looks like "let x = [ 1; 2; 3; 4; ]", which isn't all that far off from what a dynamically-typed language looks like, despite very very different things happening under the compiler's hood. Until you, the Statically-Typed Fundamentalist, have written code in a Haskell/ML-derived language, you have no right arguing the merits of static typing. (In fact, that's probably also true if you've never written code in Ruby, Python, or PowerShell, either.)
• There's lots more arguments the Static-Typing Fundamentalist can throw, by the way. I'm disappointed Rob never mentioned performance, for one—that's a classic line of attack, too. Never mind the fact that most of those guys are still looping down and doing other silly micro-optimizations because that's way C++ taught them to do it....
• Oh, and never ever show the Static Typing Fundamentalist an XML document and using something like XPath to extract data from it. They inevitably fall into XML Schema and the "if we just write the schema flexibly enough" and.... The last time I did that.... I still visit his gravesite, all these years later, and it still hurts, losing him that way.
• Java guys argued against dynamic typing for years, too... until they tried Groovy and JRuby and Clojure. Now.... not so much.

Peace out.

Cruising the Web late last night, I ran across "10 things you can do to advance your career as a developer", summarized below:

1. Build a PC
2. Participate in an online forum and help others
3. Man the help desk
4. Perform field service
5. Perform DBA functions
6. Perform all phases of the project lifecycle
7. Recognize and learn the latest technologies
8. Be an independent contractor
9. Lead a project, supervise, or manage
10. Seek additional education

I agreed with some of them, I disagreed with others, and in general felt like they were a little too high-level to be of real use. For example, "Seek additional education" seems entirely too vague: In what? How much? How often? And "Recognize and learn the latest technologies" is something like offering advice to the Olympic fencing silver medalist and saying, "You should have tried harder".

So, in the great spirit of "Not Invented Here", I present my own list; as usual, I welcome comment and argument. And, also as usual, caveats apply, since not everybody will be in precisely the same place and be looking for the same things. In general, though, whether you're looking to kick-start your career or just "kick it up a notch", I believe this list will help, because these ideas have been of help to me at some point or another in my own career.

10: Build a PC.

Yes, even developers have to know about hardware. More importantly, a developer at a small organization or team will find himself in a position where he has to take on some system administrator roles, and sometimes that means grabbing a screwdriver, getting a little dusty and dirty, and swapping hardware around. Having said this, though, once you've done it once or twice, leave it alone—the hardware game is an ever-shifting and ever-changing game (much like software is, surprise surprise), and it's been my experience that most of us only really have the time to pursue one or the other.

By the way, "PC" there is something of a generic term—build a Linux box, build a Windows box, or "build" a Mac OS box (meaning, buy a Mac Pro and trick it out a little—add more memory, add another hard drive, and so on), they all get you comfortable with snapping parts together, and discovering just how ridiculously simple the whole thing really is.

And for the record, once you've done it, go ahead and go back to buying pre-built systems or laptops—I've never found building a PC to be any cheaper than buying one pre-built. Particularly for PC systems, I prefer to use smaller local vendors where I can customize and trick out the box. If you're a Mac, that's not really an option unless you're into the "Hackintosh" thing, which is quite possibly the logical equivalent to "Build a PC". Having never done it myself, though, I can't say how useful that is as an educational action.

9: Pick a destination

Do you want to run a team of your own? Become an independent contractor? Teach programming classes? Speak at conferences? Move up into higher management and get out of the programming game altogether? Everybody's got a different idea of what they consider to be the "ideal" career, but it's amazing how many people don't really think about what they want their career path to be.

A wise man once said, "The journey of a thousand miles begins with a single step." I disagree: The journey of a thousand miles begins with the damn map. You have to know where you want to go, and a rough idea of how to get there, before you can really start with that single step. Otherwise, you're just wandering, which in itself isn't a bad thing, but isn't going to get you to a destination except by random chance. (Sometimes that's not a bad result, but at least then you're openly admitting that you're leaving your career in the hands of chance. If you're OK with that, skip to the next item. If you're not, read on.)

Lay out explicitly (as in, write it down someplace) what kind of job you're wanting to grow into, and then lay out a couple of scenarios that move you closer towards that goal. Can you grow within the company you're in? (Have others been able to?) Do you need to quit and strike out on your own? Do you want to lead a team of your own? (Are there new projects coming in to the company that you could put yourself forward as a potential tech lead?) And so on.

Once you've identified the destination, now you can start thinking about steps to get there.

If you want to become a speaker, put your name forward to give some presentations at the local technology user group, or volunteer to hold a "brown bag" session at the company. Sign up with Toastmasters to hone your speaking technique. Watch other speakers give technical talks, and see what they do that you don't, and vice versa.

If you want to be a tech lead, start by quietly assisting other members of the team get their work done. Help them debug thorny problems. Answer questions they have. Offer yourself up as a resource for dealing with hard problems.

If you want to slowly move up the management chain, look to get into the project management side of things. Offer to be a point of contact for the users. Learn the business better. Sit down next to one of your users and watch their interaction with the existing software, and try to see the system from their point of view.

And so on.

8: Be a bell curve

Frequently, at conferences, attendees ask me how I got to know so much on so many things. In some ways, I'm reminded of the story of a world-famous concert pianist giving a concert at Carnegie Hall—when a gushing fan said, "I'd give my life to be able to play like that", the pianist responded quietly, "I did". But as much as I'd like to leave you with the impression that I've dedicated my entire life to knowing everything I could about this industry, that would be something of a lie. The truth is, I don't know anywhere near as much as I'd like, and I'm always poking my head into new areas. Thank God for my ADD, that's all I can say on that one.

For the rest of you, though, that's not feasible, and not really practical, particularly since I have an advantage that the "working" programmer doesn't—I have set aside weeks or months in which to do nothing more than study a new technology or language.

Back in the early days of my career, though, when I was holding down the 9-to-5, I was a Windows/C++ programmer. I was working with the Borland C++ compiler and its associated framework, the ObjectWindows Library (OWL), extending and maintaining applications written in it. One contracting client wanted me to work with Microsoft MFC instead of OWL. Another one was storing data into a relational database using ODBC. And so on. Slowly, over time, I built up a "bell curve"-looking collection of skills that sort of "hovered" around the central position of C++/Windows.

Then, one day, a buddy of mine mentioned the team on which he was a project manager was looking for new blood. They were doing web applications, something with which I had zero experience—this was completely outside of my bell curve. HTML, HTTP, Cold Fusion, NetDynamics (an early Java app server), this was way out of my range, though at least NetDynamics was a little similar, since it was basically a server-side application framework, and I had some experience with app frameworks from my C++ days. So, resting on my C++ experience, I started flirting with Java, and so on.

Before long, my "bell curve" had been readjusted to have Java more or less at its center, and I found that experience in C++ still worked out here—what I knew about ODBC turned out to be incredibly useful in understanding JDBC, what I knew about DLLs from Windows turned out to be helpful in understanding Java's dynamic loading model, and of course syntactically Java looked a lot like C++ even though it behaved a little bit differently under the hood. (One article author suggested that Java was closer to Smalltalk than C++, and that prompted me to briefly flirt with Smalltalk before I concluded said author was out of his frakking mind.)

All of this happened over roughly a three-year period, by the way.

The point here is that you won't be able to assimilate the entire industry in a single sitting, so pick something that's relatively close to what you already know, and use your experience as a springboard to learn something that's new, yet possibly-if-not-probably useful to your current job. You don't have to be a deep expert in it, and the further away it is from what you do, the less you really need to know about it (hence the bell curve metaphor), but you're still exposing yourself to new ideas and new concepts and new tools/technologies that still could be applicable to what you do on a daily basis. Over time the "center" of your bell curve may drift away from what you've done to include new things, and that's OK.

7: Learn one new thing every year

In the last tip, I told you to branch out slowly from what you know. In this tip, I'm telling you to go throw a dart at something entirely unfamiliar to you and learn it. Yes, I realize this sounds contradictory. It's because those who stick to only what they know end up missing the radical shifts of direction that the industry hits every half-decade or so until it's mainstream and commonplace and "everybody's doing it".

In their amazing book "The Pragmatic Programmer", Dave Thomas and Andy Hunt suggest that you learn one new programming language every year. I'm going to amend that somewhat—not because there aren't enough languages in the world to keep you on that pace for the rest of your life—far from it, if that's what you want, go learn Ruby, F#, Scala, Groovy, Clojure, Icon, Io, Erlang, Haskell and Smalltalk, then come back to me for the list for 2020—but because languages aren't the only thing that we as developers need to explore. There's a lot of movement going on in areas beyond languages, and you don't want to be the last kid on the block to know they're happening.

Consider this list: object databases (db4o) and/or the "NoSQL" movement (MongoDB). Dependency injection and composable architectures (Spring, MEF). A dynamic language (Ruby, Python, ECMAScript). A functional language (F#, Scala, Haskell). A Lisp (Common Lisp, Clojure, Scheme, Nu). A mobile platform (iPhone, Android). "Space"-based architecture (Gigaspaces, Terracotta). Rich UI platforms (Flash/Flex, Silverlight). Browser enhancements (AJAX, jQuery, HTML 5) and how they're different from the rich UI platforms. And this is without adding any of the "obvious" stuff, like Cloud, to the list.

(I'm not convinced Cloud is something worth learning this year, anyway.)

You get through that list, you're operating outside of your comfort zone, and chances are, your boss' comfort zone, which puts you into the enviable position of being somebody who can advise him around those technologies. DO NOT TAKE THIS TO MEAN YOU MUST KNOW THEM DEEPLY. Just having a passing familiarity with them can be enough. DO NOT TAKE THIS TO MEAN YOU SHOULD PROPOSE USING THEM ON THE NEXT PROJECT. In fact, sometimes the most compelling evidence that you really know where and when they should be used is when you suggest stealing ideas from the thing, rather than trying to force-fit the thing onto the project as a whole.

6: Practice, practice, practice

Speaking of the concert pianist, somebody once asked him how to get to Carnegie Hall. HIs answer: "Practice, my boy, practice."

The same is true here. You're not going to get to be a better developer without practice. Volunteer some time—even if it's just an hour a week—on an open-source project, or start one of your own. Heck, it doesn't even have to be an "open source" project—just create some requirements of your own, solve a problem that a family member is having, or rewrite the project you're on as an interesting side-project. Do the Nike thing and "Just do it". Write some Scala code. Write some F# code. Once you're past "hello world", write the Scala code to use db4o as a persistent storage. Wire it up behind Tapestry. Or write straight servlets in Scala. And so on.

5: Turn off the TV

Speaking of marketing slogans, if you're like most Americans, surveys have shown that you watch about four hours of TV a day, or 28 hours of TV a week. In that same amount of time (28 hours over 1 week), you could read the entire set of poems by Maya Angelou, one F. Scott Fitzgerald novel, all poems by T.S.Eliot, 2 plays by Thornton Wilder, or all 150 Psalms of the Bible. An average reader, reading just one hour a day, can finish an "average-sized" book (let's assume about the size of a novel) in a week, which translates to 52 books a year.

Let's assume a technical book is going to take slightly longer, since it's a bit deeper in concept and requires you to spend some time experimenting and typing in code; let's assume that reading and going through the exercises of an average technical book will require 4 weeks (a month) instead of just one week. That's 12 new tools/languages/frameworks/ideas you'd be learning per year.

All because you stopped watching David Caruso turn to the camera, whip his sunglasses off and say something stupid. (I guess it's not his fault; CSI:Miami is a crap show. The other two are actually not bad, but Miami just makes me retch.)

After all, when's the last time that David Caruso or the rest of that show did anything that was even remotely realistic from a computer perspective? (I always laugh out loud every time they run a database search against some national database on a completely non-indexable criteria—like a partial license plate number—and it comes back in seconds. What the hell database are THEY using? I want it!) Soon as you hear The Who break into that riff, flip off the TV (or set it to mute) and pick up the book on the nightstand and boost your career. (And hopefully sink Caruso's.)

Or, if you just can't give up your weekly dose of Caruso, then put the book in the bathroom. Think about it—how much time do you spend in there a week?

And this gets even better when you get a Kindle or other e-reader that accepts PDFs, or the book you're interested in is natively supported in the e-readers' format. Now you have it with you for lunch, waiting at dinner for your food to arrive, or while you're sitting guard on your 10-year-old so he doesn't sneak out of his room after his bedtime to play more XBox.

4: Have a life

Speaking of XBox, don't slave your life to work. Pursue other things. Scientists have repeatedly discovered that exercise helps keep the mind in shape, so take a couple of hours a week (buh-bye, American Idol) and go get some exercise. Pick up a new sport you've never played before, or just go work out at the gym. (This year I'm doing Hopkido and fencing.) Read some nontechnical books. (I recommend anything by Malcolm Gladwell as a starting point.) Spend time with your family, if you have one—mine spends at least six or seven hours a week playing "family games" like Settlers of Catan, Dominion, To Court The King, Munchkin, and other non-traditional games, usually over lunch or dinner. I also belong to an informal "Game Night club" in Redmond consisting of several Microsoft employees and their families, as well as outsiders. And so on. Heck, go to a local bar and watch the game, and you'll meet some really interesting people. And some boring people, too, but you don't have to talk to them during the next game if you don't want.

This isn't just about maintaining a healthy work-life balance—it's also about having interests that other people can latch on to, qualities that will make you more "human" and more interesting as a person, and make you more attractive and "connectable" and stand out better in their mind when they hear that somebody they know is looking for a software developer. This will also help you connect better with your users, because like it or not, they do not get your puns involving Klingon. (Besides, the geek stereotype is SO 90's, and it's time we let the world know that.)

Besides, you never know when having some depth in other areas—philosophy, music, art, physics, sports, whatever—will help you create an analogy that will explain some thorny computer science concept to a non-technical person and get past a communication roadblock.

3: Practice on a cadaver

Long before they scrub up for their first surgery on a human, medical students practice on dead bodies. It's grisly, it's not something we really want to think about, but when you're the one going under the general anesthesia, would you rather see the surgeon flipping through the "How-To" manual, "just to refresh himself"?

Diagnosing and debugging a software system can be a hugely puzzling trial, largely because there are so many possible "moving parts" that are creating the problem. Compound that with certain bugs that only appear when multiple users are interacting at the same time, and you've got a recipe for disaster when a production bug suddenly threatens to jeopardize the company's online revenue stream. Do you really want to be sitting in the production center, flipping through "How-To"'s and FAQs online while your boss looks on and your CEO is counting every minute by the thousands of dollars?

Take a tip from the med student: long before the thing goes into production, introduce a bug, deploy the code into a virtual machine, then hand it over to a buddy and let him try to track it down. Have him do the same for you. Or if you can't find a buddy to help you, do it to yourself (but try not to cheat or let your knowledge of where the bug is color your reactions). How do you know the bug is there? Once you know it's there, how do you determine what kind of bug it is? Where do you start looking for it? How would you track it down without attaching a debugger or otherwise disrupting the system's operations? (Remember, we can't always just attach an IDE and step through the code on a production server.) How do you patch the running system? And so on.

Remember, you can either learn these things under controlled circumstances, learn them while you're in the "hot seat", so to speak, or not learn them at all and see how long the company keeps you around.

2: Administer the system

Take off your developer hat for a while—a week, a month, a quarter, whatever—and be one of those thankless folks who have to keep the system running. Wear the pager that goes off at 3AM when a server goes down. Stay all night doing one of those "server upgrades" that have to be done in the middle of the night because the system can't be upgraded while users are using it. Answer the phones or chat requests of those hapless users who can't figure out why they can't find the record they just entered into the system, and after a half-hour of thinking it must be a bug, ask them if they remembered to check the "Save this record" checkbox on the UI (which had to be there because the developers were told it had to be there) before submitting the form. Try adding a user. Try removing a user. Try changing the user's password. Learn what a real joy having seven different properties/XML/configuration files scattered all over the system really is.

Once you've done that, particularly on a system that you built and tossed over the fence into production and thought that was the end of it, you'll understand just why it's so important to keep the system administrators in mind when you're building a system for production. And why it's critical to be able to have a system that tells you when it's down, instead of having to go hunting up the answer when a VP tells you it is (usually because he's just gotten an outage message from a customer or client).

1: Cultivate a peer group

Yes, you can join an online forum, ask questions, answer questions, and learn that way, but that's a poor substitute for physical human contact once in a while. Like it or not, various sociological and psychological studies confirm that a "connection" is really still best made when eyeballs meet flesh. (The "disassociative" nature of email is what makes it so easy to be rude or flamboyant or downright violent in email when we would never say such things in person.) Go to conferences, join a user group, even start one of your own if you can't find one. Yes, the online avenues are still open to you—read blogs, join mailing lists or newsgroups—but don't lose sight of human-to-human contact.

While we're at it, don't create a peer group of people that all look to you for answers—as flattering as that feels, and as much as we do learn by providing answers, frequently we rise (or fall) to the level of our peers—have at least one peer group that's overwhelmingly smarter than you, and as scary as it might be, venture to offer an answer or two to that group when a question comes up. You don't have to be right—in fact, it's often vastly more educational to be wrong. Just maintain an attitude that says "I have no ego wrapped up in being right or wrong", and take the entire experience as a learning opportunity.

But too impatient to read a whole book on it? Try the 6-panel RefCard that Chance Coble and I put together for DZone. Free download.

Or, for the more patient type, wait for the books that Chance and I (Professional F#) are each writing; they're remarkably complementary, at least from what Chance has told me about his.

Which reminds me.... if you've not already noticed, Pro F# is now up in Amazon. Call me a romantic fool, but I get just a little thrill run down my spine every time a new book of mine shows up on Amazon, and just a slightly bigger one when it shows up on a shelf (which will happen shortly after VS 2010 hits the streets). Nothing like that little surge of energy to give you the boost you need to cross the finish line. :-)

Here we go again—another year, another set of predictions revisited and offered up for the next 12 months. And maybe, if I'm feeling really ambitious, I'll take that shot I thought about last year and try predicting for the decade. Without further ado, I'll go back and revisit, unedited, my predictions for 2009 ("THEN"), and pontificate on those subjects for 2010 before adding any new material/topics. Just for convenience, here's a link back to last years' predictions.

Last year's predictions went something like this (complete with basketball-scoring):

• THEN: "Cloud" will become the next "ESB" or "SOA", in that it will be something that everybody will talk about, but few will understand and even fewer will do anything with. (Considering the widespread disparity in the definition of the term, this seems like a no-brainer.) NOW: Oh, yeah. Straight up. I get two points for this one. Does anyone have a working definition of "cloud" that applies to all of the major vendors' implementations? Ted, 2; Wrongness, 0.
• THEN: Interest in Scala will continue to rise, as will the number of detractors who point out that Scala is too hard to learn. NOW: Two points for this one, too. Not a hard one, mind you, but one of those "pass-and-shoot" jumpers from twelve feet out. James Strachan even tweeted about this earlier today, pointing out this comparison. As more Java developers who think of themselves as smart people try to pick up Scala and fail, the numbers of sour grapes responses like "Scala's too complex, and who needs that functional stuff anyway?" will continue to rise in 2010. Ted, 4; Wrongness, 0.
• THEN: Interest in F# will continue to rise, as will the number of detractors who point out that F# is too hard to learn. (Hey, the two really are cousins, and the fortunes of one will serve as a pretty good indication of the fortunes of the other, and both really seem to be on the same arc right now.) NOW: Interestingly enough, I haven't heard as many F# detractors as Scala detractors, possibly because I think F# hasn't really reached the masses of .NET developers the way that Scala has managed to find its way in front of Java developers. I think that'll change mighty quickly in 2010, though, once VS 2010 hits the streets. Ted, 4; Wrongness 2.
• THEN : Interest in all kinds of functional languages will continue to rise, and more than one person will take a hint from Bob "crazybob" Lee and liken functional programming to AOP, for good and for ill. People who took classes on Haskell in college will find themselves reaching for their old college textbooks again. NOW: Yep, I'm claiming two points on this one, if only because a bunch of Haskell books shipped this year, and they'll be the last to do so for about five years after this. (By the way, does anybody still remember aspects?) But I'm going the opposite way with this one now; yes, there's Haskell, and yes, there's Erlang, and yes, there's a lot of other functional languages out there, but who cares? They're hard to learn, they don't always translate well to other languages, and developers want languages that work on the platform they use on a daily basis, and that means F# and Scala or Clojure, or its simply not an option. Ted 6; Wrongness 2.
• THEN : The iPhone is going to be hailed as "the enterprise development platform of the future", and companies will be rolling out apps to it. Look for Quicken iPhone edition, PowerPoint and/or Keynote iPhone edition, along with connectors to hook the iPhone up to a presentation device, and (I'll bet) a World of Warcraft iPhone client (legit or otherwise). iPhone is the new hotness in the mobile space, and people will flock to it madly. NOW: Two more points, but let's be honest—this was a fast-break layup, no work required on my part. Ted 8; Wrongness 2.
• THEN: Another Oslo CTP will come out, and it will bear only a superficial resemblance to the one that came out in October at PDC. Betting on Oslo right now is a fools' bet, not because of any inherent weakness in the technology, but just because it's way too early in the cycle to be thinking about for anything vaguely resembling production code. NOW: If you've worked at all with Oslo, you might argue with me, but I'm still taking my two points. The two CTPs were pretty different in a number of ways. Ted 10; Wrongness 2.
• THEN: The IronPython and IronRuby teams will find some serious versioning issues as they try to manage the DLR versioning story between themselves and the CLR as a whole. An initial hack will result, which will be codified into a standard practice when .NET 4.0 ships. Then the next release of IPy or IRb will have to try and slip around its restrictions in 2010/2011. By 2012, IPy and IRb will have to be shipping as part of Visual Studio just to put the releases back into lockstep with one another (and the rest of the .NET universe). NOW: Pressure is still building. Let's see what happens by the time VS 2010 ships, and then see what the IPy/IRb teams start to do to adjust to the versioning issues that arise. Ted 8; Wrongness 2.
• THEN: The death of JSR-277 will spark an uprising among the two leading groups hoping to foist it off on the Java community--OSGi and Maven--while the rest of the Java world will breathe a huge sigh of relief and look to see what "modularity" means in Java 7. Some of the alpha geeks in Java will start using--if not building--JDK 7 builds just to get a heads-up on its impact, and be quietly surprised and, I dare say, perhaps even pleased. NOW: Ah, Ted, you really should never underestimate the community's willingness to take a bad idea, strip all the goodness out of it, and then cycle it back into the mix as something completely different yet somehow just as dangerous and crazy. I give you Project Jigsaw. Ted 10; Wrongness 2;
• THEN: The invokedynamic JSR will leapfrog in importance to the top of the list. NOW: The invokedynamic JSR begat interest in other languages on the JVM. The interest in other languages on the JVM begat the need to start thinking about how to support them in the Java libraries. The need to start thinking about supporting those languages begat a "Holy sh*t moment" somewhere inside Sun and led them to (re-)propose closures for JDK 7. And in local sports news, Ted notched up two more points on the scoreboard. Ted 12; Wrongness 2.
• THEN: Another Windows 7 CTP will come out, and it will spawn huge media interest that will eventually be remembered as Microsoft promises, that will eventually be remembered as Microsoft guarantees, that will eventually be remembered as Microsoft FUD and "promising much, delivering little". Microsoft ain't always at fault for the inflated expectations people have--sometimes, yes, perhaps even a lot of times, but not always. NOW: And then, just when the game started to turn into a runaway, airballs started to fly. The Windows7 release shipped, and contrary to what I expected, the general response to it was pretty warm. Yes, there were a few issues that emerged, but overall the media liked it, the masses liked it, and Microsoft seemed to have dodged a bullet. Ted 12; Wrongness 5.
• THEN: Apple will begin to legally threaten the clone market again, except this time somebody's going to get the DOJ involved. (Yes, this is the iPhone/iTunes prediction from last year, carrying over. I still expect this to happen.) NOW: What clones? The only people trying to clone Macs are those who are building Hackintosh machines, and Apple can't sue them so long as they're using licensed copies of Mac OS X (as far as I know). Which has never stopped them from trying, mind you, and I still think Steve has some part of his brain whispering to him at night, calculating all the hardware sales lost to Hackintosh netbooks out there. But in any event, that's another shot missed. Ted 12; Wrongness 7.
• THEN: Alpha-geek developers will start creating their own languages (even if they're obscure or bizarre ones like Shakespeare or Ook#) just to have that listed on their resume as the DSL/custom language buzz continues to build. NOW: I give you Ioke. If I'd extended this to include outdated CPU interpreters, I'd have made that three-pointer from half-court instead of just the top of the key. Ted 14; Wrongness 7.
• THEN: Roy Fielding will officially disown most of the "REST"ful authors and software packages available. Nobody will care--or worse, somebody looking to make a name for themselves will proclaim that Roy "doesn't really understand REST". And they'll be right--Roy doesn't understand what they consider to be REST, and the fact that he created the term will be of no importance anymore. Being "REST"ful will equate to "I did it myself!", complete with expectations of a gold star and a lollipop. NOW: Does anybody in the REST community care what Roy Fielding wrote way back when? I keep seeing "REST"ful systems that seem to have designers who've never heard of Roy, or his thesis. Roy hasn't officially disowned them, but damn if he doesn't seem close to it. Still.... No points. Ted 14; Wrongness 9.
• THEN: The Parrot guys will make at least one more minor point release. Nobody will notice or care, except for a few doggedly stubborn Perl hackers. They will find themselves having nightmares of previous lives carrying around OS/2 books and Amiga paraphernalia. Perl 6 will celebrate it's seventh... or is it eighth?... anniversary of being announced, and nobody will notice. NOW: Does anybody still follow Perl 6 development? Has the spec even been written yet? Google on "Perl 6 release", and you get varying reports: "It'll ship 'when it's ready'", "There are no such dates because this isn't a commericially-backed effort", and "Spring 2010". Swish—nothin' but net. Ted 16; Wrongness 9.
• THEN: The debate around "Scrum Certification" will rise to a fever pitch as short-sighted money-tight companies start looking for reasons to cut costs and either buy into agile at a superficial level and watch it fail, or start looking to cut the agilists from their company in order to replace them with cheaper labor. NOW: Agile has become another adjective meaning "best practices", and as such, has essentially lost its meaning. Just ask Scott Bellware. Ted 18; Wrongness 9.
• THEN: Adobe will continue to make Flex and AIR look more like C# and the CLR even as Microsoft tries to make Silverlight look more like Flash and AIR. Web designers will now get to experience the same fun that back-end web developers have enjoyed for near-on a decade, as shops begin to artificially partition themselves up as either "Flash" shops or "Silverlight" shops. NOW: Not sure how to score this one—I haven't seen the explicit partitioning happen yet, but the two environments definitely still seem to be looking to start tromping on each others' turf, particularly when we look at the rapid releases coming from the Silverlight team. Ted 16; Wrongness 11.
• THEN: Gartner will still come knocking, looking to hire me for outrageous sums of money to do nothing but blog and wax prophetic. NOW: Still no job offers. Damn. Ah, well. Ted 16; Wrongness 13.

A close game. Could've gone either way. *shrug* Ah, well. It was silly to try and score it in basketball metaphor, anyway—that's the last time I watch ESPN before writing this.

For 2010, I predict....

• ... I will offer 3- and 4-day training classes on F# and Scala, among other things. OK, that's not fair—yes, I have the materials, I just need to work out locations and times. Contact me if you're interested in a private class, by the way.
• ... I will publish two books, one on F# and one on Scala. OK, OK, another plug. Or, rather, more of a resolution. One will be the "Professional F#" I'm doing for Wiley/Wrox, the other isn't yet finalized. But it'll either be published through a publisher, or self-published, by JavaOne 2010.
• ... DSLs will either "succeed" this year, or begin the short slide into the dustbin of obscure programming ideas. Domain-specific language advocates have to put up some kind of strawman for developers to learn from and poke at, or the whole concept will just fade away. Martin's book will help, if it ships this year, but even that might not be enough to generate interest if it doesn't have some kind of large-scale applicability in it. Patterns and refactoring and enterprise containers all had a huge advantage in that developers could see pretty easily what the problem was they solved; DSLs haven't made that clear yet.
• ... functional languages will start to see a backlash. I hate to say it, but "getting" the functional mindset is hard, and there's precious few resources that are making it easy for mainstream (read: O-O) developers make that adjustment, far fewer than there was during the procedural-to-object shift. If the functional community doesn't want to become mainstream, then mainstream developers will find ways to take functional's most compelling gateway use-case (parallel/concurrent programming) and find a way to "git 'er done" in the traditional O-O approach, probably through software transactional memory, and functional languages like Haskell and Erlang will be relegated to the "What Might Have Been" of computer science history. Not sure what I mean? Try this: walk into a functional language forum, and ask what a monad is. Nobody yet has been able to produce an answer that doesn't involve math theory, or that does involve a practical domain-object-based example. In fact, nobody has really said why (or if) monads are even still useful. Or catamorphisms. Or any of the other dime-store words that the functional community likes to toss around.
• ... Visual Studio 2010 will ship on time, and be one of the buggiest and/or slowest releases in its history. I hate to make this prediction, because I really don't want to be right, but there's just so much happening in the Visual Studio refactoring effort that it makes me incredibly nervous. Widespread adoption of VS2010 will wait until SP1 at the earliest. In fact....
• ... Visual Studio 2010 SP 1 will ship within three months of the final product. Microsoft knows that people wait until SP 1 to think about upgrading, so they'll just plan for an eager SP 1 release, and hope that managers will be too hung over from the New Year (still) to notice that the necessary shakeout time hasn't happened.
• ... Apple will ship a tablet with multi-touch on it, and it will flop horribly. Not sure why I think this, but I just don't think the multi-touch paradigm that Apple has cooked up for the iPhone will carry over to a tablet/laptop device. That won't stop them from shipping it, and it won't stop Apple fan-boiz from buying it, but that's about where the interest will end.
• ... JDK 7 closures will be debated for a few weeks, then become a fait accompli as the Java community shrugs its collective shoulders. Frankly, I think the Java community has exhausted its interest in debating new language features for Java. Recent college grads and open-source groups with an axe to grind will continue to try and make an issue out of this, but I think the overall Java community just... doesn't... care. They just want to see JDK 7 ship someday.
• ... Scala either "pops" in 2010, or begins to fall apart. By "pops", I mean reaches a critical mass of developers interested in using it, enough to convince somebody to create a company around it, a la G2One.
• ... Oracle is going to make a serious "cloud" play, probably by offering an Oracle-hosted version of Azure or AppEngine. Oracle loves the enterprise space too much, and derives too much money from it, to not at least appear to have some kind of offering here. Now that they own Java, they'll marry it up against OpenSolaris, the Oracle database, and throw the whole thing into a series of server centers all over the continent, and call it "Oracle 12c" (c for Cloud, of course) or something.
• ... Spring development will slow to a crawl and start to take a left turn toward cloud ideas. VMWare bought SpringSource for a reason, and I believe it's entirely centered around VMWare's movement into the cloud space—they want to be more than "just" a virtualization tool. Spring + Groovy makes a compelling development stack, particularly if VMWare does some interesting hooks-n-hacks to make Spring a virtualization environment in its own right somehow. But from a practical perspective, any community-driven development against Spring is all but basically dead. The source may be downloadable later, like the VMWare Player code is, but making contributions back? Fuhgeddabowdit.
• ... the explosion of e-book readers brings the Kindle 2009 edition way down to size. The era of the e-book reader is here, and honestly, while I'm glad I have a Kindle, I'm expecting that I'll be dusting it off a shelf in a few years. Kinda like I do with my iPods from a few years ago.
• ... "social networking" becomes the "Web 2.0" of 2010. In other words, using the term will basically identify you as a tech wannabe and clearly out of touch with the bleeding edge.
• ... Facebook becomes a developer platform requirement. I don't pretend to know anything about Facebook—I'm not even on it, which amazes my family to no end—but clearly Facebook is one of those mechanisms by which people reach each other, and before long, it'll start showing up as a developer requirement for companies looking to hire. If you're looking to build out your resume to make yourself attractive to companies in 2010, mad Facebook skillz might not be a bad investment.
• ... Nintendo releases an open SDK for building games for its next-gen DS-based device. With the spectacular success of games on the iPhone, Nintendo clearly must see that they're missing a huge opportunity every day developers can't write games for the Nintendo DS that are easily downloadable to the device for playing. Nintendo is not stupid—if they don't open up the SDK and promote "casual" games like those on the iPhone and those that can now be downloaded to the Zune or the XBox, they risk being marginalized out of existence.

And for the next decade, I predict....

• ... colleges and unversities will begin issuing e-book reader devices to students. It's a helluvalot cheaper than issuing laptops or netbooks, and besides....
• ... netbooks and e-book readers will merge before the decade is out. Let's be honest—if the e-book reader could do email and browse the web, you have almost the perfect paperback-sized mobile device. As for the credit-card sized mobile device....
• ... mobile phones will all but disappear as they turn into what PDAs tried to be. "The iPhone makes calls? Really? You mean Voice-over-IP, right? No, wait, over cell signal? It can do that? Wow, there's really an app for everything, isn't there?"
• ... wireless formats will skyrocket in importance all around the office and home. Combine the iPhone's Bluetooth (or something similar yet lower-power-consuming) with an equally-capable (Bluetooth or otherwise) projector, and suddenly many executives can leave their netbook or laptop at home for a business presentation. Throw in the Whispersync-aware e-book reader/netbook-thing, and now most executives have absolutely zero reason to carry anything but their e-book/netbook and their phone/PDA. The day somebody figures out an easy way to combine Bluetooth with PayPal on the iPhone or Android phone, we will have more or less made pocket change irrelevant. And believe me, that day will happen before the end of the decade.
• ... either Android or Windows Mobile will gain some serious market share against the iPhone the day they figure out how to support an open and unrestricted AppStore-like app acquisition model. Let's be honest, the attraction of iTunes and AppStore is that I can see an "Oh, cool!" app on a buddy's iPhone, and have it on mine less than 30 seconds later. If Android or WinMo can figure out how to offer that same kind of experience without the draconian AppStore policies to go with it, they'll start making up lost ground on iPhone in a hurry.
• ... Apple becomes the DOJ target of the decade. Microsoft was it in the 2000's, and Apple's stunning rising success is going to put it squarely in the sights of monopolist accusations before long. Coupled with the unfortunate health distractions that Steve Jobs has to deal with, Apple's going to get hammered pretty hard by the end of the decade, but it will have mastered enough market share and mindshare to weather it as Microsoft has.
• ... Google becomes the next Microsoft. It won't be anything the founders do, but Google will do "something evil", and it will be loudly and screechingly pointed out by all of Google's corporate opponents, and the star will have fallen.
• ... Microsoft finds its way again. Microsoft, as a company, has lost its way. This is a company that's not used to losing, and like Bill Belichick's Patriots, they will find ways to adapt and adjust to the changed circumstances of their position to find a way to win again. What that'll be, I have no idea, but historically, the last decade notwithstanding, betting against Microsoft has historically been a bad idea. My gut tells me they'll figure something new to get that mojo back.
• ... a politician will make himself or herself famous by standing up to the TSA. The scene will play out like this: during a Congressional hearing on airline security, after some nut/terrorist tries to blow up another plane through nitroglycerine-soaked underwear, the TSA director will suggest all passengers should fly naked in order to preserve safety, the congressman/woman will stare open-mouthed at this suggestion, proclaim, "Have you no sense of decency, sir?" and immediately get a standing ovation and never have to worry about re-election again. Folks, if we want to prevent any chance of loss of life from a terrorist act on an airplane, we have to prevent passengers from getting on them. Otherwise, just accept that it might happen, do a reasonable job of preventing it from happening, and let private insurance start offering flight insurance against the possibility to reassure the paranoid.

See you all next year.

Why study new and different programming languages? To change your programming mindset. Not sure what I mean by that? Check this out.

Ever done one of these?

               1:
              public
              interface IService

               2: {

               3: DateTime
GetDate();

               4:
              int CalculateSomethingInteresting(int lhs, int rhs);

               5: }

               6:  

               7:
              public
              class OneServiceImpl
: IService

               8: {

               9:
              public DateTime
GetDate()

               10: { return DateTime.Now;
}

               11:
              public
              int CalculateSomethingInteresting(int lhs, int rhs)

               12: { return lhs
+ rhs; }

               13: }

               14:  

               15:
              public
              class AnotherServiceImpl
: IService

               16: {

               17:
              public DateTime
GetDate()

               18: { returnnew DateTime();
}

               19:
              public
              int CalculateSomethingInteresting(int lhs, int rhs)

               20: { return lhs
* rhs; }

               21: }

               22:  

               23:
              public
              class ServiceFactory

               24: {

               25:
              public
              static IService
GetInstance(string which)

               26: {

               27:
              if (which
== "One") returnnew OneServiceImpl();

               28:
              else
              if (which
== "Another") returnnew AnotherServiceImpl();

               29:
              else
              throw
              new ArgumentException();

               30: }

               31: }

               32:  

               33:
              public
              class App

               34: {

               35:
              public
              static
              void Main(string[]
args)

               36: {

               37:
              foreach (string s in args)

               38: {

               39: IService
serv = ServiceFactory.GetInstance(s);

               40: Console.WriteLine("serv
calc = {0}", serv.CalculateSomethingInteresting(3, 3));

               41: }

               42: }

               43: }

So has my client this week. In fact, it's fair to say that they're infatuated with them—they've got services all over the place, including at their communication layer, where they use configuration files to decide which of the two service implementations to use, either a "native" .NET implementation or the "real" Web services implementation that they're supposed to be using. (They end up going back to the native implementation because sometimes—which is to say, apparently a lot of times—the Web services implementation is broken in some fundamental way. Go figure.)

The problem is, very bluntly, that the interfaces they're defining (the IService definition above) are ever-so-slightly different from the communications-based proxy interfaces that they use to communicate outside of this process, so some poor schmuck ends up having to write the service implementation (OneServiceImpl) that simply takes the parameters passed in, translates them into a call through the communications-based interface, then takes the response and hands it back. Tedious, mind-numbing coding, particularly painful when there are dozens of interfaces with (in some cases) hundreds of methods per interface. Ouch.

There had to be a better way.

Based on some of the work/research/play I've been doing with both dynamic and functional programming languages, it occurred to me that what they really wanted was some kind of "forwarding" or "delegating" behavior that certain languages have baked in as a feature. In those languages, it's possible to nominate a "delegate" object to which method calls are automatically forwarded if no such method is implemented on this object; in this particular case, what I'd do to replace all of the above is simply create an IService object instance that has either a OneServiceImpl or a AnotherServiceImpl instance (depending on the value in the configuration file) set up as the "delegate" object. That way the method calls remain statically type-checked, but none of this service interface/service implementation/service factory nonsense has to be created just to switch between the two.

(By the way, all of this pain goes away completely in a language that supports deferred checking of signatures until runtime. In other words, if the client had been programming in IronPython or IronRuby or even Visual Basic, we could get away with not having to do any of the above, and just use Reflection to access the appropriate method on whichever of the two service implementations they want to use at the time. Fan would let us do it if we used "->" instead of "." to invoke the method; Cobra would switch between the two automatically; and so on.)

Now, this is C# 2.0 that they're using, and they're pretty entrenched on that point, so I can't simply suggest that they use a new language, but if we take the basic idea and adapt it to C#, we can get pretty much the same behavior without having to force the poor schmuck on the bottom of the totem pole to write all those service implementations by hand.

We start by transforming the IService interface into an IService "interface" (meaning it's not really an interface anymore, but it'll sure look like one to anybody who's not paying attention):

               1:
              public
              class IService

               2: {

               3:
              public Func0<DateTime>
GetDate;

               4:
              public Func2<int, int, int>
CalculateSomethingInteresting;

               5: }

IService is now a class with fields (not properties, though I suppose if you really wanted them to be properties you could make them such, not that I see much value to doing so), where each field corresponds in name to the method of the interface it wants to replace, and the type is a delegate type parameterized to match the return type and parameter types of that same method of the original interface. Func0 and Func2 are delegate types I had to create, since nothing like them existed until C# 3.0; their definitions are pretty simple:

               1:
              public
              delegate R
Func0<R>();

               2:
              public
              delegate R
Func1<R, P1>(P1 p1);

               3:
              public
              delegate R
Func2<R, P1, P2>(P1 p1, P2 p2);

Now, assuming we have the implementation classes from before, we have two choices; one is to write a by-hand factory that fills out the fields to point to the appropriate method on the implementation class, like so:

               1:
              if (which
== "One")

               2: {

               3: servInstance.GetDate
= delegate() { return DateTime.Now;
};

               4: servInstance.CalculateSomethingInteresting
= delegate (int lhs, int rhs)
{ return lhs + rhs; };

               5: }

               6:
              else
              if (which
== "Another")

               7: {

               8: servInstance.GetDate
= delegate() { returnnew DateTime();
};

               9: servInstance.CalculateSomethingInteresting
= delegate (int lhs, int rhs)
{ return lhs * rhs; };

               10: }

               11:
              else
            

               12:
              throw
              new ArgumentException();

But, quite frankly, this defeats the point—the point was to avoid writing all this stuff by hand, not simply repeat it in a different form. So instead, we leverage Reflection, which depends on the basic assumption that the field name in the IService "interface" matches the method name on the implementation class we wish to invoke. Assuming that holds (which it does, in my client's case, anyway), we can reflect on the IService field, find the matching method name in the implementation, then construct a delegate instance around that method and assign the delegate instance to the field. Once complete, we hand back the completed service instance, and the client literally doesn't know that anything's different:

               1:
              public
              class ServiceFactory

               2: {

               3:
              public
              static IService
GetInstance(string which)

               4: {

               5: IService
servInstance = new IService();

               6:  

               7: Type
targetType = Assembly.GetExecutingAssembly().GetType(which + "ServiceImpl");

               8:  

               9:
              foreach (FieldInfo
fi in servInstance.GetType().GetFields())

               10: {

               11: MethodInfo
targetMethod = targetType.GetMethod(fi.Name);

               12:
              //Console.WriteLine("Wiring
up {0} against {1} with {2}", fi.Name, targetType, targetMethod);
            

               13: Delegate
d = Delegate.CreateDelegate(fi.FieldType, null,
targetMethod);

               14:
              //Console.WriteLine(d);
            

               15: fi.SetValue(servInstance,
d);

               16: }

               17:  

               18:
              return servInstance;

               19: }

               20: }

Remember, the client code still looks the same...

               1:
              public
              class App

               2: {

               3:
              public
              static
              void Main(string[]
args)

               4: {

               5:
              foreach (string s in args)

               6: {

               7: IService
serv = ServiceFactory.GetInstance(s);

               8: Console.WriteLine("serv
calc = {0}", serv.CalculateSomethingInteresting(3, 3));

               9: }

               10: }

               11: }

... because what the client doesn't know is that he's accessing a field, then invoking the delegate that's being returned from that field dereference.

What this permits, aside from the automated wiring up of the IService "interface", is a greater degree of flexibility—rather than having to choose which implementation to use on an interface-by-interface basis, we can now configure to use different implementations on a method-by-method basis. But considering how many interfaces and implementations my client was looking at having to write by hand, the real win is in the automated ServiceFactory wiring.

By the way, the only reason we can get away with this sleight-of-hand is because delegates are deliberately designed to act like method calls; no explicit .Invoke() call is required, it's implied with the () after the delegate instance's name. If Java7 closures and/or method handles end up with support for that kind of syntax, then we can do the same thing in Java7 (more or less).

Make sense?

Paul asked me to review this, his first book, and my comment to him was that he had a pretty high bar to match; being of the same "series" as Release It!, Mike Nygard's take on building software ready for production (and, in my repeatedly stated opinion, the most important-to-read book of the decade), Debug It! had some pretty impressive shoes to fill. Paul's comment was pretty predictable: "Thanks for keeping the pressure to a minimum."

My copy arrived in the mail while I was at the NFJS show in Denver this past weekend, and with a certain amount of dread and excitement, I opened the envelope and sat down to read for a few minutes. I managed to get halfway through it before deciding I had to post a review before I get too caught up in my next trip and forget.

Short version

Debug It! is a great resource for anyone looking to learn the science of good debugging. It is entirely language- and platform-agnostic, preferring to focus entirely on the process and mindset of debugging, rather than on edge cases or command-line switches in a tool or language. Overall, the writing is clear and straightforward without being preachy or judgmental, and is liberally annotated with real-life case stories from both the authors' and the Pragmatic Programmers' own history, which keeps the tone lighter and yet still proving the point of the text. Highly recommended for the junior developers on the team; senior developers will likely find some good tidbits in here as well.

Long version

Debug It! is an excellently-written and to-the-point description of the process of not only identifying and fixing defects in software, but also of the attitudes required to keep software from failing. Rather than simply tossing off old maxims or warming them over with new terminology ("You should always verify the parameters to your procedure calls" replaced with "You should always verify the parameters entering a method and ensure the fields follow the invariants established in the specification"), Paul ensures that when making a point, his prose is clear, the rationale carefully explained, and the consequences of not following this advice are clearly spelled out. His advice is pragmatic, and takes into account that developers can't always follow the absolute rules we'd like to—he talks about some of his experiences with "bug priorities" and how users pretty quickly figured out to always set the bug's priority at the highest level in order to get developer attention, for example, and some ways to try and address that all-too-human failing of bug-tracking systems.

It needs to be said, right from the beginning, that Debug It! will not teach you how to use the debugging features of your favorite IDE, however. This is because Paul (deliberately, it seems) takes a platform- and language-agnostic approach to the book—there are no examples of how to set breakpoints in gdb, or how to attach the Visual Studio IDE to a running Windows service, for example. This will likely weed out those readers who are looking for "Google-able" answers to their common debugging problems, and that's a shame, because those are probably the very readers that need to read this book. Having said that, however, I like this agnostic approach, because these ideas and thought processes, the ones that are entirely independent of the language or platform, are exactly the kinds of things that senior developers carry over with them from one platform to the next. Still, the junior developer who picks this book up is going to still need a reference manual or the user manual for their IDE or toolchain, and will need to practice some with both books in hand if they want to maximize the effectiveness of what's in here.

One of the things I like most about this book is that it is liberally adorned with real-life discussions of various scenarios the author team has experienced; the reason I say "author team" here is because although the stories (for the most part) remain unattributed, there are obvious references to "Dave" and "Andy", which I assume pretty obviously refer to Dave Thomas and Andy Hunt, the Pragmatic Programmers and the owners of Pragmatic Bookshelf. Some of the stories are humorous, and some of them probably would be humorous if they didn't strike so close to my own bitterly-remembered experiences. All of them do a good job of reinforcing the point, however, thus rendering the prose more effective in communicating the idea without getting to be too preachy or bombastic.

The book obviously intends to target a junior developer audience, because most senior developers have already intuitively (or experientially) figured out many of the processes described in here. But, quite frankly, I think it would be a shame for senior developers to pass on this one; though the temptation will be to simply toss it aside and say, "I already do all this stuff", senior developers should resist that urge and read it through cover to cover. If nothing else, it'll help reinforce certain ideas, bring some of the intuitive process more to light and allow us to analyze what we do right and what we do wrong, and perhaps most importantly, give us a common backdrop against which we can mentor junior developers in the science of debugging.

One of the chapters I like in particular, "Chapter 7: Pragmatic Zero Tolerance", is particularly good reading for those shops that currently suffer from a deficit of management support for writing good software. In it, Paul talks specifically about some of the triage process about bugs ("When to fix bugs"), the mental approach developers should have to fixing bugs ("The debugging mind-set") and how to get started on creating good software out of bad ("How to dig yourself out of a quality hole"). These are techniques that a senior developer can bring to the team and implement at a grass-roots level, in many cases without management even being aware of what's going on. (It's a sad state of affairs that we sometimes have to work behind management's back to write good-quality code, but I know that some developers out there are in exactly that situation, and simply saying, "Quit and find a new job", although pithy and good for a laugh on a panel, doesn't really offer much in the way of help. Paul doesn't take that route here, and that alone makes this book worth reading.)

Another of the chapters that resonates well with me is the first one in Part III ("Debug Fu"), Chapter 8, entitled "Special Cases", in which he tackles a number of "advanced" debugging topics, such as "Patching Existing Releases" and "Hesenbugs" (Concurrency-related bugs). I won't spoil the punchline for you, but suffice it to say that I wish I'd had that chapter on hand to give out to teammates on a few projects I've worked on in the past.

Overall, this book is going to be a huge win, and I think it's a worthy successor to the Release It! reputation. Development managers and team leads should get a copy for the junior developers on their team as a Christmas gift, but only after the senior developers have read through it as well. (Senior devs, don't despair—at 190 pages, you can rip through this in a single night, and I can almost guarantee that you'll learn a few ideas you can put into practice the next morning to boot.)

Those of you who've seen me speak on Java 7 at various conferences have heard me lament (in a small way) the fact that Sun decided last year (Dec 2008) to forgo the idea of including closures in the Java language. Imagine my surprise, then, to check my Twitter feed and discover that, to everyone's surprise, closures are back in as a consideration for the Java7 release.

Several thoughts come to mind:

• "WTF?!?!? This is a community effort?" Originally, when Sun created the Java Community Process, the tradeoff for a committee-based development process was against the open and fair inclusion of ideas from outside of Sun. But with the Java7 release still lacking a JSR (as of a few weeks ago, anyway; I haven't checked today to see if it was opened), and both the Modules facility and language extensions deferred to "Projects" (not JSRs), it seems Sun is now abandoning the JCP in favor of a Sun-dominant process that is certainly solicitous of the community at large, but not constrained or defined by it. And for the life of me, I can't tell if this is a good thing or a bad thing. It's good in that now we don't have to garner a critical mass of community momentum to get something included into the platform or language, but it's bad in that Sun has historically been the bigger drag on innovation there, not the community.
• "Can we please stop calling them closures?" This is a nit, but technically what we're talking about adding here are either lambda expressions or anonymous methods, depending on whose glossary you're using when you're talking. A true closure is one that will compute all referenced variables from the enclosing scope and automatically include them in the generated code, which (so far as I can tell) none of the Java anonymous method or lambda expression proposals currently include. But it's a nit, so I'll say it this once and then drop it.
• "Will Groovy, Scala, Clojure and all other JVM languages please report to the refactoring room?" People look at me quizzically when I say I'd like to see Java have closures in the language, because in general my take on language features in Java is that the Java language is more or less dead, and I could care less what happens to it; I'd vastly prefer to code in Groovy or Scala or Clojure or JRuby before writing something in Java. My rationale for wanting closures in Java, however, is this: by defining a common implementation for closures in Java, all of the above languages can refactor their implementations of anonymous methods/lambda expressions/etc into something that uses Java's closure implementation, and that'll make calling Groovy anonymous methods from Scala much much easier.
• "Why there, now?" Devoxx is apparently turning into JavaOne Winter, because Sun's been making a lot of pretty big announcements at that show, including last year's "no closures, no built-in XML support, ..." announcement about Java7, and now this year's "well, we lied, we're thinking about closures again". Fortunately I think the Devoxx folks have much better skills at keeping their conference relevant to the Java community than JavaOne's organizers did. And I say that despite the fact (or perhaps because of the fact) that I didn't speak there this year. ;-)
• "When is this all supposed to ship again?" Originally, my understanding was that JDK7 was slated to ship in the early part of 2010, but now rumor has it slipping to this time next year (2010). That is a huge postponement, and gives Microsoft a bit of an edge, since Visual Studio 2010 and .NET 4.0 are (again, according to rumor) supposed to ship somewhere around the end of 1Q2010. If Sun/Oracle keeps this up, we could very well be seeing a 2-.NET-releases-to-1-Java-release pattern, and that's disturbing in its own right. (Anybody else remember the days when Sun withdrew Java from ECMA, ISO and ANSI standardization consideration because they wanted to "innovate on the platform faster"?)
• "We really have no clue what we're talking about." Aside from rumors and hearsay (including the one that says that Mark Reinhold, who made the announcement, made up the syntax on the flight from the US to Belgium), we really don't have much by way of Sun-blessed official discussions of what this will look like or act like, at least none so far as I've been able to find, so any sort of supposition on whether it will be good or suck like an inverted hurricane is a tad premature. Trust me, I want to see where this goes, too, so I'll be keeping an eye out.

In the meantime, if you want to keep on top of the Java space, maybe it's time to consider a trip to Antwerp this time next year, since, if the new ship date rumors are to be believed,  it looks like Sun (once again) is planning to use Devoxx as the platform from which to make a large announcement, this time the release Java7 itself.

Update: Ola Bini noted that...

Two things:

• They are definitely closures. Calling them anonymous functions are incorrect, since they aren't really functions. Lambda expression is an OK name, but it has connotations that aren't really correct for a language like Java. A closure is defined as an anonymous piece of code that closes over at least one free variable, which in the case of this proposal will definitely happen. In fact, all of these will be closures, since they will be closing over the this at least.
• This is mostly on the level of compiler, syntax and type checking, and will NOT have any real implications for runtime. This means there will be no real sharing of implementation - at most JRuby, Groovy and Scala blocks will implement another interface (but all of them already implement Runnable and Callable so it's a limited win).

which prompted me to respond thusly:

First off, I actually never used the term "anonymous function"; instead, I said "anonymous method", which, as I understand it, is how the underlying implementation of these proposals will work: the syntax "#() return 42" will create an anonymous inner class instance of an interface defined by the library (in its "SimpleClosure" example, the BGGA compiler uses the interface "javax.lang.function.I", which has one method on it, "invoke()"), which, thus, makes this an anonymous method. We can't call them "anonymous functions" because Java has no function type, and probably never will. (And yes, it may seem like we're splitting hairs somewhat to differentiate between functions and methods,but once you've explored ML, Haskell, Scala, or F#, you really begin to see a huge difference in those terms, so it's important to be precise with our terminology, or else the conversation becomes almost entirely meaningless.)

Neal Gafter uses the definition "A closure is a function that captures the bindings of free variables in its lexical context." (http://gafter.blogspot.com/2007/01/definition-of-closures.html) Given that said same post also claims that Java has no function type (and therefore, by his definition, can't really have a closure), I suppose we could split the hairs even further and suggest that Java will never have closures until it has true function types. Personally, I'm happy to say that we can swap in "methods" for "functions" in this particular discussion, but my understanding is that capturing free variables also implies capturing variables referenced in the enclosing lexical context, which the current "closures" proposal (as reported by Alex Miller's closures page) will not do. (Non-final enclosing parameters will not be accessible, only those passed in formally as parameters. Stephen Colebourne reports as much: "[Mark Reinhold] also indicated that access to non-final variables was unlikely.")

Given that the current proposal suggests the new #() syntax will essentially generate an anonymous inner class with a method of the appropriate signature (though I do believe that method handles are targeted for use at some point, based on what I've been hearing through the rumor mill), to me it feels like the "closures" implementation is generating an anonymous method of an anonymous class with a few other restrictions included--hence my commentary above.

(Having said all that, the FCM proposal does provide complete capture of all referenced variables in enclosing scope, but Mark's keynote hasn't officially endorsed either the BGGA proposal or the FCM proposal, and if Sun keeps to their habits, they won't. They'll build something that's an amalgamation of all of them. Right now the current consensus seems to be to adopt the BGGA implementation behind the FCM syntax, which jives with Neal's 0.6a specification proposal.)

On top of that, the comment "all of these will be closures, since they will be closing over the this at least" is not, I don't think, entirely true. The details of the closures proposal aren't clear, but the "outer this" (which I believe is the "this" Ola refers to above) hasn't been explicitly mentioned in any of the closures proposals I've seen, nor have I seen any text suggesting that they will honor it, so I don't know that this is true. Of course, in absence of a specification or real working bits, all we can do is just speculate. However, having said that, playing around a bit with the BGGA prototype compiler (which, admittedly, is still one minor rev back from Neal's revised proposal), I saw no generated "outer this" in the generated code for the generated inner class implementation of the closure. If the comment above is meant to refer to the "this" of the inner class instance, then that would make all methods of an object-oriented language that provided an implicit "this" a closure, and somehow I doubt that's what Ola means, though I could, as always, be wrong.

As for the runtime implementation, as I said earlier I believe the plan is to use method handles (already on the table for JDK 7), which do have some runtime implications (generally good ones, from what I can tell so far), but not beyond what was already on the table for 7.

Phil Haack wrote a thoughtful, insightful and absolutely correct response to my earlier blog post. But he's still missing the point.

The short version: Phil's right when he says, "Agile is less about managing the complexity of an application itself and more about managing the complexity of building an application." Agile is by far the best approach to take when building complex software.

But that's not where I'm going with this.

As a starting point in the discussion, I'd like to call attention to one of Phil's sidebars: I find it curious (and indicative of the larger point) his earlier comment about "I have to wonder, why is that little school district in western Pennsylvania engaging in custom software development in the first place?" At what point does standing a small Access database up qualify as "custom software development"? And I take huge issue with Phil's comment immediately thereafter: "" That's totally untrue, Phil—you are, in fact, creating custom educational curricula, for your children at home. Not for popular usage, not for commercial use, but clearly you're educating your children at home, because you'd be a pretty crappy parent if you didn't. You also practice an informal form of medicine ("Let me kiss the boo-boo"), psychology ("Now, come on, share the truck"), culinary arts ("Would you like mac and cheese tonight?"), acting ("Aaar! I'm the Tickle Monster!") and a vastly larger array of "professional" skills that any of the "professionals" will do vastly better than you.

In other words, you're not a professional actor/chef/shrink/doctor, you're an amateur one, and you want tools that let you practice your amateur "professions" as you wish, without requiring the skills and trappings (and overhead) of a professional in the same arena.

Consider this, Phil: your child decides it's time to have a puppy. (We all know the kids are the ones who make these choices, not us, right?) So, being the conscientious parent that you are, you decide to build a doghouse for the new puppy to use to sleep outdoors (forgetting, as all parents do, that the puppy will actually end up sleeping in the bed with your child, but that's another discussion for another day). So immediately you head on down to Home Depot, grab some lumber, some nails, maybe a hammer and a screwdriver, some paint, and head on home.

Whoa, there, turbo. Aren't you forgetting a few things? For starters, you need to get the concrete for the foundation, rebar to support the concrete in the event of a bad earthquake, drywall, fire extinguishers, sirens for the emergency exit doors... And of course, you'll need a foreman to coordinate all the work, to make sure the foundation is poured before the carpenters show up to put up the trusses, which in turn has to happen before the drywall can go up...

We in this industry have a jealous and irrational attitude towards the amateur software developer. This was even apparent in the Twitter comments that accompanied the conversation around my blog post: "@tedneward treating the disease would mean... have the client have all their ideas correct from the start" (from @kelps). In other words, "bad client! No biscuit!"?

Why is it that we, IT professionals, consider anything that involves doing something other than simply putting content into an application to be "custom software development"? Why can't end-users create tools of their own to solve their own problems at a scale appropriate to their local problem?

Phil offers a few examples of why end-users creating their own tools is a Bad Idea:

I remember one rescue operation for a company drowning in the complexity of a “simple” Access application they used to run their business. It was simple until they started adding new business processes they needed to track. It was simple until they started emailing copies around and were unsure which was the “master copy”. Not to mention all the data integrity issues and difficulty in changing the monolithic procedural application code.

I also remember helping a teachers union who started off with a simple attendance tracker style app (to use an example Ted mentions) and just scaled it up to an atrociously complex Access database with stranded data and manual processes where they printed excel spreadsheets to paper, then manually entered it into another application.

And you know what?

This is not a bad state of affairs.

Oh, of course, we, the IT professionals, will immediately pounce on all the things wrong with their attempts to extend the once-simple application/solution in ways beyond its capabilities, and we will scoff at their solutions, but you know what? That just speaks to our insecurities, not the effort expended. You think Wolfgang Puck isn't going to throw back his head and roar at my lame attempts at culinary experimentation? You think Frank Lloyd Wright wouldn't cringe in horror at my cobbled-together doghouse? And I'll bet Maya Angelou will be so shocked at the ugliness of my poetry that she'll post it somewhere on the "So You Think You're A Poet" website.

Does that mean I need to abandon my efforts to all of these things?

The agilists' community reaction to my post would seem to imply so. "If you aren't a professional, don't even attempt this?" Really? Is that the message we're preaching these days?

End users have just as much a desire and right to be amateur software developers as we do at being amateur cooks, photographers, poets, construction foremen, and musicians. And what do you do when you want to add an addition to your house instead of just building a doghouse? Or when you want to cook for several hundred people instead of just your family?

You hire a professional, and let them do the project professionally.

The above quote was tossed off by Billy Hollis at the patterns&practices Summit this week in Redmond. I passed the quote out to the Twitter masses, along with my +1, and predictably, the comments started coming in shortly thereafter. Rather than limit the thoughts to the 120 or so characters that Twitter limits us to, I thought this subject deserved some greater expansion.

But before I do, let me try (badly) to paraphrase the lightning talk that Billy gave here, which sets context for the discussion:

• Keeping track of all the stuff Microsoft is releasing is hard work: LINQ, EF, Silverlight, ASP.NET MVC, Enterprise Library, Azure, Prism, Sparkle, MEF, WCF, WF, WPF, InfoCard, CardSpace, the list goes on and on, and frankly, nobody (and I mean nobody) can track it all.
• Microsoft released all this stuff because they were chasing the "enterprise" part of the developer/business curve, as opposed to the "long tail" part of the curve that they used to chase down. They did this because they believed that this was good business practice—like banks, "enterprises are where the money is". (If you're not familiar with this curve, imagine a graph with a single curve asymptotically reaching for both axes, where Y is the number of developers on the project, and X is the number of projects. What you get is a curve of a few high-developer-population projects on the left, to a large number of projects with just 1 or 2 developers. This right-hand portion of the curve is known as "the long tail" of the software industry.)
• A lot of software written back in the 90's was written by 1 or 2 guys working for just a few months to slam something out and see if it was useful. What chances do those kinds of projects have today? What tools would you use to build them?
• The problem is the complexity of the tools we have available to us today preclude that kind of software development.
• Agile doesn't solve this problem—the agile movement suggests that we have to create story cards, we have to build unit tests, we have to have a continuous integration server, we have to have standup meetings every day, .... In short, particularly among the agile evangelists (by which we really mean zealots), if you aren't doing a full agile process, you are simply failing. (If this is true, how on earth did all those thousands of applications written in FoxPro or Access ever manage to succeed? –-Me) At one point, an agilist said point-blank, "If you don't do agile, what happens when your project reaches a thousand users?" As Billy put it, "Think about that for a second: This agile guy is threatening us with success."
• Agile is for managing complexity. What we need is to recognize that there is a place for outright simplicity instead.

By the way, let me say this out loud: if you have not heard Billy Hollis speak, you should. Even if you're a Java or Ruby developer, you should listen to what he has to say. He's been developing software for a long time, has seen a lot of these technology-industry trends come and go, and even if you disagree with him, you need to listen to him.

Let me rephrase Billy's talk this way:

Where is this decade's Access?

It may seem like a snarky and trolling question, but think about it for a moment: for a decade or so, I was brought into project after project that was designed to essentially rebuild/rearchitect the Access database created by one of the department's more tech-savvy employees into something that could scale beyond just the department.

(Actually, in about half of them, the goal wasn't even to scale it up, it was just to put it on the web. It was only in the subsequent meetings and discussions that the issues of scale came up, and if my memory is accurate, I was the one who raised those issues, not the customer. I wonder now, looking back at it, if that was pure gold-plating on my part.)

Others, including many people I care about (Rod Paddock, Markus Eggers, Ken Levy, Cathi Gero, for starters) made a healthy living off of building "line of business" applications in FoxPro, which Microsoft has now officially shut down. For those who did Office applications, Visual Basic for Applications has now been officially deprecated in favor of VSTO (Visual Studio Tools for Office), a set of libraries that are available for use by any .NET application language, and of course classic Visual Basic itself has been "brought into the fold" by making it a fully-fledged object-oriented language complete with XML literals and LINQ query capabilities.

Which means, if somebody working for a small school district in western Pennsylvania wants to build a simple application for tracking students' attendance (rather than tracking it on paper anymore), what do they do?

Bruce Tate alluded to this in his Beyond Java, based on the realization that the Java space was no better—to bring a college/university student up to speed on all the necessary technologies required of a "productive" Java developer, he calculated at least five or six weeks of training was required. And that's not a bad estimate, and might even be a bit on the shortened side. You can maybe get away with less if they're joining a team which collectively has these skills distributed across the entire team, but if we're talking about a standalone developer who's going to be building software by himself, it's a pretty impressive list. Here's my back-of-the-envelope calculations:

• Week one: Java language. (Nobody ever comes out of college knowing all the Java language they need.)
• Week two: Java virtual machine: threading/concurrency, ClassLoaders, Serialization, RMI, XML parsing, reference types (weak, soft, phantom).
• Week three: Infrastructure: Ant, JUnit, continuous integration, Spring.
• Week four: Data access: JDBC, Hibernate. (Yes, I think you need a full week on Hibernate to be able to use it effectively.)
• Week five: Web: HTTP, HTML, servlets, filters, servlet context and listeners, JSP, model-view-controller, and probably some Ajax to boot.

I could go on (seriously! no JMS? no REST? no Web services?), but you get the point. And lest the .NET community start feeling complacent, put together a similar list for the standalone .NET developer, and you'll come out to something pretty equivalent. (Just look at the Pluralsight list of courses—name the one course you would give that college kid to bring him up to speed. Stumped? Don't feel bad—I can't, either. And it's not them—pick on any of the training companies.)

Now throw agile into that mix: how does an agile process reduce the complexity load? And the answer, of course, is that it doesn't—it simply tries to muddle through as best it can, by doing all of the things that developers need to be doing: gathering as much feedback from every corner of their world as they can, through tests, customer interaction, and frequent releases. All of which is good. I'm not here to suggest that we should all give up agile and immediately go back to waterfall and Big Design Up Front. Anybody who uses Billy's quote as a sound bite to suggest that is a subversive and a terrorist and should have their arguments refuted with extreme prejudice.

But agile is not going to reduce the technology complexity load, which is the root cause of the problem.

Or, perhaps, let me ask it this way: your 16-year-old wants to build a system to track the cards in his Magic deck. What language do you teach him?

We are in desperate need of simplicity in this industry. Whoever gets that, and gets it right, defines the "Next Big Thing".

Recently I've had the pleasure to make the acquaintance of Walter Bright, one of the heavyweights of compiler construction, and the creator of the D language (among other things), and he's been great in giving me some hand-holding on some compiler-related topics and ideas.

Thus, it seems appropriate to point out that Walter's willing to give lots of other people the same kind of attention and focus, in exchange for your presence in gorgeous Astoria, OR. The Astoria Compiler Construction Seminar is Walter teaching you about the nuts and bolts of building a compiler, from start to finish:

• Introduction to Compilers
• Lexing and Parsing
• Semantic Analysis
• Intermediate Representation
• Interpreters
• Optimization
• Code Generation
• Special Topics (thread-local storage, exception-handling, and so on)
• Building a Compiler for .NET

If you've got any interest whatsoever in building a language, but you're not sure how or where to get started, this seems like a great chance to sit down with one of the "big boys" and find out how to do it. And it doesn't hurt that Walter's an extremely pleasant guy to hang out with, either. :-) (It doesn't hurt that he was the one who created the original Empire game, either. So at least you know you'll have something to play during the breaks.)

Go. Sign up. You'll thank me later.

(Disclaimer: In the spirit of full disclosure, Stu is a friend, fellow NFJS speaker, and former co-worker of mine from DevelopMentor.)

I present this review to you in two parts.

Short version: If you want to learn Clojure, and you're familiar with at least one programming language, you'll find this a great resource. If you don't already know a programming language, or if you already know Clojure, or if you're looking for "best practices" to cut-and-paste, you're going to be disappointed.

Long version: Recently, fellow NFJS speaker Stu Halloway decided to take up a new language, and came to Clojure. He found the language interesting enough to write a book on it, something he hasn't done since his Java days, and the result is a nice walk through the language and its environment for experienced Java developers who want to understand Clojure's language, concurrency concepts, and programming model.

Now, let's be 100% honest about this: if you're coming at this book expecting it to be a language reference, you will probably be disappointed (as this guy obviously is). Stu's not like that—he's not going to re-create material that's available elsewhere, or that can be found with an easy Google search. Stu will not waste your time that way—he wants to tell you a story, one that takes you from "I'm a Java guy, but clueless about Lisp, dynamic languages, functional programming, concurrency, or macros" to "Wow. I know kung-fu." in the shortest path possible, but without trying to lobotomize you. He wants—no, expects—the readers of his book to be propping the text open with a cell phone on one side and the dinner plate on the other, craning your neck over to scan the pages and type in the examples into the REPL shell to try them out, see them work, then spend a few minutes experimenting with them before moving on to the next paragraph or page.

(Oh, I suppose you could just cut and paste them from the PDF version of the book, but where's the fun in that?)

The fact is, the concepts behind Clojure make up what's important to learn here, and readers of this book will come away like the panda from the movie, realizing that "There is no Secret Ingredient", that the power of Clojure comes not from its super-secret language sauce or special libraries, but in the way Clojure programmers approach problems and think about programming. And for that reason, if you're a programmer—even if you don't program on the JVM—you really want to take a look at what Stu's talking about (and Rich Hickey is creating).

Just remember, cellphone and dinner plate. Otherwise you'll be missing out on so much.

Well, OK, the title is trolling ever so slightly, but there is an interesting trend at work, and I'm genuinely concerned about its ultimate expression if the trend continues to its logical conclusion. Have a look and tell me if you agree or disagree.

Apparently the Rails community isn't the only one pursuing that ephemeral goal of "edginess"—another blatantly sexist presentation came off without a hitch, this time at a Flash conference, and if anything, it was worse than the Rails/CouchDB presentation. I excerpt a few choice tidbits from an eyewitness here, but be warned—if you're not comfortable with language, skip the next block paragraph.

Yesterday's afternoon keynote is this guy named Hoss Gifford — I believe his major claim to fame is that viral "spank the monkey" thing that went around a few years back.  Highlights of his talk:

• He opens his keynote with one of those "Ignite"-esque presentations — where you have 5-minutes and 20 slides to tell a story — and the first and last are a close-up of a woman's lower half, her legs spread (wearing stilettos, of course) and her shaved vagina visible through some see-thru panties that say "drink me," with Hoss's Photoshopped, upward-looking face placed below it.
• He later demos a drawing tool he has created (admittedly with someone else's code) and invites a woman to come up to try it.  After she sits back down, he points out that in her doodles she's drawn a "cock."
• Then he decides he wants to give a try at using the tool to draw a "cock" (he loves this word) — and draws a face, then a giant dick (he redraws it three times) that ultimately cums all over the face.
• A multitude of references to penises and lots of swearing — and also "If you are easily offended, fuck you!"
• And then, to top it off, a self-made flash movie of an animated woman's face, positioned as if she's having sex with you, who gradually orgasms based on the speed of your mouse movement on the page.

Wow. Just... wow. To call this unprofessional smacks of calling Hitler a "socially awkward individual"... or using a euphemism like "mild medical condition" to refer to death. This is so far "over the line" that it's unbelievable. Even Mr. Aimonetti's "CouchDB" presentation, as bad as it was, at least tried to tie the analogy together in a meaningful, if offensive, way. This is just male posturing at its worst. (I'm shocked Hoss didn't whip off his pants and demand the women in the room bow down in worship to his obviously superior manhood.)

Fortunately, according to the source, the conference organizer seems to be pretty responsive, so kudos to the one adult in the room, but....

What's worse, apparently the presenter and more than a few of his pals are (in the best traditions of assholery) blatantly unrepentant about the whole thing, claiming the moral high ground in much the same way that the Rails idiots did—it's all in good fun, if you don't find it funny you're a prude, and so on:

I checked Twitter (hashtag #flashbelt) to see what the responses were.  Here are some notable remarks:

• Fonx is reading the #flashbelt rants on Hoss offending the ladies w/ a few swear words & a penis drawing - r u really that prudish & sexist?
• nthitz lol @hoss69 "If you are easily offended, fuck you" #flashbelt
• livenootrac Ladies of #flashbelt , I am sorry for the Hoss preso, but in the flash community he gets a pass, kinda like Don Rickles - that's just Hoss.
• CujoJpn @livenootrac And there were many ladies at #flashbelt who were offended by Hoss' Preso some were thick skinned and took it as is.

So, if you didn't like it then
a) you are a prude - and sexist (?)
b) fuck you
c) suck it because Hoss gets a pass here in the boy's club known as "the flash community" and
d) you are a wimpy girl who isn't strong enough / man enough / "thick-skinned" enough  to deal with it.

Even more... wow. Talk about justification and marginalization. Amazing.

Before I figuratively smack this Hoss guy around the blog for a while, let's take a brief moment for reflection—what's going on here? Why all the misogynistic presentations recently? Is this reflective of a general trend in the programming industry? Of society in general? Is the world coming to an end?

A few possibilities present themselves:

• The lack of women in the IT industry means there's nobody around to act as a "gender filter" to keep things on an even keel. In other words, the genders constantly filter themselves based on the company they keep, and because the boys who put these presentations together don't have female input, they simply don't know where to draw the line for mixed company. This theory also presumes that an industry that's made up primarily of women will also lack such a filter and "girls will be girls" as a result. Unfortunately I have no good counterexamples at hand to examine—anybody know of an industry populated primarily by women, and can weigh in with experience there? The closest I get is my brief experience working in a restaurant with an almost-all-woman serving staff, and from that brief experience, yep, the theory holds. Solution? Easy: get more women in IT, and things will re-balance themselves naturally.
• Programmers are principally males who have no redeeming social skills. In other words, the industry gathers up exactly the kind of men who find objectifying women and reveling in late-acquired testosterone overdoses to be gratifying, and this kind of behavior is the result. If true, it leads to the conclusion that programmers are no more evolved than the Navy sailors involved in the Tailhook scandal of a few years ago. So go ahead, smack your wives and girlfriends around a little if they get a little "uppity", it's OK, 'cuz u r a l33t d00d. Personally? I find the idea ludicrous—there is definitely a strong antisocial streak that runs through the IT ecosystem (how many of you met your friends via World of Warcraft again?), but like all stereotypes, there's some elements of truth to it, and a lot of exaggeration. And frankly, anybody who believes in this theory is welcome to come with me to dinner at a No Fluff Just Stuff show and meet the other speakers, and listen in on our "boys club" conversations, including questions like, "Which movie best represents the book it was made after?" and "If given a mandate to create a programming language, what language would your language most resemble?". Oh, and the odd fart joke. We are boys, after all.
• We're hypersensitive to the subject right now. In other words, these kind of presentations have always been going on, and it's just that we notice them now, in the same way that you notice a particular brand of car on the road a lot more when you're thinking about buying that brand and model of car. Frankly, I don't buy this argument—I've been to a lot of presentations over the past decade, and I've never seen any that were anything like this.
• This is the YouTube generation, with access to everything the Internet has to offer, and this is "just how they do things". After all, how much maturity, sexual discretion and adult behavior can we expect of the generation that gave us "Girls Gone Wild" and its ilk? It's just a "generation gap" thing, and we old fogies who didn't grow up with Internet porn just a browser-click away just don't "get it". Hmm.... somehow, I just don't buy it. Sure, there may be some elements of this involved here (I'm really curious to see what all these "Girls Gone Wild" girls are going to say to their own daughters in a decade or so...), but I think that's too easy an answer, and an eminently unhelpful one.
• We have copycatters out there trying to follow the path of people they respect. If you're looking up at this Hoss character and thinking, "I want to be just like him!", you really should see a therapist and develop a sense of self, before you find yourself without friends. Hoss gets a pass because of your misguided fan-boi hero-worship. So does Paris Hilton. You want to be the Paris Hilton of your social circle? Go for it. After all, she's highly respected and loved, right? Take a clue from the next car wreck you drive past—everybody's slowing to look not because they wish they were in the body bag, folks, but because we have a ghoulish fascination with it. In the case of Ms. Hilton, that ghoulish fascination is with those who self-destruct in spectacular fashion. (Me, I'd love to be the fly on the wall at the Hoss residence when he tries to explain this whole thing to his daughter or his date/girlfriend/wife, if he ever finds one.)
• The presenters taking this tack are looking for an easy path to fame. In the grand traditions of Andrew Dice Clay ("Oh!"), the easiest way for a presenter to "stand out" from the rest of the crowd of presenters is to do something outrageous and call it "edgy", and stake out a claim on the edge of the civilization, rather than try to integrate with the rest of the crowd and build something up slowly. Don Box has already claimed "HTTP is dead", I made the analogy between a technology and a military conflict, and Matt Aimonetti claimed a data storage framework "performs like a pr0n star", so what's left but to stake out ground even further out on the fringe and just be misogynistic? Fortunately, history suggests that people with content-free/shock-heavy presentations (or even content-heavy/shock-heavy ones) don't go the distance, so to speak, and that once there's nowhere more shocking left to go, the audience comes back to the content-heavy/shock-light discussions and stays there for a while. Unfortunately, this means we're going to have to suffer through somebody's "Live YouPorn filming" talk first, which I'm not looking forward to.

And now for the smacking around... but you know, I suddenly realize that the volume of comments on the original post leave with nothing to do or say that's not already being said, so to just "pile on" would only serve to let me vent, and I have other outlets for that. But it would be inappropriate to just "walk away", so to speak, so with that in mind....

Hoss, you're an idiot. Like any sprinter, you're going to head up the pack for a bit, but soon enough, your "shtick" is going to flame out and you'll be left behind with all the other "shock jocks" of the 80's who found their material unwelcome after a while. So enjoy the spotlight (such as it is) while you can. In the meantime, I'm off to revise a few presentations, and stick with solid ideas and analogies, and maybe dropping the odd F-bomb when I want to make a point, just for emphasis, because I know something you apparently don't:

Shock makes a point because of the contrast to the rest of the talk, not because of its inherent "edginess".

Meanwhile, by all means, continue to be an idiot. You just make me look better by comparison, for which I thank you.

Update: See below, but I wanted to include the text Mike Abercrombie (DM's owner) posted as a comment to this post, in the body of the blog post itself. "Ted - All of us at DevelopMentor greatly appreciate your admiration. We're also grateful for your contributions to DevelopMentor when you were part of our staff. However, all of us that work here, especially our technical staff that write and delivery our courses today, would appreciate it if you would check your sources before writing our eulogy. DevelopMentor is open for business and delivering courses this week and we intend to remain doing so." Duly noted, Mike. Apology offered (and hopefully accepted).

An email crossed my desk today, announcing that DevelopMentor, home to so many good people and fond memories, has (at least temporarily) closed its doors.

I admit to a small, carefully-cushioned place in my heart where I mourn over this.

DevelopMentor was such a transcendent place for me. Much, if not most or all, of the acceleration that came in my career came not only while I was there, but because I was there.

So much of my speaking persona and skill I owe to Ron Sumida, who took a half-baked neophyte of intermediate speaking skill, and in an eight-hour marathon session still referred to in my mental memoirs as my "Night with Scary Ron", shaped me and taught me tricks about speaking that I continue to use to this day. That I got to know him as a friend and confidant later still to this day ranks as one of my greatest blessings.

I remember my first DM Instructor Retreat, where I met so many of the names I'd read about or heard about, and feeling "Oh, my God" fanboy-ish. I remember Tim Ewald giving a talk on transactions at that retreat that left me agape—I seriously didn't understand half of what he was saying, and rather than feeling overwhelmed or ashamed, I remember distinctly thinking, "Wow—I have found a home where I can learn SO much more." It was like waking up one morning to find that your writing workshop group suddenly included Neal Stephenson, Stephen Pinker, C.S. Lewis and Ernest Hemingway. (Yes, I know those last two are dead. Work with me here.)

I remember the day that Lorie (the ops manager at the time) called me to say that Don Box wanted me to work with him on the C# course. I was convinced that she'd called the wrong Ted, meaning instead to reach for Ted Pattison in her Rolodex and coming up a few letters shy. She tartly informed me, "No, I know exactly who I'm talking to, and are you interested or not?" How could I refuse? Help the Diety of COM write DM's flagship course on Microsoft's flagship technology for the next decade? "Hmm...", I say out loud, not because I needed time to think about it, but because a thread in the back of my head says, "Is there any scenario here where I say no?"

I still fondly recall doing a Guerilla .NET at the Torrance Hilton shortly after the .NET 1.0 release, and having a conversation with Don in my hotel room later that night; that was when he told me "Microsoft is working on an open-source version of the CLR". I was stunned—I had no idea that said version would factor pretty largely in my life later. But it opened my eyes, in a very practical way, to how deeply-connected DevelopMentor was to Microsoft, and how that could play out in a direct fashion.

When Peter Drayton joined, he asked me to do a quick review pass on the reference section of his C# in a Nutshell, and I agreed because Peter was a good guy (and somebody I'd hoped would become a friend), and wanted to see the book do well. That went from informal review to formal review to "well, could you maybe make it an editing pass?" to "Would you like to write a few chapters?" to "Well, let's sign you up as a co-author...". That project is what introduced me to John Osborn, which in turn led him to call me one day and say, "Some guys at Microsoft are working on an open-source version of the CLR, and would like to have a 'professional writer' help them write a book on it. Interested?" That led to SSCLI Internals, working with David Stutz, and wow, did I learn a helluvalot from that project, too.

Effective Enterprise Java came through DevelopMentor, thanks again to Don Box, who introduced me to the folks at Addison-Wesley that put the contract (and Scott Meyers, another blessing) in front of me.

DM got me my start in the conference circuit, as well. In 2002, John Lam pinged me over email—he'd recently become track chair for Connections down in Orlando, and was I interested in speaking there? I was such a newbie to the whole idea, but having taught classes roughly twice every month, I wasn't worried about the speaking part, but the rest of the process. John walked me through the process, and in doing so, set me down a path that would almost completely redefine my career within a year or so.

Even my Java chops got built up—the head of our Java curriculum was Stu Halloway (recently of Clojure fame), and between him, Kevin Jones, Si Horrell, Brian Maso and Owen Tallman, man, did I feel simultaneously like a small child among giants and like a kid in a candy store. Every time I turned around, they'd discovered something new about the Java platform that floored me. Bob Beauchemin has forgotten more about databases in general than I will ever learn, and he had some insights on the intersection of Java + databases that still hang with me today.

And my start with No Fluff Just Stuff came through DevelopMentor, too. Jason Whittington heard through a mutual friend (Erik Hatcher, of Ant fame) about this cool little conference being held in Denver, and maybe I should look into it. That led to an email intro to Jay Zimmerman, a dinner together while I was teaching in Denver a few weeks later, and before I knew it, I was on the Denver NFJS schedule, including the speaker panel, where I uttered the then-infamous line, "Swing sucks. Get over it."

DevelopMentor, you shaped my career—and my life—in so many ways, you will always be a source of pleasant memories and a group of friends and acquaintances that I would never have had otherwise. Thank you so much.

Rest in peace.

Update: Well, as it turns out, I have to rescind at least part of my eulogy, as the post itself generated quite a stir—the folks at DevelopMentor were pretty quick to email me, pointing out that they're still alive and well. In fact, as one of them (a friend of mine still working there) put it, "We were all kinda surprised when we came to work this morning and discovered that we could go home." Fortunately, the DevelopMentor folks were pretty gracious about what could've been a very ugly situation, and I apologize for to them for the misunderstanding—all I can say is that my "source" must've also been mistaken, and I'm glad that we're all still good. And lest it need to be said out loud, I heartily want nothing but the best for DM, and hope that I never have to write this message again.

Recently, NFJS alum and buddy Dion Almaer questioned the widespread, almost default, usage of a relational database for all things storage related:

Ian Hickson: “I expect I’ll be reverse-engineering SQLite and speccing that, if nothing better is picked first. As it is, people are starting to use the database feature in actual Web apps (e.g. mobile GMail, iirc).”

When I read that comment to Vlad’s post on HTML 5 Web Storage I gulped. This would basically make SQLite the HTML 5 for storage in the browser. You would have to be a little crazy to re-write the exact semantics (including bugs) of SQLite and its dialect. What if you couldn’t use the public domain code?

Gears lead out strong with making a relational database part of the toolbox for developers. It embedded its own SQLite, in fact one that was customized to have the very cool full text search ability. However, this brings up the point of “which SQLite do you standardize on?”

The beauty of using SQL and SQLite is that many developers already know it. RDBMS has been mainstream for donkey’s years; we have tools to manage SQL, to view the model, and to tweak for performance. It has gone through the test of time.

However, SQL has always been at odds with many developers. Ted Neward brought up ORM as the vietnam of computer science (which is going a touch far ;). I was just lamenting with a friend at Microsoft on how developers spend 90% of their time munging data. Our life is one of transformations, and that is why I am interested in a world of JavaScript on client and server AND database. We aren’t there yet, but hopefully we can make progress.

One of Vlad’s main questions is “Is SQL the right API for Web developers?” and it is a valid one. I quickly found that for most of my tasks with the DB I just wanted to deal with JSON and hence created a wrapper GearsDB to let me insert/update/select/delete the database with a JSON view of the world. You probably wouldn’t want to do this on large production applications for performance reasons, but it works well for me.

Now a days, we have interesting APIs such as JSONQuery which Persevere (and other databases) use. I would love to see Firefox and other browsers support something like this and let us live in JSON throughout the stack. It feels so much more Webby, and also, some of the reasons that made us stay with SQL don’t matter as much in the client side world. For example, when OODBMS took off in some Enterprises, I remember having all of these Versant to Oracle exports just so people could report on the darn data. On the client the database is used for a very different reason (local storage) so lets use JSON!

That being said, at this point there are applications such as Gmail, MySpace search, Zoho, and many iPhone Web applications that use the SQL storage in browsers. In fact, if we had the API in Firefox I would have Bespin using it right now! We had a version of this that abstracted on top of stores, but it was a pain. I would love to just use HTML 5 storage and be done.

So, I think that Firefox should actually support this for practical reasons (and we have SQLite right there!) but should push JSON APIs and let developers decide. I hope that JSON wins, you? I also hope that Hixie doesn’t have to spec SQLite :/

Dion's right when he says "developers spend 90% of their time munging data" and that "Our life is one of transformations", but I think he's being short-sighted and entirely narrow-minded when he says, "I am interested in a world of JavaScript on client and server AND database." Dion, I love you, man, but you're falling prey to the Fallacy of the One True Language. JavaScript (or ECMAScript, as its official name is given) is an interesting and powerful language, but why do you want to force your biases and perceptions on the rest of the world, man? You're being just as bad as the C++ or Java guys were in their heyday—remember when Java stored procedures were all the rage because "everybody knows that Java is the wave of the future"?

The fact is, from where I stand, there is no one storage solution or language solution or user-interface solution that is the Right Thing To Do in all situations. Not even inside the browser. There will be situations where a SQLite is the Right Thing, and other situations where a document-oriented JSON-like or CouchDB-like thing will be the Right Thing, and trying to force-feed one into a situation that's best solved by the other is a bad idea.

Dion alludes to my article about the Vietnam of Computer Science, but in fact, his suggestion charges right into another quagmire—how long before somebody starts trying to create a JSON-to-RDBMS adaption layer? Or JSON-to-CouchDB? Or things equally ridiculous? The fact is, data has three fundamentally different "shapes" to it, and trying to pound data from one shape into the other has all the efficacy and elegance to it just as much as pounding round pegs into square holes does. Dion even alludes to this with this paragraph:

One of Vlad’s main questions is “Is SQL the right API for Web developers?” and it is a valid one. I quickly found that for most of my tasks with the DB I just wanted to deal with JSON and hence created a wrapper GearsDB to let me insert/update/select/delete the database with a JSON view of the world. You probably wouldn’t want to do this on large production applications for performance reasons, but it works well for me.

JSON is certainly an attractive representation format for ECMAScript objects, thanks to its fundamental roots in ECMAScript's object literal syntax, and the powerful/dangerous eval() functionality offered by ECMAScript environments, but JSON also lacks a number of things a SQL-based dialect has, including a powerful query syntax for selecting individual and subsets of entities from the whole, which only becomes more and more necessary as the data base itself gets larger and larger. (Anybody who suggests that a local browser store would only remain within a certain size is clearly not thinking further ahead than the current day. Look at how cookies are outrageously abused as local storage for a lot of sites, or how Viewstate was abused in early ASP.NET apps—if you give the HTML/front-end developer a local storage mechanism, they will use it, and use it as far and as long and as hard as they can.) On top of which, JSON simply doesn't have the years of solid backing behind it than a SQL-based storage format does. And so on, and so on, and so on.

Ironically, just as JSON is a scheme for representing native objects in some kind of data format (in this case, a plain-text one), developers casually ignore the idea of storing objects in a native data format with all of the other bells-and-whistles that a database provides. Naturally, I'm referring to the idea of an object database—if JSON is appropriate for storing certain kinds of data in certain scenarios, then why isn't it appropriate to consider a native object database for some of those same certain kinds of scenarios? Not that I have anything against a JSON-based database scenario—in fact, I can easily imagine a JSON database that indexes the properties of the stored objects and takes ECMAScript functions as "native queries" in the same way that db4o doe. But let's stop with the repeated attempts at "one size fits all", and just accept that the world is a polyglot world, and that no one language—or data storage format, or data access API—will be the Right Thing To Do for all scenarios. Each language, format, API or tool has a reason to exist, a particular way it looks at the world, and optimizes itself to work best when used in that particular style. Trying to force one into the terms of the other is the road to another Computer Science quagmire.

Viva la Polyglot!

The Simple-Talk newsletter is a monthly e-zine that the folks over at Red Gate Software (makers of some pretty cool toys, including their ANTS Profiler, and recent inheritors of the Reflector utility legacy) produce, usually to good effect.

But this month carried with it an interesting editorial piece, which I reproduce in its entirety here:

When the market is slack, nothing succeeds better at tightening it up than promoting serial group-panic within the community. As an example of this, a wave of multi-core panic spread across the Internet about 18 months ago. IT organizations, it was said, urgently had to improve application performance by an order of magnitude in order to cope with rising demand. We wouldn't be able to meet that need because we were at the "end of the road" with regard to step changes in processor power and clock speed. Multi-core technology was the only sure route to improving the speed of applications but, unfortunately, our current "serial" programming techniques, and the limited multithreading capabilities of our programming languages and programmers, left us ill-equipped to exploit it. Multi-core mania gripped the industry.

However, the fever was surprisingly short-lived. Intel's "largest open-source effort ever" to provide a standard tool for writing multi-threaded code, caused little more than a ripple of interest. Various books, rushed out while the temperature soared, advocated the urgent need for new "multi-core-friendly" programming models, involving such things as "software pipelines". Interesting as they undoubtedly are, they sit stolidly on bookshelves, unread.

The truth is that it's simply not a big issue for the majority of people. Writing truly "concurrent" applications in languages such as C# is difficult, as you get very little help from the language. It means getting involved with low-level concurrency primitives, such as lock statements and so on.

Many programmers lack the skills to do this, but more pertinently lack the need. Increasingly, programmers work in a web environment. As long as these web applications are deployed to a load-balanced web farm, then page requests can be handled in parallel so all available cores will be used efficiently without the need for the programmer to be concerned with fine-grained parallelism.

Furthermore, the SQL Server engine behind these web applications is intrinsically "parallel", and can handle and use effectively about as many cores as you care to throw at it. SQL itself is a declarative rather than procedural language, so it is fundamentally concurrent.

A minority of programmers, for example games programmers or those who deal with "embarrassingly parallel" desktop applications such as Photoshop, do need to start working with the current tools and 'low-level' coding techniques that will allow them to exploit multi-core technology. Although currently perceived to be more of "academic" interest, concurrent languages such as Erlang, and concurrency techniques such as "software transactional memory", may yet prove to be significant.

For most programmers and for most web applications, however, the multi-core furore is a storm in a teacup; it's just not relevant. The web and database platforms already cope with concurrency requirements. We are already doing it.

My hope is that this newsletter, sent on April 1st, was intended to be a joke. Having said that, I can’t find any verbage in the email that suggests that it is, in which case, I have to treat it as a legitimate editorial.

And frankly, I think it’s all crap.

It's dangerously ostrichian in nature—it encourages developers to simply bury their heads in the sand and ignore the freight train that's coming their way. Permit me, if you will, a few minutes of your time, that I may be allowed to go through and demonstrate the reasons why I say this.

To begin ...

When the market is slack, nothing succeeds better at tightening it up than promoting serial group-panic within the community. As an example of this, a wave of multi-core panic spread across the Internet about 18 months ago. IT organizations, it was said, urgently had to improve application performance by an order of magnitude in order to cope with rising demand. [...] Multi-core mania gripped the industry.

Point of fact: The “panic” cited here didn’t start about 18 months ago, it started with Herb Sutter’s most excellent (and not only highly recommended but highly required) article, “The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software”, appeared in the pages of Dr. Dobb’s Journal in March of 2005. (Herb’s website notes that “a much briefer version under the title “The Concurrency Revolution” appeared in C/C++ User’s Journal” the previous month.) And the panic itself wasn’t rooted in the idea that we weren’t going to be able to cope with rising demand, but that multi-core CPUs, back then a rarity and reserved only for hardware systems in highly-specialized roles, were in fact becoming commonplace in servers, and worse, as they migrated into desktops, they would quickly a fact of life that every developer would need to face. Herb demonstrated this by pointing out that CPU speeds had taken an interesting change of pace in early 2003:

Around the beginning of 2003, [looking at the website Figure 1 graph] you’ll note a disturbing sharp turn in the previous trend toward ever-faster CPU clock speeds. I’ve added lines to show the limit trends in maximum clock speed; instead of continuing on the previous path, as indicated by the thin dotted line, there is a sharp flattening. It has become harder and harder to exploit higher clock speeds due to not just one but several physical issues, notably heat (too much of it and too hard to dissipate), power consumption (too high), and current leakage problems.

Joe Armstrong, creator of Erlang, noted in a presentation at QCon London 2007 that another of those physical limitations was the speed of light—that for the first time, CPU signal couldn't get from one end of the chip to the other in a single clock cycle.

Quick: What’s the clock speed on the CPU(s) in your current workstation? Are you running at 10GHz? On Intel chips, we reached 2GHz a long time ago (August 2001), and according to CPU trends before 2003, now in early 2005 we should have the first 10GHz Pentium-family chips.

Just to (re-)emphasize the point, here, now, in early 2009, we should be seeing the first 20 or 40 GHz processors, and clearly we’re still plodding along in the 2 – 3 GHz range. The "Quake Rule" (when asked about perf problems, tell your boss you'll need eighteen months to get a 2X improvement, then bury yourselves in a closet for 18 months playing Quake until the next gen of Intel hardware comes out) no longer works.

For the near-term future, meaning for the next few years, the performance gains in new chips will be fueled by three main approaches, only one of which is the same as in the past. The near-term future performance growth drivers are:

• hyperthreading
• multicore
• cache

Hyperthreading is about running two or more threads in parallel inside a single CPU. Hyperthreaded CPUs are already available today, and they do allow some instructions to run in parallel. A limiting factor, however, is that although a hyper-threaded CPU has some extra hardware including extra registers, it still has just one cache, one integer math unit, one FPU, and in general just one each of most basic CPU features. Hyperthreading is sometimes cited as offering a 5% to 15% performance boost for reasonably well-written multi-threaded applications, or even as much as 40% under ideal conditions for carefully written multi-threaded applications. That’s good, but it’s hardly double, and it doesn’t help single-threaded applications.

Multicore is about running two or more actual CPUs on one chip. Some chips, including Sparc and PowerPC, have multicore versions available already. The initial Intel and AMD designs, both due in 2005, vary in their level of integration but are functionally similar. AMD’s seems to have some initial performance design advantages, such as better integration of support functions on the same die, whereas Intel’s initial entry basically just glues together two Xeons on a single die. The performance gains should initially be about the same as having a true dual-CPU system (only the system will be cheaper because the motherboard doesn’t have to have two sockets and associated “glue” chippery), which means something less than double the speed even in the ideal case, and just like today it will boost reasonably well-written multi-threaded applications. Not single-threaded ones.

Finally, on-die cache sizes can be expected to continue to grow, at least in the near term. Of these three areas, only this one will broadly benefit most existing applications. The continuing growth in on-die cache sizes is an incredibly important and highly applicable benefit for many applications, simply because space is speed. Accessing main memory is expensive, and you really don’t want to touch RAM if you can help it. On today’s systems, a cache miss that goes out to main memory often costs 10 to 50 times as much getting the information from the cache; this, incidentally, continues to surprise people because we all think of memory as fast, and it is fast compared to disks and networks, but not compared to on-board cache which runs at faster speeds. If an application’s working set fits into cache, we’re golden, and if it doesn’t, we’re not. That is why increased cache sizes will save some existing applications and breathe life into them for a few more years without requiring significant redesign: As existing applications manipulate more and more data, and as they are incrementally updated to include more code for new features, performance-sensitive operations need to continue to fit into cache. As the Depression-era old-timers will be quick to remind you, “Cache is king.”

Herb’s article was a pretty serious wake-up call to programmers who hadn’t noticed the trend themselves. (Being one of those who hadn’t noticed, I remember reading his piece, looking at that graph, glancing at the open ad from Fry’s Electronics sitting on the dining room table next to me, and saying to myself, “Holy sh*t, he’s right!”.) Does that qualify it as a “mania”? Perhaps if you’re trying to pooh-pooh the concern, sure. But if you’re a developer who’s wondering where you’re going to get the processing power to address the ever-expanding list of features your users want, something Herb points out as a basic fact of life in the software development world ...

There’s an interesting phenomenon that’s known as “Andy giveth, and Bill taketh away.” No matter how fast processors get, software consistently finds new ways to eat up the extra speed. Make a CPU ten times as fast, and software will usually find ten times as much to do (or, in some cases, will feel at liberty to do it ten times less efficiently).

...  then eking out the best performance from an application is going to remain at the top of the priority list. Users are classic consumers: they will always want more and more for the same money as before. Ignore this truth of software (actually, of basic microeconomics) at your peril.

To get back to the editorial, we next come to ...

However, the fever was surprisingly short-lived. Intel's "largest open-source effort ever" to provide a standard tool for writing multi-threaded code, caused little more than a ripple of interest. Various books, rushed out while the temperature soared, advocated the urgent need for new "multi-core-friendly" programming models, involving such things as "software pipelines". Interesting as they undoubtedly are, they sit stolidly on bookshelves, unread.

Wow. Talk about your pretty aggressive accusation without any supporting evidence or citation whatsoever.

Intel's not big into the open-source space, so it doesn't take much for an open-source project from them to be their "largest open-source effort ever". (What, they're going to open-source the schematics for the Intel chipline? Who could read them even if they did? Who would offer up a patch? What good would it do?) The fact that Intel made the software available in the first place meant that they knew the hurdle that had yet to be overcome, and wanted to aid developers in overcoming it. They're members of the OpenMP group for the same reason.

Rogue Wave's software pipelines programming model is another case where real benefits have accrued, backed by case studies. (Disclaimer: I know this because I ghost-wrote an article for them on their Software Pipelines implementation.) Let's not knock something that's actually delivered value. Pipelines aren't going to be the solution to every problem, granted, but they're a useful way of structuring a design, one that's curiously similar to what I see in functional programming languages.

But simply defending Intel's generosity or the validity of an alternative programming model doesn't support the idea that concurrency is still a hot topic. No, for that, I need real evidence, something with actual concrete numbers and verifiable fact to it.

Thus, I point to Brian Goetz’s Java Concurrency in Practice, one of those “books, rushed out while the temperature soared”, which also turned out to be the best-selling book at Java One 2007, and the second-best-selling book (behind only Joshua Bloch’s unbelievably good Effective Java (2nd Ed) ) at Java One 2008. Clearly, yes, bestselling concurrency books are just a myth, alongside the magical device that will receive messages from all over the world and play them into your brain (by way of your ears) on demand, or the magical silver bird that can wing its way through the air with no visible means of support as it does so. Myths, clearly, all of them.

To continue...

The truth is that it's simply not a big issue for the majority of people. Writing truly "concurrent" applications in languages such as C# is difficult, as you get very little help from the language. It means getting involved with low-level concurrency primitives, such as lock statements and so on.

Many programmers lack the skills to do this, but more pertinently lack the need. Increasingly, programmers work in a web environment. As long as these web applications are deployed to a load-balanced web farm, then page requests can be handled in parallel so all available cores will be used efficiently without the need for the programmer to be concerned with fine-grained parallelism.

He’s right when he says you get very little help from the language, be it C# or Java or C++. And getting involved with low-level concurrency primitives is clearly not in anybody’s best interests, particularly if you’re not a concurrency guru like Brian. (And let’s be honest, even low-level concurrency gurus like Brian, or Joe Duffy, who wrote Concurrent Programming on Windows, or Mike Woodring, who co-authored Win32 Multithreaded Programming, have better things to do.) But to say that they “pertinently lack the need” is a rather impertinent statement. “As long as these web applications are deployed to a load-balanced web farm", which is very likely to continue to happen, “then page requests can be handled in parallel so all available cores will be used …”

Um... excuse me?

Didn’t you just say that programmers didn’t need to learn concurrency constructs? It would strike me that if their page requests are being handled in parallel that they have to learn how to write code that won’t break when it’s accessed in parallel or lead to data-corruption problems or race conditions when their pages are accessed in parallel. If parallelism is a fundamental part of the Web, don’t you think it’s important for them to learn how to write programs that can behave correctly in parallel?

Look for just a moment at the average web application: if data is stored in a per-user collection, and two simultaneous requests come in from a given user (perhaps because the page has AJAX requests being generated by the user on the page, or perhaps because there’s a frameset that’s generating requests for each sub-frame, or ...), what happens if the code is written to read a value from the session, increment it, and store it back? ASP.NET can save you here, a little, in that it used to establish a per-user lock on the entirety of the page request (I don’t know if it still does this—I really have lost any desire to build web apps ever again), but that essentially puts an artificial throttle on the scalability of your system, and makes the end-users’ experience that much slower. Load-balancer going to spray the request all over the farm? So long as the user session state is stored on every machine in the farm, that’ll work... But of course if you store the user’s state in the SQL instance behind each of those machines on the farm, then you take the performance hit of an extra network round-trip (at which point we’re back to concurrency in the database) ...

... all because the programmer couldn’t figure out how to make “lock” work? This is progress?

The Java Servlet specification specifically backed away from this "lock on every request" approach because of the performance implications. I heard a fair amount of wailing and gnashing during the early ASP.NET days over this. I heard the ASP.NET dev team say they made their decision because the average developer can't figure out concurrency correctly anyway.

And, by the way folks, this editorial completely ignores XML services. I guess "real" applications don't write services much, either.

The next part is even better:

Furthermore, the SQL Server engine behind these web applications is intrinsically "parallel", and can handle and use effectively about as many cores as you care to throw at it. SQL itself is a declarative rather than procedural language, so it is fundamentally concurrent.

True… and false. SQL is fundamentally “parallel” (largely because SQL is a non-strict functional language, not just a “declarative” one), but T-SQL isn’t. And how many developers actually know where the line is drawn between SQL and T-SQL? More importantly, though, how many effective applications can be written with a complete ignorance of the underlying locking model? Why do DBAs spend hours tuning the database’s physical constructs, establishing where isolation levels can be turned down, establishing where the scope of a transaction is too large, putting in indexed columns where necessary, and figuring out where page, row, or table locking will be most efficient? Because despite the view that a relational database presents, these queries are being executed in parallel, and if a developer wants to avoid writing an application that requires a new server for each and every new user added to the system, they need to learn how to maximize their use of the database’s parallelism. So even if the language is "fundamentally concurrent" and can thus be relied upon to do the right thing on behalf of the developer, the implementation isn't, and needs to be understood in order to be implemented efficiently.

He finishes:

For most programmers and for most web applications, however, the multi-core furore is a storm in a teacup; it's just not relevant. The web and database platforms already cope with concurrency requirements. We are already doing it.

This is one of those times I wish I had a time machine handy—I'd love to step forward five years, have a look around, then come back and report the findings. I'm tempted to close with the challenge to just let’s come back in five years and see what the programming language landscape and hardware landscape looks like. But that's too easy an "out", and frankly, doesn't do much to really instill confidence, in my opinion.

To ignore the developers building "rich" applications (be they being done in Flex/Flash, Cocoa/iPhone, WinForms, Swing, WPF, or what-have-you) is to also ignore a relatively large segment of the market. Not every application is being built on the web and is backed by a relational database—to simply brush those off and not even consider them as part of the editorial reveals a dangerous bias on the editor's part. And those applications aren't hosted in an "intrinsically 'parallel'" container that developers can just bury their head inside.

Like it or not, folks, the path forward isn't one that you get to choose. Intel, AMD, and other chip manufacturers have already made that clear. They're not going to abandon the multicore approach now, not when doing so would mean trying to wrestle with so many problems (including trying to change the speed of light) that simply aren't there when using a multicore foundation. That isn't up for debate anymore. Multicore has won for the forseeable future. And, as a result, multicore is going to be a fact of the developer's life for the forseeable future. Concurrency is thus also a fact of the developer's life for the forseeable future.

The web and database platforms “cope” with concurrency requirements by either making "one-size-fits-all" decisions that almost always end up being the wrong decision for high-scale systems (but I'm sure your new startup-based idea, like a system that allows people to push "micro-entries" of no more than 140 characters in length to a publicly-trackable feed would never actually take off and start carrying millions and millions of messages every day, right?), or by punting entirely and forcing developers to dig deeper beneath the covers to see the concurrency there. So if you're happy with your applications running no faster than 2GHz for the rest of the forseeable future, then sure, you don't need to worry about learning concurrency-friendly kinds of programming techniques. Bear in mind, by the way, that this essentially locks you in to small-scale, web-plus-database systems for the forseeable future, and clearly nothing with any sort of CPU intensiveness to it whatsoever. Be happy in your niche, and wave to the other COBOL programmers who made the same decision.

This is a leaky abstraction, full stop, end of story. Anyone who tells you otherwise is either trolling for hits, trying to sell you something, or striving to persuade developers that ignorance isn't such a bad place to be.

All you ignorant developers, this is the phrase you will be forced to learn before you start your next job: "Would you like fries with that?"

While playing around with a recent research-oriented project for myself (more on that later), I discovered something that I haven't seen mentioned anywhere in the Scala universe before. (OK, not really--as you'll see towards the end of this piece, it really is documented, but allow me my brief delusions of grandeur as I write this. They'll get deflated quickly enough.)

So the core of the thing was a stack-oriented execution engine; essentially I'm processing commands delivered in a postfix manner. Since some of these commands are relational operators, it's important that there be two things to relationally operate on the execution stack, after which I want to evaluate the relational operation and push its result (1 if true, 0 if false) back on the stack; this is pretty easily done via the following:

def compareOp(op : (Int, Int) => Boolean) =

{

checkStack(2)

val v1 = (execStack.pop()).asInstanceOf[Int]

val v2 = (execStack.pop()).asInstanceOf[Int]

val vr = op(v1, v2)

execStack.push(if (vr) 1 else 0)

} 

where "execStack" is a mutable.Stack[Any] held in an enclosing function.

Interestingly enough, however, when I wrote this the first time, I wrote it like this, which is a very different sequence of operations:

def compareOp(op : (Int, Int) => Boolean) =

{

checkStack(2)

def v1 = (execStack.pop()).asInstanceOf[Int]

def v2 = (execStack.pop()).asInstanceOf[Int]

def vr = op(v1, v2)

execStack.push(if (vr) 1 else 0)

}

See the difference? Subtle, is it not? But the actual code is significantly different, something that's more easily seen with a much simpler (and standalone) example:

            object App

{

def main(args : Array[String]) =

{

import scala.collection.mutable.Stack

var stack : Stack[Any] = new Stack()

stack.push(12)

stack.push(24)

def v1 = (stack.pop()).asInstanceOf[Int]

def v2 = (stack.pop()).asInstanceOf[Int]

def vr = v1 + v2

System.out.println(vr)

}

} 

When run, the console prints out "36", as we'd well expect.

But suppose we want to look at those values of v1 and v2 along the way, perhaps as part of a logging operation, or perhaps because you're just screwing around with some ideas in your head and you don't want to bother to fire up an IDE with Scala support in it. So you decide to spit those values to a console:

            object App

{

def main(args : Array[String]) =

{

import scala.collection.mutable.Stack

var stack : Stack[Any] = new Stack()

stack.push(12)

stack.push(24)

def v1 = (stack.pop()).asInstanceOf[Int]

def v2 = (stack.pop()).asInstanceOf[Int]

System.out.println(v1)

System.out.println(v2)

def vr = v1 + v2

System.out.println(vr)

}

} 

And then something *very* different happens; you get "24", "12", and then a NoSuchElementException.

If you're like me the first time I ran into this, your first reaction is, "Eh?". Actually, if you're like me, when you're programming, your profanity filters are probaby at an ebb, so your first reaction is "WTF?!?", said with great gusto and emphasis. Which has a tendency to get some strange looks when you're at a Denny's doing your research, I will admit. Particularly when it's at 3 AM in the morning. And the bar crowd is in full alcoholic haze and slightly nervous about the long-haired, goatee-sporting guy in his headphones, wearing his black leather jacket and swearing like a drunken sailor at his laptop. But I digress.

What is Scala doing here?

Turns out this is exactly as the language designers intended, but it's subtle. (Or maybe it's just subtle to me at 3AM when I'm pumped full of caffeine.)

Let's take this a different way:

            object App

{

def main(args : Array[String]) =

{

import scala.collection.mutable.Stack

var stack : Stack[Any] = new Stack()

stack.push(12)

stack.push(24)

def v1 = (stack.pop()).asInstanceOf[Int]

def v2 = (stack.pop()).asInstanceOf[Int]

System.out.println(stack)

}

} 

When run, the console prints "Stack(12, 24)", which *really* starts to play with your mind when you're a little short on sleep and a little high on Diet Coke. At first glance, it looks like Scala is broken somehow--after all, those "pop" operations are supposed to modify the Stack against which they're operating, just as the push()es do. So why is the stack convinced that it still holds the values of 12 and 24?

Because Scala hasn't actually executed those pop()s yet.

The "def" keyword, it turns out, isn't what I wanted here--what I wanted (and in retrospect it’s painfully obvious) was a "val", instead, in order to force the execution of those statements and capture the value into a local value (an immutable local variable). The "def" keyword, instead, creates a function binding that waits for formal execution before evaluating. So that when I previously said

            object App

{

def main(args : Array[String]) =

{

import scala.collection.mutable.Stack

var stack : Stack[Any] = new Stack()

stack.push(12)

stack.push(24)

def v1 = (stack.pop()).asInstanceOf[Int]

def v2 = (stack.pop()).asInstanceOf[Int]

def vr = v1 + v2

System.out.println(vr)

}

} 

… what in fact I was saying was this:

            object App

{

def main(args : Array[String]) =

{

import scala.collection.mutable.Stack

var stack : Stack[Any] = new Stack()

stack.push(12)

stack.push(24)

def v1 = (stack.pop()).asInstanceOf[Int]

def v2 = (stack.pop()).asInstanceOf[Int]

System.out.println(v1 + v2)

}

} 

… which is the same as:

            object App

{

def main(args : Array[String]) =

{

import scala.collection.mutable.Stack

var stack : Stack[Any] = new Stack()

stack.push(12)

stack.push(24)

System.out.println((stack.pop()).asInstanceOf[Int]
+ (stack.pop()).asInstanceOf[Int])

}

} 

… which, when we look back at my most recent "debugging" version of the code, substituting the "def"ed versions of v1 and v2 (and vr) where they're used, makes the reason for the NoSuchElementException become entirely more clear:

            object App

{

def main(args : Array[String]) =

{

import scala.collection.mutable.Stack

var stack : Stack[Any] = new Stack()

stack.push(12)

stack.push(24)

System.out.println((stack.pop()).asInstanceOf[Int])

System.out.println((stack.pop()).asInstanceOf[Int])

System.out.println((stack.pop()).asInstanceOf[Int]
+ (stack.pop()).asInstanceOf[Int])

}

} 

Now, normally, this would probably set off all kinds of alarm bells in your head, but the reaction that went off in mine was "COOL!", the reasons for which revolve around the concept of "laziness"; in a functional language, we frequently don't want to evaluate the results right away, instead preferring to defer their execution until actually requiring it. In fact, many functional languages—such as Haskell—take laziness to new heights, baking it directly into the language definition and assuming laziness everywhere, so much so that you have to take special steps to avoid it. There’s a variety of reasons why this is advantageous, but I’ll leave those discussions to the Haskellians of the world, like Matt Podwysocki and Simon Peyton-Jones.

From a Scalist’s perspective, laziness is still a useful tool to have in your toolbox. Suppose you have a really powerful function that calculates PI to a ridiculous number of decimal places. In Java, you might be tempted to do something like this:

            class MyMath

{
publicstatic final double PI
= calculatePiToARidiculousNumberOfPlaces();
privatestaticdouble calculatePiToARidiculousNumberOfPlaces()

{
// implementation left to the reader's imagination
// imagine it being "really cool"

}

} 

The problem with this is that if that method takes any length of time to execute, it's being done during class initialization during its ClassLoading phase, and aside from introducing a window of time where the class *could* be used before that initialization is finished (it's subtle, it's not going to happen very often, but it can, according to older versions of the JVM Spec), the problem is that the time required to do that initialization is paid for *regardless of whether you use PI*. In other words, the classic Stroustrup-ian "Don't pay for it if you don't use it" principle is being completely tossed aside.

In Scala, using the "def" keyword here, aside from avoiding the need for the additional decorators, completely eliminates this cost--people won't need the value of PI until it becomes used:

            object App

{

def PI = calculatePiToARidiculousNumberOfPlaces()

def calculatePiToARidiculousNumberOfPlaces() =

{

System.out.println("Calculating
PI")

3 + 0.14

}

def main(args : Array[String]) =

{

System.out.println("Entering
main")

System.out.println("PI
= " + PI)

}

} 

(In fact, you'd probably just write it without the calculating method definition, since it's easier that way, but bear with me.)

When you run this, of course, we see PI being calculated after main()'s been entered, thus proving that PI is being calculated only on demand, not ahead of time, as a public-static-final-constant would be.

The problem with this approach is, you end up calculating PI on each access:

            object App

{

def PI = calculatePiToARidiculousNumberOfPlaces()

def calculatePiToARidiculousNumberOfPlaces() =

{

System.out.println("Calculating
PI")

3 + 0.14

}

def main(args : Array[String]) =

{

System.out.println("Entering
main")

System.out.println("PI
= " + PI)

System.out.println("PI
= " + PI)
// prints twice! Not good!

}

} 

Which sort of defeats the advantage of lazy evaluation.

This got me wondering--in F#, we have lazy as a baked-in concept (sort of), such that when I write

#light

let sixty = lazy (30 + 30)

System.Console.WriteLine(sixty) 

What I see on the console is not 60, but a Lazy<T> type instance, which effectively defers execution until it's Force() method is invoked (among other scenarios). This means I can write things like

let reallyBigList = lazy ([1..1000000000000] |> complexCalculation |> anotherComplexCalcuation)

without fear of blowing the stack or heap apart, since laziness means the list won't actually be calculated until it's forced; we can see this from the following (from the F# interactive console):

> let sixtyWithSideEffect = lazy (printfn "Hello
world"; 30+30);;

val sixtyWithSideEffect: Lazy<int>

> sixtyWithSideEffect.Force();;

Hello world

val it : int = 60

> sixtyWithSideEffect.Force();;

val it : int = 60 

(Examples taken from the excellent Expert F# by Syme/Granicz/Cisternino; highly recommended, if a touch out-of-date to the current language definition. I expect Chris Smith’s Programming F#, from O’Reilly, to correct that before too long.)

It would be nice if something similar were doable in Scala. Of course, once I start looking for it, it makes itself visible, in the wonderful Venners/Odersky/Spoon book, Programming In Scala, p. 444:

You can use pre-initialized fields to simulate precisely the initialization behavior
of class constructor arguments. Sometimes, however, you might prefer
to let the system itself sort out how things should be initialized. This can
be achieved by making your val definitions lazy. If you prefix a val definition
with a lazy modifier, the initializing expression on the right-hand side
will only be evaluated the first time the val is used.

[...]

This is similar to the situation where x is defined as a parameterless
method, using a def. However, unlike a def a lazy val is never evaluated
more than once. In fact, after the first evaluation of a lazy val the result of the
evaluation is stored, to be reused when the same val is used subsequently.

Perfect! The key, then, is to define PI like so:

            object App

{

lazy val PI = calculatePiToARidiculousNumberOfPlaces()

def calculatePiToARidiculousNumberOfPlaces() =

{

System.out.println("Calculating
PI")

3 + 0.14

}

def main(args : Array[String]) =

{

System.out.println("Entering
main")

System.out.println("PI
= " + PI)

System.out.println("PI
= " + PI)
// prints once! Awesome!

}

} 

That means, if I apply it to my Stack example from before, I should get the same deferred-execution properties of the "def"-based version ...

def main(args : Array[String]) =

{

import scala.collection.mutable.Stack

var stack : Stack[Any] = new Stack()

stack.push(12)

stack.push(24)

lazy val v1 = (stack.pop()).asInstanceOf[Int]

lazy val v2 = (stack.pop()).asInstanceOf[Int]

System.out.println(stack)
// prints out "Stack(12,24)

} 

... but if I go back to the version that blows up because the stack is empty, using lazy val works exactly the way I would want it to:

def main(args : Array[String]) =

{

import scala.collection.mutable.Stack

var stack : Stack[Any] = new Stack()

stack.push(12)

stack.push(24)

lazy val v1 = (stack.pop()).asInstanceOf[Int]

lazy val v2 = (stack.pop()).asInstanceOf[Int]

System.out.println(v1)

System.out.println(v2)

lazy val vr = v1 + v2

System.out.println(vr)
// prints 12, 24, then 36
// and no exception!

} 

Nice.

So, it turns out that my accidental use of "def" inside the compareOp function behaves exactly the way the language designers wanted it to, which is not surprising, and that Scala provides nifty abilities to defer processing or extraction of values until called for.

Curiously, the two languages differ in how laziness is implemented; in F#, the lazy modifier defines the type to be a Lazy<T> instance, an ordinary type that we can pass around from F# to C# and back again as necessary (in much the same way that C# defined nullable types to be instances of Nullable<T> under the hood). We can see that from the interactive console output above, and from the fact that we call Force() on the instance to evaluate its value.

In Scala, however, there is no corresponding Lazy[T] type; instead, the PI() method is defined to determine whether or not the value has already been evaluated:

            public
            double PI();

Code:

0: aload_0

1: getfield #135; //Field bitmap$0:I

4: iconst_1

5: iand

6: iconst_0

7: if_icmpne 48

10: aload_0

11: dup

12: astore_1

13: monitorenter

14: aload_0

15: getfield #135; //Field bitmap$0:I

18: iconst_1

19: iand

20: iconst_0

21: if_icmpne 42

24: aload_0

25: aload_0

26: invokevirtual #137; //Method calculatePiToARidiculousNumberOfPlaces:()D

29: putfield #139; //Field PI:D

32: aload_0

33: aload_0

34: getfield #135; //Field bitmap$0:I

37: iconst_1

38: ior

39: putfield #135; //Field bitmap$0:I

42: getstatic #145; //Field scala/runtime/BoxedUnit.UNIT:Lscala/runtime/BoxedUnit;

45: pop

46: aload_1

47: monitorexit

48: aload_0

49: getfield #139; //Field PI:D

52: dreturn

53: aload_1

54: monitorexit

55: athrow

Exception table:

from to target type

14 48 53 any

If you look carefully at the bytecode, the implementation of PI is checking a bitmask field (!) to determine if the first bit is flipped (!) to know whether or not the value is held in the local field PI, and if not, calculate it and store it there. This means that Java developers will just need to call PI() over and over again, rather than have to know that the instance is actually a Lazy[T] on which they need to call Value or Force (such as one would from C# in the F# case). Frankly, I don’t know at this point which approach I prefer, but I’m slightly leaning towards the Scala version for now. (If only Java supported properties, then the syntax “MyMath.PI” would look like a constant, act lazily, and everything would be great.)

(It strikes me that the F# developer looking to write something C#-accessible need only tuck the Lazy<T> instance behind a property accessor and the problem goes away, by the way; it would just be nicer to not have to do anything special on either side, to have my laziness and Force() it, too. Pipe dream, perhaps.)

In retrospect, I could wish that Scala weren't *quite* so subtle in its treatment of "def" vs "val", but now that I'm aware of it, it'll (hopefully) not bite me quite so subtly in the sensitive spots of my anatomy again.

And any experience in which you learn something is a good one, right?

An interesting blog post was forwarded to me by another of my fellow ThoughtWorkers, which suggests a new software stack for building an enterprise system, acronymized as “JOSH”:

The Book Of JOSH

Through a marvelous, even devious, set of circumstances, I'm presented with the opportunity to address my little problem without proscribed constraints, a true green field opportunity.

Json OSGi Scala HTTP

Json delivers on what XML promised. Simple to understand, effective data markup accessible and usable by human and computer alike. Serialization/Deserialization is on par with or faster then XML, Thrift and Protocol Buffers. Sure I'm losing XSD Schema type checking, SOAP and WS-* standardization. I'm taking that trade.

OSGi a standardized dynamic, modular framework for versioned components and services. Pick a logger component, a HTTP server component, a ??? component, add your own internal components and you have a dedicated application solution. Micro deployment with true replacement. What am I giving up? The monolithic J2EE application servlet loaded with 25 frameworks, SCA and XML configuration hell. Taking the trade.

HTTP is simple, effective, fast enough, and widely supported. I'm tired of needlessly complex and endless proprietary protocols to move simple data from A to B with all the accompanying firewall port insanity. Yes, HTTP is not perfect. But I'm taking this trade where I can as well.

All interfaces will be simple REST inspired APIs based on HTTP+JSON. This is an immediate consequence of the JOSH stack.

Scala is by far the toughest, yet the easiest selection in the JOSH stack. I wrestled far more with the JSON or XML or Thrift or Protocol Buffers decision.

And, let’s be honest, the stack sounds a lot better than what he was working with before....

[...] Yes, you see, I have a small problem.

So whats the issue, you say? I write a whole blog about nothing, you say? We all know the right answer, you're pointing out? Yea, I know, its intuitively obvious to the casual observer.

We'll rewrite it from scratch.

Course we'll need a cluster of WebSphere Application Servers, and an Oracle RAC cluster for all that data. Don't forget the middleware needed to transition over from the legacy systems, so toss in an ESB cluster, and what heck a couple of BPEL servers too.

Need a SOA Center of Excellence of course too. Can't integrate without some common XML Business Object Schemas. Also need to roll the Rational RUP suite and some beefy IDE environments and for that shiny look, sprinkle the works with lots of WS-* sparkly dust. Bake 3-5 years or until done, whenever.

My presentation slides for all this will be killer. I can sell this stuff. I'm good at it. I'll look like a bloody genius. I'll have Vendors fawning all over me. And the best part is the bubble on this mess won't pop for YEARS, when I'll have plenty of plausible deniability. "Hey the plan was perfect, the business, IT managers and their people were incapable of executing it."

I feel like the enterprise IT equivalent of an AIG trader pocketing ill gotten gains from writing Credit Default Swaps that we can't pay off.

Ewww... even thinking about all that makes me want to go upstairs, step into the shower, turn the water as hot as it will go, and wash. Scrub my skin raw with soap and sponge until the top five layers of epidermis are gone, and still not feel clean.

On the surface of things, the stack sounds pretty good. OSGi is a pretty solid spec for managing versioning and modularity within a running Java system, and more importantly, it’s well-known, relatively reliable, and pretty well-proven to handle the classic problems well. And of course, anybody who knows me knows that I’m a fan of the Scala language as a potential complement or supplement to the Java programming language, so that’s hardly in debate.

But there are a few concerns. JSON is a simple wire protocol, granted, but that is both a good thing and a bad thing (it’s object-centric, for one, and will run into some of the same issues as objects do with certain relationships), and it lacks the ubiquity that XML provides. Granted, XML clearly suffered from an overabundance of adoption, but it still doesn’t take away the fact that ubiquity is really necessary if you’re building a baseline for something that will talk to a variety of different systems. Which, I admit, may not be in his list of requirements, I don’t know. And HTTP is great for long-haul, client-initiated communication, but it definitely has its limitations (which he acknowledges, openly, to his credit), at least to internal-facing consumers. There is no peer for external-facing consumers, that’s a given.

And the stack is clearly also missing something else...

The JOSH stack is lacking a letter, because a solution for persisted data is missing in the stack.

A great deal of what needs to be done does not require a ACID RDB cluster. Some of it does and I'm kicking that can down the road.

For the rest, either the data is ReadOnly and loaded a 1-3 times a day or is best persisted by a distributed Key-Value storage system. A number of these are now available as open source solutions and at the right moment I'll need to pick one and add that letter to the JOSH stack.

As a commenter suggested, CouchDB might be a solution here, or I’ll even throw db4o into the ring for discussion as an option. Again, it’ll depend on how far-and-wide the data will be seen by other systems—the more other systems need to see it, the less further away from a “regular” RDBMS we can go.

Certainly, it’s a great start for discussion, even if the acronym is likely to give those named Joshua an unhealthy ego boost. :-)

Part of me wonders, though... what would the equivalent on .NET look like? JSON + Assemblies + F# + HTTP = JAFH?