Stu demonstrates one of the basic problems with an all-dynamic language: "I just spent an hour figuring out why some carefully-tested code went no-op after adding RSpec to a project." As much as I berate Stu at times (both in person and in blog), the fact is, I deeply respect and admire his programming skill, and if he can lose an hour to something that (I submit for your consideration) could have been caught by a static analysis tool fairly easily, then clearly that was a wasted hour of Stu's life. Worse, the problem is not yet solved, since now he has to make a hard choice about which definition to use, or else find a way to hack around the two definitions and create a third. Or perhaps something even uglier than this....

And this presumes that all developers using Ruby will have Stu's skill and his sense of responsibility when coming up with the solution. Asking that of all programmers across the globe is simply too much.

But clearly we cannot simply abandon the power of the dynamic language, either. Quoting again from the same source, Stu points out the very reason why dynamic languages are so powerful: "Once you start treating code as data, the elegance of your code is dependent on your skill. You cannot hide behind the limitations of your programming language anymore, because there aren't any."

What's a language designer left to do?

Choose both, of course.

The more I think about it, the more I think Cobra (and other languages) has it right: a programming language should have both static and dynamic features within it, simultaneously. This is the first "modern" language I've seen come along that espouses the "static when you can, dynamic when you want" principle as a first-class concept. Even at that, I imagine that there's much more that could be done than what Cobra espouses. Imagine combining the power of Scala's type inferencing system with the flexibility of a Groovy or Ruby.

Shivers.

During the Lang.NET Symposium, a couple of things "clicked" all simultaneously, giving me one of those "Oh, I get it now" moments that just doesn't want to leave you alone.

During the Intentional Software presentation, as the demo wound onwards I (and the rest of the small group gathered there) found myself looking at the same source code, but presented in a variety of new ways, some of which appealed to me as the programmer, others of which appealed to the mathematicians in the room, others of which appealed to the non-programmers in the room. (I heard one of the Microsoft hosts, a non-technical program manager, I think, say, "Wow, even I could understand that spreadsheet view, and that was writing code?")

During the spreadsheet-written-in-IronPython presentation (ResolverOne), we were essentially looking at new ways of writing IronPython code, thus leveraging all the syntactic power of a programming language with a nicer front end.

During the aspect-oriented talk (the one by Stefan Wenig and Fabian Schmeid), we found ourselves looking at a tool that essentially takes compiled assemblies and weaves in additional code based on descriptors from outside that codebase; in essence, just another aspect-oriented tool.

But combine this with my own investigations into Soot, LLVM, Parrot, and Phoenix, alongside the usual discussions around the DLR, CLR, JVM and DaVinci machine, couple that with the presentation Harry gave about parser expression grammars and the research in the functional community into parser combinators, throw in the aspect-oriented and metaprogramming facilities that the Rubyists and other dynamic linguists go on for days about, and what do you end up with?

Folks, the future is in modular toolchains.

This is an oversimplification, and a radical oversimplification at that, but imagine for a moment:

1. A parser takes your source code (let's assume it is Java, just for grins) and builds an AST out of it. Not an AST that's inherently deeply coupled to the Java language, mind you, but a general-purpose one that stands as a union of Java, C#, C++, Perl, Python, Smalltalk, and other languages. (Note that some of the linguistic concepts in some of those languages may not end up in this AST, but instead operate on the AST itself, a la C++'s template facilities.) Said parser is now finished, and can either output a binary (or potentially XML, though it'd probably be hideously verbose) version of this AST to disk for later consumption, or would more than likely be passed directly along to the next beast in the chain.
2. In the simplest scenario, the next beast would be a code generator, which takes the AST and seeks to export some kind of back-end code out of it. Here, since we're working with a general-purpose AST, we can assume that this back-end is flexible and open, a la the Phoenix toolkit (where either native or MSIL can be generated).
3. In a slightly more complicated scenario, verification of the correctness of the AST (against whatever libraries are specified) is checked, usually prior to code-gen, thus making this particular toolchain a statically-checked chain; were verification left out, it would need to happen at runtime, in which case we'd be talking about a dynamically-checked chain.

   Note that I stay away from the term "statically-typed" or "dynamically-typed" for the moment. That would be a measurement of the parser, not the verifier. Verification still occurs in a lot of these dynamically-typed languages, just as it does in statically-typed languages.

   Assuming the verification process succeeds, the AST can be again, written out or passed to the next step in the chain.
4. Another potential step in the process, usually post-parser and pre-verification, would be an "aspect" step, in which a tool takes the AST, consults some external descriptors, and modifies the AST based on what it finds there. (This is how most of your non-AspectJ-like AOP tools work today, except that they have to rebuild the AST from compiled .class files or assemblies first.)
5. Naturally, another step in the process would be an optimize step, but this has to be considered carefully, since some "high-level" optimizations can be done without regard to code-gen backend, and some will need to be done with regard to code-gen backend; for example, register spill is (from what I've heard, can't say I know too much about this) generally only useful if you know how many registers you're targeting. Plus, it's not hard to imagine certain optimizations that are only generally useful on the x86 architecture, versus those that are useful on other CPU platforms. Even operating systems I would imagine would have an impact here. (It turns out that many compiler toolchains go through a dozen or so optimization steps today, so it's not hard to imagine a "code-gen backend" being a series of a half-dozen or so targeted optimization steps before actually generating code.)
6. Bear in mind, too, that these ASTs should have enough information to be directly executable, thus giving us an interpreter back-end instead of a code-generation back-end, a la the DLR instead of the CLR.
7. Also, given the standard AST format, it would be relatively trivial to create a whole series of different "parser"s to get to the AST, along the lines of what the Intentional Software guys have created, thus blowing open the whole concept of "DSL" into areas that heretofore have only been imagined. You still get the complete support of the rest of the toolchain, which is what makes the whole DSL concept viable in the first place, including aspects and verification and your choice of either interpretation or compilation.
8. While we're at it, bear in mind that this AST could/should also be reachable from within the code itself, thus giving languages that want to operate on their own AST at runtime the ability to do so, because the AST is in a standard format and the interpreter could be bundled as part of the generated executable, thus providing a compile-when-you-can-interpret-when-you-must flavor that is currently the reigning meme in language/platform environments like JRuby. (It would also have the happy side effect of making Paul Graham shut up about Lisp, at least for a while. Yes, Paul, code-as-data, it's brilliant, it's wonderful, we get it.)
9. Nothing says this toolchain needs be one-way, by the way: many of the toolkits I mentioned before (LLVM, Phoenix, Soot) can start from compiled binary and work back to AST, thus offering us the opportunity to do surgery of either the exploratory kind (static analysis) or the manipulative kind (aspect-weaving, etc) on compiled code in a relatively clean way. Reflector demonstrates the power of being able to go "back and forth" in this way (even in the relatively limited way Reflector does so), so imagine how powerful it would be to do this from end-to-end throughout the toolchain.

How likely is this utopian vision? I'm not sure, honestly--certainly tools like LLVM and Phoenix seem to imply that there's ways to represent code across languages in a fairly generic form, but clearly there's much more work to be done, starting with this notion of the "uber-AST" that I've been so casually tossing around without definition. Every AST is more or less tied to the language it is supposed to represent, and there's clearly no way to imagine an AST that could represent every language ever invented. Just imagine trying to create an AST that could incorporate Java, COBOL and Brainf*ck, for example. But if we can get to a relatively stable 80/20, where we manage to represent the most-commonly-used 80% of languages within this AST (such as an AST that can incorporate Java, C#, and C++, for starters), then maybe there's enough of a critical mass there to move forward.

Now all I need to do is find somebody who'll fund this little bit of research... anybody got a pile of cash they don't know what to do with?

Update: By the way, in case you want a graphical depiction of what I'm thinking about, the Phoenix page has one (though obviously it's limited to the Phoenix scope of vision, and you may have to be a Microsoft CONNECT member to see it).

Long-time readers of this blog know that as a general rule, I try not to include much in the way of personal stuff here; I try (sometimes with more success than others) to keep the subject material focused on the technology space: Java, .NET, Ruby, languages, XML services, and so on.

This, however, is a deviation from that norm.

A near and dear friend of mine has asked that I help spread the word about the disappearance of a family member (a cousin, in fact). I don't know the details of the disappearance other than what anybody else can read on the website, but I do know that if someone in my family were to go missing for an inexplicable reason, I would want the help of anybody and everybody I knew to try and find them.

I would like to ask everyone's help in finding my brother John. He went missing January 28, 2008 and official search efforts were called off last Friday, even though the family has mounted their own search.   Please go to www.findjohnglasgow.com to see a picture of John and print off a flyer.   If you could put it in your car window or some other visible place, it would help us a lot.   It is possible that he could have traveled out of the area where he went missing, so we are trying to get the word out on a national level, to cover all possible scenarios.   Thank you all for your help.

Donna Jean Glasgow

February 6, 2008

If you reside near the Little Rock, Arkansas area in particular, please take a look at the photo below ...

... and let somebody (either me or through the above-mentioned website) know if you've seen him, one way or another.

I won't ask you to forward this to everyone you know; instead I just ask that if you feel a twinge of sympathy for a missing family member that's connected to you through less than two degrees of separation, then do what you think would help.

Thanks for your time.

Wow. Giants 17, Patriots 14, when just about everybody had the Patriots by two touchdowns or so.

Just goes to show, shouldn't count the little guy out 'til the fat lady sings and the cows come home.

Also just goes to show, I shouldn't be blogging after an emotional heart-jerker like that one.

OK, after a week of getting the Internet equivalent of Bad Mojo being sent my way by every Perl developer on the planet, I have to admit something that may strike readers as inconsistent and incongruous.

I want Parrot to work.

I don't really care about Perl 6, per se. As I've said before, the language has a lot of linguistic inconsistencies and too many violations of the the Principle of Least Surprise to carry a lot of favor with me. Whether Perl-the-language lives or dies really doesn't make a significant dent in my life.

But Parrot.... now there's something I care about.

Following the open debate on Perl (a surprising side-effect, given the subject matter of the post that spawned it), and chromatic's insistence that Parrot development was moving along, I decided to give in to my secret hopes, and pull the Parrot bits down again for a look-see.

In the spirit of the OpenJDK post last month, this is a quick chronicle of how I got Parrot to build on a Win32 system.

Installation details

Just for the record, I'm doing this in a VMWare image (one in which I keep all the languages I play with) with both Visual Studio 2008 and Visual Studio 2005 installed. The Parrot docs explicitly reference using Visual Studio 2003 (or the free Visual C++ Toolkit, which has since turned into Visual C++ 2005 Express), but I'm going to first have a shot at it with VS 2008 before falling back to VS 2005. This shouldn't make any difference, because 2008 is supposed to be a superset of 2005, but... well, you know how that old chestnut goes.

svn co parrot

Checking Parrot's code out is easy: just svn co https://svn.perl.org/parrot/trunk parrot-svn . (I use the -svn suffix on directories to distinguish between svn-pulled source trees and downloaded source trees. Helps in case I ever need/want to pull down a named release and keep the svn-pulled source at the same time.) I pull all this into a directory underneath C:\Prg, so the total path to Parrot's source base is C:\Prg\parrot-svn.

Configure

From there, as with many Unix-based projects, you have to run the "Configure.pl" script. I opened up a VS 2008 Command Prompt, and used ActiveState's Perl [1] to run the Configure script. It chugs away and comes back with this message:

C:\Prg\parrot-svn>perl Configure.pl
Parrot Version 0.5.2 Configure 2.0
Copyright (C) 2001-2008, The Perl Foundation.

Hello, I'm Configure. My job is to poke and prod your system to figure out
how to build Parrot. The process is completely automated, unless you passed in
the `--ask' flag on the command line, in which case I'll prompt you for a few
pieces of info.

Since you're running this program, you obviously have Perl 5--I'll be pulling
some defaults from its configuration.

Checking MANIFEST.....................................................done.
Setting up Configure's default values.................................done.
Setting up installation paths.........................................done.
Tweaking settings for miniparrot...................................skipped.
Loading platform and local hints files................................done.
Finding header files distributed with Parrot..........................done.
Determining what C compiler and linker to use.........................done.
Determining whether make is installed..................................yes.
Determining whether lex is installed...............................skipped.
Determining whether yacc is installed..............................skipped.
Determining if your C compiler is actually gcc..........................no.
Determining whether libc has the backtrace* functions (glibc only)......no.
Determining Fink location on Darwin................................skipped.
Determining if your C compiler is actually Visual C++..................yes.
Detecting compiler attributes (-DHASATTRIBUTE_xxx)....................done.
Detecting supported compiler warnings (-Wxxx)......................skipped.
Enabling optimization...................................................no.
Determining flags for building shared libraries.......................done.
Determine if parrot should be linked against a shared library..........yes.
Determining what charset files should be compiled in..................done.
Determining what encoding files should be compiled in.................done.
Determining what types Parrot should use..............................done.
Determining what opcode files should be compiled in...................done.
Determining what pmc files should be compiled in......................done.
Determining your minimum pointer alignment......................... 1 byte.
Probing for C headers.................................................done.
Determining some sizes................................................done.
Computing native byteorder for Parrot's wordsize.............little-endian.
Test the type of va_ptr (this test is likely to segfault)............stack.
Figuring out how to pack() Parrot's types.............................done.
Figuring out what formats should be used for sprintf..................done.
Determining if your C library has a working S_ISREG.....................no.
Determining CPU architecture and OS...................................done.
Determining architecture, OS and JIT capability.......................done.
Generating CPU specific stuff.........................................done.
Verifying that the compiler supports function pointer casts............yes.
Determining whether your compiler supports computed goto................no.
Determining if your compiler supports inline...........................yes.
Determining what allocator to use.....................................done.
Determining if your C library supports memalign.........................no.
Determining some signal stuff.........................................done.
Determining whether there is socklen_t..................................no.
Determining if your C library has setenv / unsetenv...............unsetenv.
Determining if your platform supports AIO...............................no.
Determining if your platform supports GMP...............................no.
Determining if your platform supports readline..........................no.
Determining if your platform supports gdbm..............................no.
Testing snprintf......................................................done.
Determining whether perldoc is installed...............................yes.
Determining whether python is installed.........................yes, 2.5.1.
Determining whether GNU m4 is installed................................yes.
Determining whether (exuberant) ctags is installed......................no.
Determining Parrot's revision.......................................r25452.
Determining whether ICU is installed................................failed.
Generating C headers..................................................done.
Generating core pmc list..............................................done.
Generating runtime/parrot/include.....................................done.
Configuring languages.................................................done.
Generating makefiles and other build files............................done.
Moving platform files into place......................................done.
Recording configuration data for later retrieval......................done.
Okay, we're done!

You can now use `nmake' to build your Parrot.
After that, you can use `nmake test' to run the test suite.

Happy Hacking,
        The Parrot Team

C:\Prg\parrot-svn>

Looks good so far. I kick off nmake (which is still running as I write this). Note that the Configure script discovers ActiveState's Perl as part of its rummaging around on my system, so that's what it uses to do the build steps that require execution of Perl. I have no idea what the least-acceptable version of AS Perl is, but the version I pulled down was probably about a year ago.

(Note: I have to admit, the Configure stuff is slick. I don't like opening those files and looking at what's in there, but you'll never hear me criticize the existence of Perl, for this reason alone: having a scripting language that can rummage around your machine and figure out the paths to all the cr*p it needs to build is a hideously useful thing. I do admit to wishing those scripts were written in something I feel better about reading, though, like Ruby, but this is a practice that far pre-dates me, so I'll just shut up and ride along because I find it useful when it works. As it does here.)

Note to the Parrot guys: under VS 2008, the build generates a ton of warnings. Most of all, VS 2008 complains about the use of the Wp64 flag, which it says is deprecated and will be removed in a future release. (Chromatic, if you want a full build log, I can clean-and-build again and send you the piped output, if it'll help.)

After about 10 minutes of disk churn and a ton of warnings reported (most of which seem to be just three or four warnings being repeated throughout the code, so either it's something in a couple of headers files that're included from everywhere, or these are spurious warnings that could be turned off via a #pragma)... success! I have a parrot.exe, along with a few other .exe utilities, in the root of the parrot-svn directory.

Next step: "nmake test".

Well, clearly parrot must be working pretty well, because it's churning through a ton of tests with "ok" results for everything except that which is platform-specific (a la the Fink tests intended for Darwin/Mac OS X, which are obviously going to fail on my XP box and therefore get skipped). A couple of tests get skipped (in the compilers tree?) with explanations that I don't quite understand, but it doesn't look like these are errors, per se, so I'm willing to accept on faith that we're all kosher. So while the tests are still running, I'll post this and offer up kudos to chromatic and the crew for something that at least builds, runs, and passes a whole slew of unit tests. Now for the fun part--finding out how extensive PMC, PIR and PASM are, and thinking about how this VM fits in the Grand Scheme of Things against the Da Vinci Machine and the DLR and the JVM and the CLR....

(Note to self: must suggest to John Lam and the guys on the DLR team to invite chromatic up to the Lang.NET 2009 Symposium. If the Sun folks can be made to feel welcome on the Microsoft campus for this kind of event, then surely the Parrot guys can come and feel welcome and--hopefully--carry away some interesting ideas, too.)

Update: Well, might have spoken too soon, looks like the tests failed after all. To be exact, the tests hung for a while, and I Ctrl-C'ed the process because it didn't look like it was going anywhere; this is the last few lines:

t/library/cgi_query_hash.....................ok
t/library/coroutine..........................ok
t/library/data_escape........................ok
        1/22 skipped: test not written
t/library/dumper.............................ok
t/library/File_Spec..........................ok
t/library/getopt_obj.........................ok
t/library/iter...............................ok
t/library/md5................................ok
t/library/mime_base64........................ok
t/library/parrotlib..........................ok
t/library/pcre...............................
t/library/pcre...............................NOK 1#     Failed test (t/library/p
cre.t at line 48)
# Exited with error code: 1
# Received:
# ok 1
# ok 2
# Null PMC access in invoke()
# current instr.: 'parrot;PCRE;compile' pc 118 (C:\Prg\parrot-svn\runtime\parrot
\library\pcre.pir:127)
# called from Sub 'main' pc 83 (C:\Prg\parrot-svn\t\library\pcre_1.pir:49)
#
# Expected:
# ok 1
# ok 2
# ok 3
# ok 4
# ok 5
#
# Looks like you failed 1 test of 1.
t/library/pcre...............................dubious
        Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 1
        Failed 1/1 tests, 0.00% okay
t/library/pg.................................Terminating on signal SIGINT(2)
NMAKE : fatal error U1077: NMAKE : fatal error U1058: terminated by user
Stop.

C:\Prg\parrot-svn>

Not sure what this means, but bear in mind, this is off today's tip, so it may be a temporary thing.

[1] Why, you may ask, do I have Active State's Perl installed if I so despise the language? Rotor (SSCLI 2.0) uses it as part of its build process, and I like spelunking with Rotor, as some of you will have noticed.

Normally, I like to stay out of these kinds of wars, but this post by Stu (whom I deeply respect and consider a friend, though he may not reciprocate by the time I'm done here) just really irked me somewhere sensitive. I'm not entirely sure why, but something about it just... rubbed me the wrong way, I guess is the best way to say it.

Let's dissect, shall we?

Stu begins with the following two candidates:

1. Joe has a problem to solve. The problem is specific, the need is immediate, and the scope is well-contrained.
2. Jane has a problem to solve. The problem is poorly understood, the need is ongoing, and the scope is ambiguous.

For starters, Joe doesn't exist. Or rather, exists only in the theoretical. Of course, neither does Jane really exist, either. Fact is, almost all projects are a combination of Joe and Jane. More importantly, Stu's efforts here to force people into the "either/or" approach to categorization is a subtle (or perhaps not so) ploy to force people into the decision-making path he thinks should be taken.

It's sort of like saying, most people fall into two categories:

1. Joe lives in Ghettopia, where all the men are dumb, the women are ugly, and the children are rejects from the ADHD Clinic.
2. Jane lives in Utopia, where all the men are smart, the woman are good-looking, and the children are well-behaved.

Think about it: you're at work, you have a project, and you happen across Stu's page. Faced with the typical project (too little time, too few resources, too vague in the understanding of requirements and domain comprehension), with whom are you likely to identify? Disturblingly happy Joe, who has a specific problem in a well-constrained scope? Hardly. So from the beginning, you're expected to identify with Jane, which (not surprisingly) leads you into Stu's preferred conclusion.

He goes on:

How should Joe and Jane think differently about software platforms?

   1. Joe's platform needs to be mainstream. It needs to offer immediate productivity, and the toolset should closely match the problem. Also, Joe doesn't want to climb a learning curve.
   2. Jane's needs are quite the opposite. Jane needs flexibility. She needs glue that doesn't set. She needs a way to control technical debt (Joe doesn't care.)

For my part, I am interested in Jane's problems. (And anyway, Joe often discovers he is actually Jane midway through projects.)

Hey, Stu, quick reality check for ya: most developers want all of the above. It's not a binary choice, productivity and toolset vs. flexibility and dynamism. The fact is, the Java language has a degree of flexibility, just not as much as is offered by the Ruby language. For that matter, if you want real flexibility, maybe you oughta look into Lisp, or even Smalltalk, since it (ST) can get at the underlying stack frames from the ST language itself! Now that's flexibility you Ruby guys can only dream of. (Oh, I know, Rubinius will give you that flexibility. Someday. Justin even alludes to how Rubinius is essentially an attempt to recapture that dynamism from Smalltalk. Ironic, then, isn't it, that the guys who wrote the fastest Smalltalk VM on the planet (Strongtalk, which is open-source now, by the way) ended up working at Sun... on the thing that later came to be called Hotspot? You think maybe they have a little familiarity and experience with VMs?)

And that crack about "control technical debt (Joe doesn't care)"?

Bullshit.

Let me repeat that in case you missed it: BULL-SHIT.

Joe and Jane both care about technical debt. Each may be willing to spend their currency on different problems, granted, but both of them care about technical debt. Not caring about technical debt is what got Chandler into trouble, and it had nothing to do with language or tools whatsoever. It's insulting to suggest that either of them don't care about technical debt, particularly the guy that chooses differently than you.

(Shame on you, Stu. You know better. Quit trolling.)

We continue:

So how does this affect platform choice? If you are Joe, you care about specific details about what a toolset can do right now. Most of Graeme's Top 10 reasons are in the "Right here, right now" category. This is true regardless of whether you think he is right. (Sometimes he is, sometimes not.)

I'll grant you, some of Graeme's Top 10 reasons are a bit spurious, and Stu-and-company do a good job of pointing those out. Frankly, anybody who makes a technical selection based on version numbers or whether or not a book exists for it seems to be missing the point, if you ask me. Of far greater concern is the stability of the language/tool, or the wealth of documentation for it. (And yes, this may seem to fly in the face of my arguments against Parrot a few posts ago; actually, it's not. If Parrot were more stable and/or more fully fleshed out, and the version updates just kept going, I'd be happy to say, "Go get this thing and give it a spin". But it doesn't feel stable to me, so I can't.)

But Stu's argument here is spurious: I don't care if you're Joe or if you're Jane, you always care about specific details about what a toolset can do, right now or otherwise. Certain concerns may be concerns that you can put off until later, but those concerns are always a part of the platform selection. Consider a hypothetical for a second: you currently are developing on Windows, and your project will run on Windows servers, with a possibility that it may need to run on non-Windows servers at some point in the future. Do you consider .NET or not? This is exactly the kind of detail that needs to be discussed--how likely is the move to a non-Windows server going to be? If it's <25%, then the CLR and ASP.NET might be a good choice, particularly if your developers are less "plumbing wonk" than "GUI designer", and you rely on being able to move the assemblies to a non-Windows server later via Mono.

Note: I'm not suggesting this a good choice in all scenarios. I'm making the point that the details of the toolset matter in your choice of toolsets, based on what your particular project needs are.

Jane cares just as much about toolset details as Joe does. I can't imagine a scenario where either of them don't care.

To continue:

My advice to Joe: Know exactly what you need, and then pick the platform that comes closest to solving it out of the box. Depending on Joe's needs, either Rails or Grails might be appropriate (or neither!). A particular point in Grails' favor would be an established team of Spring ninjas.

"Know exactly what you need"? Ah, right, because Joe belongs to that .01% of projects that have "specific problems, immediate need, and well-constrained scope". Nothing like conceding a point to the other guys, in preparation for the "killer blow":

If you are Jane, you care more about architecture. I mean this term in two senses:

   1. Architecture: the decisions you cannot unmake easily.
   2. Architecture: the constraints on how you think and work.

If you are Jane, you care about how and why the platform was assembled, because you are likely to have to adapt it quite a bit.

You know, I don't think I've ever been on a project where I didn't care about architecture or in having to "adapt it quite a bit". Of course, back in the days when I was writing C++, this meant either subclassing CWnd or TWindow in interesting ways, or else sometimes even going so far as to reach into the source code and making some tweaks, either at compile-time or through some well-established hackery. (Yes, I wrote a template class called THackOMatic that allowed me to bang away on private fields. Sue me. It worked, I documented the hell out of it, and ripped the hack back out once the bug was fixed.) Point is, both Joe and Jane care about the architecture.

Now, I think what Stu means here is that the architecture of the web framework is more malleable in Rails than it is in Grails, because Rails is written on top of Ruby and Grails is written on top of Groovy, Spring, the JEE container architecture, and Java:

Most of the commenters on my earlier post (and Graeme in his addendum) correctly identified the real architectural difference between Grails and Rails. Rails builds on Ruby, while Grails builds on Groovy and Spring.

Yes! I agree with this so far. (In fact, everybody should, because these are simple statements of fact.) But then Stu takes the cake for the Best Parting Non-Supported Shot Ever:

Rails wins this architecture bakeoff twice:

    * Ruby is a better language than Groovy.
    * Spring does most of its heavy lifting in the stable layer, which is not the right place.

Huh?

Ruby is perhaps a more flexible language than Groovy (and that's an arguable point, folks, and one which I really don't care to get into), but Ruby also runs on a less-flexible and less-scalable and less-supported platform than Groovy. I dunno that this makes Ruby better. It simply makes it different. Try convincing your IT guys to add yet another platform into their already-overwhelmingly complex suite of tools, particularly given the surprisingly sparse amount of monitoring information that Ruby platform offers. Stu may want to argue that Ruby-the-language is more flexible, regardless of what platform it runs on, and if so, then we're arguing languages not platforms, and while he might win much of his "Ruby is a better language than Groovy" argument, he's going to lose the "Ruby is more dynamic than Groovy", because on the JVM they have to be implemented under the same set of restrictions. You can't have it both ways.

(By the way, if you're one of those Ruby/Rails enthusiasts who's going to counterclaim that "Ruby-meaning-MRV is fast enough", I've heard the argument, and I think it's specious and ignorant. "Fast enough" is an argument that rests on your project being able to remain within the expected performance and scalability curve known at the beginning of the project, and remember, Jane's problem is that she doesn't know those sorts of things yet. So either you know, and have some better scope around the problem than Stu gives credit to Jane for having, or else you don't know, and can't assume that the Ruby interpreter will be able to handle the load.)

And WTF is up with the idea that "Spring does most of its heavy lifting in the stable layer, which is not the right place"? I think Stu means to say that Spring is a static layer, not stable layer[1], because hey, stability is kinda important to a few folks. (I'll give Stu the benefit of the doubt here and assume he cares about stability, too. I know his customers do.) Spring has its flaws, mind you, but arguing that it's not up to the heavy lifting seems to be like arguing that Java cannot scale. (Even Microsoft has given up on that argument, by the way.)

The worst part of this is, I've had discussions like this with Stu in the past, and he's much more articulate about it in person than he is in this blog post. Frankly, I think the most interesting space here is the intersection of Graeme's and Stu's positions, which is to say JRuby (and IronRuby or Ruby.NET, but that's for a different platform and out of the scope of this discussion entirely... yet still compelling and relevant, strangely enough). At the end of the day, these arguments about "my web framework is better than your web framework" are really just stupid. (As long as you're not trying to claim that Perl is the best web framework, anyway. Yes, Perl enthusiasts, I'm picking on you.)

My advice to Jane: Rails over Grails.

My advice to Jane: pick a consulting firm that doesn't have preconceived dogma about which web framework... or language, or any other toolset... to use. [2]

And if Jane can't afford a consulting firm, then Jane needs to do the research on her own and make her own decision based on the problem set, the context, and the whole range of tools available to her. (Anybody making a decision based solely on the basis of a blog-post-flame-war deserves what they get, regardless.)

As for Joe? Well, Joe could probably benefit from the goodness inherent in the dynamic languages that are popping up all over the place, too, not to mention the goodness inherent in the type-inferred languages that are starting to poke their heads through the Barrier of Adoption, all the while not ignoring the fact that he could probably benefit from the inherent performance and scalability of the major virtual machine technologies that have been a decade or more in production...

Meaning Joe probably needs to go through the same decision-making criteria Jane does. Thank God both of them, it turned out, work on the same project, as is often the case.

Meanwhile, I'm done with this thread. It's a pointless, stupid argument. Use the right tool for the job. Or, if you prefer, "From each language, according to its abilities, to each project, according to its needs."

Just remember that both shipping and supporting are features, too. Don't neglect the other in favor of the one.

[1] Yes, I saw the hyperlink to Ola's post about languages, and his definitions therein. Ironically, Ola's own comments there state that "Java is really the only choice here", which directly contradicts Stu's choice of MRV (the native Ruby interpreter). More importantly, I think Stu's point is resting on the static nature of the Java layer in Groovy, and while it's certainly more flexible to be able to hack at any layer of the stack, this is only realistically possible in small applications--this isn't my opinion, it's the opinion of Gregor Kiczales, who spent many years in CLOS and determined that CLOS's extremely flexible MOP system (more so than what Ruby currently supports, in fact) led to inherent problems in larger-scale projects. It was this thought that led him to create AspectJ in the first place.

[2] By the way, if there's any temptation in you[3] to post commentary and say, "Dude, you just don't understand Ruby" or "How can you agree with Graeme this way?", just don't. I do understand Ruby, and I like the language. (Much more than I do Rails, anyway.) And I'm not intrinsically agreeing that Grails is better than Rails, because I don't believe that, either. I believe in the basic equation that says the solution you pick is the one that is the right solution to the given problem in the stated context that yields the most desirable consequences.

[3] This includes you, Stu. Or Justin, or Graeme, or anybody working for Relevance, or anybody working for G2One, Inc.

... here. (Yes, it's an MSDN web page, but the article itself--as have all of its brethren in the series--is actually quite technology-neutral.) Enjoy and flame away....

ROFL!

Update: Apparenty, this post (and two more referencing it) pushed me to #'s 1-4 on the "perl lover" Google list, out of 250,000. That is just so wrong, on so many levels....

My Mac froze when I tried to hook it up to the projector in the afternoon to do a 15-minute chat on Scala, thus losing the running blog entry in its entirety. Crap. This is my attempt to piece this overview together from memory--accordingly, details may suffer. Check the videos for verification when they come out. Of course, details were never my long suit anyway, so you probably want to do that for all of these posts, come to think of it...

I got to the conference about a half-hour late, owing to some personal errands in the morning; as I got there, Wayne Kelly was talking about his work on the Ruby.NET compiler.

Wayne Kelly: Parsing Ruby is made much harder by the fact that there is no Ruby specification to work from, which means the parser can't easily be generated from a parser generator. He tried, but couldn't get it to work cleanly and finally gave up in favor of getting to the "fun stuff" of code generation. Fortunately, the work he spent on the parser was generalized into the Gardens Point Parser Generator tools, which are also used in other environments and are included (?) as part of the Visual Studio SDK download. Good stuff. Ruby.NET uses a "wrapper" class around the .NET type that contains a hash of all the symbols for that type, which permits them to avoid even constructing (or even knowing!) the actual .NET type behind the scenes, except in certain scenarios where they have to know ahead of time. Interesting trick--probably could be used to great effect in a JSR-223 engine. (I know Rhino makes use of something similar, though I don't think they defer construction of the Java object behind the Rhino object.)

In general, I'm hearing this meme that "Ruby's lack of a specification is making X so much harder". I hate to draw the parallel, but it's highly reminiscent of the state of Perl until Perl 6, when Larry decided it was finally time to write a language specification (and the language has languished ever since), but maybe it's time for Matz or another Ruby digerati to sit down and write a formal specification for the Ruby language. Or even just its grammar.

Luke Hoban: Luke is the PM on the F# team, which is a language that I've recently been spending some quality time with, so I'm looking forward to this talk and how he presents the language. (Note to self: steal slides... I mean leverage slides... for my own future presentations on F#.) Not surprisingly, he makes pretty heavy use of the F# Interactive window in Visual Studio, using a trick I hadn't known before this: swipe some text in the editor, then press Alt-Enter, and it sends it to the Interactive window for execution. Nifty.

Then he starts showing off F#'s fidelity to the underlying CLR, and just for effect creates a DirectX surface and starts graphing functions on it. Then he starts playing with the functions, while the graph is still up, which has the neat effect of changing the function's graph in the DirectX surface without any explicit additional coding. Then he tightens up the mesh of the graph, and adds animation. (Mind you, these are all one-to-four lines of F# at a time he's pasting into the Interactive window.) What gets even more fun is when he pastes in a page and a half more of F# code that introduces balls rolling on the graphed surface. Very nifty. Makes Excel's graphing capabilities just look silly by comparison, in terms of "approachability" by programmers.

I will say, though, that I think that the decision to use significant whitespace in F# the same way Python does is a mistake. We don't have to go back to the semicolon everywhere, but surely there has to be A Better Way than significant whitespace.

Harry Pierson: Harry works in MS IT, so he doesn't play with languages on a regular basis, but he likes to explore, and recently has been exploring Parser Expression Grammars, which purport to be an easier way to write parsers based on an existing grammar. He shows off some code he wrote in F# by hand to do this (a port of the original Haskell code from the PEG paper), then shows the version that Don (Syme) sent back, which made use of active patterns in F#. (Check out Don's Expert F# for details.)

Harry predicated this talk with his experience talking with the creators of Glassbox (a C#-based tool that wanted to do something similar to what the C# mixins guys were doing from yesterday), and when he heard how much pain they were going through taking the Mono C# compiler and hacking it to introduce their extensions, he realized that compilers needed to be more modular. I had an interesting thought on this today, which I'll talk about below.

Magnus ???: Again, this was a lightning talk, a quick-hit lecture on the tool that his company is building, and I can't tell if the name of the tool was Intentional Software, or the name of the company was Intentional Software, or both. It's a derivative of what Charles Simonyi was working on at Microsoft (Intentional Programming), and basically they're creating programming language source trees in various ways while preserving the contents of the tree. So, for example, he takes some sample code (looked like C#, I don't think he said exactly what it was--assume some random C-family language), and presto, the curly braces are now in K&R style instead of where they belong (on new lines). Yawn. Then he presses another button, and suddenly the mathematical expressions are using traditional math "one over x" (with the horizontal line, a la MathML-described output) instead of "one slash x". That got a few peoples' attention. As did the next button-press, which essentially transformed whole equations in code into their mathematical equivalents. Then, he promptly button-presses again, and now the if/else constructs that are part of the equation are displayed inside the equation as "when"/"otherwise" clauses. Another button press, and suddenly we have a Lisp-like expression tree of the same function. Another button press, and we have a circuit diagram of the same function.

Wow. I'm floored. With this, completely non-programmer types can write/edit/test code, with full fidelity back to the original textual source. And, in fact, he promptly demonstrates that, with a table-driven representation of some business rules for a Dutch bank. It's a frickin' spreadsheet we're looking at, yet underneath (as he shows us once or twice), it's live and it's really code.

Combine this with some unit tests, and you have a real user-friendly programming environment, one that makes Rails look amateurish by comparison.

Now, if this stuff actually ships.... but this talk leads me to some deeper insight in conjunction with Harry's comments, which I'll go into below.

Wesner Moise: Wesner presents his product, NStatic, which is a static analysis tool that scans .NET assemblies for violations and bugs, much in the same way that FindBugs does in the Java space. It operates on binary assemblies (I think), rather than on source files the way FxCop does (I think), and it has a very spiffy GUI to help present the results. It also offers a sort of "live" view of your code, but I can't be certain of how it works because despite the fact that he takes the time to fire it up, he doesn't actually walk us through using it. (Wesner, if you read this, this is a HUGE mistake. Your slides should be wrapped around a demo, not the other way around. In fact, I'd suggest strongly ditching the slides altogether and just bring up an assembly and display the results.)

As readers of this blog (column?) will know, I'm a big fan of static analysis tools because I think they have the advantageous properties of being "always on" and, generally, "extensible to include new checks". Compilers fall into the first category, but not the second, in addition to being pretty weak in terms of the checks they do perform--given the exposure we're getting to functional languages and type inferencing, this should change pretty dramatically in the next five years. But in the meantime, I'm curious to start experimenting with the AbsIL toolkit (from MS Research) and F# or a rules engine (either a Prolog variant or something similar) to do some of my own tests against assemblies.

Unfortunately, it's a commercial product, so I don't think source will be available (in case you were wondering).

Chuck ...: Chuck stands up and does a quick-hit lecture on his programming language, CORBA... er, sorry about that, flashback from a bad acid trip. I mean of course, the language Cobra, which according to his blog he was working on at the Lang.NET 2006 Symposium. It's a Python derivative (ugh, more significant whitespace) with some interesting features, including a combination of static and dynamic typing, contracts, a "squeaky-clean" syntax, first-class support for unit tests (directly in the method definition!), and uses source-to-source "compilation", in this case from Cobra to C#, rather than compilation directly to IL.

It's a fascinating little piece of work, and I'm planning on playing with it some.

Miguel de Icaza: Miguel is another one of those who has more energy than any human being should have right to, and he spends the entire talk in fast-forward mode, speaking at a rapid-fire pace. He talks first of all about some experiences with Mono and the toolchain, then gets around to the history of Moonlight ("Can you give us a demo in 3 weeks?") and their (Mono's/Novell's) plans to get Moonlight out the door. They're already an impressive amount of the way there, but they have to make use of a "no-release" codecs library that also (potentially) contains some copywrit stuff, so they're instead going to incorporate Microsoft codecs, which they have rights to thanks to the Microsoft/Novell agreement of n months ago.

The thought of all these Linux devs running Microsoft code in their browser as they work with Moonlight just tickles my demented funny bone to no end.

He then switches tacks, and moves into gaming, because apparently a number of game companies are approaching Novell about using Mono for their gaming scripting engine. (Apparently it is being adopted by SecondLife, but the demo tanks because the SecondLife servers aren't up, apparently. That, or the Microsoft firewall is doing its job.) He jumps into some discussion about UnityScript, a {ECMA/Java}Script-like language for a game engine (called Unity, I think) that Rodrigo (creator of Boo) was able to build a parser for (in Boo) in 20 hours.

He then demonstrates the power of game engines and game editors by giving a short demo of the level editor for the game. He modifies the Robot bad guys to shoot each other instead of the player. If you're a game modder, this is old hat. If you're a business programmer, this is wildly interesting, probably because now you have visions of pasting your boss' face on the robots as you blast them.

Aaron Marten and Carl Brochu: I think his co-presenter's name was Carl something, but memory fails me, sorry. These two are from the Visual Studio Ecosystem team (which I think gets the prize for strangest product team name, ever), and they're here to give an overview of the Visual Studio integration API and tooling, with some sample code around how to plug into VS. This is good, because not an hour or two before, during Chuck's Cobra talk, he was talking about wanting to integrate into VS as a formal "next steps" for his language. Frankly, the whole area of IDE integration Dark Art to most folks (ranking behind custom languages, but still high up there), and the more the VSX team can do to dispel that myth, the more we'll start to see interesting and useful plugins for VS a la what we see in the Eclipse space. (Actually, let's hope the plugins we see for VS work more than a quarter of the time--Eclipse has become the dumping ground for every programmer who had an idea for a plugin, created a space on Sourceforge, wrote twenty lines of code, then got stuck and went away, leaving a nifty idea statement and a plugin that crashes Eclipse when you fire it up, not that I'm bitter or anything.)

The code demo they show off is a RegEx language colorization sample, nothing too terribly useful but still a nice small example of how to do it in VS. As VS starts to put more and more of a managed layer into place inside of VS, this sort of thing should become easier and easier, and thus a lot more approachable to the Average Mortal.

Me: I did a 15-minute presentation on Scala, since the name had come up a few times during the week, and promptly watched in horror as hooking my Mac up to the overhead projector locked the Mac completely. Ugh. Hard reboot. Ugh. Shuffle and dance about the history of Scala while waiting for the Mac to reboot and the VMWare image in which I have Scala installed to reboot. Ugh. I have no prepared slides, so I open up a random Scala example and start talking briefly about the syntax of a language whose list of features alone is so long it would take all fifteen minutes just to read aloud, much less try to explain. Cap it off with a leading question from Don Box ("Is this Sun's attempt to catch up to the C# compiler, given that Java is 'done' like the Patriots or the Dolphins?") that I try to answer as honestly and truthfully as possible, and a second question from Don (again) that forcefully reminds me that I'm out of time despite the "5 min" and "1 min" signs being held up by the guy next to him ("What would you say, in the two minutes you have left to you, is the main reason people should look at Scala?"), and I can safely say that I was thoroughly disgusted with myself at presenting what had to be the crappiest talk a the conference. *sigh*

That's it, no more presentations on technical topics, ever.

OK, not really, but a man can dream....

Don Box and Chris Andersen: I had to leave about ten minutes into their talk, so I still have no idea what Don and he are working on deep inside their incubating little cells in Microsoft. Something to do with "modeling and languages", and something that seeks to bring data to the forefront instead of code. *shrug* Not sure what to make of it, but I'm sure the video will make it more clear.

Meanwhile...

Overall: Here are some thoughts I think I think:

• A blog is not a part of your presentation, and your presentation is not part of your blog. I find it frustrating when speakers say, in their presentation, "Oh, you can find Y on my blog" and don't go into any more detail about it. I don't want to have to go look up your blog after the talk, when the context of the question or situation is swapped out of memory, and I don't want to have to go look it up during your presentation and miss whatever follows in your talk. If you blogged it, you should be able to give me a 30-second summary about the blog entry or what not, enough to tell me whether or not I want the deeper details of what's on your blog. Exception: files that contain examples of a concept you're discussing or sample code or whatnot.
• Don't hook your Mac up to the projector when you have a VMWare session on an external USB disk running. This happened to me at Seattle Code Camp, too, with the same result: Mac lockup. Dunno what the deal is, but from now on, the rule is, connect thy Mac, then fire up thy suspended VMWare VM.
• Language design and implementation is a lot more approachable now than it was even five years ago. Don't assume, for even a second, that the only way to go building a "DSL" or "little language" is by way of Rake and Rails--it's still a fair amount of work to build a non-trivial language, but between parser combinators and toolkits like the DLR and Phoenix, I'd go head-to-head against a Ruby-based DSL development process any day of the week.
• Don't go in front of Don Box at a conference. Dude may like to go long on his own talks, but man, he watches the clock like a hawk when it's time for him to start. (I may sound like I'm angry at Don--I'm not--but I'm not going to resist a chance to poke at him, either. *grin*)
• Modular tool chains are the future. Actually, this is a longish idea, so I will defer that for a future post.
• This conference rocks. It's not the largest conference, you get zero swag, and the room is a touch crowded at times, but man, this little get-together has one of the highest signal-to-noise ratio of any get-together I've been to, and without a doubt, within the realm of languages and language design, this is where the Cool Kids like to hang out.

Bye for now, and thanks for listening....

Andrew Wild emails me:

I vaguely remember one of your blog posts in which you went into a bit of an exposition of 'context'.
Did you ever come up with anything solid or did you wind up talking yourself in self-referential circles?

Because that post was actually a part of the old weblog hosted at neward.net, I decided to repost it and the followup discussion to this blog in order to make it available again, although the WayBack Machine also has it and its followup tucked away.

Context

I'm not normally one to promote myself as a "pattern miner"--those who "discover" patterns in the software systems around us--since I don't think I have that much experience yet, but one particular design approach, "patlet", if you will, has been showing up with frightening regularity (such as Sandy Khaund's mention of EDRA, the format of a SOAP 1.2 message, which in itself forms a Context, and more), and yet hasn't, to my knowledge, been documented anywhere, that I thought I'd take a stab at documenting it and see what comes out of it. Treat this as a alpha, at best, and be brutal in your feedback.

Context (Object Behavioral)

Define a wrapper object that encapsulates all aspects of an operation, including details that may not be directly related to that operation. Context allows an object or graph of objects to be handled in a single logical unit, as part of a logical unit of work.

Motivation

Frequently an operation, which consists fundamentally of inputs and a generated output, requires additional information by which to carry out its work. In some cases, this consists of out-of-band information, such as historical data, previous values, or quality-of-service data, which needs to travel with the operation regardless of its execution path within the system. The desire is to decouple the various participants working with the operation from having to know everything that is being "carried around" as part of the operation.

In many cases, a Context will be what is passed around between the various actors in a Chain of Responsibility (223).

Consequences

I'm not sure yet.

Known Uses

Several distributed communication toolkits make use of Context or something very similar to it. COM+, for example, uses the notion of Context as a interception barrier, allowing for a tightly-coupled graph of objects to be treated as an atomic unit, synchronizing multi-threaded calls into the context, also called an apartment. Transactions are traced as they flow through different parts of the system, such that each Context knows the transaction ID it was associated with and can allow that same transaction ID (the causality) to continue to flow through, thus avoiding self-deadlock.

Web Services also make use of Context, using the SOAP Message format as a mechanism in which out-of-band information, such as security headers and addressing information, can be conveyed without "polluting" the data stored in the message itself. WS-Security, WS-Transaction, WS-Routing, among others, are all examples of specifications that simply add headers to SOAP messages, so that other "processing nodes" in the Web service call chain can provide the appropriate semantics.

(I know there are others, but nothing's coming to mind at the moment.)

Related Patterns

Context is often the object passed along a Chain of Responsibility; each ConcreteHandler in the Chain examines the Context and potentially modifies it as necessary before handing it along to the next Handler in the Chain.

Context is also used as a wrapper for a Command object, providing additional information beyond the Command itself. The key difference between the two is that Context provides out-of-band information that the Command object may not even know is there, for processing by others around the Command.

The followup looked like this:

Wow--lots of you have posted comments about Context. Let's address some of them and see what comes up in the second iteration:

• Michael Earls wrote:

  Very timely. I'm building a system right now that fits this pattern. We spent about five minutes determining what to call "it" (the little "black box" that holds the core command, entity, and metadata information). We settled on "nugget". Now there's prior art I can refer to. I'm using Context with WSE 2.0 and SOAP extensions for the pipeline in exactly the way you describe. Nice.

  and

  Another Related Pattern: Additionally, the Context may also be an container/extension/augmentation/decoration on the UnitOfWork (???).

  I suspect you're right--Context can be used to hold the information surrounding a UnitOfWork, including the transaction it's bound to (if the transaction is already opened). This is somewhat similar to what the MTS implementation does, if I'm not mistaken.

• Kris wrote:

  The HTTP pipeline in ASP.NET comes to mind, with the HttpContext being passed through for various things like session state, security, etc. One possible side effect that I can see (hopefully you can drop some thoughts on this one), is how to manage dependencies between the chain, as well as order of invocation of chain elements. The MS PAG stuff I believe talks about this somewhat with the Pipelines & Filters pattern, but I'd love to hear your thoughts as well.

  The PAG stuff (Sandy Khaund's post) was part of what triggered this post in the first place, but I want to be careful not to rely too much on Microsoft prior art (WSE, Shadowfax, HttpContext, COM/MTS/COM+) since in many cases those systems were designed by people who had worked together before and/or