Recently, while engaging in my other passion (international relations), I was reading the latest issue of Foreign Affairs, and ran across an interesting essay regarding the increasing outsourcing--or, the term they introduce which I prefer in this case, "offshoring"--of technical work, and I found some interesting analysis there that I think solidifies why I think programmers shouldn't fear offshoring, but instead embrace it and ride the wave to a better life for both us and consumers. Permit me to explain.

The essay, entitled "Offshoring: The Next Industrial Revolution?" (by Alan S. Blinder), opens with an interesting point, made subtly, that offshoring (or "offshore outsourcing"), is really a natural economic consequence:

In February 2004, when N. Gregory Mankiw, a Harvard professor then serving as chairman of the White House Council of Economic Advisers, caused a national uproar with a "textbook" statement about trade, economists rushed to his defense. Mankiw was commenting on the phenomenon that has been clumsily dubbed "offshoring" (or "offshore outsourcing")--the migration of jobs, but not the people who perform them, from rich countries to poor ones. Offshoring, Mankiw said, is only "the latest manifestation of the gains from trade that economists have talked about at least since Adam Smith. ... More things are tradable than were tradable in the past, and that's a good thing." Although Democratic and Republican politicians alike excoriated Mankiw for his callous attitude toward American jobs, economists lined up to support his claim that offshoring is simply international business as usual.

Their economics were basically sound: the well-known principle of comparative advantage implies that trade in new kinds of products will bring overall improvements in productivity and well-being. But Mankiw and his defenders underestimated both the importance of offshoring and its disruptive effect on wealthy countries. Sometimes a quantitative change is so large that it brings qualitative changes, as offshoring likely will. We have so far barely seen the tip of the offshoring iceberg, the eventual dimensions of which may be staggering.

To be sure, the furor over Mankiw's remark was grotesquely out of proportion to the current importance of offshoring, which is still largely a prospective phenomenon. Although there are no reliable national data, fragmentary studies indicate that well under a million service-sector jobs have been lost to offshoring to date. (A million seems impressive, but in the gigantic and rapidly churning U.S. labor market, a million jobs is less than two weeks' worth of normal gross job losses.)¹ However, constant improvements in technology and global communications will bring much more offshoring of "impersonal services"--that is, services that can be delivered electronically over long distances, with little or no degradation in quality.

That said, we should not view the coming wave of offshoring as an impending catastrophe. Nor should we try to stop it. The normal gains from trade mean that the world as a whole cannot lose from increases in productivity, and the United States and other industrial countries have not only weathered but also benefited from comparable changes in the past. But in order to do so again, the governments and societies of the developed world must face up to the massive, complex, and multifaceted challenges that offshoring will bring. National data systems, trade policies, educational systems, social welfare programs, and politics must all adapt to new realities. Unfortunately, none of this is happening now.

Why Offshoring

Blinder first describes the basics of "comparative advantage" and why it's important in this context:

Countries trade with one another for the same reasons that individuals, businesses and regions do: to exploit their comparative advantages. Some advantages are "natural": Texas and Saudi Arabia sit atop massive deposits of oil that are entirely lacking in New York and Japan, and nature has conspired to make Hawaii a more attractive tourist destination than Greenland. Ther eis not much anyone can do about such natural advantages.

But in modern economics, nature's whimsy is far less important than it was in the past. Today, much comparative advantage derives from human effort rather than natural conditions. The concentration of computer companies around Silicon Valley, for example, has nothing to do with bountiful natural deposits of silicon; it has to do with Xerox's fabled Palo Alto Research Center, the proximity of Stanford University, and the arrival of two young men named Hewlett and Packard. Silicon Valley could have sprouted up anywhere.

One important aspect of this modern reality is that patterns of man-made comparative advantage can and do change over time. The economist Jagdish Bhagwait has labeled this phenomenon "kaleidoscopic comparative advantage", and it is critical to understanding offshoring. Once upon a time, the United Kingdom had a comparative advantage in textile manufacturing. Then that advantage shifted to New England, and so jobs were moved from the United Kingdom to the United States.² Then the comparative advantage in textile manufacturing shifted once again--this time to the Carolinas--and jobs migrated south within the United States.³ Now the comparative advantage in textile manufacturing resides in China and other low-wage countries, and what many are wont to call "American jobs" have been moved there as a result.

Of course, not everything can be traded across long distances. At any point in time, the available technology--especially in transportation and communications⁴--largely determines what can be traded internationally and what cannot. Economic theorists accordingly divide the world's goods and services into two bins: tradable and non-tradable. Traditionally, any item that could be put in a box and shipped (roughly, manufactured goods) was considered tradable, and anything that could not be put into a box (such as services) or was too heavy to ship (such as houses) was thought of as nontradable. But because technology is always improving and transportation is becoming cheaper and easier, the boundary between what is tradable and what is not is constantly shifting. And unlike comparative advantage, this change is not kaleidoscopic; it moves in only one direction, with more and more items becoming tradable.

The old assumption that if you cannot put it in a box, you cannot trade it is thus hopelessly obsolete. Because packets of digitized information play the role that boxes used to play, many more services are now tradable and many more will surely become so. In the future, and to a great extent already, the key distinction will no longer be between things that can be put in a box and things that cannot. Rather, it will be between services that can be delivered electronically and those that cannot.

Then came the second Industrial Revolution, and jobs shifted once again--this time away from manufacturing and toward services. The shift to services is still viewed with alarm in the United States and other rich countries, where people bemoan rather than welcome the resulting loss of manufacturing jobs⁵. But in reality, new service-sector jobs have been created far more rapidly than old manufacturing jobs have disappeared. In 1960, about 35 percent of nonagricultural workers in the United States produced goods and 65 percent produced services. By 2004, only about one-sixth of the United States' nonagricultural jobs were in goods-producing industries, while five-sixths produced services. This trend is worldwide and continuing.

We are now i nthe arly stages of a third Industrial Revolution--the information age. The cheap and easy flow of information around the globe has vastly expanded the scope of tradable services, and there is much more to come. Industrial revolutions are big deals. And just like the previous two, the third Industrial Revolution will require vast and usettling adjustments in the way Americans and residents of other developed countries work, live, and educate their children.

So what does this mean?

But now we start to come to the interesting part of the article.

But a bit of historical perspective should help temper fears of offshoring. The first Industrial Revolution did not spell the end of agriculture, or even the end of food production, in the United States. It jus tmean that a much smaller percentage of Americans had to work on farms to feed the population. (By charming historical coincidence, the actual number of Americans working on farms today--around 2 million--is about what it was in 1810.) The main reason for this shift was not foreign trade, but soaring farm productivity. And most important, the massive movement of labor off the farms did not result in mass unemployment. Rather, it led to a large-scale reallocation of labor to factories.

Therein lies the flaw in the argument: the amount of productivity required to achieve the desired ends is constant in the agriculture industry, yet a constantly-changing dynamic value in software. This is also known as what I will posit as the Groves-Gates Maxim: "What Andy Groves giveth, Bill Gates taketh away."

The Groves-Gates Maxim

The argument here is simple: the process of growing food is a pretty constant one: put seed in ground, wait until it comes up, then harvest the results and wait until next year to start again. Although we might have numerous tools that can help make it easier to put seeds into the ground, or harvesting the results, or even helping to increase the yield of the crop when it comes up, the basic amount of productivity required is pretty much constant. (My cousin, the FFA Farmer of the Year from some years back and a seed hybrid researcher in Iowa might disagree with me, mind you.) Compare this with the software industry: the basic differences between what's an acceptable application to our users today, compared to even ten years ago, is an order of magnitude different. Gains in productivity have not yielded the ability to build applications faster and faster, but instead have created a situation where users and managers ask more of us with each successive application.

The Groves-Gates Maxim is an example of that: every time Intel (where Andy Groves is CEO) releases new hardware that accelerates the power and potential of what the "average" computer (meaning, priced at somewhere between $1500-$2000) is capable of, it seems that Microsoft (Mr. Gates' little firm) releases a new version of Windows that sucks up that power by providing a spiffier user interface and "eye-candy" features, be they useful/important or not. In other words, the more the hardware creates possibilities, the more software is created to exploit and explore those possibilities. The additional productivity is spent not in reducing the time required to produce the thing desired (food in the case of agriculture, an operating system or other non-trivial program in the case of software), but in the expansion of the functionality of the product.

This basic fact, the Groves-Gates Maxim, is what saves us from the bloody axe of forced migration. Because what's expected of software is constantly on the same meteoric rise as what productivity gains provide us, the need for programmer time remains pretty close to constant. Now, once the desire for exponentially complicated features starts to level off, the exponentially increasing gains in productivity will have the same effect as they did in the agricultural industry, and we will start seeing a migration of programmers into other, "personal service" industries (which are hard to offshore, as opposed to "impersonal service" industries which can be easily shipped overseas).

Implications

What does this mean for programmers? For starters, as Dave Thomas has already frequently pointed out on NFJS panels, programmers need to start finding ways to make their service a "personal service" position rather than an "impersonal service" one. Blinder points out that the services industry is facing a split down the middle along this distinction, and it's not necessarily a high-paying vs low-paying divide:

Many people blithely assume that the critical labor-market distinction is, and will remain, between highly educated (or highly-skilled) people and less-educated (or less-skilled) people--doctors versus call-center operators, for example. The supposed remedy for the rich countries, accordingly, is more education and a general "upskilling" of the work force. But this view may be mistaken. Other things being equal, education and skills are, of course, good things; education yields higher returns in advanced societies, and more schooling probably makes workers more flexible and more adaptable to change. But the problem with relying on education as the remedy for potential job losses is that "other things" are not remotely close to equal. The critical divide in the future may instead be between those types are work that are easily deliverable through a wire (or via wireless connections) with little or no diminution in quality and those that are not. And this unconventional divide does not correspond well to traditional distinctions between jobs that require high levels of education and jobs that do not.

A few disparate examples will illustrate just how complex--or, rather, how untraditional--the new divide is. It is unlikely that the services of either taxi drivers or airline pilots will ever be delivered electronically over long distances. The first is a "bad job" with negligible educational requirements; the second is quite the reverse. On the other hand, typing services (a low-skill job) and security analysis (a high-skill job) are already being delivered electronically from India--albeit on a small scale so far. Most physicians need not fear that their jobs will be moved offshore, but radiologists are beginning to see this happening already. Police officers will not be replaced by electronic monitoring, but some security guards will be. Janitors and crane operators are probably immune to foreign competition; accountants and computer programmers are not. In short, the dividing line between the jobs that produce services that are suitable for electronic delivery (and are thus threatened by offshoring) and those that do not does not correspond to traditional distinctions between high-end and low-end work.

There is currently not even a vocabulary, much less any systematic data, to help society come to grips with the coming labor-market reality. So here is some suggested nomenclature. Service that cannot be delivered electronically, or that are notably inferior when so delivered, have one essential characteristic: personal, face-to-face contact is either imperative or highly desirable. Think of hte waiter who serves you dinner, the doctor you gives you your annual physical, or the cop on the beat. Now think of any of those tasks being performed by robots controlled from India--not quite the same. But such face-to-face human contact is not necessary in the relationship you have with the telephone operator who arranges your conference call or the clerk who takes your airline reservation over the phone. He or she may be in India already.

The first group of tasks can be called personally-delivered services, or simply personal services, and the second group of impersonally delivered services, or impersonal services. In the brave new world of globalized electronic commerce, impersonal services have more in common with manufactured goods that can be put in boxes than they do with personal services. Thus, many impersonal services are destined to become tradable and therefore vulnerable to offshoring. By contrast, most personal services have attributes that cannot be transmitted through a wire. Some require face-to-face contact (child care), some are inherently "high-risk" (nursing), some involve high levels of personal trust (psychotherapy), and some depend on location-specific attributes (lobbying).

• Face-to-face contact. Agile methodologies demand that customers be easily accessible in order to answer questions regarding implementation decisions or to resolve lack of understanding of the requirements. Instead of demanding customers be present at your site, you may find yourself in a better position if you put yourself close to your customers.
• "High-risk". This is a bit harder to do with software projects--either the project is inherently high-risk in its makeup (perhaps this is a mission-critical app that the company depends on, such as the e-commerce portal for an online e-tailer), or it's not. There's not much you can do to change this, unless you are politically savvy enough to "sell" your project to a group that would make it mission-critical.
• High levels of personal trust. This is actually easier than you might think--trust in this case refers not to the privileged nature of therapist-patient communication, but in the credibility the organization has in you to carry out the work required. One way to build this trust is to understand the business domain of the client, rather than remaining aloof and "staying focused on the technology". This trust-based approach is already present in a variety of forms outside our industry--regardless of the statistical ratings that might be available, most people find that they have a favorite auto repair mechanic or shop not for any quantitatively-measurable reason, but beceause the mechanic "understands" them somehow. The best customer-service shops understand this, and have done so for years. The restaurant that recognizes me as a regular after just a few visits and has my Diet Coke ready for me at my favorite table is far likelier to get my business on a regular basis than the one that never does. Learn your customers, learn their concerns, learn their business model and business plan, and get yourself into the habit of trying to predict what they might need next--not so you can build it already, but so that you can demonstrate to them that you understand them, and by extension, their needs.
• Location-specific attributes. Sometimes, the software being built is localized to a particular geographic area, and simply being in that same area can yield significant benefits, particularly when heroic efforts are called for. (It's very hard to flip the reset switch on a server in Indiana from a console in India, for example.)

But even if this analysis doesn't make you feel a little more comfortable, consider this: there are 1 billion people in China alone, and close to the same in India. Instead of seeing them as potential competition, imagine what happens when the wages from the offshored jobs start to create a demand for goods and services in those countries--if you think the software market in the U.S. was hot a decade ago, where only a half-billion (across both the U.S. and Europe) people were demanding software, now think about it when four times that many start looking for it.

Footnotes

¹ Which in of itself is an interesting statistic--it implies that offshoring is far less prevalent than some of people worried about it believe it to be, including me.

² Interesting bit of trivia--part of the reason that advantage shifted was because the US stole (yes, stole, as in industrial espionage, one of the first recorded cases of modern industrial espionage) the plans for modern textile machinery from the UK. Remember that, next time you get upset at China's rather loose grip of intellectual property law....

³ Which, by the way, was a large part of the reason we fought the Civil War (the "War Between the States" to some, or the "War of Northern Aggression" to others)--the Carolinas depended on slave labor to pick their cotton cheaply, and couldn't acquire Northern-made machinery cheaply to replace the slaves. Hence, for that (and a lot of other reasons), war followed.

⁴ An interesting argument--is there any real difference between transportation and communications? One ships "stuff", the other "data", beyond that, is there any difference?

⁵ And, I'd like to point out, the shrinking environmental damage that can arise from a manufacturing-based economy. Services rarely generate pollution, which is part of the clash between the industrialized "Western" nations and the developing "Southern" ones over environmental issues.

Resources

"Offshoring: The Next Industrial Revolutoin?", by Alan S. Blinder, Foreign Affairs (March/April 2006), pp 113 - 128.

Scott Hanselman has diabetes. So does a good friend of mine, who discovered it during our third year of college. Take a moment and spare US$20 to support Scott in his fight against it. Do it because you probably know somebody who has it, or will before long. If nothing else, do it because it's a cheap way to support somebody who's indirectly responsible for this blog (Scott maintains dasBlog, which is the blogging engine I use now).

Out.

CLASSPATH now supports wildcards to pick up multiple .jar files. Finally. But given that the AppClassLoader is a derivative of the standard UrlClassLoader, I still don't see why I can't put full URLs on the CLASSPATH and expect them to be resolved correctly....

Oh, and while we're at it, you shouldn't be using CLASSPATH-the-environment variable anymore, anyway. There's far too many ways to manage .jar file resolution to be falling back to that old hack. Prefer instead to put your .jar file dependencies inside the manifest of your application's .jar file, or at the very least, specify the classpath to the java launcher when you kick it off. If you're still doing "set CLASSPATH=..." at the command-line, you're about ten years behind the times.

Mustang, the latest JDK (to be called Java6 on its release), fixes a minor but very annoying nit that's bugged Java developers for years: "if a JDBC driver is packaged as a service, you can simply (leave out the call to Class.forName() to bootstrap the driver class into the JVM). ... The DriverManager code will look for implementations of java.sql.Driver in classpath and do Class.forName() implicitly." It's never been a huge deal in Java, to have to explicitly bootstrap the JDBC driver into the JVM before being able to obtain a Connection from it, but it's always been annoying, and inexplicable, given the Service Provider mechanism that's been there for a couple of releases now.

Next up: do the same for JNDI and XML parsers (and do away with the extensions directory while we're at it!), and let's start making all of these factories a bit more practical to use on a larger basis. It would be nice if this mechanism had percolated through other tools and areas, such as servlets, but it's probably too late to correct for that now.

From the "I'm not sure why you'd want to do this, but..." Department: How to install Windows XP onto your brand new Intel Mac Pro. (Given that so much of the Windows O/S relies on the right-mouse-button, and given that it'd be so much cheaper to buy a Dell or even a Thinkpad of equivalent power... why would you bother to buy a Mac Pro, then turn around and install--or even dual-boot install--XP on it? Maybe I'm just not "getting it" or something....)

Needless to say, despite the hefty geekiness factor inherent in the idea of doing .NET demos with a Mac Pro, I'll probably not be buying one any time soon... Instead, somebody point me to Tiger running in a VMWare image and then stand back.

Bruce Tate is at it again, this time writing for the Pragmatic Press, called "Java to Ruby", on... well, the title kinda says it all, on migrating from Java to Ruby. It's not just a "blind adoption" book, meaning it just blindly advocates the transition, but instead Bruce discusses the risks involved with making the switch, and how to justify it to upper management. Don't get me wrong, he's operating from the basic conclusion that you want to make the switch, so if you're not yet convinced that Ruby or Ruby-on-Rails is the way to go, you're not necessarily going to be any more convinced by this book. But if you are already convinced (and in Bruce's defense, there's lots of good reasons to be), this looks to be the book to help you convince others, most notably your management.

While training last week, the group I was training asked for some help in interviewing candidates for some openings. I came up with the following, and thought I'd post it in the interests of giving teams looking to hire some new folks. This was created specifically to find candidates with 2-3 years' experience with some familiarity with web applications.

• What was the most interesting computer/technical book you read last year? Why?
• What was the last new programming language you learned?
• How do you convince a customer that what they want is wrong?
• Describe the last elegant hack you wrote. Why, where, how, when, ...?
• Write a servlet that prints all parameters and HTTP headers to the browser. Build an Ant script that compiles, and creates the appropriate .war file for deployment into Tomcat.
• Name all of the verbs in the HTTP/1.1 specification.
• Given a tag library, "utilitytags.jar" which is described by the tag library descriptor file "utilitytags.tld", please write a JSP that uses the "foobar" tag from that library.
• What keyword in Java do we use to grant package-level access to class members?
• When do you think you should use an abstract base class as opposed to using an interface?
• How do I protect code in a multithreaded environment from being executed by more than one thread at a time?
• Please fill in the necessary code to return the contents of the "last name" column using the following SQL statement; make all necessary changes so that this code will compile:

  public void printUsers(Connection conn)
  {
    String sql = "SELECT last_name FROM users WHERE first_name = 'Sean'";
    ...
  }
  

• What is a design pattern? Please describe three patterns that you have used and why you used them and what the consequences of using them were, both good and bad.

Update: A couple of commenters have pointed out that a few of the questions are answered by simply looking up stuff (in the HTTP specification, for example). Two answers come to mind why I want people to know this without having to look it up--one, sometimes I want to pitch a slowball just to see how they'll react and answer, and two, if they can answer the question without having to look it up, it means they know the spec, which is a far, far different thing than being able to look something up.

Much of the power, it seems, of languages like Ruby or Nemerle or LISP derives from the ability to create chunks of code that operate in turn on the code itself; a long-standing meme of the LISP world is that "code is data", meaning it can be manipulated and twisted and tweaked in structural ways before being executed. And this isn't the first time we've experimented with this idea: CLOS, Common List Object System, was where Gregor Kiczales, of AspectJ fame, cut his teeth on the AOP concepts, largely because it seemed to him that having a completely open meta-object protocol was too dangerous--but that's another story.

So given that everybody's going ga-ga over Ruby's MOP/metaprogramming facilities, it occurs to me, is what we're really after an MOP for Java? Because such things exist already in the academic world--see OpenJava, for example. Is that what Java would need to do to evolve and take back the productivity label? Is the lack of productivity in Java (the chief complaint of Java today, according to Bruce Tate) due directly to its statically-typed nature, or is it simply the inability to twist the language in ways that are closer to what the programming staff really needs and wants? After all, if you take away some of the MOP features that Ruby uses to make the Person class pretty brief, such as the attr_reader and attr_writer manipulators, then a lot of Ruby's terseness goes away, too.

Not that functional languages aren't still a good way to go--and I will continue the discussion of Scala, largely because there's some nifty stuff there we haven't touched yet--but I'm becoming more and more convinced that the problem with Java isn't in its statically-typed nature, but in the language we use to generate bytecode for the platform. (.NET may have some better answers here, given Microsoft's greater friendliness to languages on their platform, but it's just another platform similar to the JVM in a lot of ways, and as such has the same drawbacks and benefits as Java does in this regard.)

So where are all the good macro-friendly tools/languages for Java? (And that means, "macros as from LISP", not "macros as from C". Frankly, C++ could use a good MOP, too, but that's another story for another day...)

"The request could not be completed because the service is too busy. Please try again later." (From MSMessenger, about thirty seconds ago.)

The new home page is alive and kicking...

My dad sent me this:

In today's world, we typically spend a good deal of time traveling, and with a lot of that travel by air, I thought you might enjoy the attached. Jerry Cosley is an acquaintance of mine, someone I worked with at TWA, and among his other positions he was the Staff Vice President of Public Relations. Now that TWA is gone, Jerry is involved with others in sifting through some of the memorabilia and historical items. He ran across this item, provided by Stout Air Services, a predecessor of TWA, and wanted to share it. In 1929, TWA was the hot ticket - you could get from New York to California in only 48 hours, flying by day and riding on the train through the night. #4 of course is difficult on today's airplanes. By way of comparison, in 1979 I flew in a 747 from LA to JFK in 3 hours, 58 minutes.

How to Get the Maximum Enjoyment Out of Flying

(These simple rules are provided by the Stout Air Services, Inc.)

1. DONT WORRY. Relax. Settle back and enjoy life. If theres any worrying to do let the pilot do it. Thats what hes paid for.
2. The pilot always takes off and lands into the wind. Be patient while the plane taxies to a corner of the field before taking off.
3. The pilot always banks the plane when turning in the air. Just as a race track is banked at the corners, so is an airplane tilted when making a perfect turn. Take the turns naturally with the plane. Dont try to hold up the lower wing with the muscles of your abdomen; its unfair to yourself and an unjust criticism of your pilot.
4. The atmosphere is like an ocean. It supports the plane just as firmly as the ocean supports a ship. At the speed you are traveling the air has a densitypractically equivalent to water. To satisfy yourself, put your hand out of the window and feel the tremendous pressure. That ever-present pressure is your guarantee of absolute safety.
5. The wind is similar to an ocean current. Once in a while the wind is gusty and rough, like the Gulf Stream off the coast of Florida. These gusts used to be called air pockets, but they are nothing more than billows of warm and cool air and are nothing to be alarmed over.
6. The air pressure changes with the altitude. Some people have ears that are sensitive to the slightest change in air density at different altitudes. If yours are, swallow once in a while, or breathe a little through the mouth. If you hold your nose and swallow, your will hear a little crack in your ears, caused by the suction of air on the ear drums. Try it.
7. Dizziness is unknown in airplanes. There is no discomfort in looking downward while flying, because there is no connection with the earth. Owing to the altitude you may think you are moving veryslowly, although the normal flying speed is above 105 miles an hour.
8. WHEN ABOUT TO LAND: The pilot throttles the engine preparatory to gliding down to the airport. The engine is not needed in landing and the plane can be landed perfectly with the engine entirely cut off. FROM AN ALTITUDE 0F 2,500 FEET IT IS POSSIBLE TO GLIDE WITH ENGINE STOPPED TO ANY FIELD WITHIN A RADIUS OF FOUR MILES!

In the Scala documentation, they make a point of calling out the idea that "everything's an object", including numbers and (most importantly) functions. Smalltalk had this same perception/assumption in its design, and Scala, as a result, sometimes feels very Smalltalk-ish.

For example, when Scala sees this expression:

2 + 4 * 7

2.+(4.*(7))

val nums = 1 :: 2 :: 3 :: 4 :: Nil;

val nums = 1.::(2.::(3.::(4.::(Nil))));

> val msg = "Hello World"
val msg: java.lang.String("Hello World") = Hello World
> msg equals "Hello World"
true: scala.Boolean

Thus far, this concept of everything being an object may not seem all that powerful; in fact, arguably, the above section should probably have been mentioned in the previous post than this one, since the ability to recognize new operators is itself an extension of the idea of expressing exactly what you want, nothing more. In fact, in the "Scala by Example" document that comes with the Scala download, they show a traditional Java (but which could easily be written to read in C++ or C#) quicksort implementation:

def sort(xs: Array[int]): unit = {
  def swap(i: int, j: int): unit = {
    val t = xs(i); xs(i) = xs(j); xs(j) = t;
  }
  def sort1(l: int, r: int): unit = {
    val pivot = xs((l + r) / 2);
    var i = l, j = r;
    while (i <= j) {
      while (xs(i) < pivot) { i = i + 1 }
      while (xs(j) > pivot) { j = j - 1 }
      if (i <= j) {
        swap(i, j);
        i = i + 1;
        j = j - 1;
      }
    }
    if (l < j) sort1(l, j);
    if (j < r) sort1(i, r);
  }
  sort1(0, xs.length - 1);
}

def sort(xs: List[int]): List[int] =
  if (xs.length <= 1) xs
  else {
    val pivot = xs(xs.length / 2);
    sort(xs.filter(x => x < pivot))
    ::: xs.filter(x => x == pivot)
    ::: sort(xs.filter(x => x > pivot))
  }

• If the list is empty or consists of a single element, it is already sorted, so return it immediately.
• If the list is not empty, pick an an element in the middle of it as a pivot.
• Partition the lists into two sub-lists containing elements that are less than, respectively greater than the pivot element, and a third list which contains elements equal to pivot.
• Sort the first two sub-lists by a recursive invocation of the sort function.
• The result is obtained by appending the three sub-lists together.

As hinted, functions are full objects in of their own right, and are just as easily accessible as parameters as any other object passed into a method. So, for example, consider the above sort implementation again. The only thing that really "ties" it to sorting lists of integers is the comparison that goes on to determine if the item inside the list is less-than, equal-to, or greater-than other elements in the list. If we could somehow genericize that decision-making, we could make the quicksort be entirely generic and applicable to lists-of-anything. (As it turns out, it's sometimes easier to do this by simply having any types that wish to be sorted implement the <, == and > methods in Scala, and this is possible to enforce via interfaces and mixins and such, but bear with me on this example.)

We'll start by making sort generic:

def sort(xs: List[T]): List[T] =
  if (xs.length <= 1) xs
  else {
    // ...
  }

def sort(xs: List[T]): List[T] =
  if (xs.length <= 1) xs
  else {
    val pivot = xs(xs.length / 2);
    sort(xs.filter(x => (x less-than pivot) )
    ::: xs.filter(x => (x equal-to pivot) )
    ::: sort(xs.filter(x => (x greather-than pivot) )
  }

def sort[T](xs: List[T], lt: (T, T) => boolean, 
            eq: (T, T) => boolean, gt: (T, T) => boolean) : List[T] =
  if (xs.length <= 1) xs
  else {
    val pivot = xs(xs.length / 2);
    sort(xs.filter(x => lt(x, pivot)), lt, eq, gt)
      ::: xs.filter(x => eq(x, pivot))
      ::: sort(xs.filter(x => gt(x, pivot)), lt, eq, gt)
  }

object App with Application {

  def lessThan(lhs: int, rhs: int) : boolean =
    if (lhs < rhs) true else false;

  def equalTo(lhs: int, rhs: int) : boolean =
    if (lhs == rhs) true else false;

  def greaterThan(lhs: int, rhs: int) : boolean =
    if (lhs > rhs) true else false;

  val nums : List[int] = 1 :: 4 :: 3 :: 2 :: Nil;
  val sorted = Test.sort(nums, lessThan, equalTo, greaterThan);
  System.out.println(sorted);
}

This is where the notion of an anonymous function becomes important, however. Instead of defining those three functions outright and referencing them by name in the sort call, we can instead define them "on the fly" in the call itself:

object App with Application {
  val nums : List[int] = 1 :: 4 :: 3 :: 2 :: Nil;
  val sorted = Test.sort(nums, (lhs:int, rhs:int) => if (lhs < rhs) true else false, 
                               (lhs:int, rhs:int) => if (lhs == rhs) true else false,
                               (lhs:int, rhs:int) => if (lhs > rhs) true else false );
  System.out.println(sorted);
}

Er... maybe.

If you're like a lot of developers, you're looking at the above and you're not necessarily won over. There's a couple of things that could be red-flagging at the back of your head:

• "I thought that the Don't-Repeat-Yourself principle says it's better to collect this stuff into one place, for easier maintenance?" True. But consider this: how often have you written a method in a class because you HAD to, not because it satisfied the DRY principle? Java's use of nested inner classes (and C++'s inability to allow you to define functions in-line) forces the use of methods, even in those situations where you know, without a doubt, that you will never execute or reference this code more than once.
• "Shouldn't this be making the sort shorter to use?" False. Anytime you genericize something, you run a (strong) risk that you're actually making it more complicated, and therefore more difficult to use. If we really wanted to make sort easier, I'd ask for a function parameter "compare" that does the traditional C-style comparison (return -1 for less-than, 0 for equal, 1 for greater-than), and write the necessary code inline in the sort() to use that method to determine if something fit the less-than, equal, or greater-than filters. That's not really the point, though... so I didn't.
• "That syntax is just ugly." True. To a Java, C++ or C# programmer. To a LISP programmer, though, there's clearly some parentheses missing. Seriously, the syntax isn't what you're used to, perhaps, but like most syntactic decisions, it has deeper meaning and rules around it that makes it look that way. The same is true of Java, C++ (remember the ">>" rule in multiple template usage?) and C#, and will remain so for as long as computers are what's interpreting our source code.
The thing is, the use of functions-as-objects is just the tip of the iceberg. Turns out there's some more interesting tidbits that we can make use of when using functions as objects, one of which is called "currying".

One frequently useful idiom in functional languages is to return a function, rather than the results of applying that function. This means that we can delay actually executing the function until later--this is what we're doing (sort of in reverse, passing it in rather than returning it) in the sort example above. We pass in the comparator function into the filter routine, who then executes it. Returning a function is of the same mindset--hand back a function to be executed by the caller (either implicitly or explicitly) that produces the results desired.

In some situations, however, the full inputs of the function aren't known at the time the function is returned. Or, as is often the case, some of the inputs are known, but others aren't. So the compiler, when handed a partially-called function, defers execution and uses parameters found in the caller's scope (wherever that may be) to fill out the remainder of the necessary functional inputs and carry out execution. Make sense?

Probably not; currying takes a while to ingest. At least it did for me. An example may serve to help cement this down.

def sum(f: int => int) = {
  def sumF(a: int, b: int): int =
    if (a > b) 0 else f(a) + sumF(a + 1, b);
  sumF
}

Here, we see a function "sum", which takes a function "f" that takes an int and returns an int. It in turn uses a nested function, "sumF", that takes two integer arguments "a" and "b" and applies the function "f" to them so long as "a > b". Notice, however, that sum neither takes the parameters "a" and "b", nor does it manufacture them from someplace in order to pass them in; in fact, it noticeably excludes them when it returns, without decoration, the function "sumF" as the return value of the "sum" function. (Notice again how we don't need to specify the return value of "sum", because Scala's type inference can figure it out without additional help from the programmer.)

So how does one use the sum function? By applying both parameters--the function to apply to each argument, and a pair of ints to supply bounds to be summed up:

sum(x => x * x)(1, 10)

Which, in this case, summarizes the expression 1^1 + 2^2 + 3^3 + ... + 10^10. (Readers with a background in mathematics will recognize it as a sequence, the "big E" notation, as I used to call it back in sophomore Advanced Algebra. Probably has a more formal name than that, but my background isn't in math.) To understand where the currying takes place, look at how the compiler sees this expression:

(sum(x => x * s))(1,10)

Which means, of course, pass the function "x * x" into sum, which then returns the sumF function with f(a) replaced by "a * a". sumF still expects two integer parameters, however, so the compiler takes the next expression "(1, 10)" and applies those as "a" and "b", respectively. From there, it's a simple exercise in recursion to arrive at the answer.

The power of currying becomes more apparent when you see that because the compiler is willing to accept partially-evaluated functions as first-order types, we can partially-define functions in terms of other functions, as in:

def sumInts = sum(x => x);
def sumSquares = sum(x => x * x);
def sumPowersOfTwo = sum(powerOfTwo);

and then use them as top-level functions without any special syntax:

> sumSquares(1, 10) + sumPowersOfTwo(10, 20)
267632001: scala.Int

Now, if for some reason the definition of sum() needed to change, it would ripple throughout this tiny framework by making one change in one place. (Don't know why sum() would need to change, mind you, but that's the problem with simple examples--it's sometimes hard to see the really positive benefits unless you get more complicated, but more complicated examples are harder to use to present the concept.) And I'd be lying to you if I said that I "get" how to use this in code more practical than summations yet--I still have a lot of internalizing to do. But I can see the outskirts of where it might be useful, and if I can get working samples of how and where it would, believe me, they're going up here.

In the meantime, next is traits and mixins, which are both features that are definitely easier to see applicability.



Java/J2EE

Saturday, March 04, 2006 7:52:49 PM (Pacific Standard Time, UTC-08:00)
Comments [5]  | 

My father, whom I've often used (somewhat disparagingly...) as an example of the classic "power user", meaning "he-thinks-he-knows-what-he's-doing-but-usually-ends-up-needing-me-to-fix-his-computer-afterwards" (sorry Dad, but it's true...), often forwards me emails that turn out to be one hoax or another. This time, though, he found a winner--he sent me this article, warning against the latest caller identity scam: this time, they call claiming to be clerks of the local court, threatening that because the victim hasn't reported in for jury duty, arrest warrants have been issued. When the victim protests, the "clerk" asks for confidential info to verify the records. Highly credible attack, if you ask me.

Net result (from the article):

• Court workers will not telephone to say you've missed jury duty or that they are assembling juries and need to pre-screen those who might be selected to serve on them, so dismiss as fraudulent phones call of this nature. About the only time you would hear by telephone (rather than by mail) about anything having to do with jury service would be after you have mailed back your completed questionnaire, and even then only rarely.
• Do not give out bank account, social security, or credit card numbers over the phone if you didn't initiate the call, whether it be to someone trying to sell you something or to someone who claims to be from a bank or government department. If such callers insist upon "verifying" such information with you, have them read the data to you from their notes, with you saying yea or nay to it rather than the other way around.
• Examine your credit card and bank account statements every month, keeping an eye peeled for unauthorized charges. Immediately challenge items you did not approve.

Remember, it's always acceptable to ask for verification of THEIR identity if they're asking for confidential information. And most credible organizations are taking great pains to not ask for that information over the phone in the first place. Practice the same discretion over the phone that you would over IM or email; the phone can be just as anonymous as the Internet can.

Apparently, I touched a nerve with that last post; predictably, people started counting the keystrokes and missing my point. For example, Mark Blomsma wrote:

Looks to me like you're comparing apples and pears.

C# does not force you to use accessors. The following is already a lot closer to Scala.

public class Person
{
public string firstName; public string lastName; public Person spouse;

public Person(string fn, string ln, Person s)
{
firstName = fn; lastName = ln; spouse = s;
}

public Person(string fn, string ln) : this(gn, ln, null) { }

public string Introduction()
{
return "Hi, my name is " + firstName + " " + lastName +
(spouse != null ?
" and this is my spouse, " + spouse.firstName + " " + spouse.lastName + "." :
".");
}
}

This is only 356 keystrokes, compared to 287 for Scala. Now in Scala the default accessor for classes and members seems to be public, if this were not the case then you'd need 323 keystrokes in Scala.
Only a very minor difference. And definately not enough to make a case that Scala is more efficient for a developer.

Another consideration if you start talking keystrokes is that the tooling suddenly becomes a factor. With C# and VS2005 I only type 'prop,tab,tab' and then the type and name info. Skipping quite some keystrokes.

All that sort of misses the point, though: the purpose of the comparison was not to count keystrokes, per se, but to look at the expressiveness of the language and how concisely the language can express a concept without requiring a great deal of scaffolding. C, for example, could always be used to build object-oriented systems... but you had a lot of work to do on your own to do it. As a result, a huge amount of complexity was spent in manaing the relationships between "classes" by hand (by tracking pointer relationships and so on). C++ solved a lot of that by baking those concepts in as a first-class concept, thus reducing the surface area requirment in the programmer's mind devoted to "plumbing", and making room for more business-level complexity. Java did the same to C++ by introducing GC and other VM-level support, and so on. Scala and Ruby (and other hybrid and/or dynamic languages) are now seeking to do the same to Java and .NET.

The question of tooling is an interesting one, though: is a language just the language by itself, or the language plus the tools that support it? Is Lisp still Lisp if you take Emacs out of the equation? Or is Smalltalk interesting without the Smalltalk environment and/or browser? Can we separate the two? Should we? That's a question to which I don't have an easy answer.

While speaking at a conference in the .NET space (the patterns & practices Summit, to be precise), Rocky Lhotka once offered an interesting benchmark for language productivity, a variation on the kLOC metric, what I would suggest is the "CLOC" idea: how many lines of code required to express a concept. (Or, since we could argue over formatting and style until the cows come home, how many keystrokes rather than lines of code.)

Let's start with a simple comparison. The basic concept we want to express is that of a domain object type, my favorite example, that of a Person type. In domain lingo,

A Person has a first name, a last name, and a spouse. Persons always have a first and last name, but may not have a spouse. Persons know how to say hi, by introducing themselves and their spouse.

In Java, we express this class like so:

public class Person
{
    private String lastName;
    private String firstName;
    private Person spouse;
    
    public Person(String fn, String ln, Person s)
    {
        lastName = ln; firstName = fn; spouse = s;
    }
    public Person(String fn, String ln)
    {
        this(fn, ln, null);
    }
    
    public String getFirstName()
    { 
        return firstName;
    }
    
    public String getLastName()
    { 
        return lastName;
    }
    
    public Person getSpouse()
    { 
        return spouse;
    }
    public void setSpouse(Person p) 
    { 
        spouse = p;
            // We ignore sticky questions of reflexivity and
            // changing last names in this method for simplicity
    }
    
    public String introduction()
    {
        return "Hi, my name is " + firstName + " " + lastName +
            (spouse != null ? 
            " and this is my spouse, " + spouse.firstName + " " + spouse.lastName + "." :
            ".");
    }
}

This isn't just a Java problem; the C# version of this type isn't much better:

public class Person
{
    private string lastName;
    private string firstName;
    private Person spouse;
    
    public Person(string fn, string ln, Person s)
    {
        lastName = ln; firstName = fn; spouse = s;
    }
    public Person(string fn, string ln)
        : this(fn, ln, null)
    {
    }
    
    public string FirstName
    {
        get { return firstName; }
        set { firstName = value; }
    }
    
    public string LastName
    {
        get { return lastName; }
    }
    
    public Person Spouse
    {
        get { return spouse; }
        set { spouse = value; }
    }
    
    public string Introduction()
    {
        return "Hi, my name is " + firstName + " " + lastName +
            (spouse != null ? 
            " and this is my spouse, " + spouse.firstName + " " + spouse.lastName + "." :
            ".");
    }
}

Class Person
  Dim _FirstName As String
  Dim _LastName As String
  Dim _Spouse As Person

  Public Sub New(ByVal FirstName As String, ByVal LastName As String, ByVal Spouse As Person)
    Me._LastName = LastName
    Me._FirstName = FirstName
    Me._Spouse = Spouse
  End Sub

  Public Sub New(ByVal FirstName As String, ByVal LastName As String)
    Me.New(FirstName, LastName, Nothing)
  End Sub

  Public ReadOnly Property LastName() As String
    Get
      Return _LastName
    End Get
  End Property

  Public Property FirstName() As String
    Get
      Return _FirstName
    End Get
    Set (ByVal Value As String)
      Me._FirstName = Value
    End Set
  End Property

  Public Property Spouse() As String
    Get
      Return _Spouse
    End Get
    Set (ByVal Value As Person)
      Me._Spouse = Value
    End Set
  End Property

  Public Function Introduction As String
    Dim temp As String
    temp = "Hi, my name is " & _FirstName & " " & _LastName
    If _Spouse <> Nothing Then
      temp = temp & " and this is my spouse, " & _Spouse.FirstName() & " " & _Spouse.LastName() & "."
    Else
      temp = temp & "."
    End If
    Return temp
  End Function
End Class

class Person
  def initialize(firstname, lastname, spouse = null)
    @firstname = firstname
    @lastname = lastname
    @spouse = spouse
  end

  attr_reader :lastName
  attr_writer :firstName, :spouse
  
  def introduction
    if spouse == nil
      "Hello, my name is #{firstName} #{lastName}"
    else
      "Hello, my name is #{firstName} #{lastName} and this is my spouse, #{spouse.firstName} #{spouse.lastName}"
    end
  end
end

class Person(ln : String, fn : String, s : Person)
{
    def lastName = ln;
    def firstName = fn;
    def spouse = s;
    
    def this(ln : String, fn : String) = { this(ln, fn, null); }

    def introduction() : String = 
        return "Hi, my name is " + firstName + " " + lastName +
            (if (spouse != null) " and this is my spouse, " + spouse.firstName + " " + spouse.lastName + "." 
             else ".");
}

This is hardly an exhaustive comparison of the languages, but it does give you a little taste of Scala's object flavor. Ruby's syntax is arguably of the same length as Scala's (and frankly, to my mind, they're too close to call... or care), but clearly Scala's length is much much smaller than that of the equivalent C#, Java, Visual Basic or C++ class. (C++ could make things interesting with judicious use of templates to handle backing store, accessor and mutator, but that's considered too advanced by many C++ devs, and therefore too obscure to use in common practice, rightly or wrongly.)

When next we look at this, we'll look at what Scala means when they say "everything's an object"... and how that, in many ways, this means that Scala is more object-oriented than Java itself.

Update: Glenn Vanderburg pointed out that my Ruby wasn't quite correct, and also suggested a bit more "Rubification":

     class Person
       def initialize(firstname, lastname, spouse = null)
         @firstname, @lastname, @spouse = firstname, lastname, spouse
       end

       attr_reader :lastName
       attr_accessor :firstName, :spouse  # attr_writer *just* makes a writer.  You really want this.

       # I would typically use the more explicit "if" that you used here, but for terseness I've
       # put this in the form you used with the Scala version:
       def introduction
         "Hello, my name is #{firstName} #{lastName}" + (spouse ? " and this is my spouse, #{spouse.firstName} #{spouse.lastName}" : "")
       end
     end

Thanks, Glenn. Again, I'm struck by how Ruby's strength lies not in the core language itself, but the various "macros" that they've defined (such as attr_reader and attr_accessor or attr_writer). This notion of "core language with user-defined extensions" is a powerful one, and I hope to show how Scala does much the same in its language definition.

As Joel before me, I'm going to be in beautiful Victoria, British Columbia, on April 4th to present at the Victoria .NET Developers Association, and as usual, the topic of what to present has come up.

Normally, this is a subject that the user group lead and I sort of hash out in private beforehand, but in this case, Nolan Zak (the user group lead) suggested I post here and call for suggestions. So, here's a list of topics I can present on, send me your thoughts.

1. Pragmatic XML Services: you know about SOA, you've heard the Four Tenets.... now what?
2. Intro to WCF: All you need to know about Microsoft's latest communication stack.
3. Web Services: Overview of the specs, the stacks, and the standards. What's critical, what's useful, what's vendor hype and fluff.
4. C# 3/LINQ: What's in their heads for C# v.Next
5. (Or suggest your own idea.)

I know that many readers of this blog complain when I take time out from technical topics to talk about personal stuff, so if you're one of those folks, move along. This is about as personal as it gets, and fair warning: if you're going to complain about this post, I'm going to ignore you, at best.

Daniel Steinberg, a fellow No Fluff Just Stuff speaker, lost his seven-year-old daughter not too long ago, and he wrote the story up in a really poignant and moving piece he called "Dear Elena".

Dan, you can't imagine how terrible I feel for you right now--that's every parent's worst nightmare. I wish there were something I could do or say to make this time easier or less tragic for you, but of course there isn't. She sounds like she was a wonderful little girl, and I'm saddened by the fact that I didn't get the chance to meet her. My thoughts and prayers are with you and your family right now.

Now, if you'll all excuse me, I'm going to Skype my six-year-old son at home. For what I would hope to be a fairly obvious reason, I feel the need to give him a hug from here in London.

Several people have commented on my recent posting about Spring, and I want to make something clear: I'm not saying that Spring (or Hibernate, or EJB, or anything else) is a bad technology. I'm saying that walking up to every project, assuming that Spring will be THE answer, is bad. This kind of dogmatic approach--which, by the way, more than anything else is what led to the downfall of EJB as a popular technology--is bound to bite you in an uncomfortable place sooner or later.

One commenter, in particular, chastised me for not providing specific examples regarding where Spring may fail; I'm not going to stand here and make an exhaustive analysis of Spring's strengths and weaknesses. Besides being something that's already being done elsewhere, it would be beside the point that I'm trying to make--that dogma of any form is bad.

Look, so you've been successful with Spring on a few projects--that's good, and I encourage you to consider Spring again for your next couple. But don't make the dangerous assumption that using Spring will always yield success. In fact, let's take this out of the realm of Spring entirely and restate the point: "Look, so you've been successful with [[TECHNOLOGY-X]] on a few projects--that's good, and I encourage you to consider [[TECHNOLOGY-X]] again for your next couple. But don't make the dangerous assumption that using [[TECHNOLOGY-X]] will always yield success." (Where [[TECHNOLOGY-X]] can be, but isn't limited to, one of Spring, Hibernate, EJB, J2EE, COM+, WCF, CORBA, XML services, relational databases, stored procedures, managed-code-inside-the-database, highly denormalized relational data, highly normalized relational data, ....)

Recently, while cruising the Internet (and, in particular, the Lambda-the-Ultimate site), I ran across the Scala programming language, latest brainchild of Martin Odersky (of GJ fame, which of course was derived from Pizza, among others). It's another entry in the hybrid functional/object language space, and as such, has a lot of interesting features that Ruby holds, but runs on the JVM (and can actually cross-compile into a .NET assembly, though it does require some slightly different mappings), and as such means developers don't have to make a wholesale commitment to the Ruby interpreter.

I thought I'd share some of the more interesting bits of Scala in this and a few more blog posts.

The high-level stuff

First of all, from the Scala website, let's get the high-level overview stuff out of the way:

• Scala is object-oriented. Scala is a pure object-oriented language in the sense that every value is an object. Types and behavior of objects are described by classes and traits. Class abstractions are extended by subclassing and a flexible mixin-based composition mechanism as a clean replacement for multiple inheritance.
• Scala is functional. Scala is also a functional language in the sense that every function is a value. Scala provides a lightweight syntax for defining anonymous functions, it supports higher-order functions, it allows functions to be nested, and supports currying. Scala's case classes and its built-in support for pattern matching model algebraic types used in many functional programming languages. Furthermore, Scala's notion of pattern matching naturally extends to the processing of XML data with the help of regular expression patterns. In this context, sequence comprehensions are useful for formulating queries. These features make Scala ideal for developing applications like web services.
• Scala is statically typed. Scala is equipped with an expressive type system that enforces statically that abstractions are used in a safe and coherent manner. In particular, the type system supports generic classes, variance annotations, upper and lower type bounds, inner classes and abstract types as object members, compound types, explicitly typed self references, views and polymorphic methods. A local type inference mechanism takes care that the user is not required to annotate the program with redundant type information. In combination, these features provide a powerful basis for the safe reuse of programming abstractions and for the type-safe extension of software.
• Scala is extensible. The design of Scala acknowledges the fact that in practice, the development of domain-specific applications often requires domain-specific language extensions. Scala provides a unique combination of language mechanisms that make it easy to smoothly add new language constructs in form of libraries: any method may be used as an infix or postfix operator, and closures are constructed automatically depending on the expected type (target typing). A joint use of both features facilitiates the definition of new statements without extending the syntax and without using macro-like meta-programming facilities.

Enough overview. Let's look at code.

Hello, Scala

OK, Scala really isn't all that interesting as a Hello World program, but it does highlight one of the more interesting elements of Scala that I already like:

object Hello {
  def main(args: Array[String]): Unit = {
    Console.println("Hello, Scala!");
  }
}

Some of what's interesting to see here is that the Scala compiler actually produced two .class files--one entitled App.class, another called App$.class, the second App$ class apparently to provide "module" behavior (which I suspect is related to the singleton-ness of the object declaration in the code). As you might expect, Scala injects some additional support methods into both classes, including getScalaType, which is obviously intended to return the type of the object to Scala, just as the .class or getClass does for Java. Which brings up another interesting point.

Scala presents a unified type hierarchy, such that scala.Any is the root of the type system, and (like the CLR) is bifurcated into two basic elements, one being the object-family of types (java.lang.Object, known to Scala as scala.AnyRef) and the "primitive type" family of types, known to Scala as scala.AnyVal. Scala calls these reference classes and value classes, respectively--the same monikers the CLR uses. There's also reference to a type scala.All, which the introduction/tutorial page puts at the bottom of the type hierarchy, apparently inheriting from everything, but I'm can't find documentation on it or what purpose it serves. *shrug* More on that later, I guess.

Another interesting tidbit is that we can run Scala interpretively, the same way we can do to Groovy:

> scalaint -nologo HelloWorld.scala
> HelloWorld.main(null)
Hello, world!
(): scala.Unit
>:q

Update: Forgot to mention, Scala has another interesting element to it that makes it very interesting to Ruby in much the same way:

object HelloWorld2 with Application { 
  Console.println("Hello, world!"); 
}

I consider myself a relative newbie to Scala, but as I progress through the language and see some useful applications of features, I will blog more. I'll also blog some of the features themselves, but you can find that for yourself by working through the Scala tutorial material on the site, if you're so inclined. In the meantime, catch the presentation I'm doing on Scala at the No Fluff Just Stuff symposiums, starting 2Q this year.

And, by the way, for those of you in the .NET space, Scala does, as I mentioned before, cross-compile to .NET assemblies, though I haven't spent much time exploring this. Frankly, I'd be more comfortable using Scala in the .NET world if there was a .NET-based compiler for it, rather than having to install a JRE just to run the compiler, but F# serves much the same space in the .NET world that Scala does here, and that's another language I'm pursuing with some vigor, as well. More on that later.

There are a few things, I've found, that are fun about being a speaker and general rabble-rouser, but none of them are nearly as much fun as when I get an opportunity to interview industry icons and ask them all my questions on camera. In this case, while at JavaPolis2005, my victims were the well-known pair Joshua Bloch and Neal Gafter, who, more than anyone else in the world, are most directly responsible for the language features that came in Java5. I tried to keep the interview pleasant and friendly, but I did ask the questions that've bothered me for a while, like "Why did generics end up the way they did?", "Is Java too complicated and hard to use now?" and "What are you doing at Google these days, anyway?"

Online (registration required) at the JavaPolis2005 site. Keep an eye on the site for the other interviews Dion did, as well as one more I did with Brian Goetz, who's got a GREAT book on Java Concurrency coming out in 2006.

CrazyBob just wrote about how he "doesn't get Spring", and although it runs the risk of sounding like something from the "Me, too" bandwagon, I have to say, I agree with him (and have been saying this in conferences and panels for a while now):

Even worse, I've noticed what I consider to be a dangerous and blind increase in the rate of Spring adoption. I've yet to read a critical article or book on Spring. It seems like everyone loves Spring except me.

Maybe Spring adoption is a knee-jerk reaction to J2EE. "J2EE is bad, and the Spring guys say their stuff is better, so Spring must be good." It doesn't work that way.

For starters, I'm with Bob on the statement that blind adoption of Spring is dangerous. I wrote once before that dogma of any form is bad, and Spring dogma is just as bad and just as dangerous as J2EE dogma ever was, for much the same reason: dogma discourages thinking. Walking onto a project, prepared already to believe that Spring is the best solution to the problem, without considering the context, is just as bad as when we did that with J2EE. In fact, any technology can fall into that trap, be it Ruby, .NET, J2EE, Spring, LAMP, Vista, COM/DCOM/COM+, you name it. ANY kind of dogma that allows developers to shut off the analytical part of their brain is dangerous. Spring is a useful technology, no question. But so is J2EE, and so is .NET, and so is LAMP, and...

Don't ever make the mistake of letting dogma drive your technology decisions, period. No matter who justifies them. I thinke states it best when he says

If you do [adopt Spring], go in with your eyes wide open. Be skeptical, critical. Just because someone has a popular Open Source framework, they have slick marketing, and they're supported by a big vendor (IBM pushed Struts on me for a number of years after all), it doesn't necessarily mean they know what's best for you or even that they know better than you.

Bob then goes into how he likes the setter-injection that Spring provides; I personally don't have the same degree of fondness for dependency injection, to be honest: I find JNDI (and Service Locator) to be a superior approach, mostly because with the Service Locator, I can control when the resource moniker is resolved, meaning that if the resource should fail somewhere during the call, I can control where and how I go back to the Service Locator for a failover attempt. More importantly, I can re-resolve the resource as my failover policies permit, and I'm not held hostage to how--or mre importantly when--the container decides to inject the dependency.

At the end of the day, it's important to remember that "lightweight" and "testable" doesn't have to mean "Spring". In-process testing of EJB components is possible thanks to the in-proc nature of the OpenEJB stack. Testability of JNDI is easily accomplished with unit tests that use the Hashtable JNDI provider that Sun makes available in the Java Tutorial, if you want or need to test the Service Locator code itself. Or, you take the "black box" approach (as some recommend for servlet containers), and test your code through the container itself by doing the heavierweight communication through the communication stack from out-of-process calls. In the end, it's not the APIs that define the tool's testability, but the ability to embed the tool inside a unit-test environment.

Would I recommend Spring? Certainly, under the same circumstances and for the same reasons I'd recommend J2EE: when it's appropriate, because there's some good stuff there, and it's well-known and an official (in J2EE's case) or de-facto (in Spring's case) standard.

Oh, and let's not forget, this applies to any technology, including the upcoming rise of dynamic languages...

Billy Hollis, famed Visual Basic lecturer and secret programming language anthropologist, has compiled a succinct history of programming languages. As he puts it,

If you like VB, look at the history of the C family [of languages] first. If you like C#, Java or C++, look at the history of the BASIC family first.

Happy Friday.

A number of you have made comments about my LINQ paper, and rather than respond in comments in turn, I thought I'd gather them up and respond to them en masse. So, without ado....

Stu Smith said:

Nice article. Two things occur to me immediately...

1. Based on my current understanding of LINQ, it's purely for querying, and so compared to most O/R systems it lacks caching support. (ie its queries may be optimal but that's not much consolation if it keeps re-executing them).
2. The O/R system we use (which admittedly only needs to support a particular kind of application) solves the 'lazy load vs N+1' issue by generating joined queries based on either the path taken to the data, or on particular routes to a marked-up tables. ie, if you navigate from a Customer to a collection of Orders, that's a single select. If you then start iterating the Orders and inner-iterating the OrderDetails, then on the first one a second joined select is issued, and for subsequent iterations the data is already cached and thus no further SQL statements are emitted.

1. No, LINQ can, in fact, do any sort of relational manipulation, including INSERTs, UPDATEs and DELETEs, but the real strength of the language integration is in the query aspects, particularly the fact that LINQ can do these queries across any rectangular data store, so it's fair to say that LINQ is mostly about query.
2. I'd be interested to hear more about your solution to solve the lazy-load issues, partcularly how you handle the situation where you need to display only a small part of the full OrderDetails data--remember, part of the criticism of O/R is that they either eager-load too much, or lazy-load too much, and can't infer the amount or areas of data to retrieve that's "just right".

Bryant wrote:

I thought the article was well written and informative. While I think it's cool that I might one day be able to use the same object model to query databases, XML documents, and even the file system I still feel compelled to look back upon my days as a DBA. This looks a lot like building ad hoc SQL statements in the code except we're a little more type safe here. LINQ still does not answer the question of storage abstraction. The developer still needs an intimate knowledge of the database structure. So, LINQ appears to cover "Conflicting type systems" and maybe "Transactional boundaries" and "Query/access capabilities". There are still four more items in your list that I don't see being solved with LINQ. Your article was a great read but sorry, I am still not excited.

James commented:

I thought the article was great! I'm a little unclear as to why it's not getting ranked better on MSDN but for someone who didn't really understand/appreciate the problem domain LINQ is serving, your article really got me thinking and cleared up a lot of fogginess in my mind. Excellent work!

Andy Maule said:

Very interesting! I liked the discussion of Rail's ActiveRecord which I think is an approach that most people miss when talking about OR Mappings.

There's a good research paper discussing the same stuff here. It mentions something recently developed for doing statically typed queries in Java 'Native Queries' which is an interesting comparison to LINQ. Anyone interested in this area should take a look.

I'm currently doing a PhD in this area, and I have to say that LINQ is making things very interesting.

Karmencita is my lightweight alternative for in memory object querying.

As for the rest of you who offered kudos (Bart De Boeck, Dan Kahler, Paul Wilson, Eric Bachtal), thanks; every author likes to know that their work is appreciated, particularly when so much of what they say seems to stir up more controversy than discussion.

Matt Morton commented, "One might be able to say that Ted Neward is cynical about any new technology. You might also say he puts himself in the position of the "old" kermudgeon (sp) who opposes anything new and cool." Yep, guilty as charged, for a very specific reason.

Ages ago, when EJB first shipped, I was one of the first who looked at it with stars in my eyes. It seemed like such a great, easy solution to all the problems of developers building server-side systems (and I'd done a C++-based 2-tier, CORBA-based 2-tier and Java/NetDynamics-based 3-tier system before this, so I kinda fit into that space already). I was excited. I was ready to swear it to all my friends. And then....

It was a lunch with Don Box and Kevin Jones, shortly after I'd joined DevelopMentor, and I asked Don and Kev about EJB. Don looked at Kev with this silly grin on his face, and Kev just shook his head and said, "It's a thing on a thing. That's always slower than just a thing." Slowly, the light dawned. Over the course of several conversations with Kev and Don later, I came to realize that EJB wasn't a distributed object technology, but a component technology focused on transactions. Over time, developers' stupidity in using EJB for their single-database simple-HTML-form apps drove me to be widely proclaimed as an "EJB expert who consults against EJB", which isn't exactly correct--I've recommended EJB in scenarios, but only where it seems appropriate.

Which, if you think about it, is what we're supposed to do: recommend tools where they're appropriate.

What does this have to do with being a curmudgeon? Simple: I trust no technology until it proves itself to me. Our industry is SO filled with hype, it's surprisingly easy to get caught up in the energy and excitement that surrounds a new tool, particularly those that deal with presentation issues. (Face it, folks, the "jazz factor" surrounding Avalon/WPF or Ajax is exponentially higher than that surrounding RIFE/Continuations, despite the fact that the latter is far more interesting from a technology perspective. Why is this? Because you can SEE the niftiness in Ajax; the continuations story isn't something you can show off to your mom or impress your significant other.) I look at new tools, new technologies and deliberately look to find the flaws, the various fallacies they fall prey to, and I routinely caution people against them JUST because they're new. I would much rather err on the side of caution and hestiation than fall into the trap of hyperbole and bandwagon, because I think, ultimately, it's a more responsible position to my clients and audience.

I don't think I'm unjustified in this position: there's an unhealthy absence of cynicism in our world right now. My two big examples: the terrible tragedy of the miners in West Virginia (why didn't anybody in the media CONFIRM the story of their rescue before reporting it?) and the South Korean human cloning story. Or the supposed report of "cold fusion" from a decade or two ago. Our industry could use, I think, from a large dose of, "OK, so you've created a new framework. What's it to me?" right now.

Matt went on to say,

I dont totally agree with #4 though. When a large scale Ruby project fails it wont be because of the language. Just as it is faulty to claim a language will reduce project failures, it is just as faulty to claim that a language or platform will be the cause. Projects fail because of people, plain and simple. People in general have trouble being honest especially when they have something vested emotionally (or financially) in a project. Perhaps this is what he is getting at. The proponents are so emotionally involved with Ruby that when less experienced folks try to apply it to a larger scale the project, it will blow up.
Personally I have found that Rails and Ruby are a joy to use. I guess then you could say that I get emotionally involved with things that bring me joy. Perhaps Rails strength is its weakness. It is such a joy that it blinds you to its true uses.

Which, if you think about it, is exactly the same thing that happened with EJB before it.

Lego, those folks who brought you Mindstorms in the first place, have done it again--they've announced a next-generation programmable brick, the NXT, that offers some really cool enhancements, including Bluetooth capability. Suddenly, the old Robotwar clones from ages ago take on an entirely new meaning....

More interestingly, supposedly they've announced that the VM itself will be documented, which leads me to wonder how long before we see J-NXT and NXT#. It's amazing how the managed environment is just spreading like wildfire through the world, if'n you ask me....

Hey, Santa, if you're listening, this just went on my list in a BIG way.

So the December CTP of $g{Vista} is available, and I'm finding a recurring problem trying to install it into VMWare. When I tried this with the Beta1 of Vista into VMWare 5.0, the same problem occurs as what I'm getting now: when trying to boot off of the .ISO inside of VMWare (in other words, not physically off a burned CD/DVD, but out of the .ISO image itself mapped into the virtual CD in VMWare), I get a BSOD every... single... time. Is this indicative of a f-ed up installation of my VMWare image, is this a known bug in Vista (sidenote: yes, I know about the raw partition issues with Vista, but doing an XP-first install of Vista gives the same BSOD on reboot), or is it just the combination of ISO-image-and-VMWare? Anybody got ideas for a workaround or a fix?

I've been contacted by a third party (not a recruiting agency) who's having a hard time finding a J2EE architect with familiarity/expertise with concepts of architecture and the low-level chops not to lose sight of the code. To put it in their words:

This is strongly influenced by component-based methodologies such as OpenDoc, but would be extending the RCP Platform developed by Eclipse, and encompassing other concepts from systems such as Squeak... We are having trouble finding someone senior enough to understand the conceptual architectural issues, but technical enough to know important lower-level details such as how design choices will effect performance and scalability.

By the way, DON'T send me your resumes--in your email to me, tell me what your favorite alternative language or platform is, and why. I'll use that to know if you're somewhere in the ballpark.

And no, job postings are NOT going to become a regular part of this blog; I'm doing this as a once-off. (My motivation? I just want to help these guys; this isn't an easy requirement to fill, and I'm hoping I get to meet whomever it is they end up selecting, 'cuz I'm thinking they'll be a kindred spirit. )

Update: They sent me a formal job description, and it seems the desires changed just a bit. *shrug* See for yourself:

A lead software architect for [[name snipped]]. Applicant must be able to take a high level conceptual framework and architect a conceptually sound implementation using open standards in collaboration with an existing team. Experience with Java, Eclipse, and the Rich Client Platform (RCP) sub-project in Eclipse is a significant advantage. Experience with Aspect Oriented Programming is a plus. Participation in the open standards community a plus (e.g. W3C and Eclipse standards communities). Five or more years of experience required with experience as a lead software architect, industry experience preferred. Opening is for a full-time onsite position.

Jim Hugunin strikes again: IronPython 1.0 beta 1 is out. I think Jim should get an award for "first language designer to operate on both the JVM and CLR". Now he just needs to do one more implementation to get the hat trick.

Flame away...

The Pragmatic guys are at it again... This time it's a whole course, taught by two of the finest instructors I have had the privilege to know (and, quite honestly, argue with), on everybody's favorite presentation-layer hot topic, Ajax.

By the way, dear audience, this is one class you can attend regardless of which camp you prefer--both Stu and Justin are equally adept on both enterprise platforms (Java and .NET) and the new hot language, as is clear when they say that they will show you how "to use frameworks such as Rails, Spring, and ASP.NET"; Justin, for example, co-authored "Better, Faster, Lighter Java" and the "Spring Developer's Handbook", as well as built DevelopMentor's ASP.NET website and infrastructure. Stu was one of the COM cognoscenti back in the day (his poems on the subject (towards the bottom, search for Stu) are legend), and then took over as Java curriculum lead when DevelopMentor... well, created a Java curriculum. Two brighter guys--and better instructors--you're not likely to meet.

Oh, and, uh, they seem to know a fair amount about Ajax, too. Enough that I'd attend the course, were I not already busy that week.... Do yourself the favor, if you want to know more about Ajax, go see them. I can think of a lot worse ways to spend a grand in cash and 3 days...

An interesting question emerged during a discussion with some buddies/co-workers/peers/whatever-you-want-to-call-them today:

"Which conferences have you attended in the past that you thought was really good, and why? Which sessions were your favorites, and why? What made them that way?"

Yes, this is somewhat selfish, since the new conference season is amping up, and obviously I'd like to make sure my sessions are ones that people find interesting and recommend to others, but the question actually stemmed from an unrelated discussion to that. I promise.

Microsoft has done it again--this time, they're previewing the next release of C# planned for after LINQ/C# 3.0. They call it, for obvious reasons, the YODA programming langauge. Judge me by my size, do you? As well you should not, for my ally is the Source, and a powerful ally it is, indeed.

Remember: A Jedi uses the Source only for knowledge and defense, never for a hack.

In a recent thread on TheServerSide.com, Rick Hightower, a fellow NFJS speaker, commented on the JSR-175/annotations specification, and I felt a little obligated to respond, since this is a common critique/criticism:

Why don't you like the implementation? I hate the fact that your code has to import the annotations and then your code is tied to the annotation. It does not seem that different than depending on a interface (i.e., a marker interface). I'd like to see a soft import for annotations that does not impact compilation. (from TheServerSide.com)

The basic problem was this: if the compiler runs across an annotation, and it doesn't match an annotation type defined anywhere in the compilation classpath of imported symbols, what is the proper behavior? In any other scenario, such as:

public class App
{
  public static main(String[] args)
  {
    Systme.out.println("Hello, world!");
  }
}

public class App extends Object
{
  @Overide public String toString(Object obj) { return "App"; }
}

The big IDE vendors were particularly upset at this idea, leading one to claim, "If we cannot solve this problem we will consider JSR 175 to have been a failure." Unfortunately, then, we failed--there is no good way to solve this problem without breaking the fundamental vision of the Java platform. We tried a variety of ideas, including a few centered around the JDK 1.4 assertion idea (some kind of runtime flag indicating which annotations were safe to ignore), but couldn't work out the basic semantics of such without requiring a definition of the annotation to be present on the compilation path. And frankly, in the grander scheme of things, it makes sense to me that annotations ARE required at compilation time--just as interfaces, helper classes, member field types, and other types are required to be present at compilation time, as well.

Rod Johnson continued his critique of annotations by citing the following two reasons:

1. No proper mechanism for overriding annotations at runtime, despite the fact that just about any framework that uses annotations is going to need to consider doing that.
2. Inability for an annotation to extend an existing interface (even if that interface is simple enough to sit within annotations). Of course there are implementation issues around this one, I guess. But it means that it's hard to avoid code duplication when working with annotations and alternative metadata sources--something that's going to be particularly important until everyone and their dog uses Java 5, and anyway will remain important to work with existing code that may not have the right annotation.

Nutshell version of all this, the JSR 175 EG did, in fact, think long and hard about "soft imports" and "runtime annotation modification", and both ideas were shot down for what we felt were good reasons.

In keeping with the tradition, I'm suggesting the following will take place for 2006:

1. The hype surrounding Ajax will slowly fade, as people come to realize that there's really nothing new here, just that DHTML is cool again. As Dion points out, Ajax will become a toolbox that you use in web development without thinking that "I am doing Ajax". Just as we don't think about "doing HTML" vs "doing DOM".
2. The release of EJB 3 may actually start people thinking about EJB again, but hopefully this time in a more pragmatic and less hype-driven fashion. (Yes, EJB does have its place in the world, folks--it's just a much smaller place than most of the EJB vendors and book authors wanted it to be.)
3. Vista will be slipped to 2007, despite Microsoft's best efforts. In the meantime, however, WinFX (which is effectively .NET 3.0) will ship, and people will discover that Workflow (WWF) is by far the more interesting of the WPF/WCF/WWF triplet. Notice that I don't say "powerful" or "important", but "interesting".
4. Scripting languages will hit their peak interest period in 2006; Ruby conversions will be at its apogee, and its likely that somewhere in the latter half of 2006 we'll hear about the first major Ruby project failure, most likely from a large consulting firm that tries to duplicate the success of Ruby's evangelists (Dave Thomas, David Geary, and the other Rubyists I know of from the NFJS tour) by throwing Ruby at a project without really understanding it. In other words, same story, different technology, same result. By 2007 the Ruby Backlash will have begun.
5. Interest in building languages that somehow bridge the gap between static and dynamic languages will start to grow, most likely beginning with E4X, the variant of ECMAScript (Javascript to those of you unfamiliar with the standards) that integrates XML into the language.
6. Java developers will start gaining interest in building rich Java apps again. (Freely admit, this is a long shot, but the work being done by the Swing researchers at Sun, not least of which is Romain Guy, will by the middle of 2006 probably be ready for prime-time consumption, and there's some seriously interesting sh*t in there.)
7. Somebody at Microsoft starts seriously hammering on the CLR team to support continuations. Talk emerges about supporting it in the 4.0 (post-WinFX) release.
8. Effective Java (2nd Edition) will ship. (Hardly a difficult prediction to make--Josh said as much in the Javapolis interview I did with him and Neal Gafter.)
9. Effective .NET will ship.
10. Pragmatic XML Services will ship.
11. JDK 6 will ship, and a good chunk of the Java community self-proclaimed experts and cognoscente will claim it sucks.
12. Java developers will seriously begin to talk about what changes we want/need to Java for JDK 7 ("Dolphin"). Lots of ideas will be put forth. Hopefully most will be shot down. With any luck, Joshua Bloch and Neal Gafter will still be involved in the process, and will keep tight rein on the more... aggressive... ideas and turn them into useful things that won't break the spirit of the platform.
13. My long-shot hope, rather than prediction, for 2006: Sun comes to realize that the Java platform isn't about the language, but the platform, and begin to give serious credence and hope behind a multi-linguistic JVM ecosystem.
14. My long-shot dream: JBoss goes out of business, the JBoss source code goes back to being maintained by developers whose principal interest is in maintaining open-source projects rather than making money, and it all gets folded together with what the Geronimo folks are doing. In other words, the open-source community stops the infighting and starts pulling oars in the same direction at the same time. For once.

Whilst perusing the latest VMWare Workstation offering from their website, I noticed that not only does VMWare offer a free VMWare player (in other words, take a VMWare disk image created by somebody else and use it), but the VMWare site also has links to various pre-built VMWare disk images, including one for BEA's complete WebLogic 8.1 environment.... Whoever thought this idea up deserves to be knighted--what a great way to make it trivially simple for somebody to get started with a rather intimidating task (be that either installing a new O/S or a new app server).

Are you listening, Microsofties? VPCs of Vista, Visual Studio Team System and, heck, even just a base Visual Studio Express (pick a language, C# and VB sound like good starters) image are definitely something to consider if you want to make it easy for dev's to play with your tools.... Particularly people who DON'T want to install Windows just to play with Microsoft's implementation of .NET....

If you read no other book this coming year, you must read "Blink", by Malcolm Gladwell, the same author who wrote "The Tipping Point" (which is about why certain trends seem to just "take off" with no prior warning--case in point, the incredible rise of certain fashion trends, such as "Hush Puppies")..

I won't tell you what it's about except to quote the back cover; to do so would ruin the book's effect, to be blunt. The inside jacket reads,

In his landmark bestseller The Tipping Point, Malcolm Gladwell redefined how we understand the world around us. Now, in Blink, he revolutionizes the way we understand the world within. Blink is a book about how we think without thinking, about choices that seem to be made in an instance--in a blink of an eye--that actually aren't as simple as they seem. Why are some people brilliant decision-makers, while others are consistently inept? Why do some people follow their instincts and win, while others end up stumbling into error? How do our brains really work--in the office, in the classroom, in the kitchen and in the bedroom? And why are the best decisions often the ones that are impossible to explain to others?

In Blink we meet the psychologist who has learned to predict whether a marriage will last, based on a few minutes of observing a couple; the tennis coach who knows when a player will double-fault before the racket even makes contact with the ball; the antiquities experts who recognize a fake at a glance. Here, too, are great failures of "blink": the election of Warren Harding; New Coke; and the shooting of Amadou Diallo by police. Blink reveals that great decision makers aren't those who possess the most information or spend the most time deliberating, but those who have perfected the art of "thin-slicing"--filtering the very few factors that matter from an overwhelming number of variables.

Drawing on cutting-edge neuroscience and psychology and displaying all of the brilliance that made The Tipping Point a classic, Blink changes the way you understand every decision you make. Never again will you think about thinking the same way.

You make the call on this one... cut & pasted directly out of the email (after the horizonal rule):

Mark Michaelis posted a challenge: modify a string such that the following would print "Smile":

class Program
{
  static void Main()
  {
      string text;

      // ...
      // Place code here
      // ...
      text = "S5280ft";
      System.Console.WriteLine(text);
  }
}

His solution?

class Program
{
  static void Main()
  {
      string text;
      unsafe {
          fixed (char* pText = text) {
              pText[1] = 'm';
              pText[2] = 'i';
              pText[3] = 'l';
              pText[4] = 'e';
          }
      }
      text = "S5280ft";
      System.Console.WriteLine(text);
  }
}

My answer; note that I believe mine to be cleaner, more elegant, and far far more dangerous, since it never uses any sort of unsafe code:

class Program
{
  static void Main()
  {
      string text;

      string internedText = "S5280ft";
      String.Intern(internedText);
      MethodInfo mi = typeof(string).GetMethod("InsertInPlace", 
        BindingFlags.NonPublic | BindingFlags.Instance, null,
        new Type[] { typeof(Int32), typeof(string), typeof(Int32), typeof(Int32), typeof(Int32) }, null);
      mi.Invoke(internedText, new object[] {0, "Smile", 1, 7, 5});      

      text = "S5280ft";
      System.Console.WriteLine(text);
  }
}

The point? Playing with Reflection can be dangerous... oh, and it helps to know that strings are only as immutable as the platform forces them to be. In this case, my little hack would only be possible because under the covers, .NET doesn't really have immutable strings--it just doesn't let YOU modify them.

(By the way, same trick is available in Java, using the same approach. Or you could write JNI code to sort of duplicate Mark's trick, but who'd want to do that? Brrr.)

Matt Cassell is putting on an Academic .NET radio show (something in the vein of .NET Rocks! but aimed at students), and asked me to be the opening episode. It's up online now, so have a listen and see if I managed to steer the kids straight....

A good friend of mine and I are looking at taking on a new project together, and as part of the discussion we were exploring some of the differences of taking a relational perspective against an object perspective, and one of the comments she made was that in a relational model, you can always "filter" the data you want based on some predicate. "Ha!", I said, "If that's what you want, I can give you that over objects, too!" What's more, thanks to generics, I can do this for any collection type in the system without having to introduce it on some kind of base class:

    static class SetUtils
    {
        public static List<T> Project<T>(List<T> list, Predicate<T> pred)
        {
            List<T> results = new List<T>();

            foreach (T p in list)
                if (pred(p))
                    results.Add(p);

            return results;
        }

        // Not too hard to imagine the other relational operators here, too
    }

    // Usage:
    class Person
    {
        private string firstName;
        private string lastName;

        public Person(string fn, string ln, int age) {
            this.firstName = fn;
            this.lastName = ln;
        }

        public string FirstName {
            get { return firstName; }
            set { firstName = value; }
        }
        public string LastName {
            get { return lastName; }
            set { lastName = value; }
        }
        public override string ToString() {
            return "[Person [" + firstName + "]" + " " + "[" + lastName + "]" + "]";
        }
    }

    class Program {
        static void Main(string[] args) {
            Person cg = new Person("Cathi", "Gero", 35);
            Person tn = new Person("Ted", "Neward", 35);
            Person sg = new Person("Stephanie", "Gero", 12);
            Person mn = new Person("Michael", "Neward", 12);

            List<Person> list = new List<Person>();
            list.Add(cg);
            list.Add(tn);
            list.Add(sg);
            list.Add(mn);

            List<Person> newards = 
                SetUtils.Project<Person>(list, 
                    delegate (Person p) { if (p.LastName == "Neward") return true; else return false; } );
            foreach (Person p in newards)
                Console.WriteLine(p);
        }
    }

Now if I could just figure out how C# 3.0 manages to differentiate/overload between delegate instances and Expression objects in LINQ/DLinq, I might be able to backport that to C# 2.0, too, and be able to pass these Predicate instances across the wire for execution on other machines.

In a lot of ways, the Predicate delegate type is an example of using C#'s anonymous methods as a form of closure or lambda expression. (It's been argued that anonymous methods-as-delegates aren't "true" closures, since the local variables referenced in a closure will only be references to the objects, not complete copies, but to my mind that's exactly as it should be, as any time you pass a reference to an object, you're passing just that--a reference to an object, not a complete copy of the object. To do otherwise in anonymous methods would violate the Principle of Least Surprise, IMHO.) The Ruby syntax arguably isn't any more elegant or terse, and I suspect similar things could be done in C++ using templates; probably something along these lines already exists in Boost. But alas, I see no way to do this in Java given the current state of the JVM, namely the aforementioned lack of "managed functors" and type-preserving generics. If any out there in Java-land know otherwise, please holler, because I would really love to know how to do this as elegantly.

This is a bit of old news, but the discussion came up during the Seattle Code Camp, so I thought I'd go through the problem, and use it as an example of the issues that can come up when trying to map language concepts on top of a platform that doesn't support the idea natively. Hopefully, this will cause developers looking to build DSLs or other languages on top of the .NET (or JVM) platform to see some of the edge cases a bit more clearly and a bit sooner.

To lay down the background first: dealing with NULLs has always been somewhat problematic; the most obvious example of this is the mapping between relational databases, where even an INTEGER column can either have a value, or be empty, or be NULL, each of those being separate and distinct states. Trying to map NULL integer column values to integer values in the language has always been difficult in Java. C++, and C#, since primitive types / value types generally don't support null values, and Anders (among others) decided that it was time to try and integrate nullability more deeply into the language. The .NET team saw an opportunity to support nullability by creating a generic/templatized type to represent the possibility of nullability, and the C# language team took it further to try and make nullability feel "more at home" within the language. It was a bold, if at first seemingly-trivial, step.

Initially, the Nullable<T> type was pretty simple: it captured an instance of T internally, and if T was null it tripped an internal flag such that the IsNull property would return true. So, using a nullable int would work something like this:

Nullable<int> ni = new Nullable<int>(null);
if (ni.IsNull)
  Console.WriteLine("It's null!");
else
  Console.WriteLine(ni.Value);

int? ni = null;
if (ni == null)
  Console.WriteLine("It's null!");
else
  Console.WriteLine((int)ni);

int? ni = null;
int i = (int)ni;

OK, so far, so good. The problem is, however, that people were going to ask these nullable types to do things that subtly were different from what they'd ask of Nullable<T> instances. For example, the following snippet of code wouldn't behave as expected:

int? ni = null;
object o = ni; // What should this conversion be?
if (o == null) {
  // Should we be in this block?
}

Somasegar's weblog describes what happened next in some detail:

Clearly this had to change. We had a solution in Visual Studio 2005 Beta2 that gave users static methods that could determine the correct null-ness for nullable types in these more or less untyped scenarios. However, these methods were costly to call and difficult to remember to use. The feedback you gave us was that you expected it to simply work right by default.

So we went back to the drawing board. After looking at several different workarounds and options, it became clear to all that no amount of tweaking of the languages or framework code was ever going to get this type to work as expected.

The only viable solution was one that needed the runtime to change.

int? x = 10;
object y = x;
int? z = (int?)y; // unbox into a Nullable<int>

The lessons here? When building languages to run on top of another platform or runtime, the decisions that runtime makes often put some serious constraints around what you can do within your language. For example, looking to support first-class functors on a JVM or CLR will run into the fact that functions aren't first-class in the runtime, but instead have to be handled with object wrappers around the functions. Hiding those differences in language semantics can only get you so far, and that sometimes you need to involve the runtime team a bit more deeply if you want to close all those edge cases. (Hint to Sun: you really need to start thinking about revising and extending the JVM, instead of this current policy that essentially describes the JVM as perfect as-is. The changes made to support annotations were minor, but a good first step; it's time to open that Pandora's box wider if you want to keep up with the CLR, to be blunt about it.)

Dion Almaer made an interesting point recently:

A friend ... talked about how it is interesting that HTML is not statically typed, yet it has scaled pretty well. The internet architecture has made this happen. We are loosely coupled and modules (pages/site) are seperated out.

Ruby has its merits, Dion--you don't need to make spurious comparisons to try and justify it. Let programmers discover the beauty that is dynamically-typed programming on their own.