Thursday, November 29, 2007
Proving That, Once Again, "Corporate Management" Equals "Idiots"...

Now, I've had my issues with corporate management before (some of you may recall my run-in with Sun lawyers over the domain ""), and I've seen corporations behave badly with cease-and-desist letters (Hey, Microsoft... you and RedHat can come out of the corner now...), but this action on Sun's part to effectively muscle out the four project leads on the open-source OpenDS project definitely deserves some kind of award.

(For those of you who want the short version: four Sun employees, including Neil Wilson, the poster, have been working on the OpenDS project for the last n number of years. Sun decides to offshore its directory services development to Europe (WTF?!?), and pink-slips the four project owners. The project owners decide to continue working on the project, and decide that there should be a fifth owner, a Sun employee, in order to make sure Sun keeps a hand in a project it vested time & energy & money into. Sun decides instead to require that ALL the project owners be Sun employees. Project owners, a tad distrubed, circle back to discuss things. Sun management threaten to yank all four ex-employees' severance benefits if they fight this. Project owners basically decide to bail entirely, and not anger the Sun Gods any further, until after the benefits have been paid out. Project owner Wilson blogs the aforementioned story, which annoys a few dark corners of the blogosphere, including yours truly.)

In fact, in honor of the book "In Search of Stupidity", I hereby nominate Sun for a Stupidity Award, in this case, a two-parter: one, "For having the incredible temerity to threaten with termination the very core group of people who built the product into what it was", and two, "For having the absolute chutzpah to effectively backstab the very community-oriented efforts that Sun upper management are seeking to build as a way to make the company once again relevant in the computing industry".

I dunno if anybody at Sun reads my blog, but if you do, forward this over to Simon Phipps and Jonathan Schwartz, because if they want to be taken seriously as an open-source company, they need to find the morons who acted with this kind of heavy-handedness and defenestrate them at the earliest opportunity. An open letter of apology and response should be posted on one or both blogs. Then, they need to go back to Mr. Wilson, apologize profusely, and ask him and the other three--very nicely--if they'll agree to take up the reins of the OpenDS project again and allow Sun to demonstrate--by actions, not words--that they are a community-focused company.

Or, I suppose, they could do none of this, and prove--by actions, not words--that they don't give a sh*t. (Frankly, I could care less about directory services projects, but I would think Sun cares far more about its reputation at this point than the project itself.)

'Tis your move, Sun. Are you "profoundly involved in the open source world", or not?

Update: Simon Phipps writes, in a comment here (which may not be accessible to some RSS readers, so I will quote in full):

Ted: I have investigated this situation. It's very regerttable that Sun had to let some great engineers go as part of a staff reduction, unrelated to the governance issue, but sadly that happens in every company. The real issue I was concerned about here was the discussion of governance. I discovered that Neil did not tell the whole story.

The original governance of OpenDS said this:

<blockquote>The OpenDS project has single, overall Project Lead. This Project Lead, who is appointed by Sun Microsystems, is responsible for managing the entire project, and is the final arbiter of all decisions.</blockquote>

At some point, without any community discussion I can find, and without management discussion, while still Sun employees, the leaders of OpenDS changed the governance to say this:

<<The OpenDS project has single, overall Project Lead. This Project Lead, who is appointed and removed by a majority vote of the Project Owners, is responsible for managing the entire project, and is the final arbiter of all decisions in the development process.>>

You can see the change in the Subversion system for OpenDS here.

When Neil says:

<<On November 14, 2007, a member of executive management within Sun’s software division contacted one of the recently-laid-off OpenDS project owners and demanded that the owners approve a governance change that would grant Sun full control of the OpenDS project.>> that is misleading. What actually happened is the leaders were asked to revert their arbitrary change of the governance (they also removed the phrase "Project Owners will always attempt to reach consensus before making decisions") so that the matter could be openly discussed.

I'm continuing to investigate and monitor, but right now the evidence I see is that Sun has not acted in bad-faith towards the governance. I'd welcome evidence to the contrary if it exists, but that SVN dif seems pretty conclusive to me.

There are, I think, two issues here that are getting conflated, and I want to tease them apart. What is not an issue, as far as I'm concerned, was Sun's decision to let these engineers go in the first place. Business decisions that involve letting people go are always hard decisions to make, and I won't second-guess the thought process that led up to it. I'm willing to believe that if Sun management felt they could have kept them in place, they would have. (You may choose to disagree but that's your own value judgement to make.) That doesn't take away from the two issues I want to discuss here.

First, there is the issue of whether or not Sun's actions were in accordance with the project governance model for the OpenDS project, which Simon seems to focus on above. Frankly, I could care less about this aspect of things; whether that change was appropriate, necessary, legal, whatever, is not the issue I want to discuss or point out. Yes, there is the possibility that Neil-and-company's actions were regarded as being contrary to the charter of the project by the legitimately-selected Project Owner, and Sun felt it necessary to retaliate legally. Even in that scenario, it strikes me as a heavy-handed reaction.

(Quite footnote: on a whim, after posting my update, I checked the SVN revisions beyond what Simon posted. The revision change in question, from 1622 to 1739, occured on April 28th, 2007, which is close enough to have been part of this whole fiasco, but obviously I'm wasn't there, I can't know that for certain. Needless to say, if all these changes occurred without any community input, that's clearly not playing to the spirit of the open source community, IMHO.)

Secondly, there is the issue of Sun's threat to revoke their severance benefits. To me, this is unconscionable. Severance is a way of providing valuable employees with a "cover period" while they seek new employment ("we were encouraged to use this time to find a new job", Neil reports). This is a nice gesture, a way of saying "thank you" for the valuable efforts of a valued employee. But to turn around and threaten to remove those benefits if the employees don't play ball on a job-related issue when those employees are seeking to further the company's stated cause seems heavy-handed and unnecessary.

Look, in order to be seen as effective without being heavy-handed, keep the scope of the response within the frame of the supposed offense: If they are violating the governance model, then threaten to remove them from the project. If they are seeking to use the code in a way that is violating the license of the original, then threaten legal action over the licensing model and those actions. To go straight for the severance benefits here seems akin to Pres. Bush reaching for the nuclear codes when a foreign national jaywalks. The punishment must fit the crime. And the crime has to be proven first, in any event. The risks of a "preventative first strike" are only acceptable when the liabilities are equally as high, if not higher.

Look at it this way: the way this played out, Sun...

  1. ... kept control over the OpenDS project... which they were going to have anyway, under either side's story.
  2. ... still paid out the severance benefits for those engineers.
  3. ... earned a negative PR hit among the very people they're trying to convince, the open-source community.
  4. ... earned the enmity of the core developers working on the OpenDS project, which I presume they would have preferred to have continue working on the project, even if they no longer paid their salary.
  5. ... forced their Chief Open Source Officer to spend time investigating this whole sordid mess.

What might have happened if Sun had simply approached the engineers in question and instead of threatening to yank their benefits, had said "Look, we understand what you're trying to do, but as we understand it, the project's governance statement says otherwise... We value you guys, we value what you did, we want you to continue to contribute, how do we resolve this amicably?" Sun might have had to compromise--noting, by the way, that they might have done this even when legally or morally they didn't have to--and would have instead...

  1. ... kept control over the OpenDS project.
  2. ... still paid out the severance benefits for those engineers.
  3. ... earned a positive PR boost at best, no PR at worst, among the community.
  4. ... kept the goodwill of the developers who represented Sun's efforts on the project.
  5. ... had a "feel-good" story their Chief Open Source Officer could use when addressing the OpenDS group at the next directory services conference. ("Look, Sun is committed to open source, even when we can't contribute the resources financially, we can prove this by...")

Sometimes, even when you're right, you get more from taking the position that there is room to compromise on your end. Flies, honey and vinegar, and all that. I applaud Simon's efforts to get involved and track this one down, but I don't think this is an open-source issue.

I think this is a management issue.

.NET | Java/J2EE

Thursday, November 29, 2007 12:04:33 AM (Pacific Standard Time, UTC-08:00)
Comments [2]  | 
 Friday, November 23, 2007
When is something not ready for Prime-Time?

When you can't do something simple with it.

Check out this absurdly simple Groovy script:

import as JMXFactory
import as JMXUrl

def serverUrl = 'service:jmx:rmi:///jndi/rmi://'
def serv = JMXFactory.connect(new JMXUrl(serverUrl))
def on = new ObjectName('Catalina:type=Server')
def gmb = new GroovyMBean(serv, new ObjectName('Catalina:type=Server')).serverInfo

For those of you not up on your JMX, this is a simple connection via an RMI connector to the JMX server running at port 9004 (which happens to be my local Tomcat installation). This is straight off the Groovy-JMX documentation page, but slimmed down because the GroovyMBean constructor throws an exception, claiming that the desired constructor cannot be found:

C:\Projects\Exploration\Groovy>groovy GroovyJMX.groovy
Caught: groovy.lang.GroovyRuntimeException: Could not find matching constructor
for: groovy.util.GroovyMBean(,
at GroovyJMX.main(GroovyJMX.groovy)

C'mon, folks. I have no idea what the problem is, and debugging this is a nightmare. I've looked around various Groovy forums, and nobody appears to have any real idea what's going on. Either nobody is really using Groovy as a JMX client (in which case, just remove the GroovyMBean from the library), or else Groovy has a bug within it (thus reducing its efficicacy as a production-ready language).

I'm fully willing to accept that the problem is with me or my environment somehow. The challenge, however, is for somebody to take a stock JDK 1.6 and Groovy 1.0 download (oh, and 1.1-rc2 fails with the same error, so that's not the issue, either), run the above 8-line script, and tell me why mine isn't working. (I've already done the suggested step of removing the mx4j jar out of the groovy-1.0/lib directory, so that doesn't help, either.)

Oh, and if you're going to write in claiming that this is a ClassLoader issue or something, you'd be wrong--all of the JMX types being loaded are coming out of rt.jar (which I verified using -verbose:class, doing which required me to edit the Groovy launcher scripts, which I find to be just silly--don't make it hard for me to use the basic management & monitoring facilities of the JVM). The only ClassLoader player I don't know for certain is the org.codehaus ClassLoader that's established by Groovy itself, so if the problem is in ClassLoaders, it's inside of the Groovy implementation, which means it's a Groovy problem, not mine.

I've even taken the step to compile the Groovy code into .class files, and run those:

C:\Projects\Exploration\Groovy>groovyc -d compiled GroovyJMX.groovy
C:\Projects\Exploration\Groovy>cd compiled

C:\Projects\Exploration\Groovy\compiled>java -classpath .;libg\groovy-1.0.jar;li
bg\asm-2.2.jar GroovyJMX
Exception in thread "main" groovy.lang.GroovyRuntimeException: Could not find ma
tching constructor for: groovy.util.GroovyMBean(
at groovy.lang.MetaClassImpl.invokeConstructor(
at groovy.lang.MetaClassImpl.invokeConstructor(
at org.codehaus.groovy.runtime.Invoker.invokeConstructorOf(
at org.codehaus.groovy.runtime.InvokerHelper.invokeConstructorOf(Invoker
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeNewN(ScriptBy
at gjdk.GroovyJMX_GroovyReflector.invoke(Unknown Source)
at groovy.lang.MetaMethod.invoke(
at org.codehaus.groovy.runtime.MetaClassHelper.doMethodInvoke(MetaClassH
at groovy.lang.MetaClassImpl.invokeMethod(
at groovy.lang.MetaClassImpl.invokeMethod(
at org.codehaus.groovy.runtime.Invoker.invokeMethod(
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.
at org.codehaus.groovy.runtime.InvokerHelper.runScript(InvokerHelper.jav
(Unknown Source)
at groovy.lang.MetaMethod.invoke(
at org.codehaus.groovy.runtime.MetaClassHelper.doMethodInvoke(MetaClassH
at groovy.lang.MetaClassImpl.invokeStaticMethod(
at org.codehaus.groovy.runtime.Invoker.invokeMethod(
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(Scrip
at GroovyJMX.main(GroovyJMX.groovy)


And here we can obviously see, by the light of the .jars passed in, that the mx4j jars are nowhere to be found (unless, of course, Groovy is explicitly searching the hard drive for them, because I deleted them entirely out of Groovy's lib directory).

I've asked a couple of the Groovy heavyweights (not mentioning anybody by name) if they know what's up. Silence. Not a good sign.

I'm about to go off and try the same thing using JRuby. If it works, out of the box, then the problem is with Groovy, not with me. If you're a Groovy expert, or know someone who is, then have them email me the solution (assuming they can find one), because this is a point where I'm about to close the door on Groovy forever: if you can't do something simple with it, it's not ready for prime-time. It may be OK to whip up dirt-simple websites where 90% of the stuff is pre-generated, but if I can't use it for something like being a JMX client, then it ain't worth my time.

'Nuff said.


Update: OK, I may have to eat my words.

Playing around with some JRuby/JMX stuff (which is absurdly simple, even without jmx4r, which I'm trying to get gem to install right now), I went back and decided to try a little JMX without using GroovyMBean:


def serverUrl = 'service:jmx:rmi:///jndi/rmi://'
def connector = JMXConnectorFactory.connect(new JMXServiceURL(serverUrl))
def mbsc = connector.mBeanServerConnection

def memory_mbean =
ManagementFactory.newPlatformMXBeanProxy(mbsc, "java.lang:type=Memory", MemoryMXBean.class)

This, by the way, is almost a straight port of Jeff's corresponding JRuby/JMX example code. Works just fine. From there, I thought, "Let's see if I can get past the problem I was having with GroovyMBean a few minutes ago." So I add the one-line GroovyMBean constructor (taking the mBeanServerConnection as the first parameter):

def gmb = new GroovyMBean(mbsc, new ObjectName('Catalina:type=Server')).serverInfo

Voila! Success! A quick execution test verifies that I'm all good:

def query = new ObjectName('Catalina:*')
String[] allNames = mbsc.queryNames(query, null)
def modules = allNames.findAll{ name ->
}.collect{ new GroovyMBean(mbsc, it) }

println "Found ${modules.size()} web modules. Processing ..."

modules.each{ m ->
    println "Found ${} at ${m.path} (${m.processingTime})"


C:\Projects\Exploration\Groovy>groovy GroovyJMX.groovy
Found 5 web modules. Processing ...
Found Catalina:j2eeType=WebModule,name=//localhost/docs,J2EEApplication=none,J2E
EServer=none at /docs (0)
Found Catalina:j2eeType=WebModule,name=//localhost/host-manager,J2EEApplication=
none,J2EEServer=none at /host-manager (0)
Found Catalina:j2eeType=WebModule,name=//localhost/examples,J2EEApplication=none
,J2EEServer=none at /examples (0)
Found Catalina:j2eeType=WebModule,name=//localhost/,J2EEApplication=none,J2EESer
ver=none at (0)
Found Catalina:j2eeType=WebModule,name=//localhost/manager,J2EEApplication=none,
J2EEServer=none at /manager (0)

Now if that ain't cool, I don't know what is.

Now for the hard part: What happened? Why'd my earlier code fail?

Looking back at my example script above, I clearly see that somewhere along the way in my debugging/exploration, I accidentally left out the call to obtain the MBeanServerConnection from the Connector and pass that in to the GroovyMBean constructor. Ugh.

Formal apologies to the Groovy crowd. However, I stipulate, for your own consideration, that

  1. Originally my script was not so--I copied it line-for-line from the Groovy/JMX page, and I got the GroovyCastException that led me down this path. (You decide whether you believe me or not. :-) )
  2. The GroovyMBean constructor could (and should) be overloaded to take a Connector and extract the MBeanServerConnection from it and proceed. In fact, I'd suggest that the GroovyMBean should be written to take a JMXServiceURL as a parameter and handle all the details of connecting to the remote server.
  3. Exploration tests would have yielded a better record of my efforts, and maybe found the point where my coding got led astray. It would also help track down the problem for others, if there was a language issue at stake here. (There may be--one noticeable difference is that in the working version, I don't use the import-redeclaration feature, whereas in the non-working version I did. More research is required.) At the time, I was just trying to debug a simple script problem, but at some point, greater rigor on my part should have kicked in so I could have better results to work from. Sigh.
  4. Debugging this was way too hard. As we move into an era of more languages-atop-VMs, better debugging/spelunking facilities has to be a top priority.
  5. Groovy (and JRuby) need to make sure they honor--somehow--the various flags I want to pass to the Java VM, such as -verbose or -client/-server, or even -ms or -mx. Java does have the JAVA_TOOLS_OPTIONS environment variable that will be picked up by the Java launcher (java.exe), but that only works if the language in question uses the launcher itself as part of its startup script. Both Groovy and JRuby seem to fail this test (though I admit I didn't look super-hard for undocumented/underdocumented ways to do it).


Friday, November 23, 2007 4:18:07 AM (Pacific Standard Time, UTC-08:00)
Comments [7]  |