Sunday, February 22, 2009
As for Peer Review, Code Review?

Interesting little tidbit crossed my Inbox today...

Only 8% members of the Scientific Research Society agreed that "peer review works well as it is". (Chubin and Hackett, 1990; p.192).

"A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research." (Horrobin, 2001)

Horrobin concludes that peer review "is a non-validated charade whose processes generate results little better than does chance." (Horrobin, 2001). This has been statistically proven and reported by an increasing number of journal editors.

But, "Peer Review is one of the sacred pillars of the scientific edifice" (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and "Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process" (Horrobin, 2001).


Chubin, D. R. and Hackett E. J., 1990, Peerless Science, Peer Review and U.S. Science Policy; New York, State University of New York Press.

Horrobin, D., 2001, "Something Rotten at the Core of Science?" Trends in Pharmacological Sciences, Vol. 22, No. 2, February 2001. Also at and (both pages were accessed on February 1, 2009)

Goodstein, D., 2000, "How Science Works", U.S. Federal Judiciary Reference Manual on Evidence, pp. 66-72 (referenced in Hoorobin, 2000)

I know that we don't generally cite the scientific process as part of the rationale for justifying code reviews, but it seems to have a distinct relationship. If the peer review process is similar in concept to the code review process, and the scientific types are starting to doubt the efficacy of peer reviews, what does that say about the code review?

(Note: I'm not a scientist, so my familiarity with peer review is third-hand at best; I'm wide open to education here. How are the code review and peer review processes different, if in fact, they are different?)

The Horrobin "sacred pillars" quote, in particular, makes me curious: Don't we already apply "scientific [and engineering] methods" to the peer review process? And can we honestly say that we in the software industry apply "scientific [and engineering]" methods to the code review process? Can we iterate the list? Or do we just trust that intuition and "more eyeballs" will help spot any obvious defects?

The implications here, when tied up next to the open source fundamental principle that states that "more eyeballs is better", are interesting to consider. If review is not a scientifically-proven or "engineeringly-sound" principle, then the open source folks are kidding themselves in thinking they're more secure or better-engineered. If we conduct a scientific measurement of code-reviewed code and find that it is "a non-validated charade whose processes generate results little better than does chance", we've at least conducted the study, and can start thinking about ways to make it better. (I do wish the email author had cited sources that provide the background to the statement, "This has been statistically proven", though.)

I know this is going to seem like a trolling post, but I'm genuinely curious--do we, in the software industry, have any scientifically-conducted studies with quantifiable metrics that imply that code-reviewed code is better than non-reviewed code? Or are we just taking it as another article of faith?

(For those who are curious, the email that triggered all this was an invitation to a conference on peer review.

This is the purpose of the International Symposium on Peer Reviewing: ISPR ( being organized in the context of The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009 (, which will be held on July 10-13, 2009, in Orlando, Florida, USA.

I doubt it has any direct relevance to software, but I could be wrong. If you go, let me know of your adventures and conclusions. ;-) )

Conferences | Development Processes | Security | Windows

Sunday, February 22, 2009 10:36:43 AM (Pacific Standard Time, UTC-08:00)
Comments [9]  | 
 Wednesday, February 18, 2009
Woo-hoo! Speaking at DSL DevCon 2009!

Just got this email from Chris Sells:

For twelve 45-minute slots at this year’s DSL DevCon (April 16-17 in Redmond, WA), we had 49 proposals. You have been selected as speakers for the following talks. Please confirm that you’ll be there for both days so that I can put together the schedule and post it on the conference site. This DevCon should rock. Thanks!

Martin Fowler - Keynote

Paul Vick + Gio - Mgrammar Deep Dive

Tom Rodgers - Domain Specific Languages for automated testing of equity order management systems and trading machines

Paul Cowan - DSLs in the Horn Package Manager

Guillaume Laforge - How to implement DSLs with Groovy

Markus Voelter - Eclipse tooling for Model-Driven stuff

Dionysios G. Synodinos - JavaScript DSLs for the Client Side

Ted Neward, Bradford Cross - Functional vs. Dynamic DSLs: The Smackdown

Gilad Bracha - embedding EBNF in a general purpose language

Umit Yalcinalp, Tilman Giese - RUMBA: RUby Managed Business data for Applications

Bob Archer - A DSL for Cool Effects in Adobe Pixel Blender

Chance Coble - Language Oriented Programming in F#

As my 15-year-old son Michael has grown fond of saying... w00t! The list of topics is fascinating, and I'm really looking forward to most, if not all, of them. Chance's talk on LOP in F# should be good, I'm really curious to see Gilad's discussion of EBNF (and wondering if this is Newspeak we'll be seeing), and Guillaume is always fun to watch when he's going on about Groovy. Of course, I'm also excited to be paired up with Brad, who's an insanely smart guy--I have a feeling I'll learn a lot just by standing next to him. (Sort of a speakers' osmosis.)

If you're not planning to be here for this (and the Lang.NET Symposium), either you have life-saving surgery scheduled that can't be pushed back, or you're clearly not interested in DSLs. For your own sake, I hope it's the latter. ;-)

Seriously, come for the full week. The Lang.NET Symposium last year was an amazing event, for a number of reasons, not the least of which is that it saw Sun celebrities John Rose, Charlie Nutter and Brian Goetz step on to the Microsoft campus, deliver a great presentation on the JVM, MLVM/invokedynamic, and JRuby, and get good feedback and discussion from Microsoft engineers and other notables. You don't get to see that every day. :-)

.NET | C# | C++ | Conferences | F# | Flash | Java/J2EE | Languages | Ruby | Visual Basic | Windows | XML Services

Wednesday, February 18, 2009 4:29:25 PM (Pacific Standard Time, UTC-08:00)
Comments [0]  | 
 Tuesday, February 17, 2009
What do beer, London, Alt.NET and ThoughtWorks have in common?

Answer: "I don't know, but I'm damn well going to find out!"

(Now I really wish I were in London. Ah, well, will just have to go see Ward Cunningham speak at Alt.NET Seattle, instead.)

.NET | C# | Conferences | F# | Languages | Social | WCF | Windows | XML Services

Tuesday, February 17, 2009 10:26:00 AM (Pacific Standard Time, UTC-08:00)
Comments [0]  | 
 Saturday, February 14, 2009
NOW you know why you want to learn Haskell

Matt Podwysocki makes it all clear:


Hey, I'd have learned Haskell a LONG time ago if I'd known it could yield up a beer!

F# | Java/J2EE | Languages | Reading | Social

Saturday, February 14, 2009 12:41:48 AM (Pacific Standard Time, UTC-08:00)
Comments [2]  | 
 Friday, February 6, 2009
Nice little montage from JDD08

Last year I had the opportunity to return to the land of my roots, Poland, and speak at Java Developer Days (JDD). Just today, the organizers from JDD sent me a link with a nice little photo montage from the conference. (I did notice a few photos from the after-party were selectively left out of the montage, however, which is probably a good thing because that was the first time I'd ever met a Polish Mad Dog, and boy did they all go down easy...)

If you're anywhere in the area around Krakow in March, you definitely should swing by for their follow-up conference, 4Developers--it sounds like it's going to be another fun event, and this time it's going to reach out to more than just the Java folks, but also the .NET crowd (and a few others), as well.

(I don't really expect any of the readers of this blog living outside Poland to really pack up and head over to Krakow for a weekend, mind you, but if you're a technology speaker and you're interested in hanging with an extremely good group of people, the people who put these shows on--ProIdea--are top-notch, take great care of the speakers, and overall make the entire experience well worth the trip.)

.NET | C# | Conferences | F# | Java/J2EE | Languages | Parrot | Ruby | WCF | Windows | XML Services

Friday, February 6, 2009 2:17:15 AM (Pacific Standard Time, UTC-08:00)
Comments [1]  |