Monthly Archives: March 2008

Sorting by Schwartzian transform in Scala

The term “Schwartzian transform” was named in honor of Randall Schwartz by Tom Christiansen. (If you recognize either name, you might be thinking, “What would Perl have to do with Scala?” I’ll simply observe that the idea of a multi-paradigm language is an old one.)

At JPR 2008, Dianne Marsh mentioned an article that included a description of the orderby keyword in LINQ. The resulting conversation explored the strategy described below.

The example

Let’s use a minimal Employee class for our discussion:

  class Employee(
      val id:    Int,
      val dept:  String,
      val first: String,
      val last:  String
  ) {
    override def toString() =
      "(" + id + ") " + first + " " + last + " in " + dept

Here’s a list of sample data, used as the basis for a list of employees:

  val data = List(
      (314, "Payroll", "Ursula", "Ficus"),
      (159, "Sales", "Ville", "Eave"),
      (265, "Development", "Wilbur", "Dim"),
      (358, "Sales", "Xeris", "Confabulation"),
      (979, "Development", "Yolanda", "Borg"),
      (323, "Development", "Ziggy", "Ableton")
  val employees =
    for ((id, dp, fn, ln) <- data) yield new Employee(id, dp, fn, ln)

Note that we could mix in the Ordered trait, implementing compare and equals methods, to create a standard ordering for Employee. We won’t use that approach here because we intend to sort employee lists in a variety of ways: alphabetically by name, by employee ID, or even by length of name.

The goal

Our immediate objective is to create a function to sort a list of objects by identifying the sort criteria. The result of the discussion is the sortBy function, which allows us to sort our employees by first name and employee ID by writing as little as:

  sortBy ((e: Employee) => (e.first, (employees)

The remainder of this post describes how we compose such a function. There’s certainly room for further enhancement, but getting this far was enough for one conversation (and more than enough for one article).

The standard sort method

Scala’s List class has a pre-defined sort method which uses a caller-provided function to determine whether a pair of elements is in order. To obtain a list sorted by a single Employee attribute, such as id, we can write the expression:

  employees.sort((a, b) => <

Because it takes only a single argument, the sort method can be written as…

  employees sort ((a, b) => <

…which is the style I’ll use for the remainder of this article.

Scala also doesn’t require us to specify the type for arguments a and b; the compiler can infer that they have type Employee because they are coming from employees, which has type List[Employee]. (In this case, we could go further and use placeholder syntax to abbreviate the above to…

  employees sort ( <

…but we can’t use that abbreviation for functions that refer to the same parameter more than once.)

As our sorting criteria are more complex (i.e. sorting by length of name, with last name and then first name as tie-breakers), the in-line syntax becomes cumbersome. We can define and use a method which returns Boolean, as in:

  def lessByNameLength(a: Employee, b: Employee): Boolean = {
    val bylength = (a.first + a.last).length - (b.first + b.last).length
    if (bylength == 0) {
      val bylastname = a.last compare b.last
      if (bylastname == 0) {
        a.first < b.first
      } else {
        bylastname < 0
    } else {
      bylength < 0
  // ...
  val employeesByNameLength = employees sort lessByNameLength

That approach provides the per-call flexibility we want, but it has two drawbacks:

  1. it is “boilerplate-heavy”, and
  2. it re-derives the ordering data for each operand, regardless of how many times the operand has been examined previously.

With respect to point #1, the multiple-key comparison requires much more typing, hides our intention in a mound of implementation detail, and is therefore much more error-prone. (For example, if a.first < b.first had been mistyped as a.first > b.first, how long would it take to find the typo-bug?)

We’ll address the boilerplate issue after we discuss the other drawback.

Schwartzian transform

A typical, well-chosen sorting algorithm requires O(n log n) comparisons. The key idea (pun intended) of the Schwartzian transform is to:

  • compute the sort key for each value in the collection and attach it to that value,
  • sort the key/value combinations based on the pre-computed keys, and
  • extract the sorted values from the key/value combinations.

This is a nice application of the time-honored mathematical strategy of transforming a problem to a representation which makes work easier, doing the work, and then transforming the result back (f = g · h · g-1). Each stage can be written and examined individually, a core idea in functional programming. Re #2 above, this approach performs the key computation exactly once per original element, during the setup stage, rather than repeating it every time a pair of elements is compared.

So what do we do with the keys, and how do we use them in the sorting process?

Tuples as keys

A nice feature of Scala tuples is the implicit conversion to Ordered for a tuple whose fields have Ordered types. For example…

  val data = List(
    ("my", 9),
    ("ukulele", 8),
    ("has", 7),
    ("my", 6),
    ("dog", 5),
    ("has", 4),
    ("fleas", 3)
  println(data sort (_ < _))

…produces output of…

  List((dog,5), (fleas,3), (has,4), (has,7), (my,6), (my,9), (ukulele,8))

…in which the values are ordered by the first element of each tuple, with the second element serving to break ties. Because String and Int are both Ordered, Scala can treat Tuple2[String,Int] as Ordered as well.

It’s easy to write a function that produces a key tuple from an instance of Employee, especially if the key is simply made up of fields! Examples of this include:

  def byName       (e: Employee) = (e.last, e.first)
  def byDeptId     (e: Employee) = (e.dept,
  def byNameLength (e: Employee) = ((e.first + e.last).length, e.last, e.first)

That idea, along with map, zip, and the sort method described earlier, provide all the raw materials we need.

Bolting it together

After reviewing the map and zip methods, we’re ready to start assembling a solution.

If Visitor is the standard illustration for OO design patterns, then map probably fills that role for functional programming. It’s the first syllable in “mapReduce“, and also makes an occasional guest appearance when closures are being discussed.

The expression

  employees map byDeptId 

applies byDeptId to each element of employees and returns the results as a List (which I’ve wrapped to fit on the page):

  List((Payroll,314), (Sales,159), (Development,265),
       (Sales,358), (Development,979), (Development,323))

Associating each key with the corresponding employee is the job of zip, which combines two sequences (as if matching the two sides of a zipper). Given two lists, aList and bList, the expression aList zip bList returns a list of pairs (instances of Tuple2), each of which contains an element of aList and the corresponding element of bList. In our case, that means that…

  employees map byDeptId zip employee

…gives us the list of key/value pairs we need to sort our employees by department and then by ID. Because we want to sort only by the key part of each pair, we need to use sort with a function that compares just the keys. The _1 method retrieves the first element of a Tuple2 instance, so we’ll write:

  employees map byDeptId zip employees sort (_._1 < _._1)

With key/value pairs in the desired order, we are ready to extract the values to form the result, which we can do by adding another map to the pipeline:

  employees map byDeptId zip employees sort (_._1 < _._1) map (_._2)

Evaluating that expression gives us a sorted list. Printed one element per line (via the toString method) it contains:

  (265) Wilbur Dim in Development
  (323) Ziggy Ableton in Development
  (979) Yolanda Borg in Development
  (314) Ursula Ficus in Payroll
  (159) Ville Eave in Sales
  (358) Xeris Confabulation in Sales

Getting functional

Now that we can write an expression that does what we want in a specific instance, we want a general function that captures the core pattern. Looking at the last expression above, it’s clear that the key-extraction function and the list of values are the two parts that need to vary as we re-use the idea. The general template is…

  employees map ? zip ? sort (_._1 < _._1) map (_._2)

…so we just need to fill in the blanks in the following function definition:

  def sortBy ? (f: ?) (vs: ?): ? = {
    (vs map f zip vs) sort (_._1 < _._1) map (_._2)

If the values (the vs parameter) are provided in a list of some type, then the result should be a list of the same type. Because that type can very, we will use type parameters. Choosing V for our value type, we get:

  def sortBy[V, ?] (f: V => ?) (vs: List[V]): List[V] = {
    (vs map f zip vs) sort (_._1 < _._1) map (_._2)

Specifying the remaining type parameter is a bit tricky; we need for f to map from type V to type K (for key)…

  def sortBy[V, K ?] (f: V => K) (vs: List[V]): List[V] = {
    (vs map f zip vs) sort (_._1 < _._1) map (_._2)

…however, K must be a type we can sort. As mentioned in the “Tuples as keys” section, Scala has an implicit conversion to Ordered for tuples (for Tuple2 through Tuple9, actually). That means that we must specify that K can be converted implicitly to Ordered[K], which give us the last bit of type incantation:

  def sortBy[V, K <% Ordered[K]] (f: V => K) (vs: List[V]): List[V] = {
    (vs map f zip vs) sort (_._1 < _._1) map (_._2)

While that last detail is not the sort of thing you’d likely want to write throughout your application code, you don’t have to! That last definition is the sort of thing (pun intended 😉 that you’d put in a library (utility jar, etc.) and just use throughout your application. With that function in hand, we’re now free to write…

  val sortedEmps = sortBy ((e: Employee) => (e.first, (employees)


  def byNameLength (e: Employee) = ((e.first + e.last).length, e.last, e.first)
  // ...
  val sortedEmps = sortBy (byNameLength)(employees)

…or even…

  def byNameLength (e: Employee) = ((e.first + e.last).length, e.last, e.first)
  def sortEmployeesByNameLength = sortBy (byNameLength) _
  // ...
  val sortedEmps = sortEmployeesByNameLength(employees)

…if you’ve read the post on currying, that is.


Why functional programming?

The canonical answer to that question is probably “Why functional programming matters“, but here’s a specific example that makes the case nicely.

Neil Mitchell is working on Supero, an optimizing compiler for Haskell which includes some ideas from supercompilation. But that’s not important right now.

What is important is the technique Mitchell uses in the blog post at the second link above. Algebra. As in junior-high-Algebra-I-algebra. The old “if a = b then you can substitute either one for the other” kind of Algebra.

Functional programming is based on defining functions.

  def square(i: Int) = i * i

In pure functional programming, where side-effects are not allowed (or at least kept securely behind bars), you can treat that definition as an equation; wherever you see square(foo) you can rewrite it as (foo * foo) and vice versa. This applies even when foo is an arbitrarily complex expression (don’t worry about operator precedence or extra parentheses right now; that’s not the point).

The point is that you can’t do that with confidence for impure (i.e., having side-effects) languages. Consider trying that replacement in something as simple as…

  n = square(++i);

…which clearly is not the same as…

  n = ++i * ++i;

…at least in most cases. Are there any special cases where it’s the same? Would the answer be different if the enclosed expression had been i++ instead?

If you even had to think about it for one millisecond (or, like me, thought “That’s ugly! I’ll think about it later!”) that’s exactly the point. In a pure language the question can’t even come up. You’re free to do the rewriting, either way, any time.

(Fair warning: You don’t need to know Haskell to read Mitchell’s post, but it helps. Even if you don’t, it’s worth the little bit of time it takes to work through his example. The cheat sheet at the bottom of this post is a quick introduction for non-Haskell-speaking readers, or non-Haskell-reading speakers.)

Mitchell takes a simple task, breaking a string into a list of words, and shows a simple and obvious function for the task. However, that simple and obvious function ends up with redundant “is it a space?” tests against some characters. Mitchell does a bit of refactoring, using simple substitute-equals-for-equals steps, to track down and eliminate the redundance, resulting in an equivalent (still simple) function that runs faster.

All of this is based on the facts that:

  • more powerful functions can be composed out of simpler ones, and
  • pure functions ensure that the obvious substitutions work.

For me this makes a powerful demonstration of the benefits of the functional approach, whether I’m using Haskell, Scala, or even a non-FP language such as Java. (Not only does functional programming matter, knowing why functional programming matters matters as well.)

Cheat sheet

For newcomers to Haskell notation, here’s a bit of explanation. Please don’t hold the length (or any shortcomings) of my explanation against Haskell. I’m assuming a reader who’s never seen Haskell before (or any similar language), and so say something about almost every token. Feel free to skip the obvious, with my apologies.

Mitchell’s analysis works on this original function…

  words :: String -> [String]
  words string = case dropWhile isSpace string of
                     [] -> []
                     s -> word : words rest
                         where (word, rest) = break isSpace s

…which breaks a string into a list of words (strings).

The special-character soup isn’t really that bad, if you use your Captain Zowie decoder ring. It helps to know three facts:

  • indentation shows nested structure, instead of using curly braces,
  • function application doesn’t require parentheses, and
  • a Haskell String can be considered as a list of characters.

The first line…

  words :: String -> [String]

…declares the type of words as function from string to list of strings. Specifically:

  • the double-colon can be read “has type”,
  • the arrow indicates a function taking the type on the left, yielding the type on the right, and
  • the square brackets on the right read “list of”.

The second line…

  words string = case dropWhile isSpace string of

…starts the definition of words, using a parameter named string (which we already know is of type String from the first line). The phrase…

  dropWhile isSpace string

…returns a left-trimmed version of string. It applies isSpace to successive elements of string, repeating as long as the result is true, and returns what’s left (either because there’s nothing left or because a character is found for which isSpace is false). It does so without modifying the original string, just as String#substring does in Java. That left-trimmed (sub-)string is then subjected to a case analysis, by way of pattern matching.

Remember that the result of left-trimming was produced either because there were no characters left, or because a non-space character was found. Therefore, there will be two cases to consider. The line reading…

  [] -> []

… reads “if you have an empty list (of characters) then return an empty list (of Strings)”. We (and the Haskell compiler!) can infer the parenthesized types because we know that trimming a string produces the String (list of characters) being inspected, and the type declaration on the first line says that the function produces a list of String.

The last two lines address the case in which there actually were some characters left after trimming.

  s -> word : words rest
      where (word, rest) = break isSpace s

For a non-empty string s… Wait! What’s s and how do we know it’s a non-empty string? Remember that we’re in a case (glance back at the original above to see the indentation) and we’ve already eliminated the possibility that the trimmed string we’re examining is empty. So the value in hand must be non-empty. And pattern matching give us the ability to name things on-the-fly, more or less the equivalent of defining a local variable in e.g. Java. So, as I was saying before I so rudely interrupted myself…

For a non-empty string s, the result is a word followed by the words from the rest of the string, where those two things (word and rest) are constructed by breaking s at the first space (the first character for which the function isSpace returns true). The single-colon is often read as “cons”, in homage to Lisp. It constructs a list (of some type) by gluing a value (of the correct type) onto the beginning of a list (of the same type). Becuase word is a String and words rest is a list of String, the entire expression word : words rest is also a list of String.

Notice that both dropWhile and break have two arguments: another function (isSpace in both uses here) and a list. This works because dropWhile and break are higher-order functions, which take functions as arguments and use them whenever needed. If you cross your eyes and squint a little, this begins to look like design patterns in OO.

The way to take notes

OK, I enjoyed taking notes at JPR08 with the EeePc, but this guy’s notes are the best I’ve ever seen! I just had to point you to them.

Scala currying without lab rats

What is it?

In normal function evaluation, you supply all of the arguments of a function and get back a result. Currying is a technique for specializing a function by providing some of the arguments, in which case you get back a function with fewer arguments. This technique is widely used in functional programming, and is named after logician and mathematician Haskell Curry (who is also honored by a programming language using his first name).

As the well-known joke goes, “I’m glad we don’t have to call it Schoenfinkeling!”

A practical example

Some US states compute sales tax by a scheme similar to the following: there are two rates applied, one to the entire taxable value, the other to some part of the value up to a limit. (And it obviously gets more complicated than that, but this is enough real-world complexity for our example.)

For example, here are the sales tax policies for three hypothetical states.

Abbr. State City Tax computation
TF Terrafirma n/a 2% of the entire value
TC Terracotta n/a 6% on the first $1500
CF Confusion Gotham 5% on the entire value plus 4.5% on the first $1000
CF Confusion Ocean City 5% on the entire value plus 3% on the first $1000
CF Confusion Hometown 5% on the entire value plus 1.5% on the first $1000

Without currying

With a helper to compute a rounded percentage…

    def pct(rate: Double, amt: Long) =
(rate * amt / 100.0D + 0.5D).toLong

… we could fit all these cases into the single method below. (To keep things simple, we’re representing money amounts as whole cents.)

    def tax(rateA: Double, limit: Long, rateB: Double, amt: Long) =
pct (rateA, amt) + pct (rateB, limit min amt)

With that general method in hand, we can define functions for the various locations.

    def taxTF         (amt: Long) = tax(2.0D,      0, 0.0D, amt)
def taxTC         (amt: Long) = tax(0.0D, 150000, 6.0D, amt)
def taxGothamCF   (amt: Long) = tax(5.0D, 100000, 4.5D, amt)
def taxOceanCityCF(amt: Long) = tax(5.0D, 100000, 3.0D, amt)
def taxHometownCF (amt: Long) = tax(5.0D, 100000, 1.5D, amt)

The significant changes are hidden in the mass of boilerplate code, which means a large amount of redundant typing (and reading). Let’s look a the difference when we revise the implementation to use currying.

Now with a dash of curry

First, we replace the method above with a curried function. (The extra line-breaks aren’t significant; they’re just fitting the slightly longer line into a narrow blog page layout.)

    def tax =
(rateA: Double) =>
(limit: Long  ) =>
(rateB: Double) =>
(amt:   Long  ) =>
pct (rateA, amt) + pct (rateB, limit min amt)

The first improvement shows up when we define the sales tax functions for Terrafirma and Terracotta:

    def taxTF = tax( 2.0D)(     0)(0.0D)
def taxTC = tax( 0.0D)(150000)(6.0D)

Currying lets us define the specialized functions directly and simply, without the need for all the boilerplate. Providing only the first three arguments to tax, we get back functions that take the one remaining argument, the amount of the sale.

It gets even nicer when we start to deal with the great state of Confusion. We begin by defining a state-level function…

    def taxCF = tax( 5.0D)(100000)

… which we then use in the definitions of the individual locales.

    def taxGothamCF    = taxCF( 4.5D)
def taxOceanCityCF = taxCF( 3.5D)
def taxHomeTownCF  = taxCF( 1.0D)

Could we do this without currying? Of course! But the point is that the curried version eliminates almost all of the typing except for the part that’s significant: the arguments that change for each specialized version. By supporting currying and the use of functions as first-class values (e.g., as arguments of other functions, as returned values from functions, etc.), Scala and other functional languages encourage us to think of composing programs as a much finer-grained level.

As a small example, we can create a function such as:

    def report(taxf: (Long => Long), amts: List[Long]) = {
println("Tax on " + amts + " is " + taxf(amts.foldRight(0L)(_ + _)))

Given a list of item prices…

    val items = List[Long](20000, 40000, 60000, 80000)

…we can apply shotItemsTax with a defined tax function:

    report(taxHometownCF, items)

We can also apply it to an-inline function literal based on our curried tax functions:

    report(taxCF(0.5D), items)

And that reflects the larger goal of making the definition, customization, and application of functions as painless as possible.

What about those lab rats?

There was a recent discussion on Artima of an article on currying. The original author used illustrations based on simple numerical calculations, something I’ve done in many conversations or classes.

My rationale has always been this: I want to illustrate a technique without requiring that the reader/audience spend time learning the problem domain. Presumably anyone who would read an article about a programming technique would already know arithmetic, and therefore could concentrate on the technique instead of the example. Biased by my own background, I tend to think of the Euclidean algorithm, factorials, and Fibonacci numbers as the standard lab rats of computing. But I’ve slowly begun to appreciate that people whose learning/thinking style is more inductive than deductive may prefer concrete examples that resemble their day-to-day programming tasks.

Language(s) for teaching programming

I’ve seen (and participated in) a number of discussions recently about the selection of a first programming language. I don’t think the choice is a trivial matter, and don’t necessarily think there’s a single right answer, depending on the overall goals of a curriculum. Here are some choice strategies I’ve seen discussed recently:

  • Commercially-popular language

    • students see a connection to the job market,
    • prospective employers may like it,
    • students may have had prior exposure (especially non-traditional students), and
    • a wealth of related materials (books, web pages, etc.) is available.

    Cons (especially if the curriculum doesn’t provide multi-language experience):

    • risks biasing the student’s perception of programming,
    • risks putting the student into the “knowledge half-life” problem, and
    • risks the nasty choice, as language trends change, between churn in the curriculum or becoming less “relevant”.
  • Academically-popular research language

    • professors and grad students may like it, and
    • some of these languages explore ideas that may be important by the time the students graduate.


    • similar to the “cons” list above(no pun intended!).
  • Teaching-oriented language (remember Pascal?)


    • can risk separation of “theory” and “practice”,
    • teaching-oriented aspects may shield students from the “pain” of production programming, and
    • prospective employers may strongly prefer “practical” language experience.

Following the line of thinking from the previous post, I believe that a language can serve as a bride from one style of thinking and problem solving to another, if it is properly designed for that purpose. Going further, the book Concepts, Techniques, and Models of Computer Programming makes use of a multi-paradigm approach in which a kernel language is extended in various ways to expose the student/reader to a variety of styles of programming. I’m currently focusing my spare time on Scala, but I’m intrigued by Van Roy and Haridi’s use of Oz to explore the style “spokes” out from a simple “hub” of basic computation concepts.

Maybe I can get back to Mozart/Oz after I’ve mastered Scala. Of course, I’d have to make friends with emacs

Scala tutorial on IBM developerWorks

IBM developerWorks has the first two parts of an introduction to Scala by Ted Neward. The series looks good so far, and I’m eager to see more.

I was particularly interested to see in the second article that Neward described Scala as “a better Java“, a theme that Dianne Marsh and I explored in the Scala workshops we did at Java Posse Roundup 2008. I’m tidying up the material we used in the first presentation to make it more blog-friendly, and will post it soon.

For me, one key point of that description is the way Scala provides a graceful transition from Java to functional programming, just as Java offered a graceful transition from C to object-oriented programming.


Java’s similarity to C allowed a developer to quickly begin writing procedural Java “with a C accent”, and then transition to an object-oriented style. In the same sense, a developer can quickly begin writing Scala “with a Java accent”, and then begin to shift to a more functional style.

Scala makes this an attractive proposition by offering some immediate benefits. The canonical example is type inference, which allows the programmer to skip writing redundant type specifications. But more about that as I get the workshop material posted.

JPR 2008: Tuesday afternoon

During the afternoon break, Dianne Marsh took Dick Wall, Mike Levin, and me to the Crested Butte Nordic Center for a bit of cross-country skiing. (Dick and Mike had tried xc before, so I was the rank newbie.) Dianne was a very patient and helpful babysitter. I didn’t fall down once. (*)

I took the picture below…


…looking back down a small hill toward Dianne and Dick standing by the Nordic Center building. The snow on the right side of the picture (between the building and the ice rink) really is as deep as the second-floor balcony. The next picture…


…was taken only a few dozen yards further on (after Mike had joined us and everyone had passed me, which wasn’t hard 😉 ). This shot illustrates that the beauty of nature is only a few steps away, almost no matter where you are in Crested Butte.

(*) The statement above is true, strictly speaking. I actually fell down about a half-dozen times. 😉

JPR 2008: Tuesday morning, part 3

“70% Rowing Backward: Why is software development so messed up?”

Bruce Eckel convened this session, inspired a statement he heard from Cameron Purdy of Tangosol:

“70% of programmers are writing code that is going to have to be fixed to move forward.”

(Purdy is known for other insightful statements in the past…)

Bruce immediately clarified that he was not referring to deliberate misbehavior, but rather to effort expended by people who thought they were doing the right thing. The question behind the question was, “Why is it so hard for developers to keep in synch with each other and with the organization’s goals?”

As we discussed the issues, answers included:

  • education out of touch with the current software development, because of:
    • a focus on theory to the exclusion of application and best practice,
    • a focus on short-half-life skills without conveying the underlying concepts, or
    • a focus on a single language or platform without showing the bigger, longer-term picture;
  • curricula with no practice-oriented components;
  • no (or infrequent) code reviews, so that large amounts of code get written before anyone but the (single!) author sees it;
  • division of labor (e.g. framework dev vs. application dev.) with inadequate knowledge-sharing between dev groups; and
  • production environments and architectures which make rapid deployment (and rollback, when necessary) painful.

There was a fair bit of discussion about code reviews, especially about the fact that the most effective reviews are done by (or with) the most experienced developers. Under that assumption, how can code reviews be done without saturating the senior people? Several options were suggested, including:

  • on-line code reviews, via such tools as Google‘s Mondrian and Atlassian‘s Crucible;
  • frequent code walk-throughs (i.e. take small bites often, don’t try to swallow the elephant all at once);
  • pair programming (of course); and
  • post mortem discussions per load or milestone (common at Google, especially for problem cases).

The post mortem topic generalized into a the question of how to increase communication, both within a team and across teams (even including non-developers). Google has “grouplets” that meet to discuss common issues, concerns, and techniques. Some companies use reading rooms or tech talks for the same goal. Dianne Marsh‘s company holds lightning talks every other Friday. Others reported officially-sanctioned “long coffee breaks” as an opportunity for cross-pollinating conversations. Barry Hawkins mentioned weekly talks (any topic allowed: technology, post mortem, etc.) held on Friday mornings right before lunch as an effective stimulus for informal follow-up conversations.

A more formal approach added to the mix was the use of a structured panel discussion. A request for topics was made a month in advance, the collected list was redistributed for voting, and the winning topic(s) provided the subject matter for one or more panels.

Several people responded to the point that personality and personal responsibility are crucial. It’s easy for a programmer (responding either to schedule pressure or a desire for higher productivity) to go “heads-down”, but there’s a risk that (s)he will lose sight of the big picture by doing so. Worse, some developers seem to want to “stop studying and just do it”, while others see learning as a career-long process. A popular, related theme was the need to retain humility and a beginner’s mind, rather than becoming a self-perceived expert (a small boy with a hammer, who sees everything as a nail). Regardless of what our environments offer us, we are responsible for our own skills and development, and for our influence on those with whom we work. Carl Quinn admitted, “We drag the quiet guys to lunch.”

Returning to the environment and organization, we also discussed physical set-up (cubes, offices, open space), technology (using a chat server to support constant, open communication), employee life-cycle (aptitude testing for applicants, “how we do it” training for new-hires, mentoring, and internships), and relationships with other parts of the organization (e.g. having a project-level “war room” in common with development, QA, marketing, etc.)

There were also complaints about interruptions and noise from non-development co-workers who don’t respect the times when a long attention span is needed. This was balanced by recognizing that developers can’t “go dark” for so long that they loose communication and shared vision.

In fact, I perceived balance as a constant theme throughout the discussion. A successful software development organization must be alert and flexible enough to maintain a dynamic balance among a number of forces:

  • short-term demands vs. long-term vision,
  • flexibility vs. formality,
  • constant communication vs. quiet “think time”,
  • moving among the roles of craftsman vs. engineer vs. scientist,
  • popular “building trades” and “factory” metaphors vs. the unique pressures of fast-moving technology,
  • the legitimate business needs of cost and schedule predictability vs. the equally legitimate difficulties of dealing with highly-complex systems and incomplete or ad hoc requirements, and
  • moving the art forward vs. dealing with what we have today.

Of course no one argued with the strategy, voiced late in the session, “Hire smart people and turn them loose on a problem!”

Suggested follow-up reading:

Pair Programming Illuminated
QBQ! The Question Behind the Question: Practicing Personal Accountability in Work and in Life

JPR 2008: Tuesday morning, part 2

“Startup Mistakes Not to Repeat”

Although I’m not in a startup software company, I found much of the discussion to be relevant to software developers in other environments. The participants described a variety of startup experiences, with the expected aspects of stress, excitement, and occasional burnout. I was impressed by how often the discussion included:

  • the roles and contributions of all parties – developers, marketing, executives, and venture capitalists;
  • the importance of mutual respect;
  • the critical role of simple (and frequent) communication between developers and non-developersn;
  • the variety of ways that developers and non-developers were able to achieve teamwork and common cause:
    • weekly lunches,
    • video game rooms,
    • joint customer calls,
    • and crossing traditional “boundaries” to ask others what they’re doing and thinking.
  • and being able to look back and value the learning and growth achieved in the face of difficulties.

After a short break, and some Camp 4 Coffee, we were back at it. My second session of the day was, “70% Rowing Backward: Why is software development so messed up?”

JPR 2008: Tuesday morning, part 1

Given that about half of the attendees this year were alumni of JPR 2007, it’s no surprise that many sticky notes were posted even before the opening session began.

IMG_0288 IMG_0289 IMG_0291 IMG_0292

Bruce called the opening session to order with the by-now-traditional warnings about the risks of altitude sickness and dehydration. After a few more suggestions and a discussion of optional activities for the week, he explained the protocols for open space conferences, including The Law of Two Feet (near the bottom of the linked page).

After a bit more posting, shuffling, and negotiation, we reassembled ourselves for the first topics of the morning. I sat in on the session entitled, “Startup Mistakes Not to Repeat”.