• I have been programming long enough to have forgotten what seemed clear or opaque when I was a beginner.
• Our backgrounds (artistic versus mathematical) provided different sets of expectations and metaphors.
• He can take advantage of, and for granted, an enormous variety of resources that did not exist when I started, from a wealth of alternative programming languages and open-source code to pervasive consumer-level use of the Internet.

He has also encountered a phenomenon that has frustrated me throughout my time as a student, teacher, and practitioner: lab rat code.

Illustrative code in blogs, articles, and textbooks is often unrealistic for the same reason that physics homework refers to friction-free pool tables and the stereotypical psychology lab is filled with rats and mazes. A writer who wants to illustrate a technique needs to apply it to a task that is simple enough not to create distraction. Stated from the other perspective, a reader who doesn’t understand the goal will likely not appreciate the path.

So the canonical first program in a new language prints “Hello, world!” to standard output, often soon followed by YAFG (Yet Another Fibonacci Generator).

The challenge remains for the reader to ignore distraction. I’d like to offer some strategies that I find helpful in that role.

1. If the sample task seems too trivial, then the author succeeded in picking a goal that doesn’t require much of your attention. Stop reading long enough to sketch out what you regard as the obvious solution, then resume reading to see if the author’s solution offers you any new insights.
2. If the task seems unfamiliar, then the author may have used more detail than necessary. Skim the problem statement, then examine the solution to see how many of those details actually matter for the illustration.
3. If the solution seems too heavy, the author may be illustrating it on a problem that doesn’t require its full power. Check to see whether the author follows up with a harder problem that exploits more of the solution’s capabilities. Or create your own example of a harder problem with similar characteristics. Delaying the natural tendency to think, “I can solve this more easily another way!” may provide an opportunity to understand the reasons for the apparent complexity.
4. Develop a tolerance for uncertainty. I have developed a better appreciation for some concepts only through repeated exposure. (It really doesn’t matter whether that statement is about the concept or about me. The result was worth the process.) Similarly, I have sometimes learned something interesting from a solution to a problem outside my experience or interest. Over time you may develop a sense for what you can safely ignore.

The author who wants to reduce the risk of distraction might consider some of these strategies:

1. Avoid cliches like the plague. Instead of offering YAFG, come up with a fresher example (unless you know your readers really LIKE Fibonacci numbers.) Which brings me to my next cliche…
2. Know your audience. I have heard busy practitioners dismiss a technique because (IMHO) they had never been shown its application to a problem about which they cared. A bill-of-materials example might illustrate recursion to an industrial programmer far better than parsing or binary tree traversal.
3. When using a lab-rat example, be explicit about that fact. At least then your reader will be forewarned.
4. If realism, or an agenda beyond the single example, prompts you to include more detail than necessary for the current illustration, consider the structure of your presentation. Can those details be delayed? If not, than being explicit about which ones are important to the solution may help your reader avoid bogging down on the less relevant ones.

I’ll try to apply those practices myself, both as reader and writer.

But suppose lobbyists for the convenience store industry got congress to pass legislation that would authorize the stores to:

1. Keep assault weapons under the counters;
2. Use them at will under a “shoot first, ask questions later” policy; and
3. Exempt them from responsibility for using deadly force if they could say, “he/she looked suspicious to me!”

That is the equivalent of what the backers of SOPA / PIPA are attempting to do.

Don’t just take my word for it. Please watch the video from Khan Academy that explains the reach and risks of these proposals.

Stop PIPA.

But don’t stop there.

(And now, back to our regularly-scheduled programming…)

That statement was associated with many others, such as, “All persons are equal before the law“, that emphasized that the same rules applied to everyone, regardless of economics, education, race, or any of the other attributes that in various times and places have been used to diminish the access of some to justice, opportunity, and fair treatment.

Many in this country still hold to those principles, for which I am grateful. As one who grew up during the Civil Rights Movement, I honor the courage of those who used the rule and power of law to move a society toward justice. That journey is not complete, but equality before the law has been a powerful aid along the path to freedom.

But freedom and equality, as with physical health, clean dishes, a mowed lawn, and a bug-free code base, are not persistent states. They require constant scrutiny and action, both preventative and corrective.

Legislative corruption threatens to replace the bright vision of rule by law with the dark specter of power for sale to the highest bidder. Uncontrolled lobbying and so-called “campaign contributions”, unlimited terms of office, and the apparent willful technological ignorance of many in congress, threaten to bring the grim scenarios of the cyberpunk genre to reality.

I do not advocate violating the law. I respect the need for a society to have structures in place by which creativity–artistic or technical–can be encouraged. And that includes paying fairly for its benefits.

But I regard as misguided any attempt to pervert our legal process to allow preemptive or punitive action without due process. I regard as toxic to health and progress the attempts to provide artificial protection for obsolete business or technological structures that resist change simply because people with money want to keep making more money in the same way.

And I regard as fatal to any virtue in the concept of rule by law the risk that our legislative, regulatory, and enforcement institutions are for sale.

Stop SOPA.

But don’t stop there.

(And now, back to our regularly-scheduled programming…)

Hungry and in a hurry, I dropped into the diner and ordered eggs and toast. The server returned shortly with an electric skillet, a toaster, two whole eggs, and two slices of bread. As I cracked the eggs into the skillet, the server came back along the counter with a bag of coffee beans and a grinder, asking, “Want coffee with that?”

At this point, you’re probably questioning my choice of restuarant. Especially if I make a habit of eating there. And what does this have to do with software development?

Deficit Spending

The concept of technical debt is well established by now. In the best case it means buying a near-term benefit by deliberately taking on a future responsibility, as in, “I’l take out a loan to replace my monster truck with a hybrid, and will make the payments out of what I’m saving in gas!” Sometimes we take a short-cut in the interest of advancing a project, but also know that we’ll pay it back later.

It’s more troublesome when the obligation is created by someone who will not have to deal with the consequences. To put it bluntly, separating a decision from its consequences is a recipe for bad design. It is both easier for the decision-maker to add costs and harder for the ones who will bear those costs to see the debt growing.

I propose to borrow the term “deficit spending” to refer to technical debt imposed by someone else. And to keep this from being an abstract rant, will follow with a few additional posts to identify kinds and causes of technical debt spending.

Given a type-compatible associative operator and sequence of values, we can fold the operator over the sequence to obtain a single accumulated value. For example, because addition of integers is associative, addition can be folded over the sequence:

1, 2, 3, 4, 5, 6, 7, 8

from the left:

((((((1 + 2) + 3) + 4) + 5) + 6) + 7) + 8

or the right:

1 + (2 + (3 + (4 + (5 + (6 + (7 + 8))))))

or from the middle outward, by recursive/parallel splitting:

((1 + 2) + (3 + 4)) + ((5 + 6) + (7 + 8))

A 2-D view shows even more clearly the opportunity to evaluate sub-expressions in parallel. Assuming that addition is a constant-time operation, the left fold:

and the right fold:

require linear time, but the balanced tree:

can be done in logarithmic time.

But the associative operation for the word-splitting task involves accumulating lists of words. With a naive implementation of linked lists, appending is not a constant-time operation; it is linear on the length of the left operand. So for this operation the right fold is linear on the size of the task:

the left fold is quadratic:

and the recursive/parallel version is linear:

Comparing just the “parallel-activity-versus-time” parts of those diagrams makes it clear that right fold is as fast as the parallel version, and also does less total work:

Of course, there are other ways to implement the sequence-of-words concept, and that is the whole point. This little example provides a nice illustration of how parallel execution of the wrong implementation is not a win.

The title of this post is a not-very-subtle (but respectful) reference to “Why Functional Programming Matters“, an excellent summary by John Hughes that deserves to be more widely read.

Purely Functional Data Structures, by Chris Okasaki, covers a nice collection of algorithms that avoid the trap mentioned above.

Also, video from this year’s Northeast Scala Symposium is now on-line for the session on functional data structures in Scala, presented by Daniel Spiewak.

A few miles uphill, the story was a bit different, but our Alpine Express driver got us to our door smoothly and safely.

Day Zero is traditionally reserved for explorations of new technologies, alternate languages on the JVM, and collaborative skills-building. I joined a group that focused on Scala. We talked about Guy Steele’s parallel string-splitting algorithm as presented at StrangeLoop 2010, got a demo of parallel collections in Scala 2.9, and worked on Scala koans.

After I pointed the folks at my table to Steele’s published slides and summarized the the algorithm, Michael Barker immediately re-coded it in Scala and we started looking at performance. More on that later.

Inspired by the excellent Ruby koans work led by Jim Weirich, Dick Wall had begun working on Scala koans at a previous year’s CodeMash. Dianne Marsh took up the reins; the informal team continues to refine and add to the material, and welcomes additional participants. The Scala koan collection is currently a work in progress; despite a rough edge or two, I really like this approach to ease into the thought processes of a language and very much appreciate the work of the team.

The end-of-day semi-lightning-talk presentations demonstrated a breadth of interests and subjects that will likely help shape this year’s Roundup:

• Michael and I discussed our observations on the performance of the parallel string splitter;
• Dianne and Dick described the Scala koans and their current statusy
• Joe Sundow summarized the jQuery and JavaScript MVS explorations that he had led through the day;
• James (“I’m just a Flex guy!”) Ward showed how he’s using Spring and Hibernate on a current demo project;
• Fred Simon gave a quick demo of Fantom, including its ability to compile to JavaScript.

Handerson Gomes and Jim Hurne get my “Wow!” vote for a joint presentation which really made us all sit up and take notice. In the course of one day, they downloaded the Android development kit, got themselves up to speed, and built and tested a small app which include audio and use of the touch-screen gestures.

It was a great start to what promises to be an excellent week. In other words, it was typical for a Roundup!

Goals

I got my iPad as a tool to accomplish things with more mobility and efficiency, not to spend time wandering virtual supermarket aisles looking for the shiniest variation on a theme. Given that, I was immediately attracted by this title from O’Reilly. Both the author, Peter Meyers, and the publisher have the right credentials and reputation to address my need.

The book

The content is well organized, with chapters (and sections) that support both leisurely browsing and focused navigation: At Work, At Leisure, Creative Corner, At Play, At Home, Out and About, For Your Health.

The reviews typically provide an app’s icon (great for quick visual reference), price, reviewed version (important in the fast-moving world of the App Store), publisher, overview, well-organized comments and usage tips, and screen shots for key points.

The rankings Meyers gives were highly consistent with my experience on key apps I regularly use. More important, he is clear about his point of view and why he evaluates as he does—a crucial feature for this type of reference. On my first reading, he introduced me to new and useful possibilities. I will be keeping this book within easy access for ongoing use.

Finally, I must confess a slightly wistful thought that turned out to be premature. I still remember the early days of the World Wide Web, when a variety of printed “yellow-pages to the Web” books appeared. Most of them had a fairly short shelf-life, as the explosive growth of the web left them quickly out of date. I immediately wondered whether this book would have such a future. But…

Horseless carriage?

In its early days, the automobile was often referred to as a “horseless carriage”; most people only thought of it in terms of what they already knew, and hadn’t realized the implications of that new technology. (How many people—and companies—are still trying to think of the web as a magazine, newspaper, radio, television, mailbox, etc. minus some physical attribute, not recognizing it as a new thing that is all and none of the previous media?)

That’s why I regard Meyers’ preface as one of the most enduring parts of this book.

He gets it.

Meyers explicitly focuses on what makes the iPad a new thing, not just a mobile phone or netbook, and uses that understanding to guide his selection and evaluation of apps that are important, note-worthy, or simply enjoyable to use. And that makes this book useful not only to a “discriminating downloader” like me, it makes it a great reference to an aspiring app developer who needs to understand what makes iPad apps different, and to any technophile (iPad owner or not) who wants to understand better the potential of this new thing.

I was particularly impressed and inspired by Must. Try. Harder. (also linked from here and here), a presentation by Keavy McMinn, who described her experiences training for Ironman. Part of Keavy’s first blog post on Ironman Cuzumel resonated strongly with something I once saw in some code, hence this article.

By way of background (with vagueness required by confidentiality considerations), I remembered looking at test code wrapped around a fairly subtle bit of business logic that had passed through multiple hands. I believed that I understood the intentions of the original designer, and wanted to confirm or correct that belief. All of which sounds like a perfect fit for test cases, right?

Hmmmmm.

The majority of the test cases dealt with various ways that something could go wrong in configuring the targeted business component. Additional tests dealt with incorrect inputs to a correctly-configured instance, and only a small fraction actually dealt with the behavior of a correctly-configured instance. In fact, there were no tests that answered the particular questions that I had in mind.

How can this be? (And what does this have to do with Keavy’s Ironman experience?)

About half-way down in her Ironman Cozumel blog entry, there’s a section entitled “Fear”. I thought about quoting a key sentence or two, but didn’t want to interfere with her well-crafted flow. So just take a minute and read it (at least—for now—the section under her “Fear” heading, between the two photos).

(Wow! You read quickly! )

So, as I looked at the test cases, a pattern began to emerge. The largest portion of the tests seemed to orbit around a particular aspect of the configuration that I took for granted, giving that configuration concept far more than its share of attention, IMHO. Which got me to thinking about similar patterns in other test cases. Which prepared me to have an “<em.Aha!</em.” moment when I encountered Keavy’s description of fear of the ocean.

Fear

Unbalanced tests that over-emphasize some issues and under-emphasize others may be a subtle hint that fear is influencing the work. I’ve observed two common responses to fear in myself and others: obsession and avoidance. On reflection, I believe that those attitudes show up in tests that I’ve looked at over the years.

Obsession

We may joke about it, but I’ll admit to going back a few times to make sure the coffee pot is off, jiggling the knob one more time to verify that I really did lock the door, or repeatedly checking my pocket to reassure myself that my airline tickets didn’t jump out and hide under the couch. (OK, forget the “hide under the couch” part.)

In this light, I see the set of test cases mentioned above as being obsessed with a particular configuration issue. Those tests were me, patting my pocket again, trying the door again, pushing the switch again (even though I can clearly see that the light is <em.not</em. glowing…) If I find that my tests appear to be obsessing over a particular aspect of the task at hand, that might be a symptom of fear—a need for comfort and reassurance in the face of of an unfamiliar domain concept, a language feature or API with which I’m uncomfortable, or a requirement that I don’t really understand. Or of moving onto the next use case, which brings me to…

Avoidance

Inadequate attention to testing an aspect of the design may indicate a reluctance to engage fully with the issue. In that connection, Keavy’s description of initial hesitation at the water’s edge brought back a vivid memory for me.

My family loves camping in the Great Smoky Mountains National Park. A popular spot along the Little River Road called “The Sinks” features a waterfall almost directly under a bridge, followed by a long, deep swimming hole with steep rocky banks on both sides. Visitors can climb to a ledge which offers a clear jump into a deep part of the pool. Although that ledge is only about 12 feet above the water level, it looks much higher to a first-time jumper. I don’t laugh at young (or not-so-young) first-timers who hesitate on that ledge, because I still remember my first leap.

Even though my head knew it must be safe—I had seen many people take the plunge before me that day—my stomach and legs hadn’t yet gotten the message. I “paused” for several long moments before stepping into the air.

(I’m not talking about truly high-risk activity here, I should point out. A friend of mine has a truly terrifying story about jumping from height into unfamiliar—and unsafe—water and impaling his foot. So let’s keep it in the pool and well-known swimming holes, kids. And don’t forget your buddy.)

Of course, we developers <em.never</em. procrastinate in the face of a fear-inspiring task! Then again, there’s that bit of legacy code that nobody wants to maintain, much less rewrite. And the lingering suspicion that “starting with the low-hanging fruit” can turn into an excuse to put off the parts we’re secretly fearing.

So, how do I fight back, when an imbalance in my tests shows evidence of fear, either positively (obsession) or negatively (avoidance)?

Suggested antidotes…

First, I should consider whether the test really are over- or under-emphasizing something. Different aspects of the code have different levels of risk and/or consequences, so I don’t expect absolutely even distribution of attention. If there is a true imbalance—even better, if I’m about to create one—I need to recognize that and apply an appropriate antidote.

…to obsession

When I start bogging down, these questions can be useful:

Is this test about the product or about me?

Let me tip my hat to Ruby Koans, which gives beautiful evidence of the power of testing as a way to explore a language or framework. If a test can confirm (or correct) my understanding of something I’m using in this project, I certainly should write it! But I’m equally certain that I should keep it outside my project’s code base.

What does/will this test teach me?

If it conveys useful new information, or affects the code in a material way, fine. But if it’s just a trivial variation on a theme, perhaps I should move on. If I’m wrong, I can always come back and add a test later.

How likely is this?

If this test is exercising a scenario that would almost certainly never occur in normal use, maybe I should take care of the more-likely cases first.

How bad can it be?

Do I really need this last little bit of yak hair?

…to avoidance

Keavy wrote about overcoming by focusing on technique, process, and details under our control. My high-school marching band director kept taking us back to the basics, with scales on our instruments and eight-to-five marching drills until our feet could hit the chalk line precisely, with no last-minute stretches or stutters. In fact, without even looking. For this side of my fears, here are some questions…

What do I not know?

If I can express it as a test, I can get moving.

How would I explain this to ___?

I have gotten past mental blocks by explaining my impasse to a friendly non-programmer. Being clear and non-technical often has amazing benefits! Fill in the blank with a name of your choice. Sometimes only imagining the conversation is enough. It also helps to imagine it out loud (but not where I could frighten or distract my colleagues!)

Am I trying to swallow an elephant?

The last time I got stalled out, I let circumstances outside the task at hand stampede me. I blurted out a train of thought into code and quickly ended up, as I once heard someone say, “crocheting in logic space“. Admitting that fact to myself allowed me to back out of my self-imposed impasse.

…and a couple of challenges:

Don’t just sit there, DO SOMETHING!

Sometimes even a stumbling start is better than stalling, because it gets me moving. As with explaining my problem to a non-programmer, thinking out loud in code may help me get a better perspective (even if I end up throwing away that first bit).

Don’t just do something, SIT THERE!

In my best ersatz-mystico-philosophical style, sometimes I need to do just the opposite. Stop fidgeting and pacing back and forth. Sit down. Close my eyes. Be still. Take a breath. Think a moment. In that jam there’s probably one log that I can shift a little, and it’s probably not the one I’ve been tugging on unsuccessfully for way too long. Now open my eyes. Look around not in the same place. Oh. Duh. There it is. 

Let me get back to you. I’ve got a test to write.

The value of persistence

After beginning this post, I learned that Keavy finished in Cozumel! Congratulations!

Code koans:

• in Java-Script;
• in Scala;
• I heard from Dianne Marsh that CodeMash featured work on koans last year; they may do so again this year.

I’m sure there are many more; please let me know.

And now it has its own O’Reilly Nutshell book, R in a Nutshell, written by Joseph Adler. I am pleased to report that Adler has risen to the challenge of the highly-regarded “Nutshell” franchise. As is traditional for the series, this title mixes introduction, tutorial, and reference material in a style that is well suited to a reader who already has a background in programming, but is a new or occasional user of R.

The book’s flow was very effective for addressing the different points of view from which I approached it.

As a curious newcomer to R who wanted to get going quickly, I was well-served by Part 1, which provided an R kickstart. Chapter 1 covers the process of getting and installing R. It is short, to the point, and just works, addressing Windows, Mac OS X, and Linux/Unix with equal attention. Chapter 2, on the R user interface, introduces the range of options for interacting with R: the GUI (both the standard version and some enhanced alternatives), the interactive console, batch mode, and the RExcel package (which supports R inside a certain well-known spreadsheet). Chapter 3 uses a set of interactive examples to provide a quick tour of the R language and environment, establishing a task-oriented theme that carries through the rest of the book. The last chapter of part 1 covers R packages. It summarizes the standard pre-loaded packages, introduces the tools to explore repositories and install additional package, and concludes by explaining how to create new packages.

As a polyglot programmer who is always interested in seeing how a new language approaches programs and their construction, I enjoyed Part 2, which described the R language. This section begins with an overview in chapter 5, and then devotes a chapter each to R syntax, R objects, symbols and environments (central to understanding the dynamic nature of R), functions (including higher-order functions), and R’s own approach to object-oriented programming. This section closes in chapter 11, with a discussion of techniques and tips for improving performance.

As a busy professional with data sitting on my hard drive that I’d like to understand better, I appreciated Part 3, with its practical emphasis on using R to load, transform, and visualize data. Chapter 12 presented alternatives for loading, editing, and saving data, from the built-in data editor, through file I/O in a variety of formats, to a mature set of database access options. Chapter 13 illustrated a range of techniques for manipulating, organizing, cleaning, and sorting data, in preparation for presentation or more detailed analysis. Chapter 14 introduces the reader to the wealth of graphical presentation options built into the R environment. There are so many charting types and details that this chapter could have been overwhelming, but Adler keeps the interest high and the mood light by drawing on an engaging variety of data: toxic chemical levels, baseball statistics, the topography of Yosemite Valley, demographic data, and even turkey prices. Chapter 15 is devoted to lattice graphics, the R implementation of the “trellis graphics” technique for data visualization developed at Bell Labs. This chapter illustrates the power of lattice graphics by exploring the question of why more babies are born on weekdays than weekends.

As a non-statistician who still occasionally needs to do some number-crunching, I’m sure I’ll be returning to Part 4, with its detailed explanations and illustrations of analysis tools and techniques–almost two-hundred pages worth. In chapters 16 through 20, Adler surveys topics in data analysis, probability, statistics, power tests, and regression modeling. As someone who has been offered too many medications and lost fortunes, I found much to enjoy in chapter 21, which used a variety of spam-detection techniques to illustrate the concepts of classification. Chapter 22, on machine learning, discusses several of the data mining techniques that R supports. Chapter 23 covers time series analysis, which may be used to identify trends or periodic patterns in data. Finally, chapter 24 offers an overview of Bioconductor, an open-source project focused on genomic data.

The book closes with a detailed reference to the standard R packages.

This is an impressive piece of work. In a volume of this size (about 650 pages), navigation is crucial, and I found both the organization of the chapters and index up to the task. I was able to follow the instructions and examples through the first several chapters of the book essentially without a hitch, and in the latter chapters the variety of illustrations and data sources added interest to what could have been very dull going.

I won’t claim perfection for this book. There were a couple of explanations that could have been clearer, and one or two odd turns of phrase or rough edits. Out of all the code examples that I tried, I found exactly one that didn’t seem to work without a minor correction. For a work of this size, that’s actually pretty amazing!

As a long-time O’Reilly reader, I see Joseph Adler’s R in a Nutshell as a welcome addition to the menagerie.

The researchers compared a variety of techniques for representing directionality of arcs: standard arrows, gradients (light-to-dark, dark-to-light, and green-to-red, and curved and tapered connections. Subjects were shown graphs rendered using different techniques, and asked questions about connections between nodes, with speed and accuracy being analyzed.

According to the summary:

• one should avoid standard arrows and curves,
• tapered arcs (from wide to narrow) produced the best results,
• dark-to-light value gradients were better than light-to-dark (no surprise there, as this is consistent with the “more intense to less intense” result of the tapering), and
• there appears to be no advantage to combining techniques (I confess a bit of surprise over that conclusion).

But most fascinating to me is the fact that when presenting the results visually, the paper and article both violated the very conclusions just reached! Superior performance among techniques was presented using a directed graph using curved arcs with standard arrowheads! (See the diagram here, or in the article and linked paper.)

It was but a few moments’ work in OmniGraffle to recreate that diagram using the recommended technique (tapered arcs). In addition, I rearranged the nodes to reduce the arc crossings, while keeping the preference flow downward from higher to lower. Here’s the result:

I’m amazed that the researchers and author(s) of the summary ignored the result when presenting the result! Does this tell us something about the force of habit, or perhaps the limitations of existing tools, or something else entirely? I wonder.