Functional Wrappers for legacy APIs

Download project source on GitHub

There are many challenges in introducing a new technology into an existing project or organization.  To be successful, you normally need to be able to achieve one or all of the following:

  • Experiment with the new technology incrementally without incurring a massive adoption cost
  • Showcase the strengths of the technology immediately
  • Give a concrete benefit to the people on the front lines of the organization

This article demonstrates techniques that can be used for such an introduction by examining a contrived example of building a functional API for JDBC1 using Scala.  While the stated example is contrived, the inspiration for it is an enterprise integration API called the Documentum Foundation Classes, which has many similarities to JDBC.

Our goal is to be able to write functional programs that efficiently interact with our legacy API. JDBC was chosen for this example because it is widely accessible and lends itself to self-contained projects via embedded databases like H2. Keep in mind that the point of this article is not to provide a wrapping service for JDBC, but rather to illustrate how to construct similar wrappers for other legacy APIs that are far less common, and often proprietary in nature.

Using our wrapper, we can write a short program that executes a SQL query and converts the results to JSON. Our answer, armed with the facilities we construct here, will look like this:

val connectionInfo = new Jdbc.ConnectionInfo("jdbc:h2:mem:test1;DB_CLOSE_DELAY=-1")

val mapper = new ObjectMapper().registerModule(DefaultScalaModule)

def queryToJSON(conn: Jdbc.ConnectionInfo, sql: String) =
	Jdbc.withResultsIterator(conn, sql, it => mapper.writeValueAsString(it))

Lines 3 sets up a Jackson mapper which can convert Scala data structures to JSON. Lines 5-6 use the functional wrapper we created to iterate over our result set performing our desired data conversion. We can then write a console app as follows:

  def main(args: Array[String]) {
    queryToJSON(connectionInfo, "SELECT * FROM EXAMPLE") match {
      case Success(json) => println(json)
      case Failure(e) => println(e.getMessage)
    }
  }

So that’s where we’re headed. Now let’s take a closer look at how to get there!

We have two design goals when creating a functional wrapper for a non-functional API:

  1. Hide state from your callers as much as possible
  2. Reshape the API in terms of a collections API as much as possible

The first goal gets at the heart of functional languages.  A program that is “purely functional” has no state and never mutates a variable.  While we won’t be able to achieve this goal ourselves, we can still allow the callers of our wrapper to do so.

Of course, you’ll have a tough time hiding state if you can’t recognize it.  Let’s start by looking at a common code block in JDBC code written in Java:

Connection conn =  DriverManager.getConnection("url", "user", "pass");
try {
	...
} finally {
	conn.close();
}

Bam! State is right there. The connection object might be open or closed, and the programmer has to know about this and manage it. This is a potential source of programmer errors and more importantly, it completely screws up Java’s “memory management” model. When you have objects like this, you can’t just wait for them to be garbage collected, so you’re basically back in C/C++ land and have to manage it yourself.

There’s a way to address this in Scala. Let’s build it up bit by bit:

case class ConnectionInfo(url: String, username: String = "", password: String = "")

This defines a structure named ConnectionInfo that represents the data necessary2 to instantiate a JDBC connection.

Syntactically there is not much to explain here. Case classes3 are special beasts in Scala, but they are not too different from immutable structures in other languages. The three member values (url, username, password) are all strings, and two of them have default values defined.

Now we can define a function:

def withConnection (connInfo: ConnectionInfo, f: Connection => Unit) { ... }

That declaration defines a new function named withConnection where:
• The first parameter is an object of type ConnectionInfo
• The second parameter is a function that takes a JDBC Connection and returns nothing (Unit is a special type which is effectively nothing)
• The function does not return a value

The second argument is the most important concept at this point. In Scala (and generally in all functional languages), a function is a “first class object” which means that a function can be stored in a variable or passed as a parameter like any other object. Also, because Scala is strongly typed, the compiler will make sure that only a function of that exact signature can be passed into withConnection.

A fair point can be made that you can accomplish this same thing using interfaces in Java. This is definitely true and, further, it is definitely possible to program in a functional style in any language. The advantage of Scala is that this technique is syntactically convenient. Using this technique in Scala reduces code size, whereas in Java it often can increase it.

Now we can examine the body of the function:

def withConnection (connInfo: ConnectionInfo, f: Connection => Unit) {
	val conn = DriverManager.getConnection(connInfo.url, connInfo.username,
			connInfo.password)
	try {
		f(conn)
	}
	finally {
		conn.close()
	}
}

This implementation is pretty much what one would expect, and looks similar to our original Java pattern. The advantage is twofold:
1. We only write this try/finally logic once for our application vs. in multiple places
2. We are hiding the existence of the connection state from our callers. They just need to operate on a connection, they will not need to know how to create or manage one.

Now let’s work on the return values of our function. In functional programming, it is frowned upon to define functions that do not return values. This is because a function that returns no value (often called a procedure) likely manipulates state or produces some other side effect — things that would violate purity. Of course, we want anyone to be able to invoke withConnection, so we’ll use Scala’s ability to allow any return type

def withConnection [T] (connInfo: ConnectionInfo, f: Connection => T): T = {
	val conn = DriverManager.getConnection(connInfo.url, connInfo.username,
			connInfo.password)
	try {
		f(conn)
	}
	finally {
		conn.close()
	}
}

Not much has changed, as we would hope. Our function is now a template4 for generic type T. withConnection will return whatever type the passed in function returns. This works almost identically to generics applied to methods in Java.

We have one more thing to address with this method before we move on. While Scala can handle Java-style exceptions in the expected fashion, it is more common to use types that convey the possible return states of a function. In this case, we can apply something called the Try monad5 to our try block. As a result, our function will return either Success or Failure (but will never actually throw) and the Scala compiler will force our callers to distinctly handle the two scenarios.

Here is our final version:

def withConnection [T] (connInfo: ConnectionInfo, f: Connection => T): Try[T] = {
	val conn = DriverManager.getConnection(connInfo.url, connInfo.username,
			connInfo.password)
	val result = Try(f(conn))
	conn.close()
	result
}

We can now do the same thing for acting on a JDBC statement:

def withStatement [T] (connInfo: ConnectionInfo, f: Statement => T): Try[T] = {
	def privFun(conn: Connection) = {
		val stmt = conn.createStatement()
		try {
			f(stmt)
		}
		finally {
			stmt.close()
		}
	}

	withConnection(connInfo, privFun)
}

The basic mechanics are the same as our withConnection, but we have introduced two new elements worth mentioning. Firstly, within the body of our function we have defined a new function privFun. This is no different from any other function declaration save for its scope. This function is only visible inside of withStatement. We pass this privation function as an argument to withConnection, chaining the two methods together.

Secondly, and more importantly, our private function invokes f, an original parameter to withStatement in the first place. This is a closure6 — an important concept in functional programming. privFun isn’t just a function declaration. It is a function that also has in-scope objects associated with it.

Completing our matched set, we provide withResultSet, which introduces no new concepts:

def withResultSet [T] (connInfo: ConnectionInfo, sql: String,
		f: ResultSet => T): Try[T] = {
	def privFun(stmt: Statement) = {
		val resultSet = stmt.executeQuery(sql)
		try {
			f(resultSet)
		}
		finally {
			resultSet.close()
		}
	}

	withStatement(connInfo, privFun)
}

Managing connections, statements and result sets7 accomplishes only part of our goal though and, and while we are saving our callers from dealing with the management of these objects, we have still have left our callers dealing with the native API objects.

ResultSet, in particular, presents a very unwieldy interface. It combines:
• State regarding the result set
• Iteration
• Data structure of the results
• Read access to the “current” row
• Write access to the “current” row

A result set already has many characteristics of a collection, so it is an obvious candidate to be converted to the collections API. There are three high level targets:
1. A concrete collection, like a list
2. An iterator
3. A stream

A concrete collection must be completely populated on construction. Iterators allow once-only dynamic traversal. Streams allow multiple traversals and can be used to represent infinite collections, like the set of all positive integers.

Since we have no control over the number of rows in our ResultSets, we would run the risk of memory issues were we to choose concrete collections. Streams are very difficult to use correctly in this scenario because of memoization8 , which is used internally to implement the multiple-traversal facility of streams. The once-only traversal nature shared by both Iterators and ResultSets is the key factor in deciding which collection category is the best fit.

Here’s a simple wrapper to get us started:

class ResultsIterator(resultSet: ResultSet) extends Iterator[ResultSet] {
	def hasNext = resultSet.next()
	def next() = resultSet
}

Right away, we can spot a conflict between these two APIs. The ResultSet.next method method performs two actions. It scrolls the result set to the next element AND it returns a boolean indicating if the end of the result set has been reached or not9. It is worth noting that while this iterator will work correctly without modification, that is only because of implementation details of Scala which we do not want to rely on. Secondly, this issue is very hard to solve directly because if we try to move things around, we’ll run afoul of the internal state of the result set. Without broader changes, this problem is best left alone.

Let’s tackle another problem first. Imagine a nefarious programmer decides to call “next” while iterating over our wrapper. That will completely break our iteration mechanic. We need to shield ourselves from this by hiding the result set from the users of our iterator. JDBC does not make this particularly easy for us. There is no mechanism to get a “row” of data from a result set so we will need to construct such a row object ourselves.

There are two sorts of collection objects that we can consider for a row. One is an array-like structure such as a list or vector. The other is a map. A map presents a clearer interface to our callers, because it allows the callers to reference the database columns by name rather than their order in the query.

In order to construct our map, we need to determine the names of the columns available through the ResultSet. We can do this as follows:

val columnNames: Seq[String] = {
	val rsmd: ResultSetMetaData = resultSet.getMetaData

	for (i <- 1 to rsmd.getColumnCount) yield rsmd.getColumnName(i)
}

A for comprehension in Scala allows us to extract the field (the column name) from a larger structure (the results metadata). In this case for each column index, we produce (yield) the name of the column at that index. Seq is a type that is similar to java.util.Collection or Iterable.

Armed with the list of column names for a given result set, we can construct a name-value map with the following function:

private def buildRowMap(resultSet: ResultSet): Map[String, AnyRef] = {
	(
		for (c <- columnNames) yield c -> resultSet.getObject(c)
	).toMap
}

Again, we use a for comprehension to perform the data translation. This time though, we use the -> operator in Scala to produce “tuples”. This is the same as if we manually constructed a pair of (c, resultSet.getObject(c)) for each column name. Conveniently, we can convert a collection of tuples like that into a map. The toMap routine we invoke assumes that each tuple represents a pair of (key, value).

Now we can produce an updated Iterator implementation using our new mechanism:

class ResultsIterator (resultSet: ResultSet) extends Iterator[Map[String, AnyRef]] {
	val columnNames = { … }
	private def buildRowMap(resultSet: ResultSet) = { ... }

	def hasNext = resultSet.next()
	def next() = buildRowMap(resultSet)
}

We now allow our callers to iterate over our results, but give them only an immutable structure to do so, so there is no risk of our callers disturbing the state of our JDBC ResultSet object.

We are now left with the problem we passed over before, but we are now in a far better position to address the API conflict involving hasNext and next. At the heart of the issue is that the ResultSet object has an internal state that is changing when “next” is called, so we cannot directly attach next to the iterator method hasNext, which may be called multiple times inadvertently.

Our approach to this is to manage the state of the result set in our iterator instead. For this purpose, we define the following helper method:

private def getNextRow(resultSet: ResultSet) = {
	if (resultSet.next())
		Some(buildRowMap(resultSet))
	else
		None
}

Our helper method getNextRow gives us a better model for managing the two aspects of the information we wish to collect from the results set. The Option10 structure conveys to us whether or not the next row exists. If it does exist, it can provide us with the value. Best of all, we can interrogate it multiple times without changing the state of the option we are interrogating.

Observe the final Iterator:

class ResultsIterator (resultSet: ResultSet) extends Iterator[Map[String, AnyRef]]
{
	val columnNames = { … }
	private def buildRowMap(resultSet: ResultSet) = { … }
	private def getNextRow(resultSet: ResultSet) = { … }

	var nextRow = getNextRow(resultSet)

	def hasNext = nextRow.isDefined

	def next() = {
		val rowData = nextRow.get
		nextRow = getNextRow(resultSet)
		rowData
	}
}

Blindly calling “get” on the Option nextRow is a dangerous call that could throw a runtime exception. We don’t care about this issue in our example because our next method is effectively “guarded” from misuse by hasNext. If someone blindly calls next, then they deserve whatever they get!

Armed with our iterator, we can complete our utility API. Here is the final function that builds upon withResultSet to expose our iteration mechanics to our callers:

def withResultsIterator [T] (connInfo: ConnectionInfo, sql: String,
		itFun: ResultsIterator => T): Try[T] =

	withResultSet(connInfo, sql, resultSet => itFun(new ResultsIterator(resultSet)))

That’s it! We can now write functional programs against JDBC. More importantly, we can identify non-functional characteristics in APIs that we use every day, and develop an approach for reshaping those APIs in functional terms.

Thanks for reading. If you’re looking for more information, check out the footnotes or download the project source on GitHub.

  1. Note that if this is something you actually want to, it’s been done several times already Scala Slick is one that is definitely worth checking out if you want to apply functional programming techniques to database access []
  2. It’s possible for advanced usage that more fields would be necessary, and those fields could be trivially added []
  3. Further reading: A Tour of Scala: Case Classes []
  4. Further reading: Writing generic functions that take classes as parameters []
  5. Further reading: The Neophyte’s Guide to Scala Part 6: Error Handling With Try []
  6. Further reading: Functional Scala: Closures []
  7. A general term for this technique is Automatic Resource Management. Josh Suereth wrote a library named scala-arm in this space. The project source and documentation provides all sorts of additional details on the topic. []
  8. Further reading: Scala Stream Memoization []
  9. This is exactly the sort of combination of functionality that functional programming opposes []
  10. Further reading: scala.Option Cheat Sheet []
Posted in Software Development | Leave a comment

One Day in the Wasteland

It was a strange journey that brought me to the edge of a settlement in the middle of nowhere.  The old woman wove quite a tale for me and by the time I figured out what a load it was I was already committed.  Well, committed and a little curious, but my curiosity had worn off over the past week.

The shack was just as she had described it though, right down to the faded “Electronics Repair” plank nailed above the open door.  I take off my pack and lay it on the ground outside the entrance.  Originally I’d been apprehensive about this part of her instruction, but with no one around it seemed safe enough.  Out in the desert there was some value to the collection of junk I carried, but this close to a settlement I don’t think anything was worth stealing.

With a deep breath, I cross the threshold, parcel in hand.  The old man eyes me up and down as I approached the counter and lay my parcel in front of him.  I stare at him.  He stares at me.  He sets aside the radio he had been tinkering with and begins unwrapping what I had brought.

He has a lens mounted on his head somehow, and he pulls it in front of one eye and explores the machine with his fingers.  All I have to do is stay silent and this will all be over soon.  That was her final instruction.  I can’t believe how stupid this is.  I can’t believe how stupid I was.  Finally, I can’t take the silence any longer and ask “Is it true what they say about you?”

The old man chuckles lightly to himself, but doesn’t look up from his work.  He keeps at it, his fingers deftly manipulating the object, slowly taking it apart and laying the pieces on top of the cloths I’d wrapped it in.  He removes the lens with a sighed replied “What do they say?”

“They say you are older than Cochise!”  Any fool could tell, just by looking at him, that this man was older than Cochise.  What an idiot he must take me for.  Of course the man is older than Cochise.  He is older than Dirt.  He remains silent, but continues his work.  I try again.  “They say there’s nothing you can’t fix.”

Who am I kidding?  “They” is just the one mystic whose name I don’t even know.  How would she know anyway?  Looking around his shop, I see piles of electronics, but I can’t tell what is working and what is not.  The old man remains silent, and now his hands have stopped moving as well.  After a minute of close observation, when I am sure that he is still breathing, I whisper “They say you can tell the future…”

He shakes his head, but he resumes his work.  He has the cover off now, exposing the innards.  “I can’t, unfortunately, but toasters sometimes do and you have brought me a fine specimen.  Might I ask how you found it?”  He pokes the electronics with a metal stick and I smell burnt metal.

She told me not to reveal anything about her, and she was convincing enough that I spent a month in the desert finding this piece of junk and bringing it here.  I don’t even know what a toaster is, but she told me to bring him one, and she drew me a picture.  “Don’t tell him,” I say to myself.  I repeat it, “Don’t tell him.”

He snaps his fingers in front of my face.  “Your toaster is fixed!” he says, waving his hand over the assembled device with a flourish.  “Would you like to see?”

I’m not even sure when he finished, but now my attention is drawn to the glowing block he is retrieving from under the counter.  It is the size of a fist, and produces a lazy blue light.  He attaches the wires of the toaster to the fist of light, and pushes down the lever on the side.  “Did she tell you to bring bread?  She usually forgets.”  He waves a hand in front of my eyes.  “No matter, I have some of my own,” and he drops two squares into the toaster.  I have no idea where they came from.

“This is not why you are here though,” he continues.  “You are here for this,” and he places a golden disk on the counter.  “It was inside your toaster, and now it is yours.”  He reaches over the counter and pulls my arm, placing my hand over the disk.

“The toasters tell the future, not me, and this one is saying that darkness lies ahead.  It is saying that a new evil rises in the wasteland, greater than the old.  It is armed with newer technology and greater funding.  The toaster is telling me that YOU must take up arms against this evil.  That you must do it, and that you cannot do it alone.  The toaster tells me this, and therefore I tell you.”

I contemplate this for a few moments then realize that the elder is waiting for me to respond.  I say the first words that come into my mind, “How do you know that this is for me, and not for you?”

He smiles and laughs at me for the second time.  “Because I already have one…”

Posted in So insignificant it might as well not even be here | 3 Comments

Volvo S80 Car Fire

A couple of weeks ago, just after I arrived at work, people were drawn outside by a car fire in the parking lot.  I followed the crowd, and stood amongst all the people who were wondering who’s car it was.

If you watched the video, you’ll already have solved the mystery.  It was mine.  As I was standing there, a couple of thoughts popped into my head:

Don’t open the hood

I’m not sure where or when I learned this, but it’s a good tip, right up there with not pulling the gas nozzle out of your car if a fire starts when you are filling up.  The reasons are different, but the results are the same in that the fire will spread very quickly.

That’s not going to buff out

Those words are a simplification of what I was thinking.  The amazon page below is a more accurate depiction of my thoughts at the time.

VolvoS80Amazon1

The rest was just emptiness.  It was almost relaxing.  The next hour or so was consumed mostly by me standing, transfixed while my co workers “documented” the incident and started working up the comedic avalanche that was to follow.  Look at what I have to deal with:

ComedicAvalanch

It’s a worthwhile experience, if you haven’t done it before.  By this I mean “being in shock” more than “having your car self-immolate.”  It’s one of those things you can’t possibly understand without experiencing it first hand, and it’s also unfortunately hard to simulate.  My lesson from the experience is this:

If you want anything of value to pop into your head during a moment of shock, it had better already have been there.

Posted in Completely reasonable complaints, Real life stuff | 5 Comments

Performance Enhancement

A friend of mine recently brought to my attention the use of performance enhancing drugs (PEDs) in the workplace, especially in Silicon Valley.  I suspect this is not limited to the software field and probably either has or soon will bleed over into other “knowledge worker” fields.  Caffeine is now a gateway drug.

The sports world has something to teach us here.  Back in the 70′s, bodybuilders and power lifters started experimenting with steroids and 40 years later it’s hard to imagine a sport that hasn’t had a PED scandal.  It’s not hard to see why athletes were (and continue to be) so easily tempted:

  1. Their careers are relatively short.
  2. Their earning potential after their athletic career is often limited, so they want to maximize their earnings while they can.
  3. Their short career can be cut even shorter at any time by injuries, which can be either prevented, ignored, or recovered from faster through the use of PEDs.
  4. At the highest levels, incremental increases in performance are rewarded with exponential increases in compensation.

There are circumstantial similarities, but they aren’t as exaggerated in knowledge worker industries.  For instance, a sudden industry shift can shorten or redefine your career.  This recently affected many hardware engineers working for hard drives manufacturers.  A repetitive stress injury or back issue can make it difficult to work at a computer for long periods of time though I’m not aware of any workplace injury for programmers that is on the same level as a blown ACL in terms of either frequency or impact.

The money angle matches up the best, but really only for people that are able to “exit” an equity position.  In sports, the parallel would be for those exponential earnings increases to come only if your team wins the championship.  For company founders in a tech exit, the money can be as good or better as an elite athlete.  At many companies though, a senior engineer who is “twice as good” makes less than twice as much money.

There are other competitive advantage issues in sports though, and many of the others aren’t so clearly “cheating.”  For example, Tiger Woods had Lasik eye surgery.  Twice1.  I’ve never heard any suggestion that this was improper, and I’m certainly not suggesting that now.  Still, his surgically enhanced eyes make him a better golfer.  Say that last sentence out loud.  Now, say it again but replace “surgically” with “chemically” or replaced “eyes” with some other body part.  It doesn’t seem like it would take much for this sort of thing to lead to serious objections from people in both sports and medicine.

I would now like to introduce you to Oscar Pistorius:

Oscar Leonard Carl Pistorius (born 22 November 1986) is a South African sprint runner. Known as the “Blade Runner” and “the fastest man on no legs”, Pistorius, who has a double amputation, is the world record holder in the 100, 200 and 400 metres (sport class T44) events and runs with the aid of Cheetah Flex-Foot carbon fibre transtibial artificial limbs by Ossur.

Oscar’s record times in the three primary sprinting events (100, 200, and 400 meters) are within 1-2 seconds of the world records in those events by runners with legs.  There is a debate surrounding him and his running blades concerning whether or not he should be eligible to compete in an open competition like the Olympics and whether or not these blades constitute an unfair advantage.2  That debate is complicated, I think, by the fact that the Olympics have generally allowed advanced technologies to be used in competition in ways that absolutely destroy existing records.  Two recent examples of such technologies are the LZR Racer swimsuits and Clap Skates.  In both cases, almost every related world record fell in the first Olympic Games where those technologies were introduced. For any readers that find Oscar’s story interesting, check out Anthony Robles.  he is a disabled athlete who has won in open competition.

Provigil seems to match the profile of sports-related PEDs like steroids and HGH.  There are both short and long term side effects.  There are potentially additional long term effects that are not known or understood yet.  People who use them illegally generally have a low likelihood of criminal prosecution.  Because of these similarities, I’d like to make the following predictions regarding how this is going to play out:

1. Leadership will initially have tacit approval of such things, and may even encourage it in some cases.  Ultimately, this will fade because companies will not want to expose themselves to the legal liability resulting from creating a culture that pressures their workforce into using illegal drugs and exposing themselves to long term health consequences.

2. Even though the average employee would probably rather not take such drugs, nor would they enjoy competing against other members of the talent pool, there will be resistance to drug testing in this area.  This will happen because the workforce will not want to grant control and leverage in this area to leadership.

It won’t stay that simple for long.  There’s enough money involved, that it’s not hard to imagine a market for designer drugs3 for knowledge workers.  There could be surgical and mechanical options too, just like in sports.  If you are one of the many people that missed the fine documentary Johnny Mnemonic, I recommend you track down a copy to see a completely reasonable possible future.

This may not end up manifesting itself on a person-by-person basis either.  With the global economy, certain countries could specialize in offering a “performance enhanced” workforce.  It seems dark, but there were such high levels of exploitation when manufacturing became a global market, why would it be any different here?

There’s no point in making specific predictions though.  That’s already been done, and done well by creative minds of science fiction.  In a comment to Attention to Detail, Hacker News user 6ren brought up the Verner Vinge novel A Deepness in the Sky and the “Focus” he was referring to can also be used as an example here4.  Further examples are left as an exercise to the reader.

  1. I was reluctant to use Tiger again so soon, but it fit too well.  I’m not nearly as interested in Tiger you would guess based on my initial articles []
  2. Note: His current status is eligible.  The Wikipedia article I linked lays out the details clearly []
  3. In the sports world, the best example of this is BALCO []
  4. This is also a favorite of mine.  Nice work to unknowingly choose an example that ties together consecutive articles []
Posted in Fascinating observations, just fascinating | 5 Comments

Attention to Detail

Perhaps you have been a party to the following conversation:

Person A: Remember in <substitute movie> when <subtle event happened>?

Person B: I watched that movie just last night and I didn’t notice.

If you are not the creative type, then allow me to substitute a fun example for you:

Person A: Remember in Toy Story when Mr. Potato head kisses his butt with his own lips, to mock the slinky dog?1

potato-butt-kiss

Person B: I watched that movie just last night and I didn’t notice.

Now, if you don’t have children, and don’t spend a lot of time around children, you’ll have to take my word for it on this one.  Every other conversation you have with a child is like this.

This takes some time to sink in if you haven’t lived it.  I’ve chosen to buy you a little time with a filler paragraph bereft of any useful information or subtle transition.  Parents especially will appreciate this break, I hope.  I’d also like to take this time to dispel any confusion regarding my choice of example.   The fact that I used a kids movie2 is not relevant to the point.

Bill Watterson really nailed this concept in so many of his strips.  Unfortunately, there is no legitimate Calvin and Hobbes archive for me to link to.  Roger Ebert also expressed it with great clarity in his original review of Willie Wonka and the Chocolate Factory:

“Kids are not stupid. They are among the sharpest, cleverest, most eagle-eyed creatures on God’s Earth, and very little escapes their notice.”3

I recently had an experience with my children that illustrates the point perfectly.  We were  playing Plants vs. Zombies together4.  They need a little help planting enough sunflowers and defending in the early part of each mission, so I get them going and then they take over.  We’ve unlocked all the plants, so they get to choose from the full arsenal at the start of each mission.  It’s ok if you don’t follow.

Their favorite plant is what we call “The Chomper.”  This presents a problem as it is a sub-optimal purchase and offends me as both a gamer and a game developer.  I try to support my children’s decisions though, even the suspect ones.  The way it works is it “eats” the first zombie that approaches it.  While it is chewing on the zombie (which takes a minute or so), an arm sticks out the side of the chomper’s mouth and it is vulnerable to attack.5

Almost all of the zombies are the same size, except for two.  One is a giant that the chomper cannot eat and the other is a mini guy that the chomper can eat.  In one of our games, a chomper ate a mini guy.  This is  such an unlikely occurrence that it has only happened to us once.6

Plants vs. Zombies Anomaly

So there’s all this stuff flying around the screen, and in the middle of the action a chomper eats a mini zombie and right away, my daughter asks me why the arm sticking out of the chomper was a normal-sized arm, when the chomper had only eaten a mini zombie.  Recall that the entire point of relating the story is that this sort of thing is not exceptional for a child.  This sort of thing happens all the time.

There’s something powerful here that feels familiar when you think about it.  I’m sure people in many fields have contemplated the issue, but of course I have the most personal experience in my own field.

Programmers struggle at times to grasp subtle details, even those right before their eyes.

This can manifest itself in so many ways.  The nuance of a word choice in a requirement can be lost during development.  A programmer can stare at a screen, knowing something is wrong (and where), but not being able to put their finger on it until someone leans over their shoulder to help.  Consider just a few things a software organization might do to mitigate these issues:

  1. Pair programming7
  2. Putting “extra eyes” on a problem when progress slows.
  3. Mixing young energy with veteran leadership/talent.

The loss of an eye for detail could be a result of any number of things.  It might be as simple as the same numbing effect that results from proofreading your own work8.  Maybe children have an eye for detail because everything is new to them.

There is a company named Specialist People Foundation9 that deals extensively with issues in this area.  They employ people with Autism and similar disorders and leverage the different nature of their employees’ perceptions to tackle a variety of problems.  I like a lot of things about that organization, but I think the most impressive is how effectively they created a strength from something that is perceived as a weakness:

“Our consultants have a passion for detail that is second to none, and bring unique competencies to tasks that most company employees are less motivated to perform, and therefore more prone to errors. The unique characteristics of autism and similar challenges mean that our consultants actually enjoy tasks that most employees find boring, repetitive or difficult due to the level of detail and concentration required.”

You are overlooking any number of details (at this moment even), and your ability to capture them is probably declining, not growing.  Since you only get so many chances to consume something before you’ve totally lost the eye for detail, you want to bring as much energy as possible to each attempt.  I’m not aware of any better way to approach things, unfortunately.

Of course, what you decide to do with those details after you notice them is an entirely different discussion.  Clearly though,  you can’t make meaningful decisions about things you’re not aware of.

  1. If you doubt me, start watching at 7:25.  It is also listed in the parental guide for the movie at IMDB. []
  2. Toy Story is not a kids movie.  If you agreed with that classification then watch it again. []
  3. You can go read the rest if you want.  It is not very long. []
  4. If you are not into steam for some reason, here is an Amazon Link. []
  5. No, really, this is a good game for kids.  I think. []
  6. It turns out that you can easily reproduce this in the puzzle levels where you control zombies against plants setup by the computer. []
  7. This is not an endorsement of pair programming. []
  8. Note the built-in excuse here for any proofreading errors. []
  9. I linked to their United States operation for my readers, the real deal is Specialisterne based in Denmark []
Posted in Fascinating observations, just fascinating | 13 Comments