Wednesday, July 04, 2007


In Verner Vinge's space opera A Deepness in the Sky, he proposes that one of this future's most valuable professions is that of Programmer-Archaeologist. Essentially, the layers of accreted software in all large systems are so deep, inter-penetrating, idiosyncratic and inter-dependent that it has become impossible to just re-write them for simplicity's sake - they genuinely can't be replaced without wrecking the foundations of civilization. The Programmer-Archaeologist churns through this maddening nest of ancient languages and hidden/forgotten tools to repair existing programs or to find odd things that can be turned to unanticipated uses.

"The word for all this is 'mature programming environment.' Basically, when hardware performance has been pushed to its final limit, and programmers have had several centuries to code, you reach a point where there is far more significant code than can be rationalized. The best you can do is understand the overall layering, and know how to search for the oddball tool that may come in handy -"
A Deepness in the Sky - Verner Vinge

Surfing the Web
This picture of a flexable keyboard tunnel linked to from Lord Cuauhtli Rangel's Flickr photostream, he owns the picture; made available under a creative commons license, some rights reserved.

This is not all that different from what I actually do in my current job.

I work at a moderately large corporation analyzing and fixing bugs in our production systems. When a system gets too buggy, because:
  • Over time, the other systems it interfaces with have changed and so now it can't communicate with them as smoothly
  • Elements of the server or network environment in which the programs run have changed, negatively affecting the program's performance
  • Users have developed odd work-arounds to make the program partially serve some unanticipated need, and now those have taken on real business importance
There is often a desire to rewrite the program to accommodate the new situation. Many times, rewriting the system isn't really practical, at least short term, because of the number of unknowns that much change would introduce in to the weirdly balanced ecosystem of software.

I spend a significant amount of time digging through old, partially documented or undocumented code, trying to establish relationships between systems and reconstruct reasons for the way things were done, so the fuller implications of changes to be made can be understood.

We're less than a hundred years into the history of software accretion for our civilization, and already the notion of a Programmer-Archaeologist is not so absurd.

To my knowledge there aren't any significant tools or proven working methods for this new trade, but looking forward, I'll bet this is fertile ground for innovation.


Anonymous said...

This is an interesting view of what we became. The concept of a layer of software implies that each layer's designers all want to tell the world a different story than the layers above and below... It is inherent in human nature to want to say "I am the definitive vocabulary to describe your problem" However if you take this away and try to centralize everything, software development becomes less quick and with more cost I believe. The same pattern of "I am the definitive vocabulary" exists in creating computer languages. If ADA was the perfect language... why did all these other computer languages come into being?

Bill Cunningham said...

I guess that's how you'd have to define a layer of software... certainly the programmers writing it at the time don't think they are describing a defined world-view. But if you are looking at their combined body of code and asking, "what problems were they trying to solve?", patterns should emerge.

Sometimes knowing what problems they were trying to solve in different defined strata can at least narrow down your field of investigation when you are digging through a library of in-house code looking for some useful class that you can borrow or adapt to some similar but new issue.

P.S. - Sorry your post was held for so long - I didn't realize I had anonymous comments moderated. That's been fixed now.

Cuauhtli said...
This comment has been removed by a blog administrator.
Bill Cunningham said...
This comment has been removed by the author.