the double-edged sword of efficiency

The big software defect story of the past couple of days is definitely Vassar’s accidental sending of acceptance notifications to several students. It’s a great example of one of the consequences of putting an algorithm (and indirectly, a programmer), in charge of disseminating information. On the one hand, I’m sure this saved Vassar a lot time and perhaps a job or two, completely eliminating their need for post and paper. On the other hand, they’ve adopted a system that is going to fail from time to time, and not in graceful ways that paper does, but in big, dramatic, and unpredictable ways.

The unpredictability of software defects is one of the most interesting properties of software as a medium. It’s inherent complexity means that even the people who develop it are going to have a hard time knowing what part of the system will fail and how dramatically. In fact, if the developer follows best practices by modularizing the system and enabling it to scale gracefully, it will actually guarantee that the failures will be more dramatic: whether it’s a list of 1, 100, or 1,000,000, I’m sure the Vassar notification system algorithm will do the exact same thing.

I wonder how software might be built to better account for the significance of the information it transmits and computes. At the moment, I suppose this is captured in the software tests that teams perform. Perhaps a better way might be to tag the data that moves through software systems and propagate things like the confidence, credibility, and integrity of data as algorithms munge and manipulate it.