Applied equally to software

Software development tools and languages are progressively automated with heavy layers of encapsulation in the guise of freeing the developers from needing to know the underlying principle. This heavy layer of automation is fine until problem strikes and that is when people's knowledge of its underlying principle is needed to dig oneself out of the problem. I am not anti-encapsulation or things like that but I am a strong advocate of the importance to dig deeper to learn the principle behind a framework or class library so that one knows precisely what is underneath.

The situation has many similarity to this pilot's encounter with an automated plane:

The irony, she said, is that the more advanced the automated system, the more crucial the contribution of the human operator becomes to the successful operation of the system. Bainbridge also discusses the paradoxes of automation, the main one being that the more reliable the automation, the less the human operator may be able to contribute to that success. Consequently, operators are increasingly left out of the loop, at least until something unexpected happens. Then the operators need to get involved quickly and flawlessly, says Raja Parasuraman, professor of psychology at George Mason University in Fairfax, Va., who has been studying the issue of increasingly reliable automation and how that affects human performance, and therefore overall system performance.


”There will always be a set of circumstances that was not expected, that the automation either was not designed to handle or other things that just cannot be predicted,” explains Parasuraman. So as system reliability approaches—but doesn’t quite reach—100 percent, ”the more difficult it is to detect the error and recover from it,” he says.
And when the human operator can’t detect the system’s error, the consequences can be tragic.