Commercial software subjected to public scrutiny

Bruce Schneier reports a court case in which the accuracy or quality of the embedded software in an alcohol breathalyzer is being questioned in a court case and the court decides to subject the software in question to an independent proper assessment, i.e. a proper code review, despite protests naturally from the vendor.

For people interested in seeing a professionally compiled thorough code review can read the full report and a separate deeper analysis report. Both reports are very educational, particularly the second one.

When I am digging through this report, which I do not recommend people to print out the full 57 pages, it reminds me of the similarity in the quality (or lack of) in an Australian-made ERP software. To be honest, the ERP software is far worse by a large margin.

For example, it is not uncommon in the Australian-made ERP software to have far higher density of code exceeding the CC of 105, admittedly, though inexcusably, the ERP software has more LOC. I have personally seen CC higher than 150-200 after discounting the accuracy of the metric measuring tool.

Another issue identified by this review that is wide spread in this Australian-made ERP software is the frivolous and gratuitous use of global variables; this was caused by lack of training in good software practices, lack of reviews and aided by the development tool. In fact so wide spread that it was used as an excuse to avoid an architectural design correction exercise. In the company's view, it is much cheaper to let their poor users to foot the bill of their frivolous resources wastage, the result of its architectural flaws and ignorance, by paying much higher hardware costs.

Public review of commercial and proprietary software should be a normal public quality assurance process to safe-guard the welling being of the software consumers; it is similar to safety rating of cars or electrical appliances. Such public review would have unearthed the glaring mistakes committed by this Australian-made ERP system allowing the users to seek compensation.

Currently, the playing field is severely tilled towards the producers, as reported, allowing them to discharge all responsibilities and to take all rewards.

Companies have often invoked the commercial-in-confidence or proprietary IP excuses to escape such scrutiny. But this report stated in court that

Base One found that the code consists mostly of general algorithms arranged in a manner to implement the breath testing sequence. "That is, the code is not really unique or proprietary."

I doubt there are too many genuinely proprietary IP stuff in many today's commercial software. The more well established they are, the less they are and many are just a quilt of widely publicised algorithms poorly implemented to meet the so-called commercial dead lines.

In my opinion as a developers with over 20 years of experience, I believe those hiding being the veil of proprietary secrecy are too afraid to be caught using unsafe practices, not using industry best practices and ignorance of their mistakes in their own coding; the Alcotest did not even realise that they have committed buffer overruns and including others surprises in the published findings.

The review report attempts to nail down the reasons for the high CC has this to say:

source code appears to have evolved over numerous transitions and versioning, which is responsible for cyclomatic complexity.

While this is a likely cause, and is in agreement with my experience in the Australian-made ERP software, it only indicates both companies have a very poor software maintenance process. Most likely it is based on code-and-fix with little regards to refactoring during a bug fix phase.

The review findings and my own personal experience in this industry indicate that bad software practices are more universal with no geographical boundaries.