Is building a better mouse trap (Signal Private Messenger) enough to win market shares?

I am please to see the release of Signal Private Messenger for Android and iOS, a messaging application that has earned full marks in the EFF security score sheet. I am a fan of this product and I like it very much for the following reasons:

• It is an open-source project offering the service for free. WhatsApp is not a free.

• As a result, it can be reviewed by anyone capable of doing it while WhatsApp is proprietary, even though it claims to be underpinning by Open Whisper Systems but no one has reviewed that. Recent event has indicated that WhatsApp messages have been intercepted and decoded.

• It is not owned by any company while WhatsApp is owned by Facebook, Skype by Microsoft. Thus all metadata in WhatsApp and Skype belongs to Facebook or Microsoft respectively.

According to well-known security researchers, Bruce Schneier and Matt Green, Signal is developed to a very high quality to provide end-to-end encryption (E2E) not only for messaging but also for voice and their endorsement must mean something.

I am not here to raise doubt of this product which I am using admittedly with very limited users to interact with and I have great trust. I hope it will do well.

But I am here to question whether it is enough to rely on technical superiority which is so well hidden from the users to induce them to switch to Signal and to grow its market shares. That's is: is building a smarter (more secure) mouse trap enough to win market shares? Other class of software such as web browser, anti-virus, media player, or mail client can draw people to switch based of superiority of features.

Looking at the landscape of messaging applications it is difficult to see how Signal can rely on security implementation, so out of sight of the user, to win market shares. Will this become a replay of VHS (WhatsApp, Skype, etc) vs BetaMax (Signal) of the 21st Century?

Messaging applications are like clubs or cults in which they only allow club members to interact and go to great length to discourage inducement to leave and definitely providing no facility to support inter-club interaction. This produces network effect to draw people in and that also becomes disincentive to leave and its nurture of human social interaction provides a positive feedback to increase the network effect.

Looking at the EFF Security score card, most of the popular messaging applications do not use security best practices and their inferiorities do not seem to matter to the users. The anecdotal conclusion one can draw is that users do not care with online privacy and security despite well publicised massive surveillance activities. Unlike other type of application, such as web browser, there is no report of people deserting one messaging application to another, despite vulnerabilities and caught not using secure messaging mechanism when they claim to use. For those entrenched players, they must feel like in a no-loss situation. The only way they can lose to a competitor is by a total annihilation of the enterprise.

Messaging applications have another unique characteristics that it is not the features that draw users to choose a particular application; there is a great degree of peer pressure exerted by those early adapters unwittingly forcing people to form that circle of friends. This peer pressure then forms a vortex to draw more and more people in. Their only concern is to be able to communicate with the club members.

Because of the lack support for inter-application interaction, the application through using proprietary communication protocol forms a natural barrier for their user to leave. Apart from that, the user does not see any benefit for using a different application that essentially providing the same things - messaging and may be voice - and having to desert their friends. So why leave? What is the benefit to them?

Many users of messaging applications also form the mistaken belief that they can only use one messaging application in their device. Perhaps it is this mistaken belief or blind fanaticism to their favourite application they are also reluctant to install other messaging applications to increase their reach to their friends. Since Signal is so similar to WhatsApp, it is simply a matter of installing and waiting for others in the contact to install their copy of Signal to re-establish communication. Even that simple is not enticing.

I have spoken to several users of messaging applications as well as non-users and recommending to switch over to a more secure application called Signal. But telling them the benefits of Signal is like talking about wine apprecThis is particularly difficult when Signal is so similar to the operations of WhatsApp separated by a thin veneer of technical features. In view of this, users of WhatsApp (or other app) are unwilling to desert their circle of friends to use something that to them is almost the same thing with minute user base, by comparison. iation to a group of teetotalers. To them the improve security and end-to-end encryption (E2E) are not enough to sway them. Even people that has not used messaging application seems to be reluctant to get onboard with Signal because they have not heard of it being mentioned by their friends.

So I wonder how a late comer like Signal can overcome these barriers to increase its market shares? How it can base on technical superiority to entice users who are disinterested of them that Signal relies on to distinguish it from others? What is the future of Signal apart from being a niche player at best? Clearly Signal needs to improve its image and marketing.

From the analysis, users of messaging applications place extremely high premium on their ability to reach their circle of friends and ignore other issues like security and privacy. Therefore if the new comer, like Signal, wanting to rise up, it must give their users a transparent way to interact with their circle of friends without requiring them to switch en masse like the present situation. How to achieve that is the real challenge in messaging application development in view of no standard communication protocol?

Comments on using e-mail address as username for online services

I have encountered more and more online facilities using e-mail address as the user name. In my mind, this is a lazy way for the service to check or to provide a unique user name when creating an account. In some rare usage, this may be fine but generally this is a very restrictive form and the reasons are given below.

Using e-mail address has the following problems:
1) While it is unique in the universe of the Internet it does not uniquely identify a user of the service, thus unsuitable as a user name unless the service has other facility to deal with one e-mail address for multiple users.

For example if one manages several properties or funds belonging to different entities under some management agreement, it is often convenient to use one e-mail address for all these properties or funds. It is also possible that the e-mail owner owns all those properties or funds, it is unreasonable to base that identifier on an e-mail address which does not map to a unique entity; e-mail address is for correspondence - like a house address.

Who would then use a house address to identify a person living there when it can house several persons?


I have seen one service that uses the user name (aka e-mail address) as a proxy to a fund account. This then assume the owner of that e-mail address cannot have more than one funds - one may be for him and another for some other ownership arrangement with correspondence being sent to the same address. Clearly the developers have not model the usage requirement well.

This silly design is like the above house number analogy requiring a house to house just one person.

The assumption that an e-mail address uniquely maps to a particular person or entity is unsound. Don't do it. It is far better and more secure if your system generates a unique number, a la, account number, for the user.

2) The use of e-mail address as a user name can confuse user in that he/she has to supply to the online service the same password for e-mail account. This can lead to an increase (or subliminally encouraging) reuse of password, a dangerous practice.

To a less technically savvy person, he/she may be misled into believing that the e-mail provider now have access or linked to whatever materials available in the online service.

3) While it is infrequent, though not impossible or improbable, for people to change e-mail address, services that uses e-mail address for correspondence as well as for user identification inevitably prevent user from changing e-mail address. This is because it is using a very poor design pattern - one piece of data to serve two distinctly different and diverse purposes. The user name is to identify a user which an e-mail address does not and the e-mail address is for correspondence, like a house address which can be used by anyone living there to receive correspondence.

If you ask correspondence sender to simply put the address on the envelope no one in the household will know to whom is that letter addressed; you need to put the addressee's name (the user name). A person could one day moves out of that address; he/she retains the same name (user name) but simply changing the delivery address (changing the e-mail address). This happening may not be frequent but not improbable or impossible.

No right minded person would combine the two (addressee's name and the delivery address) but why do that in the computer system?

To address this kind of short coming, they then have to provide a means for the user to define an e-mail address for correspondence. In this situation which one should the system uses during account set up and validation purpose?

How to overcome this poor design as a user?

If you, as a user, are confronted with this problem - how to use one e-mail address for more than one users of the service - you may try this solution provided that:

• Your e-mail provider supports e-mail alias. GMail and Hotmail support them. If you provider does not supports this, set up a GMail account as a mail redirector.
• Your online service's user name (aka e-mail address) validation knows about RFC 822 - Section 6 Address specification. Those failing to parse this properly would reject your e-mail address with alias.

Then use e-mail alias (like Somebody+Property1@GMail.com or Somebody+Property1@Hotmail.com) to allow one e-mail address to be used for several entities. The '+' character in the local part of the e-mail address is valid and permitted under the RFC. If their developers tell you that it is an incorrect address, point them to the RFC.

Those thinking of using e-mail address as a user name to relieve them the task to validate its uniqueness needs to validate the e-mail address to conform to the RFC.

To me, the task of validating and ensuring a user name is unique within the system is far easier than validating the e-mail address because the latter needs to check:

• conformance to RFC
• that the e-mail provider supports the e-mail alias that the user enters, as the service has to make sure it is a reachable address to receive correspondence. If that alias syntax is not supported by the mail provider, conforming to RFC does not guarantee it can be used for correspondence.  

Here lies the danger of tying the two purposes to one piece of data, that is using an inappropriate design pattern.

Software is not created to serve human. A banking example.

This is a rather provocative title for a purpose. Recently I read a very good article from ACM Communication titled "Center Wheel for Success" by Poul-Henning Kamp, Communications of the ACM, Vol. 57 No. 2, Pages 52-54 in which the author questions the high costs of some of the large scale software developments and the finding so many for failing to meet the objectives while many died prematurely. He was citing the a system designed to meet US government's Obamacare as the example.

He attribute this largely to "Blindly deciding that IT be substituted for humans is unenlightened. IT is not a magic potion that makes unpleasant or inconvenient things disappear."

It is nice to find someone that shares my feeling on this topic and that they do not really serve human. At best they should be a tool to help human to deal with mundane and repetitive tasks, which computer and IT programs are apt to do. Human is a master of creating changes and master in complexity and adaptability to deal with ad hoc matters.

Let's consider a real life example.

Today, I was in a bank - it does not matter Which Bank - as I am sure this happenstance is found in any bank any time of the day.

A relatively matured lady was waiting in line patiently for being served and a bank staff approached her to ask how she could help her. She told the staff all she wanted was to get a list of recent transactions of her account that was accessible from the card in her hand. How hard could that be and I am sure that is probably the most required action from any customer.

Without a second to digest the full requirement, the staff (could she be a developer in her previous job?) shepherded her to an ATM machine protruding from the wall and attempted to show her how to get the list. After some moments of fiddling with the ATM without satisfying the customer, the staff realised how stupid she was that did not even listen carefully.

Eventually she took the lady to stand beside a bank teller using her terminal to view the list of transactions. She did not receive a print out despite her money funding the running of the bank.

Is this service? Does the banking system actually serve the customer's needs? Her needs is nothing complex and must be known to anyone in the banking industry without even involving a neuron and yet she still could not get what she wants other than 'being a bank teller' literally.

I contend that the banking system has missed the mark by a long shot. Consider not long ago, she would not have to walk to the bank, waste her time standing in queue and explaining the simple thing that she took for granted for so long  when she was clinging to an information rich bank book; she could simply look at her bankbook. Now she has been given a piece of plastic devoid of any useful information. I have a feeling that she did not enjoy such an experience that the system has successfully made her feeling useless.

Sure time has changed but human's needs and modus operandi have not. What good is a system if it cannot meet the human's basic needs other than to satisfy the fancy of the developers.

In this case, why can't the ATM has an option that allows the user to list and print the transactions one is interested in. This is a simple operation and it is definitely a good way to ease those bankbook dependent customers over to a new system. At best the current crop of ATM could only show the current balance.

In HK MTR, they have a balance checking device next to the ticket dispensing machine that allows the Octopus Card holder to list the 10 most recent trips, their costs and the balance remaining. That's a good service.

Surely any bank system analyst would have sat in a bank branch observing how customers interact with the bank over days, recording and understanding their needs and expectations before attempt to develop a system to 'serve' the customers?

It is not hard to replicate a simple bankbook and doing so will make many happier and imposing less burden on them. It is worth remember this simple phrase "You (the developers) are not the users".

If the system requires the user to buy a modem, have an internet connection, learn how to deal with a computer system and to fire up a web browser (web what? to many) to do the job of a bank teller, how can one calls that system serving the customer. It is basically a DIY operation and not serving the customers at all. The bank is passing the buck to the customers making them the pseudo-bank employee with no pay.

Common reasons in software system failure

A post by Charette in IEEE Spectrum on the failure of various government systems sums up pretty much the general reasons why they fail:

put all the blame for the fiasco on Deloitte, which Deloitte heatedly contests. News reports state that at least one Florida lawmaker is suggesting that Deloitte be barred from future Florida contracts, something that Australia’s Queensland government has done to IBM as a result of Big Blue's role in the Queensland Health payroll debacle.
[...]
The bottom line was that no one was in charge, vendors and the state did not get along, the vendors themselves did not get along, no one wanted to hear about the myriad significant technical risks, and political motivations dominated decision making. In other words, all the makings of an all-too-typical government IT project.

Could it be that we are trying to develop software to do far more complex stuff, instead of being a tool to help human, they are being developed to replace human? In many of these systems, the common theme in all these failures - hire more human to remedy the failure!

In most of these large systems development, by the time the system is ready, the world has changed; the political scene has changed causing requirements to change; rules are changing all the time before the ink is dried.

Do not accept answers that you do not understand.

Not only it is the duty of the purse string holders needing to know precisely, without being clouded by technical jargon, what a software solution can do for the enterprise, every users need to know. This ensure that the buyers and users know precisely what they are getting and what they are paying for.

It is refreshing to come across this article in Financial Times relaying the experience of a very wise and persistent director of an enterprise that "demanded the computing experts translate their plans into plain English."

Far too often, as I have observed, that the recipients of software products choose to keep quiet for fear of showing up their ignorance or failure to comprehend of the materials from the so-called "technical experts". They all should emulate this "Dennis".

In fact, it is the communication fault of the "technical experts" that fail to explain the materials, particularly the functional part, in the terms that the users can understand. I don't expect them to explain how data base schemas are designed, XML messages being passed around or how to pool connections but at least they should be able to answer user-centric questions like what is the response time of say 20,000 users accessing the system, the stability of the system in terms of how often one expects a failure.

In a theme similar to the "Why Software Sucks" but at a different level, the  messages from the FT article are worth remembering. Admitting one does not understand the gobbledegook, often silently, and then not demanding answers to one's satisfaction is an abdication of one's duty.

Crazy idea - Razer mouse and cloud

It seems a day goes pass without some one becomes seduced by the cloud - just for something to store things is enough to be called cloud, totally crazy.

Now Razer Mouse has to be registered with its Cloud activation server before its full capability can be used.

I totally agree and sympathise with the author of this post in lambasting the Razer for their crazy unfriendly scheme.

Razer's Ming-Liang Tan's justification is every more questionable:

Instead of having mouse settings limited by the space in onboard memory, Synapse 2.0 allows gamers to now have almost unlimited space for their profiles and macros.

Do they really need unlimited space? Perhaps Razer should publish some survey or statistics to convince doubters. What's wrong with such thing as USB drive. How much memory are you talking about? 16G USB memory drive is as common as mouse attached to machines.

Why not have a SD Card slot into your Razer USB drive that the user can stash their settings into that.

While Razer is free to request their user to register but should not coerce, what I have found the order in the process some what unfriendly and draconian. Accord to Razer

Once registered, Synapse 2.0 works offline and never needs to be online again. So basically, a user creates an account, saves initial settings, and if there's no internet connection, it doesn't matter - settings are saved on the client PC and are not synced to the cloud. Synapse 2.0 works offline. 

If I have bought one of these mice, I will definitely be annoyed - what the hell Razer has a right to demand my e-mail address in order for me to use the full capability of their product. Why should I have to register & activated?

I guess by now their cloud server is full of fake information and disposable e-mail addresses for people just simply eager to dismiss the pest. I do that all the time after all it is a virtual world and honesty has not place.

Perhaps their developers have not considered their users' feelings properly. A much more user- friendly way is to reverse the order of the above mentioned process. That is:

• Let the user on to the settings program - regardless. They are already your customer and you have their money, mind you!
• On saving (I hope Razer has a way to save the settings) offer the user a choice, like all good user caring program does.
• If they choose to register, they have the benefits blah blah blah - do your Razer advertising to convince your user to go your cloud way - but don't coerce.
• Or the user can choose to store the settings locally on their USB drive etc.

I have to say, your current implementation is too draconian - this is the way take it or leave. Not nice and deserves the user's wrath.

What is more difficult - to think and to do something - follow on

In my previous post, I pondered on this topic and wonder if the experiment in MIT laboratory in studying habit can explain them.

Since then I rediscovered a study [Hadar] published in May 2008 that seems to attempt to answer similar question but more specifically directed to OO design.

Apparently

the formal OO paradigm has come to sometimes clash with the very intuitions that produced it. Thus while objects, classes, and inheritance certainly have an intuitive flavour, their formal version in OOD is different in important ways from their intuitive origins.

Dual-process theory, imported from contemporary cognitive psychology, highlights the underlying mechanism of those situations where our intuitions clash with our more disciplined knowledge and reasoning..... Highly accessible features will influence decisions, while features of low accessibility will largely ignored.

The intuition, as discussed in this paper, is explained based on the Dual-Process theory, credited to D. Kahneman, which states that

our cognition and behavior operate in parallel in two quite different modes, called System 1 (S1) and System 2 (S2), roughly corresponding to our common sense notion of intuitive and analytical thinking.

... S1 processes are characterized as being fast, automatic, effortless, unconscious, and inflexible (difficult to change or overcome). In contrast, S2 processes are slow, conscious, effortful, and relatively flexible. In addition, S2 serves as monitor and critic of the fast automatic responses of S1, with the "authority" to override them when necessary. In many situations, S1 and S2 work in concert, but there are situations ... in which S1 produces quick automatic non-normative responses, while S2 may or may not intervene in is role as monitor and critic.

The paper is worth reading as it describes experiments with experienced developers rather than using students.

I wonder if this explains why it is much easier to get "experienced" people to do work (the S1 process at play or the assisted and influenced by habits) while harder to get someone to think (the S2 process and the need to counter to what habits may lead to)?

I am also wondering if S1 relates to the Accidental Tasks while S2 is to Essential Tasks in software development as elaborated long ago in the "No Silver Bullet" paper by Brooks?

---

[Hadar] "How intuitive is Object-Oriented Design?" by Irit Hadar and Uri Leron, CACM May 2008/Vol 51 No. 5, pp 41-46

[Brooks] "No Silver Bullet - Essence and Accidents of Software Engineering", Frederick P. Brooks, Jr. IEEE Computer April, 1987. Vol 20 No. 4, pp 10-19

There is no dangerous language only dangerous programmers

It seems the data and comments presented in "Software Failures, follies and fallacies" by Les Hatton provide support for the above mentioned comment.

We all extol the benefits of our favourite programming language whilst denigrating other languages less attractive to us.  In truth, published data from around the world of which Table 2 is a subset shows  that there is no clear relationship between programming language and the defect density of systems implemented in  that  language.    Ada,  for  example,  supposedly  far  more  secure  than  other  languages  produces systems of comparable defect density.  In contrast,  C is reviled by many  safety-related  developers and yet it is responsible for some of the most reliable systems ever written.   We  can  conclude  that programming language choice is at best weakly related to reliability.

However, I doubt his conclusion from

a  recent  study  comparing  two  similar  systems  of  similar  size,  (around 50,000  lines each), one  in  C  and  one  in  object-designed  C++,  the  resulting  defect  densities  were shown  to  be  around  the  same  at  2.4  and  2.9  per  1000  lines  respectively,

that

that language has little effect on reliability, object-oriented or not  and  that  the  massive  drive  to  object-orientation  is  another  giant  leap  sideways  in  which  the software industry appears to specialise.

It is unfair to conclude a programming paradigm is defective or fail to live up to its touted benefit from observation on a programming language. This is particularly true for C++ which actually is hybrid in nature - a better-C and Object-Oriented programming language.

From my personal development experience and reviewing code, a person using an object-oriented language, be it C++, C# or Java, does not necessary mean that the developer is apply sound OO principles to their creation. It is the use of these principles that give rise to the touted benefit. 

If they are not applied, C++ and other object-oriented language can result in system with far worse defects and undesirable features than say a procedural language like C. Lakos echos similar sentiment:

It is completely wrong, however, to think that just using C++ will ensure success in a large project.


C++ is not just an extension of C: it supports an entirely new paradigm. The object-oriented paradigm is notorious for demanding more design effort and savy than its procedural counterpart. C++ is more difficult to master than C, and there are innumerable ways to shoot yourself in the foot. Often you won't realize a serious error until it is too late to fix it and still meet your schedule, such as indiscriminate use of virtual functions or passing of user-defined types by value, can result in perfectly C++ programs that run ten times slower than they would have had you written them in C.
[...]
Unfortunately, the undisciplined techniques used to create small programs in C++ are totally inadequate for tackling larger projects. That's to say, a naive application of C++ technology does not scale well to larger projects. The consequences for the uninitiated are many.

---

"Large-Scale C++ Software Design" by John Lakos, page 2, Anddison Wesley Longman Inc. 1996

What a piece of gem

I am so glad that I have rediscovered this piece of gem titled "Good practice in software" by Les Hatton. It should be read by any aspiring or experienced developer. It is full of timeless advices and facts, like this:

Perhaps the biggest surprise in an industry awash with technology is that it doesn’t seem to make much difference. By far the biggest factor which emerges in most studies is the individual quality of the engineers who build a system and a few common-sense principles.  We have known this since the admirable book by Fred Brooks, [6] which every aspiring software producer should read. Rather than absorbing this powerful lesson, the computing industry became obsessed with the notion that the process or bureaucracy of building software was the most important part.  The evidence is to the contrary.  No matter how well- defined a process, the quality of the product still depends mostly on the quality of the engineers who build it. To this, a good process can bring consistency and accountability.

So true! People are forever darting from one language to another or from one technology to another in an escapist attitude to avoid the hard slog of learning the trade!

No use to defend an indefensibly bad user-interface

I was shown a web application designed to manage project and progress report, a la MS Project minus the Gantt Chart and time lines. I am not a user of any project management software and hence this post is not about its capability, though my shallow knowledge of this topics tells me that it is rather incomplete.

As a software developer with low tolerance of terrible user-interface that only the creator loves, this package has an example that really enrages me into highlighting it here. Consider the following screen capture showing the 'Select All' check box circled in red:

A shaped-eye reader will instinctively spotted something amiss in this diagram; the 'Select All' check box is checked but the other check boxes in that column aren't. Well according to the developer, this is a feature. Let me describe how this 'Select All' check boxes works in this application.

Normally it was not checked and the user can check the relevant row by using the check box which operates in the standard manner.

The minute you click on the 'Select All' check box, it selects all the check boxes in that column. Nothing strange about that and it is the same behavior as in Google's GMail or Hotmail, just to name a few.

However, the user-interface becomes non-intuitive and distorted when you try to de-select all the check boxes by clicking on the 'Select All' check box, which by now has a check mark on it, an operation that comes naturally for all users. In this crazy illogical scheme, the developer literally creates a different class of check box but with the same look and few as the standard ones. A true bastardization of the check box.

When one clicks on the 'Select All' check box the second time, instead of extinguishing the check mark and de-selecting all the check boxes in that column as everyone (bar the developer) expects, this check box maintains the check mark - a kind of one shot check box but not exactly. If you click it in vain attempt to de-select all, it in fact selects all check boxes for you. There is no way to de-select all and the 'Select All' check box is not disabled or grayed out. Because there is no way to uncheck all the check boxes in one operation, experimentation with this 'Select All' check box brings frustration and curses.

The standard check box has a binary state - click it to check and click it to uncheck - and that is the behavior any user expects when one sees a check box and not a distorted one as highlighted.

The developer maintains that it is not a bug but a feature just like so many when cornered into this indefensible situation. I think the developer is wise to read this book and constantly reminding himself that "Your user is not you". In clear vain attempt to defend the indefensibly wrong user-interface, the developer offers this way to extinguish the 'Select All' check mark, which is an other brilliant example of his failure to understand "Your user is not you":

• Uncheck each check box in the column manually
• Then press the update
• The round trip back from the server will clear the 'Select All' check box.

I am surprise that he did not suggest to log out of the application and relog in as that would have done the same job more quickly saving you manually unchecking each check box. Interesting no other users think that this is a brilliant design! My advice: listen to your users as you software is as useful as seen through your users' eyes. Your user is not you or there are more users than developers.

The correction of this distorted user-interface is technically very simple but psychologically difficult as the developer has to admit that it is a bug first; they already have JavaScript code to check each check boxes in the column and it is just a simple process to run the code through to uncheck them. In fact, they expand more efforts in creating this only-creator-loves distorted user interface.

 This software also has another time wastage feature, which clearly has not gone through any user design mock up such as "Paper Prototyping" by Carolyn Snyder.

There is a page where the user has to indicate the progress of each milestone and at which date. The date must be entered by using 3 combo boxes - one for day, month and year as shown here:

This means to enter a date, one needs a minimum of 6 mouse clicks. If the date can be entered via a edit box with an calendar icon to invoke it if necessary, the user could enter the date much more quickly.

But this will require more developer's input validation effort. Since it is already using client side JavaScript, it is not such a big deal with the most difficult task for determining the client's locale.

When you have a dozen or so dates to enter, every saving is a real bonus and the developer should consider redesigning. The current design is really for the benefit of the developer - saving code. In fact, they still need to validate the input. What if someone select 31/04 or 30/02?

Definition of a "Bad Design" in software

I came across this succinct definition of what constitutes a "Bad Design" in software:

1. It is hard to change because every change affects too many other parts of the system. (Rigidity)
2. When you make a change, unexpected parts of the system break. (Fragility)
3. It is hard to reuse in another application because it cannot be disentangled from the current application. (Immobility).

Surprisingly, software having these traits are very difficult or even impossible to be unit tested. Hence unit test is often used to a gauge of these bad traits. Don't believe, try it.

Applied equally to software

Software development tools and languages are progressively automated with heavy layers of encapsulation in the guise of freeing the developers from needing to know the underlying principle. This heavy layer of automation is fine until problem strikes and that is when people's knowledge of its underlying principle is needed to dig oneself out of the problem. I am not anti-encapsulation or things like that but I am a strong advocate of the importance to dig deeper to learn the principle behind a framework or class library so that one knows precisely what is underneath.

The situation has many similarity to this pilot's encounter with an automated plane:

The irony, she said, is that the more advanced the automated system, the more crucial the contribution of the human operator becomes to the successful operation of the system. Bainbridge also discusses the paradoxes of automation, the main one being that the more reliable the automation, the less the human operator may be able to contribute to that success. Consequently, operators are increasingly left out of the loop, at least until something unexpected happens. Then the operators need to get involved quickly and flawlessly, says Raja Parasuraman, professor of psychology at George Mason University in Fairfax, Va., who has been studying the issue of increasingly reliable automation and how that affects human performance, and therefore overall system performance.


”There will always be a set of circumstances that was not expected, that the automation either was not designed to handle or other things that just cannot be predicted,” explains Parasuraman. So as system reliability approaches—but doesn’t quite reach—100 percent, ”the more difficult it is to detect the error and recover from it,” he says.
And when the human operator can’t detect the system’s error, the consequences can be tragic.

Good for the Industry

More of this will be good for the IT industry.

Imporantce to knows what is behind the facade

A well written essay argues

there is still a need for "old-fashioned gumption" — a need to retain some mastery over tools we rely on and to explore others' potential.
[snip]
... "it can be very disempowering because people aren't really understanding what's going on in their business".
Of course, the most common tools in our business and personal lives have user-friendly interfaces, enabling the least tech-savvy to use them. An unfortunate side-effect of this, Mr Gelme says, is that "general users of technology have no idea about how an appliance really works, which means they may not know how to use it most effectively or repair it".
He adds: "Hiding the internal operation with an intuitive interface is compounded by the manufacturer's natural tendencies to protect their product from competitors and to increase sales by ensuring broken appliances have to be replaced, not repaired." This is achieved mainly through proprietary software.

Far too many developers lack the understanding of the mechanics behind the tools they use. Many relies on the tools as an escape from learning the fundamentals. As a consequence, they are at loss when problem occurs, which according to Murphy's law will inevitably occur.

Difference between expert programmer and novice

Very succinctly put:

An expert can tell a non-expert to check for such conditions, and with those specifications, the non-expert can probably code them, but only the expert anticipates them. Just like driving, what makes a good programmer is not only the ability to solve the problems that occur, but the ability to foresee (and avoid) problems that haven't occurred yet. Unfortunately, experts learn how to do that by making mistakes. It's a sad commentary on the human condition. Each generation acquires expertise primarily by repeating the mistakes of the past generations. To paraphrase Neils Bohr, "An expert is someone who has made all the possible mistakes in a very narrow field."

But when you're riding in a car with a novice driver, you'll probably appreciate P. J. Plauger's version more, "My definition of an expert in any field is a person who knows enough about what's really going on to be scared."

Comments:
Unfortunately, many novice developers or ones that have just acquired a taste of a new technology immediately delude into writing something only an expert is capable. It does not matter if it is just to store data in a file as illustrated by the example used. Or using COM, C++ framework such as ATL/STL, .Net or Java.

Many development managers lacking the experience and expertise fail to distinguish between act of stupidity and stroke of an expert, thus concur blindly allowing novice to use their product as the training ground, often leading to failure to exploit opportunities and creating laughable products.

Often these managers confuse the expert's time and effort to "foresee (and avoid) problems that haven't occurred yet" as being a case of gold plating and then wondering why they are having recurring problems, lack of respect and trust from their customers, and complaints.

Tools to help you to make your code secure

Microsoft has released two tools BinScope and MiniFuss.

these tools to show you how easy it is to use them to improve the security of your software. BinScope Binary Analyzer is a Microsoft verification tool that analyzes binaries on a project-wide level to ensure that they have been built in compliance with the requirements and recommendations of the Microsoft Security Development Lifecycle (SDL). At Microsoft, use of BinScope is a requirement of the Verification Phase of the SDL.

The best way to learn them is to start from this article.

I just love this quote...

Some times ago, I by accident came across this quote by a Poet Janet Minor:

"I have a spelling checker
It came with my PC;
It plainly marks four my revue
Mistakes I cannot sea.
I've run this poem threw it,
I'm sure your pleased too no,
Its letter perfect in its weigh,
My checker tolled me sew.

I frequently come across code that resembles something like this passage; all syntactically correct but full of runtime error.

Is application of McCabe Complexity just another length reduction technique?

I was reading a book by Tom Demarco [1] that has made some comments about McCabe Cyclomatic Complexity [2] that I believe indicate he has misread the theory and showing off his lack of knowledge in this area. He said this:

... But when you let it guide you to produce alternate designs with lower values, you almost always end up dividing the modules into smaller ones. Now that is a pretty sensible thing to do in general, but it is not really a complexity reduction technique. It's a length reduction technique. You could do the same thing with a much simple metric, like line of code: If a module has too many lines divide it up. V(G) is, most of all, a length metric. Longer modules have higher V(G). If you factor out the hidden effect of length, V(G) is relatively meaningless.

Before I discuss many incorrect assertions in the above paragraph, Tom DeMarco also made the following remark:

... Tom McCabe's Cyclomatic Complexity - also called V(G) for some reason now forgotten.

Well sorry DeMarco, it has not been forgotten. For DeMarco's information V(G) is a standard notation used in Graph theory [3] and it stands for vertices of G. It is not a notation invented by McCabe. In fact, possibly the lack of familiarity with this notation and the graph theory explain why DeMarco makes such a rash and incorrect observation and from these mistakes one can safely assume he has not even used McCabe Complexity metric in even toy projects.

Let's consider his assertion: "It's a length reduction technique. You could do the same thing with a much simple metric, like line of code". Could you? What is better to start this debate than by stating Tom McCabe's expression of V(G) [2]:

V(G) = e - n + p
where
e = number of edges in a graph
n = number of vertices
p = number of connected components.

A graph is not about the number of assignment, line of expressions or function calls but it is made up of branching and logical expressions that causes transitions from one block of code to another, thus the edges. Where is line of code in the above expression, Tom DeMarco?

The following block of code gives some idea of complexity & LOC:

public static Int32 f1( Int32 a, Int32 b, Int32 c, Int32 d, Int32 e )
{
Int32 t = a;

t = a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + 3;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
t = t + a + 2 * b + c + d;
return t;
}

Which contains 23 statements, thus LOC=23, but according to McCabe, it only yields a CC=1. It does not matter how complex or how simple is the expression on the right hand side of the assignment it remains a CC=1. It could be all repeatedly initializing t to say 0 or it could be a very complicate polynomial expression of any length involving the function arguments CC remains 1 while LOC can increases to very large number.

No matter how Tom DeMarco slices and dices the above block, even down to functions with one line, thus LOC=1, CC is still 1. Hence this invalidates his wild claim.

Now consider this block containing 4 statements, thus LOC=4:

public static Int32 f2( Int32 a, Int32 b, Int32 c, Int32 d, Int32 e )
{
if( (a == b && (c != 5 || d != 6 ) )  ||
    (c == e && (e == 2 * a && c > b && d  == 3 * a ) ) )
    return 23;
else
    return 0;
}

Which yields a CC=9.

If Tom DeMarco's assertion that "You could do the same thing with a much simple metric, like line of code" is right, then the first block of code is more complex than the second, according to DeMarco. Clearly that is not true! It does not take much brain power to scan through say 23 initialization statements but to come up with an idea of what is doing in the second block takes a bit of comprehension.

In general, the technique to reduce CC is to make the code more readable to human as machine could not careless by rewriting the logical or control expressions using meaningful function names so that instead of just stating the logical expression in its raw form, the function names help the reader to read the meaning of the expression. It is kind of like literate programming. Or one could use this to make your code more readable and to reduce the CC.

To prove this point and to show that CC and LOC are unrelated, let's refactor the function f2() above in accordance to the recommendation to produce f3() as follows:

public static Int32 f3( Int32 a, Int32 b, Int32 c, Int32 d, Int32 e )
{
  bool danger = ( a == b && ( c != 5 || d != 6 ) );
  bool risky = (e == 2 * a);
  bool makesLotOfMoney = (c > b && d == 3 * a);
  if ( danger ||
      ( c == e && ( risky && makesLotOfMoney ) ) )
          return 23;
  else
      return 0;
}

This function has a LOC=7 and increase from 4 but a reduction of CC from 9 to 6. In according to DeMarco's claim, I have in fact increases the complexity read LOC. This simple experiment clearly refutes DeMarco's unsubstantiated wild claims and his remarks should be discarded.

The above demonstration clearly shows the lack of relationship between LOC and CC. As a result, we can safely discard DeMarco's assertion as being without foundation.

I have great respect for many of his other writings and therefore I am shocked by so many false statements in just one paragraph of this book [1].

---

[1] "Why does software cost so much? And other puzzles of the information age" Tom DeMarco, Dorset House Publishing 1995

[2] "A Complexity Measure" by Thomas J. McCabe, IEEE Trans. Software Engineering, Vol SE-2, No. 4, Dec 1976, pp 308-320.

[3] "Discrete Mathematics, 2nd Edition", K.A. Ross and C.R.B. Wright, Chapter 8, Prentice-Hall International Editions, 1988

Do software systems age analogously to humans?

Having worked in software industries for a long time, this question, posted in an introduction to a paper by the noted and respected Computer Scientist, David Parnas, in a book seems kind of obvious to me and no doubt to many practitioners. However, there are software companies that still hold this belief of "Design once last forever" philosophy and need not change at all. They are also unfortunately pursuers of this illusive Silver Bullet that another noted and respected scientist, Fredrick Brooks, has warned. They naturally do not believe software can age.

Having worked in such a nescient organisation, I am interested to read what Parnas has to say when I come across his paper "Software Aging". There are specifics that I believe worth mentioning here.

One of the good ways to slow down or even to deal with changes, is to design to cater for changes using well-known design principles. In his research and experience, sadly "I do not see much software that is well designed from this point of view" and he gave some of the reasons why:

... The principle is simple; applying it properly requires a lot of thoughts about the application and the environment. The textbooks do not make that clear.

Many programmers are impatient with such considerations; they are so eager to get the first version working or to meet some imminent deadline, that they do not take the time to design for change. Management is so concerned with the next deadline (and so eager to get to a higher position) that future maintenance costs don't have top priority.

Designs that result from a careful application of information hiding are quite different from the "natural" designs that are the result of most programmers' intuitive work.....

Programmers tend to confuse design principles with languages.... Even worse, they think that one has applied the techniques, if one has used such a language. [Sadly, in my professional experience many managers are the big champions of this (mis)belief creating massive software problems and wastes.]

Many people who are doing software development, do not have an education appropriate to the job.

Italics text are my comments.

One of the topic that he commented on that I share passionately that is often not done in Software Industry as compared with others, such as other forms of engineering and medicines is the topic of reviews despite its effectiveness supported by abundance of well publicized materials. Robert Glass even proclaims that it is "being a breakthrough as anything we have". Parnas said

it is quite astonishing to see how often commercial programs are produced without adequate reviews. There are many reasons for this:

• Many programmers have no professional training in software at all. Some are engineers from other fields; some are "fallen scientists" who learned programming incidentally while getting education... In many of those areas, the concept of preparing and holding a design review is nonexistent.
• Even among those that have Computer Science degrees many have had an education that neglected such professional concerns as the need for design documentation and reviews....
• Many practitioners (and many researchers) do not know how to provide readable precise documentation of a design, as distinct from an implementation. No precise description, other than the detailed code, is available for review. Design reviews early in a project, when they would do the most good, are often reduced to chat sessions....
• Much of software is often produced as a cottage industry, where there are no people who could serve as a qualified reviewers ....
• Software is often produced under time pressure that misleads the designers into thinking that they have no time for proper reviews.
• Many programmers regard programming as an "art" and resent the idea that anyone could or should review the work that they have done.

I also share Parnas' concern that "it is common to hear a programmer saying that the code is its own documentation; even highly respected language researchers take this position, arguing that if you use their language, the structure will be explicit and obvious"

My argument with proponent of this view is that code does not document a design; it merely documents an implementation of a design decision. There are plenty of materials a designer has discarded that are vitally important to the maintenance of the implementation but are not and cannot be translated to code. For example, technologies or algorithms that a designer has considered by discarded and for what reasons are not found in code and cannot be for practical reasons. Design decisions why it is done this way and not another are not found in code; in fact, lack of supplemental information that we called documentation one could be misled from reading code. They cannot be deduced from code no matter what language one uses or how clever one uses the language construct to aid documentation.

Parnas went on to say "Even if we take all reasonable preventive measures, and do so religiously, aging is inevitable." and in that paper he offers many practical advices to deal with software aging, including "start taking documentation more seriously".

---

Daniel M. Hoffman and David M. Weisss, "Software Fundmentals - Collected Papers by David L. Parnas" 2001. Chapter 29 "Software Aging"

Robert L. Glass, "Facts and Fallacies of Software Engineering", page 104

Commercial software subjected to public scrutiny

Bruce Schneier reports a court case in which the accuracy or quality of the embedded software in an alcohol breathalyzer is being questioned in a court case and the court decides to subject the software in question to an independent proper assessment, i.e. a proper code review, despite protests naturally from the vendor.

For people interested in seeing a professionally compiled thorough code review can read the full report and a separate deeper analysis report. Both reports are very educational, particularly the second one.

When I am digging through this report, which I do not recommend people to print out the full 57 pages, it reminds me of the similarity in the quality (or lack of) in an Australian-made ERP software. To be honest, the ERP software is far worse by a large margin.

For example, it is not uncommon in the Australian-made ERP software to have far higher density of code exceeding the CC of 105, admittedly, though inexcusably, the ERP software has more LOC. I have personally seen CC higher than 150-200 after discounting the accuracy of the metric measuring tool.

Another issue identified by this review that is wide spread in this Australian-made ERP software is the frivolous and gratuitous use of global variables; this was caused by lack of training in good software practices, lack of reviews and aided by the development tool. In fact so wide spread that it was used as an excuse to avoid an architectural design correction exercise. In the company's view, it is much cheaper to let their poor users to foot the bill of their frivolous resources wastage, the result of its architectural flaws and ignorance, by paying much higher hardware costs.

Public review of commercial and proprietary software should be a normal public quality assurance process to safe-guard the welling being of the software consumers; it is similar to safety rating of cars or electrical appliances. Such public review would have unearthed the glaring mistakes committed by this Australian-made ERP system allowing the users to seek compensation.

Currently, the playing field is severely tilled towards the producers, as reported, allowing them to discharge all responsibilities and to take all rewards.

Companies have often invoked the commercial-in-confidence or proprietary IP excuses to escape such scrutiny. But this report stated in court that

Base One found that the code consists mostly of general algorithms arranged in a manner to implement the breath testing sequence. "That is, the code is not really unique or proprietary."

I doubt there are too many genuinely proprietary IP stuff in many today's commercial software. The more well established they are, the less they are and many are just a quilt of widely publicised algorithms poorly implemented to meet the so-called commercial dead lines.

In my opinion as a developers with over 20 years of experience, I believe those hiding being the veil of proprietary secrecy are too afraid to be caught using unsafe practices, not using industry best practices and ignorance of their mistakes in their own coding; the Alcotest did not even realise that they have committed buffer overruns and including others surprises in the published findings.

The review report attempts to nail down the reasons for the high CC has this to say:

source code appears to have evolved over numerous transitions and versioning, which is responsible for cyclomatic complexity.

While this is a likely cause, and is in agreement with my experience in the Australian-made ERP software, it only indicates both companies have a very poor software maintenance process. Most likely it is based on code-and-fix with little regards to refactoring during a bug fix phase.

The review findings and my own personal experience in this industry indicate that bad software practices are more universal with no geographical boundaries.