Product SW quality (part 2): The quality factor Reliability

Dear reader,

Today I will drill down more into the details of the quality factor Reliability.

The following three measures are of importance:

• Mean time between failure (MTBF),


• S-curve (accumulated number of found defects over time)


• SW complexity metrics (i.e. Cyclomatic complexity).

MTBF:
For this measure I think you should included “everything” that is a failure. Not just a crash or restart of the SW, but also when the product hangs in some way. It can also be very good to include a sub-measure like mean time to fist failure (MTTFF). Since MTTFF impact very much on the customer perceived reliability of your product.

S-curve:
The S-curve is mostly used to understand the quality maturity (or reliability status) of your product. The accumulated number of found defects over time is a measure that I think tells you the most when it is 100% connected with the correctness quality metrics like number of open defects, pass rate and test coverage. You also need testing done consistently and defect needs to be registered consistently in your ALM tools otherwise you might be fooled.

SW complexity metrics (i.e. Cyclomatic complexity)
This measure is not very often used but I believe many organization would be able to increase the SW reliability a lot if used the right way.
Cyclomatic complexity will tell you how error prone a function, module, method or classes is. If you would like to know more details look at this Wikipedia link: http://en.wikipedia.org/wiki/Cyclomatic_complexity.
The important part here is to make sure that your cyclomatic complexity is lower than 10 for all modules within your product. Otherwise the risk of faults increases a lot and your SW reliability is at risk. A number of studies have investigated cyclomatic complexity's correlation to the number of defects contained in a module. Most such studies find a strong correlation between cyclomatic complexity and defects: modules that have the highest complexity tend to also contain the most defects. For example, a 2008 study by metric-monitoring software supplier Enerjy analyzed classes of open-source Java applications and divided them into two sets based on how commonly faults were found in them. They found strong correlation between cyclomatic complexity and their faultiness, with classes with a combined complexity of 11 having a probability of being fault-prone of just 0.28, rising to 0.98 for classes with a complexity of 74 (Rich Sharpe. "McCabe Cyclomatic Complexity: the proof in the pudding". Enerjy.).

As mentioned before there are a lot of automatic tools available and I am looking forward to TFS 2010 that will also have this measure included.

Happy Ester!
Anders