Product SW quality (part 2): The quality factor Reliability

Dear reader,

Today I will drill down more into the details of the quality factor Reliability.

The following three measures are of importance:

• Mean time between failure (MTBF),


• S-curve (accumulated number of found defects over time)


• SW complexity metrics (i.e. Cyclomatic complexity).

MTBF:
For this measure I think you should included “everything” that is a failure. Not just a crash or restart of the SW, but also when the product hangs in some way. It can also be very good to include a sub-measure like mean time to fist failure (MTTFF). Since MTTFF impact very much on the customer perceived reliability of your product.

S-curve:
The S-curve is mostly used to understand the quality maturity (or reliability status) of your product. The accumulated number of found defects over time is a measure that I think tells you the most when it is 100% connected with the correctness quality metrics like number of open defects, pass rate and test coverage. You also need testing done consistently and defect needs to be registered consistently in your ALM tools otherwise you might be fooled.

SW complexity metrics (i.e. Cyclomatic complexity)
This measure is not very often used but I believe many organization would be able to increase the SW reliability a lot if used the right way.
Cyclomatic complexity will tell you how error prone a function, module, method or classes is. If you would like to know more details look at this Wikipedia link: http://en.wikipedia.org/wiki/Cyclomatic_complexity.
The important part here is to make sure that your cyclomatic complexity is lower than 10 for all modules within your product. Otherwise the risk of faults increases a lot and your SW reliability is at risk. A number of studies have investigated cyclomatic complexity's correlation to the number of defects contained in a module. Most such studies find a strong correlation between cyclomatic complexity and defects: modules that have the highest complexity tend to also contain the most defects. For example, a 2008 study by metric-monitoring software supplier Enerjy analyzed classes of open-source Java applications and divided them into two sets based on how commonly faults were found in them. They found strong correlation between cyclomatic complexity and their faultiness, with classes with a combined complexity of 11 having a probability of being fault-prone of just 0.28, rising to 0.98 for classes with a complexity of 74 (Rich Sharpe. "McCabe Cyclomatic Complexity: the proof in the pudding". Enerjy.).

As mentioned before there are a lot of automatic tools available and I am looking forward to TFS 2010 that will also have this measure included.

Happy Ester!
Anders

Product SW quality (part 1)

Dear reader,

Nice to be back after a relaxing vacation. In this first post about product SW quality I want to focus on number 2 from my post on the 22nd of February. Lack of quality control and good quality code measures (software Metrics). Most companies measure SW quality in some form, for example number of open defects often connected to a ranking system. This is one simple way of measuring product quality, even if it standalone will not tell you very much. To be able to get a full overview of your products quality you need more than just one measure, you need to find a number of measures that give you the full picture of your product SW quality. However, first I want to elaborate some on what I mean with SW quality. There are of course a lot of things that can fit into this term. My number one focus is on product SW quality and the main quality factors for me are reliability, usability and correctness. Of course you also have many other factors like maintainability, reusability, portability etc but it is not as important as the other three in succeeding with a good customer "perceived" quality level (looking from the out side an in).

Below you can see the metrics I think you need to use on a top level:

• Reliability: Mean time between failure (MTBF), the S-curve (accumulated number of found defects over time) and SW complexity metrics (i.e. Cyclomatic complexity).


• Correctness: Open defects rank of A, B and C, % pass rate for test cases (all test areas: i.e. function test, integration test etc), test coverage, performance metrics.


• Usability: Performance metrics (latency), effectiveness, efficiency, user satisfaction (or perceived quality).

If you make sure you get reliable data on the above quality factors you are getting very good input into how to improve you product quality. More importantly you will also have the right information on your product quality status to be able to make the often hard decision on when to release you product to real customers.

Even the most mature companies I worked at best case use most of the correctness quality factor and MTBF from the reliability quality factor. This often lead to products that are released too early (i.e. with too low quality), since missing the S-curve and a good complexity metrics you miss 2 out of 3 metrics for reliability. The SW complexity metrics has been around for over 20 years, but I have yet see any company using them, even if there has been automated tools available since at least 15 years back. I myself did my master thesis at Volvo Cars in 1995 on SW complexity metrics (I do not know if they ever went ahead and implemented my suggestions :-). However in TFS 2010 you will find a automated tool for measuring Cyclomatic complexity, maybe that will make it more frequently used?

That’s it for today. In coming posts I will drill down to details for each quality factor.

Kind regards
Anders