Product SW quality (part 2): The quality factor Correctness

Dear reader,

Quality factor correctness is maybe the most common measure for SW quality, and it might be kicking in open doors writing about them? However as Henrik quite correctly pointed out in his comment on my first post about Product SW quality, “correctness is measured at a specific point in time”. This fact makes these quality metrics less trustworthy the more changes you make to the code without performing the right regression testing. Therefore I use them for making release decisions on the maturity of the product together with the other quality factors at a specific point in time. Normally it should be measured against a preset targets for beta SW or final SW. Only looking at one or two of the quality factors correctness can give you a false sense of good quality. For example: You might have low number of open defects and a good pass rate. Without knowing the test converge and the S-curve you will still be in the dark.

Here are some details that I think of when using the correctness quality metrics.

Open defects rank of A, B and C :
This metric is very straight forward, nevertheless the truly hard part is to set the targets that is right for your product. It can be very different for a mobile game compared to a heart-lung machine. The only way I have found that works so far is actually to learn the right limit as you go ahead (maybe not the best way if you do heart-lung machines :-) and releases your product and get customer feedback (if possible before market launch using beta testers). I.e. is zero A-ranks something your customers expect or is 10 A-rank okay? If you have any better way of doing the target setting please let me know? Also what is important is that you use a good ranking system and that everyone gets trained and good at using the ranking system when reporting defects. In addition to this it is very important that everyone buy-in on always adding all defect that is found to your defect tracking system to get traceability. Without 99,9% buy-in form the organization you are in trouble.

% pass rate for test cases:
This metric is also quite straight forward. You just measure the passed test case divided by the total amount of available test cases. This should be measure for at least unit tests, integration testing and system testing.

% test coverage:
Again this metric is not rocket science .You measure % test converge of all the available test cases you have, to validate that the % pass rate tells you the true story. This should be done for all test cases. The targets in an optimal world is of course 100% if you have no blocked areas. For for example unit test it is also good to try to accomplish as close to 100% code converge as possible for the metric to be even better.

Performance metrics:
It is of great help for the development organization to receive targets for none functional requirements like performance metrics from product management in the product specification, before the development starts. Then it is much easier to spend time on estimating the needed work to archive the targets. One example could be the start up time for the product. Without targets you often end-up arguing before launch what is good or bad. If product management do a good job they benchmark with competitors or set good metric targets based on customer expectations.

Thank you for reading and please give some feedback?

Anders

Product SW quality (part 2): The quality factor Reliability

Dear reader,

Today I will drill down more into the details of the quality factor Reliability.

The following three measures are of importance:

• Mean time between failure (MTBF),


• S-curve (accumulated number of found defects over time)


• SW complexity metrics (i.e. Cyclomatic complexity).

MTBF:
For this measure I think you should included “everything” that is a failure. Not just a crash or restart of the SW, but also when the product hangs in some way. It can also be very good to include a sub-measure like mean time to fist failure (MTTFF). Since MTTFF impact very much on the customer perceived reliability of your product.

S-curve:
The S-curve is mostly used to understand the quality maturity (or reliability status) of your product. The accumulated number of found defects over time is a measure that I think tells you the most when it is 100% connected with the correctness quality metrics like number of open defects, pass rate and test coverage. You also need testing done consistently and defect needs to be registered consistently in your ALM tools otherwise you might be fooled.

SW complexity metrics (i.e. Cyclomatic complexity)
This measure is not very often used but I believe many organization would be able to increase the SW reliability a lot if used the right way.
Cyclomatic complexity will tell you how error prone a function, module, method or classes is. If you would like to know more details look at this Wikipedia link: http://en.wikipedia.org/wiki/Cyclomatic_complexity.
The important part here is to make sure that your cyclomatic complexity is lower than 10 for all modules within your product. Otherwise the risk of faults increases a lot and your SW reliability is at risk. A number of studies have investigated cyclomatic complexity's correlation to the number of defects contained in a module. Most such studies find a strong correlation between cyclomatic complexity and defects: modules that have the highest complexity tend to also contain the most defects. For example, a 2008 study by metric-monitoring software supplier Enerjy analyzed classes of open-source Java applications and divided them into two sets based on how commonly faults were found in them. They found strong correlation between cyclomatic complexity and their faultiness, with classes with a combined complexity of 11 having a probability of being fault-prone of just 0.28, rising to 0.98 for classes with a complexity of 74 (Rich Sharpe. "McCabe Cyclomatic Complexity: the proof in the pudding". Enerjy.).

As mentioned before there are a lot of automatic tools available and I am looking forward to TFS 2010 that will also have this measure included.

Happy Ester!
Anders

Product SW quality (part 1)

Dear reader,

Nice to be back after a relaxing vacation. In this first post about product SW quality I want to focus on number 2 from my post on the 22nd of February. Lack of quality control and good quality code measures (software Metrics). Most companies measure SW quality in some form, for example number of open defects often connected to a ranking system. This is one simple way of measuring product quality, even if it standalone will not tell you very much. To be able to get a full overview of your products quality you need more than just one measure, you need to find a number of measures that give you the full picture of your product SW quality. However, first I want to elaborate some on what I mean with SW quality. There are of course a lot of things that can fit into this term. My number one focus is on product SW quality and the main quality factors for me are reliability, usability and correctness. Of course you also have many other factors like maintainability, reusability, portability etc but it is not as important as the other three in succeeding with a good customer "perceived" quality level (looking from the out side an in).

Below you can see the metrics I think you need to use on a top level:

• Reliability: Mean time between failure (MTBF), the S-curve (accumulated number of found defects over time) and SW complexity metrics (i.e. Cyclomatic complexity).


• Correctness: Open defects rank of A, B and C, % pass rate for test cases (all test areas: i.e. function test, integration test etc), test coverage, performance metrics.


• Usability: Performance metrics (latency), effectiveness, efficiency, user satisfaction (or perceived quality).

If you make sure you get reliable data on the above quality factors you are getting very good input into how to improve you product quality. More importantly you will also have the right information on your product quality status to be able to make the often hard decision on when to release you product to real customers.

Even the most mature companies I worked at best case use most of the correctness quality factor and MTBF from the reliability quality factor. This often lead to products that are released too early (i.e. with too low quality), since missing the S-curve and a good complexity metrics you miss 2 out of 3 metrics for reliability. The SW complexity metrics has been around for over 20 years, but I have yet see any company using them, even if there has been automated tools available since at least 15 years back. I myself did my master thesis at Volvo Cars in 1995 on SW complexity metrics (I do not know if they ever went ahead and implemented my suggestions :-). However in TFS 2010 you will find a automated tool for measuring Cyclomatic complexity, maybe that will make it more frequently used?

That’s it for today. In coming posts I will drill down to details for each quality factor.

Kind regards
Anders