Thursday 1 August 2013

Metric Rhetoric

I meet very few testers who argue that there is value in metrics like number of test cases executed as a measure of progress. The rhetoric for metrics such as these seems to have changed from defiant oblivion to a disillusioned acceptance; "I know it's wrong, but I don't want to fight".

The pervasive view is that, no matter what, project management will insist upon metrics in test reporting. Testers look on in awe at reporting in other areas of the project, where conversations convey valuable information. We are right to feel sad when the reporting sought is a numeric summary.

But they want numbers...

Do they? Do your project management really want numbers? I hear often that testers have to report metrics because that is what is demanded by management; that this is how things are in the real world. I don't think this is true at all. Managers want the numbers because that's the only way they know to ask "How is testing going?". If you were to answer that question, as frequently as possible, making testing a transparent activity, much of the demand for numbers would disappear.

"Even if I convince my manager, they still need the numbers for reporting further up the chain". If your manager has a clear picture of where testing is at, then they can summarise this for an executive audience without resorting to numbers. Managers are capable communicators.

Meh. It's not doing any harm.

Everytime that you give a manager a number that you know could be misleading or misinterpreted you are part of the problem. You may argue that the manager never reads the numbers, that they always ask "How's it going?" as you hand them the report, that they'll copy and paste those digits to another set of people who don't really read the numbers either, so where's the harm in that? Your actions are the reason that I am constantly re-educating people about what test reporting looks like.

Snap out of it!

We need to move past disillusioned acceptance to the final stage and become determined activists. A tester is hired to deliver a service as a professional. We should retain ownership of how our service is reported. If your test reporting troubles your conscience, or if you don't believe those metrics are telling an accurate story, then change what you are doing. When asked to provide numbers, offer an alternative. You may be surprised at how little resistance you encounter.


  1. Hi Katarina

    Thanks for an interesting article. I am interested to know how you present your testing results to your project leader?

    In the current project that i am working i am working in a agile team where after each iteration we present a test report where the tester/s in the team describe the confidence of the delivery, any know issues or dependencies that could affect other teams and mitigation to any known risks/issues that may show up. We do not present any metrics on how many test cases we have run etc.

  2. Two nice quotes on this thought:

    "Managers who don’t know how to measure what they want settle for wanting what they can measure" - Ackoff

    "the most important figures that one needs for management are unknown or unknowable" - Lloyd S. Nelson (Deming quoted him in Out of the Crisis, many people then attribute the quote to Deming).

  3. @Superbosanac

    I also work in an agile team, I'm not sure why you'd be doing your test reporting after an iteration? That doesn't sound particularly integrated? I like to keep my testing transparent right through our SCRUM cycles and like to think that everyone is aware of where testing is at.

    I intend to blog more about the reporting I use, but it basically nuts down to 3 points:

    1. Automated checks that can be understood by the business stakeholders.

    2. Visible test coverage in the form of mind maps.

    3. Constant communication.

  4. Hi Katrina,
    Your articles are really interesting.I am really impressed Please keep writing...