the testing curve

my learning curve in software testing

Why your Product Risk Analysis isn’t

Ok, going to try to keep this short and ranty (rantish?).

Typical test advice is to do a Product Risk Analysis (PRA, mind the capitalisation!) and based on that you decide what to test and how thoroughly. The most common way to do a PRA is with a workshop. Put some people in a room with a lot of stickies, let them list all the risks they can think of and then have them score them. Et voilà, Product Risk Analysis is done!

But that doesn’t really make sense, now does it? If someone were to give you an object and asked you “What could possibly go wrong with this?”, what would you do? Gather a bunch of people with some knowledge of the object, yet no actual experience with it and do a workshop imagining things that could go wrong? That’s not an Analysis (capital A!), that’s a SWAG – a scientific wild ass guess.

Yet that’s exactly what we do in software testing. After the workshop the PRA is done. It gets (at least!) version 1.0. But if the PRA is done, that is if we have a good analysis of the product risks, why do we need to test? Well of course, we didn’t do a proper analysis, we did a SWAG and we want and need to do better.
It is through testing (and checking), through interacting and experimenting with the product, we can enhance our PRA from “best we can think of” to “this is what we observed”. By testing we upgrade our risks from SWAG-status to three pieces of information: what works, what doesn’t work and what hasn’t been tested yet. Now that information is a lot more useful than what we got from our so-called PRA: “We have a moderate amount of fear it will fail and that would be fairly bad.”

Now quite often the list of stuff that doesn’t work is a bit too long, so we have a fixing phase. And that’s cool. We just keep testing, until the person who gets to make the call, is happy with the three stories in our PRA (works, doesn’t work, don’t know). And that’s when our Product Risk Analysis really is done, ready and finished: during our final test report.

3 responses to “Why your Product Risk Analysis isn’t

  1. Jackie June 24, 2013 at 3:05 pm

    Like! :-)

  2. Ard Kramer June 28, 2013 at 10:09 am

    I like to do discuss with you about your definition of SWAG in terms of risk management. Because everything about risk management is SWAG (because all kinds of business like things are SWAG’s e.g. MBA’s are also SWAG’s). So to make a point about risk managent and SWAG’s is a very easy one.
    and of course we (as people) were able to define (and sell) a process around risk management. Nothing strange about human behaviour;-)
    Besides that I disagree what the product risk analysis is done at our final test report. A good risk analysis is part of system documentation which means it doesn’t end at the final test report. It lives on, and must used and update at every change of the system and then you could make a point about risk management: it is continues awareness during the complete life cycle of a product

    [Joep’s reply: The reason I used the word ‘SWAG’ was to stress the difference between what people think the status of the PRA is at the beginning of the project (i.e. “Done!”) and what the status actually is: our best guess at that moment in time.
    But I do see your point that a PRA will never get beyond SWAG status. In some cases I have a similar response to people saying “You’re making an assumption.” Making assumptions is all I do, because that’s what knowledge is.

    On your second point I can only agree. I wrote my post from a single project perspective. Yet when there is a subsequent project (or change), it would be quite stupid no to use the information produced by the previous project.]

  3. Ard July 17, 2013 at 4:50 pm

    I didn’t see your reply upto now Joep, sorry but it all comes to make your own heuristics isn’t it ;-)

    [Joep’s reply: doesn’t it always? ;-) ]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s