Test cases, can’t do ‘m no more

Continuing the style of my previous blog post…

Some days ago I found myself no longer able to think in test cases while testing. Of course, it’s not as if I was using test design techniques to generate test cases one day and woke up the next day to find myself unable to do it anymore. But still, about a week ago I figured I had explored enough to be able to write down the test cases I wanted to execute and found myself staring at a blank page (well ok, empty Excel sheet) feeling alienated from what I was planning to do.

So what do I mean when saying “thinking in test cases”. Simply put, you take a piece of functionality, let a test design technique loose on it and there you go: a set of test cases to execute. Combine test design techniques over the different pieces of functionality as required and you’re all covered test strategy-wise. Or that’s the idea.
The problem with this approach is that it is based on reductive and deductive reasoning. It believes that we can transform a description of some piece of software to a set of actions and checks – with nothing important getting lost in that transformation. How is that ever supposed to work? Systems thinking anyone?

Yet if not test cases, than what? You model, you explore and you investigate. You don’t think in test cases; you generate test ideas and work with those. You approach the application as the complex system that it is, with all the different interactions that entails. And yes, during this process you will be using test design techniques. The difference is that they will not give you any guarantee of coverage except in a very trivial way, i.e. that you got the desired coverage for the very specific thing you were testing. That is all.
To answer the question if you tested all the important parts of the application, you do not need test design techniques, you need models. More than one, some may overlap and some may be somewhat contradictory. That’s ok. If testing weren’t such a messy business, it wouldn’t be that much fun.

3 thoughts on “Test cases, can’t do ‘m no more

  1. Interesting Read. I too believe in using Test Design Techniques to get a very visual (with some techinques) of the coverage. I have also blogged about these in my blog.

    [Joep’s reply: Thank you! I took a look at your blog and like how you changed the focus of your test cases from ‘document everything’ to ‘communicate what’s important’.]

  2. Joep,

    Interesting read and I sympathise with where you’re coming from (despite the fact that I’m a test case generating tool creator).

    As you’re suggesting, tests that you create BEFORE you really start interacting with a SUT are often less rich, less interesting, and less effective than tests that you think of “in the moment” when you’re interacting with the SUT and learning about potential problem areas within it.

    Along those lines, one area that I’m exploring with some clients is the idea of exploratory combinatorial testing. The goal of this approach is to free the tester from the confines of executing overly-prescriptive test scripts (which I am not defending) while taking advantage of the following benefits of pair-wise and/or other combinatorial test design techniques. Maximum variation, minimum repetition, and maximum coverage efficiency. The verdict of whether this approach actually works in practice, is, to be honest, still out. I’m optimistic that it will be helpful in certain contexts though if the testers using it understand it for what it is and use the approach to generate possible “test ideas” to guide their Exploratory Testing as opposed to a “silver bullet” solution. See, e.g., http://cast2013.sched.org/event/c5034344045613916f53d3359255d451#.Ud-FBT6XLmY

    Also, FWIW, here are some additional thoughts relevant to “If we don’t use traditional software tests, then what should we use?” http://www.slideshare.net/JustinHunter/documenting-software-testing-instructions-a-survey-of-successful-approaches

    [Joep’s reply: Hello Justin, my apologies for taking quite some time to reply to your comment.

    I really like the idea of exploratory combinatorial testing, as I do regularly find it a challenge to keep track of what combinations of input (or data) I did cover during exploratory testing and which ones I did not. I then end up messing around in Excel with counting formulas and conditional formatting, which works, but is far from ideal. Hopefully your explorations yield good results and I get a change to reap some of the benefits.:-) ]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s