the testing curve

my learning curve in software testing

Why the testing/checking debate is so messy – a fruit salad analogy

Five days ago James Thomas posted the following in the Software Testing & Quality Assurance group on LinkedIn:

Are Testing and Checking different or not?
This article by Paul Gerrard explains why we shouldn’t be trying to draw a distinction between checking and testing, but should be paying more attention to the skills of the testers we employ to do the job.

I posted a reply there, but I think I can do better than those initial thoughts, so here we go.

Let’s imagine the following scene: Alice and Bob are preparing a fruit salad together.
Alice: “Ok, let’s make a nice fruit salad. We need some apples and some fruit.”
Bob: “Euh, aren’t apples fruit?”
Alice: “Yes. Of course. But when I say ‘fruit’, I mean ‘non-apple fruit’.”
Bob: “So you don’t think that an apple is fruit?”
Alice: “No, I do. It’s just when I say ‘fruit’, I want to focus on the non-apple fruit.”
Bob: “Uhuh. So fruit is stuff like bananas, pears and pomegranate?”
Alice: “Exactly. And that would actually make a great fruit salad: apple and those three fruits.”
Bob: “Ok, but what if I feel like having a fruit salad. And it turns out that I only have apples and bananas at home and I don’t have time to go to the store. And, importantly, I really really don’t like bananas. So I decide to only use apple. That’s still a fruit salad, right?”
Alice: “I suppose so, technically, but still… a fruit salad without any non-apple fruit… I mean, everyone puts apples in their fruit salad and there’s so much more fruit than just apples! So when I say ‘fruit’, I just really want to focus on the non-apple fruit, ok?”
Bob: “Ok, fine. Glad we cleared that up. One more question though: what about tomatoes?”
Alice: “Don’t. Just don’t.”

Now read this piece of dialogue again and replace ‘apple’ with ‘checking’ and ‘fruit’ with ‘testing’. Bob’s confusion is exactly the reason why the whole testing/checking debate is messy: most of the time it’s about testing *versus* checking. You can see it in the title of the LinkedIn post: “Are testing and checking different or not?” You can see it in Paul Gerrard’s article: “[…] the James Bach & Michael Bolton demarcation of or distinction between ‘testing versus checking’.” You can see it in Cem Kaner’s article: “According to the new doctrine, “checking” is the opposite of “testing” and therefore, automated tests that check against expectations are not only not sapient, they are not tests.” You can also see it in the original “Testing vs. Checking” blog post by Michael Bolton dated August 2009. It’s right there in the title. Do take note however, that this post has been retired and we are directed to the new version “Testing and Checking Refined“. However, the new version still contains a sub-title “Checking vs. Testing”.

“Testing and Checking Refined” also contains a helpful diagram, that’s key to the point I want to make in this post. The diagram shows us that there’s one overarching category ‘testing’ (fruit), which contains two things: ‘checking’ (apples) and all other testing activities (non-apple fruit). This helps us to understand two things.

First of all, it shows that any discussion about testing *versus* checking is bullshit. They are on a different conceptual level, just like fruit and apples, so any direct comparison is meaningless. To throw in one more analogy, what would you answer when you were asked while visiting a friend at his home: “Would you like coffee, or something to drink?”

Secondly, it explains why my previous point is difficult to get(1). The diagram presents people with two concepts: ‘testing’ and ‘checking’. Of course there’s also “learning by experimenting, including study, questioning, modeling, observation, inference, etc.”, but that’s just too vague to register mentally as an entity. It does not coalesce into a concept. What we’re left with are only two concepts, ‘testing’ and ‘checking’, and the non-checking part of testing is gone. This is actually illustrated by the title of James Bach’s and Michael Bolton’s blog post: “Testing and Checking Refined”.

So when you present two concepts in this way, is it really that surprising that people talk about them likes apples and oranges, instead of like apples and fruit? I think not.

— — —

(1) Including for myself. See for instance how I’m struggling with this very problem in my blog post from August: “What’s the word for the part of testing that’s not checking?

11 responses to “Why the testing/checking debate is so messy – a fruit salad analogy

  1. ObServant Tester November 15, 2015 at 1:30 pm

    Hi Joep,
    I agree that the crux is that the same word (testing) is used at two conceptual levels for two different meanings. One is a container, the other describes a ‘thing’ for which definitions vary which doesn’t make it any easier.
    I see two levels of discussions though – one in the testing community, the other in your day job. What I use in the latter is up to me and if it helps in my context to have checking and testing on the same conceptual level of course I’ll use it. Everyone can decide if they take a pragmatists approach (if it works it must be the right thing to do) or rather figure out the correct meaning in each word – both approaches have merit.

    [Joep’s reply: Excellent point. The intention behind my blog post is to make sense of the confusion myself and others have encountered with “checking versus testing”. It is not my intention to tell people what the correct meaning of the words is and if they can please get with the program. Looking back at my post, however, I see how it may give the impression that *is* what I am doing, so thank you for bringing this up.]

  2. Michael Bolton November 15, 2015 at 8:31 pm

    When I first wrote about “Testing vs. Checking”, there was a bug. The bug was in my not recognizing that a distinction (like “trees vs. leaves”, “biting vs. eating”, “bathrooms vs. houses”) would be seen as inherently opposition (“Manchester United vs. Liverpool”, “Frazier vs. Ali”). I was startled by the fact some people who touted flexible thinking and multiple interpretations couldn’t appear to handle more than one interpretation of “versus”. James Bach was the first, but he got over it in a hurry, and joined me in making the distinction. Others did tool What seemed to happen after that is that some people heard about what we were saying, but were not actually reading what we were saying. That seems to persist until this day. Perhaps one day I should do a post on “hearing about vs. understanding” to emphasize the point.

    We don’t say “versus” any more; we talk about testing and checking. But it appears that once a bug is in the field, it takes a long time for people to apply the patch, so it seems. And yes, in that post, there is a “vs.”—but it seems to me that anyone who reads beyond that heading (“Testing encompasses checking, whereas checking cannot encompass testing.”) would understand the role of “versus” there.

    I must say that it blows my mind that there is controversy about the basic idea. There is such a thing as compiling, the part of programming that can be entirely automated. No one gets upset at the distinction between programming and compiling; and no one takes seriously anyone who talks of compiling as “automated programming”. Everyone allows that it takes an impressive amount of skill to develop software that performs compiling that does its job, just as everyone should allow that it takes an impressive amount of skill to develop software that performs checking. Everyone believes that the process of compiling needs to be tested, just as everyone should believe that process of checking needs to be tested. No one that I’m aware of believes that there should be a single word for “the part of programming that isn’t compiling”, since there is not single concept to represent the part of programming that isn’t compiling. I don’t think the “vs.” in describing “programming vs. compiling” would be controversial, since people would immediately see the trees-vs.-leaves sense of that heading. But again, once a bug is out there…

    Finally, let’s look at this: “we shouldn’t be trying to draw a distinction between checking and testing, but should be paying more attention to the skills of the testers we employ to do the job.” To me, that’s exactly like saying “we shouldn’t be trying to draw a distinction between drawing and using a printer, but should be paying more attention to the skills of artists we employ to do the job.” We make these distinctions precisely so we can have clearer discussions about skills and about how to help people develop them. That which can be done by a machine (checking, printing) is by definition skill-free. How we want to use the machinery skillfully to extend what we do (testing, art) requires us not to be confused about what we’re talking about.

    [Joep’s reply: Thank you, Michael, for taking the time to write this comment. I hope it helps in getting rid of the bug you mention.]

    • Shane Dennis November 15, 2015 at 11:35 pm

      The analogy of the compiler bugged me for a long time, and it wasn’t until I read Michael’s last two sentences that I couldn’t quite put my finger on what it was.

      That which is done by a compiler is automatic. In most cases it literally is “press a button”. But the same cannot be said for automated checks in testing. The commencement of the automated check may be the press of a button, but the evaluation of checks that can be converted, the coding of the automation, and the pre- and post-review of check and result are most definitely not “skill-free”.

      Michael does state above that the “process of checking needs to be tested”, and I assume that by this he means that all of the peripheral (human) parts of the check that I’ve mentioned above are indeed testing and not checking?

      [Joep’s reply: I cannot speak for Michael, but I think you are correct. All those peripheral aspects of a check that you mention, are testing. Which is also the reason I think it’s incorrect when someone is characterised as “merely doing checking”. There may be little and very bad testing going on next to the checking, but I don’t see how a human can do checking without engaging in absolutely any testing at all.]

      • Michael Bolton November 16, 2015 at 4:37 pm

        Hi, Shane…

        In http://www.satisfice.com/blog/archives/856, we say

        “Checking is a process that can, in principle be performed by a tool instead of a human, whereas testing can only be supported by tools. Nevertheless, tools can be used for much more than checking. We are not saying that a check MUST be automated. But the defining feature of a check is that it can be COMPLETELY automated, whereas testing is intrinsically a human activity.”

        Compiling is a process that can, in principle, be performed by a tool instead of a human. The defining feature of compiling is that it can be completely automated. So it is for checking.

        You say “That which is done by a compiler is automatic. In most cases it literally is ‘press a button’. But the same cannot be said for automated checks in testing.” We’re saying that exactly the same can be said for automated checks in testing, since that’s the essence of what automation is: delegating some portion of a task to a machine, while people do the things that only people can do. The programming that precedes the compilation is not mechanizable; the risk assessment and question framing and encoding that precedes the check is not mechanizable. Evaluating the outcome of compilation—a compiled program—for its value and threats to its value is similarly not mechanizable; evaluation of the outcome of the checks—a passel of check results—for their information value and threats to their information value is similarly not mechanizable. Also, creating a compiler is not a mechanizable process; creating checks is not a mechanizable process.

        You say “The commencement of the automated check may be the press of a button, but the evaluation of checks that can be converted, the coding of the automation, and the pre- and post-review of check and result are most definitely not ‘skill-free’.” We’re saying exactly that, and we’ve been saying exactly that for about six years now: http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/

        “Michael does state above that the “process of checking needs to be tested”, and I assume that by this he means that all of the peripheral (human) parts of the check that I’ve mentioned above are indeed testing and not checking?”

        YES. And if you’re aware of anything in our actual, written work that even hints at suggesting otherwise, I’d be delighted to know about it.

        —Michael B.

  3. jlottosen November 15, 2015 at 10:30 pm

    A Wise man knows that a tomato is a fruit But a smart man knows to leave it out of his fruit salad

  4. Pingback: Why the dichotomy of testing versus checking is the core of our craft | Anders Dinsen on software testing and development

  5. John Stevenson November 16, 2015 at 8:50 am

    I do like your article Joep but there is a basic flaw in your argument that jlottosen alludes to in the use of fruit. The definition of what is a fruit is quite difficult to fully define unless you use the botanical definition of a fruit.

    “Botanically speaking, a fruit is a seed-bearing structure that develops from the ovary of a flowering plant, whereas vegetables are all other plant parts, such as roots, leaves and stems”

    This has lead to strange issues such as the tomato by botanical definition being a fruit but for import tax purposes in the USA it is classed as vegetable.

    Add into that mix the bell-pepper, cucumber and eggplant. Therefore I could make a botanical fruit salad containing no apples but only bell peppers, egg plant and tomatoes how strange would it be to be served that fruit salad after your main meal in a restaurant.

    Do not get me started on the ‘what is a berry debate’ where a strawberry is not a berry and a banana is a berry!!!

    So to the point of your article I do feel there is still an important need to know the distinction between testing and checking. Checking is a sub set of testing and still has value in some contexts and using that as a discussion point I find useful.

    Maybe there is a need to frame testing and checking in the same context as the statements that are made in the Agile manifesto.

    “We prefer testing and providing information over checking and reporting pass/fail.”

    That is, while there is value in the items on the right, we value the items on the left more.

    This would then remove the ‘bug’ as Michael frames it in his comment.

    [Joep’s reply: The fruit analogy certainly does have its limits. Anders Dinsen also commented on this in his blog post: apples are a sort of fruit, checking is not a sort of testing.

    I wholeheartedly agree that there’s value in distinguishing checking as a sub set of testing. It’s a shame that’s often not how the distinction is being made.

    Thinking about your suggestion to phrase the distinction more like the Agile manifesto, I wonder if it would be helpful to create several such statements to illustrate the difference between testing and checking:
    We prefer providing information over reporting pass/fail.
    We prefer validating against different models over verifying against the requirements (document).
    etc.]

  6. Michael Bolton November 16, 2015 at 4:41 pm

    “Anders Dinsen also commented on this in his blog post: apples are a sort of fruit, checking is not a sort of testing.”

    I’m afraid that in saying that, Anders is misinterpreting our work.

    I wholeheartedly agree that there’s value in distinguishing checking as a sub set of testing. It’s a shame that’s often not how the distinction is being made.

    I don’t have a whole lot of control over people who are misrepresenting our work. I’d be happy to hear about it though, so we can at least try to put a stop to it.

    —Michael B.

  7. Pingback: Five Blogs – 17 November 2015 | 5blogs

  8. Pingback: The Testing & Checking Discussion Lives On! | bmod

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s