the testing curve

my learning curve in software testing

The Seven Basic Principles of the Context-Driven School – part two

After the introductory post (to be found here) it’s time to take a closer look at each of the basic principles. In the past weeks I found out that it’s quite possible to take any one of these principles as a starting point for several different trains of thought. More importantly I discovered a story(1) to the principles: the first five principles are ways in which software testing is intellectually challenging, as stated by principle six. And principle seven then wraps it all up.
So below you can find some of the thoughts I had on the principles and the story I discovered.

1. The value of any practice depends on its context.
To get a better understanding of this principle I started thinking: what if the value of a practice did not depend on context? What else coud it depend on?
One possibility is that practices have an intrinsic value. If that’s the case, how can we compare two practices to see which one has the most intrinsic value? Should we look at elegance, simplicity, completeness, … (2) ? The problem is that those are not the usual criteria we judge a practice on. Since a practice is something that requires application, it does not stand on itself for us to enjoy its intrinsic value. So the value of a practice depening on intrinsic value makes little sense. Although I do think it’s something people take into account: a good method also needs a good story. It needs to be convincing.
Another possibility is that pracitices do not have any value: it doesn’t really matter what you do. Or that there is no way for us to discern the differences in value – even in hindsight. As a consequence it does not matter what you do, because you will never know if you could have done any better. If that were the case, software testing would be the easiest job in the world and the most ridiculous. Of course, often it is difficult to choose between different practices, but the choices you make do matter.
So these two alternatives are not complete nonsense, but in the end they do not (or at least should not) determine the value of a practice. The context does. But are all software testing contexts sufficiently alike so that we only need one practice? To answer this we need to look at the second princple.

2. There are good practices in context, but there are no best practices.
If there are good practices in context, there are also bad practices. Secondly, there are no best practices – ‘best’ implying context-independence here. Remains the question: are there better practices? Are some practices better than others? Outside of any context, there cannot be. Otherwise the one practice that is better than all the rest would be the best practice. In context, obviously, some practices are better than others. And to take the point even further: what is a good practice on one context, may be a bad practice in a different context. Which is kind of a weird thought if you’re used to thinking in best practices and maturity models.

Another thing I wondered is if there are ways around this principle. I could think of two. Unfortunately neither work.
Firstly, there are ‘best’ practices, i.e. best practices applied with some adaptivity. So you apply your best practices and where it doesn’t work you patch it up by getting creative. The problem with this workaround is that you will always maintain the basic philosophy of your practice. So in the details your approach may not be one-size-fits-all, the fundamental parts of your approach will be. You will approach every problem from the same perspective. This makes you context-conscious, but not context-driven.
The other workaround is creating a practice that is abstract enough that it’s context-independent. I don’t think that’s possible. Sure, it is possible to define a set of principles that is abstract enough. Just look at these seven principles. But I don’t see how one could create a practice that is abstract enough to be context-independent, yet concrete enough to have any practical value.

3. People, working together, are the most important part of any project’s context.
What does ‘working together’ mean? Are factory workers at a production line working together? Or does it depend on the degree of communication and interaction between them? On a shared goal, a shared understanding of what they are doing and of what they are trying to accomplish? I think it does. Working at the same production line or being assigned to the same project, does not mean you are ‘working together’. So the most important part of a project’s context is the ways in which the people in the project interact with each other. And remember, you are one of those people.
Also, because the value of a practice is determined by context, these interactions are what the value of your testing practice depends upon. This also means that the following parts of a project’s context are less important to the value of your practice: deliverables, processes, the type of product, laws and regulations, the project management method, the planning, etc.

4. Projects unfold over time in ways that are often not predictable.
My first reaction to this one was “Well… duh!” My second reaction was “But what do we do with this knowledge?” Which got me to my third reaction “What does ‘not predictable’ mean?”
Well, we all know that unexpected stuff happens and that’s why we add an error margin to our budget and our planning. We may not be able to predict exactly which tasks are going to eat up our error margin, but we can make a fair guess how much error margin we need. Often enough that guess is wrong, but luckily the project plan and the project itself are not two seperate entities. Changing the plan changes the project and vice-versa. As a result this kind of being not predictable is manageable to some degree.
Yet the above is not what we should think about to understand this fourth principle. What we should think about is that our model of the project might be wrong. Something that happens and does not fit into your model, is truely ‘not predictable’. Which is different from ‘hard to estimate’. (See the idea of black swans and the known unknown vs the unknown unknown, or this set of blog posts by Michael Bolton.) Now why would our model be wrong? Assuming we’re not clinging on to some best practice, in most cases I think it’s simply a case of having several models fitting the observations of the project so far. So there is just no way of deciding which of these models is the correct one. Hopefully, as the project progresses, we stay alert and will be able to discard the models that no longer fit our observations.

5. The product is a solution. If the problem isn’t solved, the product doesn’t work.
The easiest interpretation here is saying this principle is about verification (Did we build the product right?) versus validation (Did we build the right product?). As a consequence, delivering a product that meets all documented requirements on time and on budget, does not mean it’s a good product. One could argue that it was a well-managed project that produced this product, but does that really matter if the product is no good? Probably not.

A more difficult question is who gets to define the problem. One way to look at it is that everyone with access to information about the problem, defines the problem by building a mental model about the problem. However, none of the models are the definitive model. So another way to look at it is that nobody gets to define the problem. And that is (at least to me) the fun part of testing: you are being challenged to build two sets of models. One set about the problem and one set about the solution; comparing those with eachother is what we call ‘testing’.

6. Good software testing is a challenging intellectual process.
The main implication of this principle is that good software testing is not defined by following a method, going through the test process step by pre-defined step. It is defined by exploration, investigation and learning.
And as I said above, one way to look at the principles is to see the first five principles as ways in which software testing is intellectually challenging:
1. Understanding the context of a project.
2. Deciding which practice to use in a specific context.
3. Interacting succesfully with people.
4. Building and evaluating models of the project.
5. Evaluating if the product solves the problem.
Those are the five skills you need to be a good software tester.

7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.
This last principle wraps it all up, referring to the other principles: judgment and skill (principle 6), cooperatively (principle 3), right things (principle 1 and 2), right times (principle 4), to efectively test our products (principle 5). And last but not least, ‘throughout the entire project’ implies there is no testing phase.

That concludes part two. In part three I will use these principles to compare the Context-driven school with the other schools of testing.
— — —
(1) There’s a reason I say ‘story’. Stories are great ways to make sense of things, but they can also be deceptive. People like stories, but not everything allows to be ‘storified’ without losing important bits.
(2) I know, one can argue none of those are actual indicators of intrinsic value. Elegant/simple/complete to whom?

2 responses to “The Seven Basic Principles of the Context-Driven School – part two

  1. David Greenlees February 5, 2012 at 11:45 pm

    Great Joep! I’ve been looking forward to the next instalment…

    A couple of points if I may…

    “This makes you context-conscious, but not context-driven.”

    Very nice. I’ve worked in a few different environments that considered themselves context-driven and I knew that there was something missing. Yes, they considered context but I just couldn’t convince myself that they were driven by it. Context-conscious is an excellent way to put it. At least they had taken the first step. ;0)
    [Joep’s reply: Credit where credit is due: the distinction between context-driven and context-conscious comes straight from the context-driven-testing website. (It’s down at the moment, though.)]

    “The other workaround is creating a practice that is abstract enough that it’s context-independent.”

    This could be considered as a ‘Test Policy’ in many companies. A lot of effort goes into these policies. Trying to create an overarching process that allows for deviation when a particular project calls for it. This means that it’s really quite hard to write one that works. To allow for that required deviation, you really can’t go to any worthwhile level of detail and it ends up being more or a testing governance document. These sit on the shelf collected a lot of dust.
    [Joep’s reply: have you read ‘Perfect software and other illusions about testing’ by Jerry Weinberg? It has a story about someone at a job interview for a test manager position. She is shown the ‘Test Policy Manual’, runs her finger over the top and decides based on the amount of the dust on her finger, this is not a place she wants to work. :-)
    Instead of writing a test policy I think most companies should rather write down some principles and check lists to guide the test process. I mean, it’s not the amount of policy you write down that matters, but the degree to which people actually use it.]

    “Working at the same production line or being assigned to the same project, does not mean you are ‘working together’.”

    I think that depends on your view. Using the production line analogy… If I was the production line manager looking out of my second floor office at all my production line workers then I could see these guys and girls as working together. If I’m one of them, then maybe not. However, what happens if one of them takes off to the toilet and leaves their post unmanned?
    [Joep’s reply: I like that. The manager thinking the workers are actually working together and the workers themselves just doing their job on what they receive from the guy/girl before them. I think most people have seen that dynamic in at least one software project.
    If one of the workers leaves their post, the others will have to fill in. The same happens when one of the workers can’t keep up the pace. So if one worker keeps struggling with the tempo, the others may start to ‘motivate’ him to pick up the pace.]

    “One set about the problem and one set about the solution; comparing those with each other is what we call ‘testing’.”

    Yep, I like it! The certified among us may compare this to static versus dynamic testing. No, no, no… it’s much more than that. This view is maintained throughout the entire project. Constantly revisiting that initial problem is key to producing valuable information from testing.
    [Joep’s reply: Thanks for mentioning static and dynamic testing. Hadn’t thought about that connection and indeed it’s not what I was talking about.
    One thing I really miss in e.g. TMap is the realiziation that comparing ‘the actual result’ with ‘the expected result’ is not that simple on an epistemological level. Hence the ‘set of models’.]

    “Evaluating if the product solves the problem.”

    Good, but should we be better at uncovering information that allows others to make that evaluation? Or maybe we keep that evaluation to ourselves and use it to identify the valuable information? Not sure.
    [Joep’s reply: Perhaps a bit of both? The main thing in my opinion is that a tester does evaluate if the product solves the problem. Most, if not all people, do that (and should do that) in a project, but for a tester it’s one of his/her main tasks. Deciding if the product solves the problem to a sufficent degree, is something else entirely, of course.]

    Excellent post mate! Bring on part 3. :0)
    [Joep’s reply: thanks! I’ll do my best to make part 3 worth the wait. ;-)]

  2. cemkaner April 10, 2012 at 3:26 pm

    Hi, Joep:

    [Joep’s reply: Hello Cem, thank you for taking the time to comment on my blog. It’s not everyday you get feedback from one of the authors of the article you blog about.]

    With respect to #5: 5. The product is a solution. If the problem isn’t solved, the product doesn’t work.

    I think you have it backwards. This is not about verification. It is about value. Depending on what someone means by “validation”, the testing to discover whether the product “solves the problem” might be called validation.

    [Joep’s reply: Rereading what I wrote, I don’t see how I got that backwards. Perhaps I should have expanded that paragraph, because I’m not explicitly saying principle no.5 is (in my opinion) about validation, but that is what I meant to say.]

    It is easy to write a specification for a product that won’t actually meet the user’s needs. In a verification-focused test, the product that meets this spec would be considered acceptable (some people would define quality in such a way as to call it high quality). But people rarely put a product into production because it meets a specification. They put it into production in order to meet their needs. If the product doesn’t do this, it has low value for them. If the tester can recognize lack of value, then (normally) the tester should report it–whether the spec says this is what the product should do or not.

    The tester is not necessarily required (or able) to model the solution. When I write a bug report (as a tester), I say “X is wrong”. I need a justification for this, a reasonable basis for making the claim. But I don’t have to say (I don’t have to know) that some other Y is “right”. I am not the designer of the software or the business that uses the software. I am the person who (sometimes) recognizes when that design is weak.

    [Joep’s reply: In that paragraph I was using ‘the solution’ and ‘the product’ interchangeably, because principle no. 5 states “The product is a solution.” Funnily enough, when I copied that sentence, I was surprised it said ‘a solution’ instead of ‘the solution’. So replacing ‘the product’ by ‘the solution’ does not completely work on a textual level here.

    Finally, you may have given me an idea for another blog post (so thank you): to what degree do the pairs verification-validation and verification-falsification match? Is “Did we build the product right?” a question to be verified and “Did we build the right product?” a question to be falsified? It has some appeal to me, but I can also think of a few exceptions, so I’ll have to think some more and see if something interesting happens.]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s