Getting [name] from “Name: [name]” in Python – an engineering problem

Today I was presented with an interesting engineering problem. (Important later: context was the code of an auto-test.) Given a string of the format “Name: [name]”, what’s the best way to get the [name] in Python?

There are several options:
– lstrip()
– split()
– replace()
– string slicing
– regex

So let’s look at each of them and then I’ll explain which one I prefer and why. All examples are in Python 3.6, using the Python Interpreter.

lstrip()

>>> our_string = "Name: this-is-the-name"
>>> our_string.lstrip("Name: ")
'this-is-the-name'

That seems to work until we try a different string:

>>> our_other_string = "Name: a name"
>>> our_other_string.lstrip("Name: ")
'name'

This may seem weird, but lstrip() is doing exactly what it says it will do: “The chars argument is not a prefix; rather, all combinations of its values are stripped.” So it will continue stripping until it encounters the first character that doesn’t match any character in the string we gave to lstrip().

To fix that, we can do:

>>> our_other_string = "Name: a name"
>>> our_other_string.lstrip("Name:").lstrip()
'a name'

Where the first lstrip() won’t get beyond the space and the second lstrip() will get rid of that space for us.

split()

>>> our_string = "Name: this-is-the-name"
>>> our_string.split(":")[1].lstrip()
'this-is-the-name'

>>> our_other_string = "Name: a name"
>>> our_other_string.split(":")[1].lstrip()
'a name'

The split() will split the string on the colon (:) into a list with two items. We get the second item with the ‘[1]‘ (list indices start at 0). We strip off the leading space.

replace()

>>> our_string = "Name: this-is-the-name"
>>> our_string.replace("Name: ", "", 1)
'this-is-the-name'

>>> our_other_string = "Name: a name"
>>> our_other_string.replace("Name: ", "", 1)
'a name'

The replace() method replaces parts of a string. So here we replace “Name: ” with an empty string – deleting it. The ‘1‘ argument at the end tells replace() to only replace the first occurrence:

>>> too_many = "Name: more Name: please"
>>> too_many.replace("Name: ", "")
'more please'

>>> too_many.replace("Name: ", "", 1)
'more Name: please'

string slicing

Python allows you to grab slices from a string similar to what you do from a list (as we did for the split() above).

>>> our_string = "Name: this-is-the-name"
>>> our_string[6:]
'this-is-the-name'

>>> our_other_string = "Name: a name"
>>> our_other_string[6:]
'a name'

The number before the colon tells Python where to start, the one after the colon where to end. So ‘[6:]‘ means to start after the 6th character (or rather from index 6, since indices start at 0) and continue until the end of the string (because no number was given).

regex

>>> import re
>>> regex = "(^Name: )(.+)"

>>> our_string = "Name: this-is-the-name"
>>> re.match(regex, our_string).group(2)
'this-is-the-name'

>>> our_other_string = "Name: a name"
>>> re.match(regex, our_other_string).group(2)
'a name'

The regular expression is defined in the regex variable. It describes a string that starts with “Name: ” and is followed by at least one other character. The parentheses define groups, which we’ll use to get the part we’re interested in, i.e. the part after “Name: “.

match() will return a match object if zero or more characters at the beginning of the string match the regular expression pattern. Which means we don’t actually need the ‘^‘ at the start of our regex to only match the start of the string.

group() will return one or more subgroups of the match. The zeroth group is the entire match. Thanks to the parentheses in our regex definition we also have a first group (“Name: “) and a second group (the thing we’re after).

which one to use

Going through the options they all have disadvantages:
lstrip() strips characters not strings, but we want to strip a certain string.
split() returns two things of which we need only one after we have lstrip()-ed it.
replace() returns a changed copy of a string, while we just want to parse the original string.
– string slicing ditches the first 6 characters not caring what they are.
– regex feels like a bit too much.

However, two of them have a distinct advantage over the others.

As I said in the introduction, the context of this problem was the code of an auto-test. More specifically, the “Name: [name]” is returned by Selenium WebDriver as the content of an html tag. So without knowing or seeing the application, just from reading the code that retrieves the content of the tag, you have no idea what this string is that we are manipulating.

So that’s the first thing we want from our solution: it should give as much information about the string we are manipulating as possible. This means that split() and string slicing are not an option. The split() solution only tells you there’s a colon followed by a whitespace in the string. The string slicing solution only tells you there are at least 7 characters in the string.

Secondly, the solution should capture our intention as clearly as possible, which is getting the [name] from the string “Name: [name]”. That means lstrip() is out, because it strips characters, not a specific string.

You could argue something similar for replace(), since the “replace with empty string” is a somewhat clever hack to use something intended for editing to retrieve a specific part of a string. On the other hand, the code is clear enough – unlike lstrip() where the characters-not-string might become a bit of a surprise down the line.

So basically we’re left with replace() and regex. At the office we hadn’t figured out that replace() has a count argument, so based on ‘But what if there’s a “Name: ” in the content of the tag?’, we went with the regex solution. Regardless, I still think that the regex solution is the clearest one.

conclusion

The Zen of Python states: “Explicit is better than implicit.” This one sentence is the reason I can write a whole blog post about what the best way is to get [name] from “Name: [name]”. Code being explicit about what it’s doing and what it is trying to do, matters. The more explicit it is, the better it can serve as documentation.

But what if you have to choose? The regex solution is indeed the clearest – assuming you know some regex, that is. So the solution that’s most explicit about what the string is and what our intentions with it are, is the least explicit about how it does what it does. (Also, if you have trouble understanding what it does, it will be hard to determine the intentions behind the code.) So what now?

Personally I will always trade the code being less explicit about what it does for being more explicit about the thing-under-test and what my intentions are. Because figuring out what the thing-under-test exactly does and what someone’s intentions were, are hard things to do. Learning some new things about a programming language or tool, significantly less so – even if it’s regex.

Your CI/CD pipeline does not run regression tests

CI/CD pipelines

The purpose of a CI/CD pipeline is to allow you to deliver small changes in a fast and controlled way. Without any tests in your pipeline you would gain a lot of speed. You’d also lose a lot control, which is why people in general do run tests in their pipeline. The purpose of these tests is to check if that stage of the pipeline meets the minimum level of acceptable quality for that stage.

For example, commit stage tests will consist of mostly unit tests, a few integration tests, and even fewer end-to-end tests, because early in the pipeline speed is more important than comprehensiveness. When I commit my changes, I want the results fast enough so that I will wait for them – ready to fix any issue that might occur.

Regression testing

There are many definitions of regression testing, as you can read in Arborosa’s blog post on the topic. I have always defined regression testing along the lines of “testing the parts that weren’t impacted by a change to see if they really weren’t impacted.” (Which is really weird if you start thinking about it: something is regression testing depending on your knowledge of the system and the change.)

The tests in your pipeline are regression tests, …

Most of the tests that run in your pipeline are regression tests. Your commits are small and you have a lot of tests, so most of those will cover parts of the system that shouldn’t have been impacted by your changes. So yes, regression tests.

The one exception is if your commit contains both changes and new or updated tests related to that change. For that one run of the pipeline those tests are not regression tests. The next commit they are.
Or, since you ran those tests before committing, perhaps they already have become regression tests when they are executed by the pipeline?

Sidenote:
A grey area is when your commit is a pure refactoring, as in: you didn’t even have to change any of the tests. On the one hand, you made a change, so the tests covering that change, are not regression tests. On the other hand, at the level these tests are defined, there should be zero impact, they shouldn’t detect any changes. So in that sense they are regression tests.

…, but that’s irrelevant.

So sure, the tests run by your pipeline are regression tests. However, they are regression tests incidentally, not essentially. They happen to be regression tests, but that’s not really relevant.

To see why, we need to revisit the start of this blog post.

The purpose of a regression test is to check if unchanged parts of the system are indeed unchanged. It’s the testing that got a name, so we could distinguish it from the other testing, which never really got a name. (Progression testing? Feature testing?) It’s the testing you do after sufficient testing and fixing, when you’re not expecting any more changes and you need to check if all the “other stuff” still works.

The purpose of a test in a CI/CD pipeline is to check the level of quality of a particular stage in the pipeline. The pipeline stages combined with all the practices that surround them, result in a continuous delivery of changes that can be deployed to production. Whether the tests at a particular stage are regression tests or not, doesn’t matter. What does matter is if they provide the information required to decide if we should proceed to the next stage or not.

And that’s why I claim that your CI/CD pipeline does not run regression tests. The definition of “regression test” may technically apply to the tests run by your pipeline; the context that comes with the term, does not. So although it might (mostly) be correct to say that your pipeline runs regression tests, doing so is not helpful in how you think about your pipeline or about your tests. It moves your mind towards thinking about changed versus unchanged things – drawing it away from the continuous delivery of a good enough product.

update August 6th:
After publishing this post, I got the following question on twitter: so how does this impact actual decisions? In response, I came up with four things you might do if you think of the tests in your pipeline as regression tests:
1. Not looking for regressions when exploratory testing because you already have so many regression tests.
2. Poorly designing the stages of the pipeline, because all it needs to do is just run those regression tests.
3. Doing exploratory testing too early in the pipeline, because you should do feature testing before regression testing.
4. Being lenient towards a failed pipeline because they’re just regressions, we can fix them later.

— — —

p.s. 1: One thing I’m glossing over is that your CI/CD pipeline can (should) have stages in which the testing involves a human. I don’t think it makes a difference for my argument. Yet I’m still conveniently limiting the scope of this post to the literal interpretation of “Your CI/CD pipeline does not run regression tests”.

p.s. 2: None of the ideas in this blog post are new, which you can see from the replies in the twitter thread that lead me to writing this blog post.

how this tester writes code

A long time ago (March 2015) I wrote a post titled “Test automation – five questions leading to five heuristics“. Later that year Rich Rogers asked for a follow-up. To which I replied I should do a follow-up post (ahum) “soon”.
Then last Wednesday Noah Sussman said on twitter: ‘I don’t know that I’ve *ever* seen “this is how testers write code‘. To which I replied “challenge accpeted”, so now here we are, me writing a blog post about how I as a tester write code.

The format of this post turned out to be advice based on my experiences, so the usual disclaimers apply. And feel free to leave a comment if you have any feedback!

the basics

use an IDE

An IDE is not just an advanced text editor. It understands your code – to a degree, since it’s not interpreting, compiling or executing the code. So not only allows an IDE you to manipulate your code as text, it also allows you to manipulate your code as code.

The first place people seem to run into this, is when renaming functions or variables. With a basic text editor you do all the renaming yourself. With a more advanced text editor you have several ways to do a find-and-replace. With an IDE, however, you can do a refactor-rename, which means the IDE will figure out all the places it needs to that rename for you.

Realizing this is the power of a good IDE, is an important step. It allows you to interact with code-as-code, with code-as-text as a fall-back option.

using libraries well takes skill

Using libraries involves several skills. There’s the skill of picking a library based on the documentation, the number of recent commits/releases, number of stars, quality of the code, quality of the tests, trying it out. There’s the skill of using the library by reading the documentation, looking at examples, reading (parts of) the library code, experimenting with it.

Finding the right library that can do most of the heavy lifting for you and being able to use it to its full potential, will make your life a lot easier. So acquire these skills.

use version control

As the saying goes: “Version control is a great way to do version control.” It’s definitely better than manually adding version numbers to your source code files (been there, done that).

As for version control tools, Git is a great one. It’s very popular and probably also complete overkill for what you need. Don’t let that discourage you. My own approach to using git consists of:
1. basic commands: add, commit, push, pull, checkout, checkout -b, merge, rebase, push -f, stash, stash apply.
2. git aliases for some commands I never remember, e.g. “git oops” for “git reset HEAD^” and “git tree” for some ridiculously long command that shows a tree-like view of commits.
2. reading whatever git, GitLab and GitHub tell me.
3. Googling stuff, which often brings me to this page.
4. whatever I remember of a video (can’t find it, sorry) about how git works (local vs remote, hashes and trees).

One more thing: if you use version control, small commits are the way to go. And don’t feel bad if you have to learn that the hard way. I suspect most of us do it that way.

next steps

mess around with different languages

Over the years I have written code in Bash, Perl, PHP (if adding a string to an array counts), VBA for Excel, Java, JavaScript, TypeScript, and Python. In some more than others and I must admit that in any of those languages except for Python I’d have trouble writing three lines of correct code without looking something up.
And that’s ok. There’s value in having a little experience in a bunch of different languages, because it gives you an idea in which ways those languages are different, and in which ways they are similar.

dive deeper into one language

James Powell his talk So you want to be a Python expert? was important to my learning in three different ways. First of all, watching it and realizing I understood most of it, made me appreciate how much I had learned already. Secondly, I learned a lot about some more advanced features of Python and more importantly the ideas behind them. Thirdly, I realized that learning that stuff is important to progress beyond basic programming.

writing good code

readability

I was making two big mistakes in this area until a developer showed me the error of my ways. The first mistake is that I used short names for everything, because I didn’t realize that IDEs with their auto-completion and refactor-rename make using longer names trivial. The second mistake is that I only used functions and methods to avoid copy-pasting code, never to provide structure to the code.
Avoid those two mistakes by giving everything a proper descriptive name and by using functions and methods for structure, and you have two of the three basics of clean code covered. The third one is: refactoring. Getting naming and structure right in your first version is incredibly hard. So iterate a few times until it’s good enough.

As soon as you have a basic grasp of the above, learn more about clean code and design patterns. Some smart people who have written a lot of code, have thought a lot about how to write good code. Learn from their insights.

separation of concerns

Separation of concerns is one of the most important design principles. If we should use functions and methods to structure code, then what should be the scope of one function or method? Ideally a function or method does one thing and one thing only. That also makes naming easier: you name it for the one thing it does.

If you write code this way, you will have separation of concerns on the lowest level. Next you might notice that some pieces of code are more similar than others. There’s the code that forms the actual tests, the code that interfaces with the software we are testing, the code that does data setup and teardown, etc. That too is separation of concerns, but one level higher: group functions and methods that do similar things together.

If this still seems a little abstract, the most widely known example of separation of concerns for test automation is the Page Object Pattern. You separate the code that knows how to interact with the DOM (the page objects) from the code that contains the test steps and asserts.

logging and debugging

In the beginning it’s fine to use print statements to get an idea of what your code is doing. At some point however, you’ll want to switch to logging. It gives you more flexibility through different log nodes, different log levels, and different log handlers.

If you find yourself adding more and more print statements and/or logging to figure out why your code is not working, switch to a debugger. Especially with a good IDE the learning curve is not that steep and nothing beats being able to inspect your code while it’s doing what it’s doing.

exploring and one-offs

Your code should allow you to easily add some new tests based on the existing ones, for the times you want to explore some behavior that’s not quite covered by the existing tests. Afterwards you can decide to clean up and commit, or to discard these tests.
It should also be easy to create one-off scripts based on your tests and framework. For example, a script to do the data setup for an exploratory test session or a script to generate load as part of a crude performance test.

If your code only allows you to run your tests as they are, you’re missing out.

building good tests

a test should do one thing and one thing only

This is separation of concerns applied to tests: a test should do one thing, not multiple. Of course, depending on the type of test, the size of that one thing will vary. A test on the unit level will be smaller than a test going through the whole stack.

Tests doing one thing only also makes naming easier. Name the test for the thing it is testing. And by this I mean: capture the intention of the test, not the implementation. The implementation I can get by reading the code, the intention I might be able to deduce from the code, but there’s a good chance I can’t.

Separation of concerns for tests also means having all setup and teardown in separate functions or methods. This makes it a lot clearer what the test is trying to cover. So when you start reading a test, you are reading the actual test immediately, not some preparation steps.
Sidenote: ideally your setup and teardown do not have asserts, but will throw exceptions when unexpected things start to happen.

interfaces should be sufficiently transparent

So we have separated our interface code from our test code and most likely we are using some library (e.g. Selenium WebDriver) to do the actual interfacing. Life is easy and code is readable. However… how exactly are we interfacing with the software we’re testing? Do we know what our test is actually doing to the thing we’re testing?
That’s why I want interfaces to strike a balance between ease-of-use and transparency. I do want to say “do a GET on this url” without having to worry about the actual HTTP implementation, but I also want to know what my HTTP library puts in the headers of the requests.

Do you have to write tests for your tests?

(After I accepted Noah’s challenge, I asked if there was any topics in particular he wanted me to cover. This is one of them. Follow the link to read Pedro Gonzalez‘s response and Noah’s reaction.)

If you feel the need to write tests for your tests, that suggests your tests are too complicated. So no, don’t write tests for your tests. I am in favor, however, of testing your tests. This testing can take many forms: a review by you or (even better) someone else, making the test fail through a change in the test or the software under test, mutation testing. You can also consider your tests as implicit tests for your other tests. Do note that how much coverage you get from this implicit testing can vary a lot depending on the kind of tests.

One thing you should consider writing tests for, is your test framework and utilities. If they’re fairly trivial, it might not be needed. As soon as some complexity creeps in, add some simple tests – rather add them a little too soon than a little too late. It will make refactoring your framework and utilities easier.

Can you apply TDD to building tests?

Thinking about writing tests for your tests, I began to wonder: what would doing TDD for tests look like? (Disclaimer: I’m familiar with the ideas behind TDD, but have never practiced it.)
You’d start by writing a meta-test, a test to test your test. You’d run it and it fails, because you haven’t written the test yet. So you write the test. You run the meta-test. It passes. (Or not, so you work on the test until the meta-test passes.)
Now what would a meta-test look like? It’s good test design to have the meta-test not be aware of the implementation details of the test. So the meta-test should only care about what the test returns, which is pass/fail/exception. But isn’t that what our test runner already does for us? Then why write a separate meta-test?

And that train of thought gave me an idea. Perhaps it could be a good practice to start building a test by giving it a name and writing the asserts you want for that test. If you’d run that test, it would fail horribly, because there’s nothing to assert yet. Then you add code to the test until the asserts can evaluate what they need to evaluate.
As I said, right now this is only an idea. If you decide to try it out, please let me know how it went.

Is it ok to have conditional logic in your tests?

(After I accepted Noah’s challenge, I asked if there was any topic in particular he wanted me to cover. This is the other one. He answers this question himself here.)

Needing conditional logic in your tests suggests to me that you’re lacking control somewhere. That your tests are not as deterministic as they should be, so you add conditional logic to remedy that. This really should be a last resort where no other solution is feasible, and the value of the test outweighs it being only half-deterministic.

Conditional logic in setup or teardown is a different matter, by the way. For instance for tests where data creation is expensive, doing a get-or-create can be a good solution.

knowing what’s going on when your test runs

Testing is investigating to evaluate a product. So you want as much information as possible – assuming it’s structured well enough you can navigate it easily. So good test reports and logging are crucial. When a test fails, you want to have more information than the name of the test, the failed assert, and its assert message.

tests-as-code versus tests-as-tests

two different perspectives

As hinted at by having a section on “writing good code” and one on “building good tests”, I approach auto-tests from two different perspectives: tests-as-tests and tests-as code.

Usually these two perspectives align nicely, for example when talking about separation of concerns. Sometimes, however, they do not.
The don’t-repeat-yourself (DRY) principle is a good example. Even when you apply separation of concerns, auto-tests tend to get a bit repetitive when you want to test different variants of the same scenario. (Test parameterization can help here too.)
Let’s say you have a bunch of tests expecting the same (or a very similar) error message. Following the DRY principle, it would make sense to put that message in one variable shared across tests. From a tests-as-code perspective this makes perfect sense. From a tests-as-tests perspective, however, it does not, because it makes the test more difficult to read.
In cases like this I prefer to have easier to read tests over easier to maintain code.

Another thing is that when you’re focused on one of the two perspectives, it sucks to be forced to switch to the other perspective. When you’re in tests-as-tests mode, you’re thinking about the software you’re testing, about the problems it’s trying to solve, the capabilities it tries to deliver. In tests-as-code mode, you’re thinking about the code you’re writing, how to make it do what you need it to and how to keep it maintainable. Switching between the two in a natural rhythm is fine; being forced to switch not so much.

a programming language is a DSL

There are quite some testing tools and libraries that introduce a domain-specific language (DSL) to make writing tests easier. Their claim is that it’s easier to learn their domain-specific language than it is to learn a general-purpose programming language.

In my opinion that’s not an entirely fair comparison. I don’t need to learn all of Java or Python or … to be able to build tests in those languages. I only need to learn the part of the language that’s relevant to the building I want to do. Hence, a programming language is a domain-specific language.

Now I must admit that most DSLs are easier to learn than the required sub-set of a programming language. Even so, I prefer general-purpose programming languages for two reasons.
First of all, a DSL is limited to its domain. If you want to do something it wasn’t designed for or something outside of that domain, you’ll either need to switch to a different DSL or to a general-purpose language. Since that’s bound to happen at some point, I’d rather learn a sub-set of a general-purpose language from the start and expand my knowledge of that language when needed.
Secondly, I think there’s value for anyone who’s part of software development, to have a basic understanding of programming. The best way to do that is to do things in code on a a regular basis – preferably as part of your job. Building tests is a great way for non-programmers to have that experience.

revisiting my blogpost from 2015

As mentioned in the introduction, in 2015 I wrote a blogpost with the title “Test automation – five questions leading to five heuristics“. So let’s see revisit those heuristics more than four years later.

Heuristic 0: Don’t call it test automation.
I don’t think I ever stopped calling it test automation. When I wrote the post, I was stricter in using the best possible words. Currently, my position is that if communication is happening, the words we are using are good enough. I do prefer the term “auto-tests” over “automated tests”, since the latter still suggests to me that perhaps we might be able to “automate all the testing”.
(And yes, I snuck six heuristics in my original post by starting with 0.)

Heuristic 1: Never trust a test you haven’t seen fail.
Yep, see “Do you have to write tests for your tests?“.

Heuristic 2: Each test should test only one thing.
Yep, see “A test should do one thing and one thing only“.

Heuristic 3: It’s better to have reliable information that doesn’t exactly tell you what you want to know, than unreliable information that does.
Not covered in this post, although it is related to having deterministic tests (“Is it ok to have conditional logic in your tests?“). And I still agree: I’d rather have a reliable test that doesn’t entirely test what I want to test, than an unreliable test that does (or more precise: tries to).

Heuristic 4: Every minute spent debugging test automation code is wasted, because you learn nothing.
I was quite amused reading this, since it seems I was finding no joy at all in writing and debugging code. I don’t think that was actually the case, but it is true that I enjoy programming more now than I did back then in 2015. The reason is quite simple: I’ve gotten better at it.
Setting aside the tone of that paragraph, I do see a link with what I have written above about tests-as-tests and tests-as-code, and about how being forced to switch between them sucks.

Heuristic 5: Epistemic testability, epistemic testability, epistemic testability.
Revisiting the definition of epistemic testability, I don’t think I was using the term the way James Bach intended. Be that as it may, I still stand by the point I was making: is your test automation helping you to do better testing? And by extension, helping you to develop and deliver software better?