Don't start with unit tests
Don't start with unit tests
MP 45: Unit tests are easy to write, but they're not always the best place to start.
I recently gave a presentation on #vBrownBag about testing. Preparing this talk pushed me to formalize some of my thoughts around testing.
When people ask how to get started with testing, they’re often told to start by writing unit tests. If you’re just learning about testing, focusing on unit tests is reasonable. Unit tests are small, and they’re relatively easy to write and run. When it comes to testing real-world projects, however, I don’t think unit tests are the right place to start.
If you write unit tests too early in a project, they’re more likely to break from intentional changes in the implementation of your project, rather than unintentional changes in the behavior of your project. In the early stages of a project’s lifetime, I’ve often found integration tests much more useful than unit tests.
The testing pyramid
There are many visualizations of the “testing pyramid”. My go-to version comes from Automation Panda:
We need to be clear about the different kinds of tests before we can consider which ones are most appropriate to start with. For this discussion, the main points are:
Unit tests cover individual units of a project’s implementation: functions, classes, and other blocks of code within a project.
Integration tests don’t deal with implementation; they verify the behavior of a project. Integration tests can cover the behavior of several units that need to work together, or they can cover the overall behavior of the project. For example an integration test can run a project, let it generate output, and then make assertions about the overall output.
End-to-end tests exercise your project exactly as your users would, including interactions with the outside world. If your project creates external resources through third-party providers, e2e tests should do that as well. If there are multiple pathways through a typical user’s experience of working with your project, there should be tests that exercise each of these main pathways.
These represent three different kinds of tests, each of which has different benefits and different costs. Keep in mind though, that there is some overlap between the different layers.1
Why people recommend starting with unit tests
Many resources recommend that if you’re new to testing, you should start by writing unit tests. If you’re just starting to learn about testing code, without any specific project in mind, it’s natural and appropriate to start with a focus on unit tests. You can work with very small examples, such as a file containing a single function that can be tested. The tests themselves are short, require little or no configuration, and quickly generate results that are relatively easy to interpret.2
But why do people recommend unit tests when someone is looking to write their first tests for a real-world project? There are a number of reasons:
Writing (and running) unit tests is easier than writing other kinds of tests.
People who are experienced at testing have worked on projects where unit tests are really useful.
Writing a single unit test feels good, and once one is written it’s even easier to add more. This is satisfying, because we can start to see test coverage increasing.
It’s natural to suggest starting with unit tests, but it’s not always the best place to start.
The problem with starting with unit tests
There’s one main reason it might not be good to start with unit tests:
Unit tests depend on a stable implementation.
If you change the implementation of your project, some of your unit tests will break, without anything being wrong with your project. Nobody experienced any bugs, but a portion of your new test suite is now broken.
What happens next? Exactly the kinds of things that make people dislike testing:
You have to stop making progress on your project, and fix your broken tests.
You’ll know you have to replace these tests again every time you change how your project is implemented.
This kind of experience sours people on testing in general. The good feeling of having test coverage is replaced with a feeling that the tests aren’t helping with the development process, or maintaining the integrity of the project. They’re just something that’s now getting in the way, without any clear benefit. This doesn’t happen because unit tests are bad, or testing is hard; it happens because the wrong kinds of tests were written too early in the project’s lifetime.
The right kind of tests to start with
I believe, for many projects, the best way to start testing is with integration tests. When I’m working on a new project and reach the point where it’s doing what I want to, it’s that behavior I care about, not the implementation. Integration tests verify that behavior. At this stage of a project, I know my implementation is likely to change. But the behaviors are absolutely something I want to preserve. Integration tests written at this point are much more likely to have a lasting positive impact on the project than unit tests.
Integration tests will almost certainly fail at times, but these failures indicate a problem with the project, not a problem with the tests. A failing integration test indicates a change in your project’s overall behavior, which is something you want to know about. These are the kinds of testing experiences that make people start to value and appreciate tests. Integration tests are a little harder to write, but they’ll almost certainly have an ongoing beneficial impact on the project.
This also holds for established projects that don’t currently have any tests. When I come into an untested but heavily-used project, I don’t care too much about the implementation right away. It usually takes some time to even understand the implementation. I care much more about the overall behavior of the project, which again leads to prioritizing integration tests over unit tests.
Building a testable project
Starting with integration tests has another significant benefit that’s worth mentioning: it ensures that your project is testable.
In order to write integration tests, you need to be able to write code that runs the overall project. If the project has a CLI or an API, that’s easy. But for some projects that have only been developed with end users in mind, it’s not at all clear how to write a script that runs the project. For testing purposes, that’s an important issue to address. The earlier you face this issue in a project, the easier it is to deal with. You might choose to implement a small CLI or limited API to assist with testing. Or, you might choose to restructure your code so it’s easier to initiate the main behaviors of the project programmatically.
Addressing testability isn’t always easy, and it’s one of the reasons that writing integration tests and end-to-end tests is more challenging than writing unit tests. But these issues won’t get easier as a project grows, and putting it off only tends to compound the issue.
Here’s one testing trap that’s nice to avoid: you write a significant number of unit tests and then decide to change the implementation of your project to better support integration and e2e tests. Now your unit tests are breaking, and you have to throw them away or make time to fix them! Putting off integration testing can create significant unintentional time sinks, that end up making people want to avoid dealing with testing as much as possible.
When to start with unit tests
There are few meaningful absolutes in programming, so of course it’s appropriate to start with unit tests in some projects. There are a number of situations I can think of where starting with unit tests is justified, and unlikely to lead to frustration:
You want to write a couple quick tests to get the overall testing infrastructure in place. Writing the first tests for a project often means installing a testing framework, setting up a tests/ directory, and maybe adding some configuration files. This is quicker and easier to do in the context of unit tests, and can be a good way to start testing.
You have units that multiple people or teams depend on. If there are functions and classes already in use by multiple people or teams, it can be helpful to start writing tests for those units. This is in line with the overall spirit of this post: Write tests that will catch meaningful changes in your project’s behavior.
Write tests that will catch meaningful changes in your project’s behavior.
You have critical units. Some units play a more critical role than others in the early phases of a project. If a unit is critical to the success of your project, writing a test for that unit is perfectly appropriate.3
You have stable units. If you have some units where you’re confident the implementation won’t change, feel free to write unit tests for those parts of your project.
You’re already getting bug reports. If you get a bug report, and you can trace that bug to a specific unit, it’s perfectly appropriate to write a unit test that will catch that same bug if it reappears.
These kinds of tests will help you continue to develop your project, without making breaking changes. These tests give meaningful information when they fail or break. They’re driven by the need to verify ongoing behavior of important parts of your project, rather than a blind attempt to gain test coverage.
What about end-to-end tests?
End-to-end tests can be useful as well, but there are some issues that make them less beneficial than integration tests early in a project’s lifetime. Like unit tests, end-to-end tests are dependent on implementation. For example if your tests call out to some remote services, they’ll break if you decide to change service providers.
End-to-end tests are also less consistent than integration tests. A failing integration test almost certainly indicates a change in your project’s behavior. A failing end-to-end test may indicate a change in your project’s behavior, or it may indicate an issue with an external resource. You’ll need to deal with this, but sorting that out in the context of building a test suite may not be the best use of your time early in a project.
Conclusions
Testing is an important part of modern software development workflows, and unit tests are an important part of a mature test suite. But just because they’re at the base of the testing pyramid doesn’t mean we should start there when adding tests to an untested project. If your tests break without any benefit, then testing will be a drag, and a sunk cost as well.
Regardless of whether you’re working on a new, unreleased project or a mature, heavily-used project, consider the benefits and drawbacks of each kind of test before deciding where to start. Integration tests offer tremendous benefit for the expense of a little more work and thought in getting started. For many projects, the benefits of starting with integration tests far outweigh the costs.
Note: This post focuses on why you should consider writing integration tests before writing unit tests. A series coming up soon will focus on how to write integration tests.
Also, thank you to the people who read an early version of this post; it is much better for the feedback you all shared.
When I first learned about these different layers, I got integration tests and e2e tests backwards in my mind. I thought integration tests integrated with the outside world. I think I first came across these terms in a resource that didn’t include anything visual, and I ended up unclear about the different levels. I still have some projects where the tests are quite useful, but the directory names (ie tests/e2e_tests/) don’t match the kinds of tests they contain.
For small projects that only run locally, there might not be much difference between an integration test and an e2e test.
I include myself in this; I have developed some resources that imply people should start testing by focusing on unit tests. I’m going to add some clarification to these resources about when people should consider writing unit tests, and when they should start with higher-level tests.
As a quick example, I was recently working on a project that helps people learn to read music. The integration tests cover a user’s experience of taking a quiz, seeing their results, and then running through a practice session of the notes they most need to work on.
When I’m working on the local version of this project, the quiz only covers a handful of musical notes, and I finish every quiz in under a minute. The integration test covers the full set of notes, but finishes the entire quiz in under a second. The first time I took the quiz using the entire set of notes, the reported time was “1 minute 93 seconds”. There was a bug in a function called getTimeString() that didn’t subtract the minutes from the seconds when generating a user-friendly message about how long the quiz took.
In this situation, I added a unit test for getTimeString(), making sure it returns correct strings for times less than and greater than one minute. It’s currently the only unit test in the project, because it meets the following criteria:
It addresses a bug that’s already come up.
It tests a stable unit of the overall project.
It tests a critical unit; users of this small project will always want an accurate report of how long it took them to pass the quiz.