After my previous post, I began a journey to discover better ways of unit testing. I had come to the realization that I had been following the cargo cult and that we as an industry don’t seem to be very good when it comes to understanding and executing automated testing. Perhaps you would disagree, but never in my career so far have I experienced what I would consider a comprehensive automated test suite for any project I’ve yet been a part of, which is a shame because I think the value of automated testing is underrated.
I am still well in the middle of my journey, but after searching for months, I finally stumbled on a path that has lead me to a better approach. The trigger for me was the book, Clean Craftmanship, which highlighted the solution to a number of questions I was unable to answer myself.
A phrase I’ve begun to say with increasing frequency is, “do what adds value. Do what makes sense.” Value may be a nebulous concept that’s difficult to quantify, but most engineers can tell when they’re doing activities that don’t add value – following procedures because that’s the process rather than a clear, stated need for doing so. The default approach to testing is full of these valueless activities because we fail to understand how to write meaningful tests, so while nearly every engineer I’ve talked with understands the value of testing on paper, it’s little surprise when they abandon the act of actually writing them when they find themselves doing what seems, and is, not useful.
Here’s what’s not useful:
@Test
public void testAdd() {
assertTrue(5, add(2, 3));
}
public int add(int x, int y) {
return x + y;
}
This is the typical, trivial example used to explain unit testing, but because it is so simplistic, it fails to be useful to anyone. So let’s create a better scenario; how do we test a multi-tier API that has a database?
Let’s say we have a controller like this:
public class DemoController {
private final DemoService service;
public DemoController(DemoService service) {
this.service = service;
}
public Person getPerson(int id) {
if(id < 1) {
throw new IllegalArgumentException("IDs must be greater than 0.");
}
return service.getPersonById(id);
}
…that calls a service like this:
public class DemoService {
private final DemoDatabaseWrapper database;
public DemoService(DemoDatabaseWrapper database) {
this.database = database;
}
public Person getPersonById(int id) {
return database.getPersonById(id);
}
…that calls a repository like this:
public class DemoDatabase implements DemoDatabaseWrapper {
private final NamedParameterJdbcTemplate template;
public DemoDatabase(NamedParameterJdbcTemplate template) {
this.template = template;
}
public Person getPersonById(int id) {
String sql = "select * from person where Id = :id";
MapSqlParameterSource map = new MapSqlParameterSource();
map.addValue("id", id);
return template.queryForObject(sql, map, BeanPropertyRowMapper.newInstance(Person.class));
}
There are a few guidelines we want to follow for testing these endpoints. I’ll summarize them here:
- Mock only in unit tests and only at the boundary layer. Overuse of mocks is bad, but not using them is also bad.
- Don’t mock what we don’t own. The behavior could change without us knowing and our tests could miss it.
- Test at the top level only. Do not isolate our code from itself.
The first two are covered in plenty of articles, but the third is the most important and the concept I call Black Box Unit Testing.
The Solution
Black Box Unit Testing is my repackaging of the idea that test classes should not have a one-to-one relationship with the classes under test. We should look to test, exclusively, the behavior and not the implementation. The behavior is the foo in, bar out relationship; in other words, the black box.
Tests in this style interact with the top level of the call stack and include all the concrete implementations of all layers, except for on the boundary with external dependencies, which are any services, databases, etc. that reside outside the project in question. These tests provide coverage for all levels of the application and still allow sending requests with any range of input we need to consider.
The advantage of this approach is the ability to change the implementation without breaking the tests as long as the behavior is unchanged. Tests that do not follow this style with tightly constrain any changes and will likely be abandoned due to the tediousness of having to constantly fix them for even minor implementation changes. While it does guarantee useful tests, this approach makes it more difficult to write valueless ones.
Let’s look at an example:
class DemoApplicationTests {
private DemoController controller;
private AutoCloseable closeable;
@Mock
private DemoDatabaseWrapper database;
@BeforeEach
void setUp() {
closeable = MockitoAnnotations.openMocks(this);
controller = new DemoController(new DemoService(database));
}
@AfterEach
void tearDown() throws Exception {
closeable.close();
}
@Test
void negativePersonIdShouldThrowIllegalArgumentException() {
assertThrows(IllegalArgumentException.class, () -> controller.getPerson(-1));
}
@Test
void validPersonIsReturned() {
// Given
Person p = new Person();
p.setId(1);
given(database.getPersonById(anyInt())).willReturn(p);
// When
var person = controller.getPerson(1);
// Then
assertEquals(1, person.getId());
}
}
You may notice that this looks quite like what many people would call an integration test. Indeed, they are going to be similar in many cases using this approach, but with one critical difference – the integration tests use the real version of any external dependencies and will refrain from using any mocks.
Black box unit tests do not preclude the use of integration tests, but in fact help to enable them. Partly due to their reliance on mocks, unit tests have notable weaknesses that make them insufficient if used as the exclusive testing method. Instead, they are a part of a greater testing infrastructure than includes integration, and likely even end-to-end tests as well. To see how integration tests might fit into this example, see the link to the code on GitHub down below.
This style of testing isn’t new and I’m far from the first to suggest it, but it is a style that is not commonly known or used. There have been many iterations of the same attempts to teach the concept, but the style I first mentioned in the beginning of this post remains the prevailing one. I encourage you to try black box unit testing and that it can help you realize that writing automated tests is not only something you should do, but hopefully something you can actually enjoy doing.
For more info, integration tests, and my detailed notes, the code is available on GitHub.
Adam Lif 