I was brought into my role at Immerse.io to uphold a high level of quality for the company’s awesome products. But where to start?

After I put together my test strategy I needed to know how much test coverage was currently in place.

Plenty of manual testing was taking place but it needed refining as not everything needed to be tested manually. Also, there were no test cases for the new system documented or being run so there was no way to know how much had been tested, what test was last run let alone what results those tests gained.

So with no test cases to look at, I turned my attention to the developers and their unit tests. The good news is that they had unit tests, the bad news is that they didn’t know how much. There were no code coverage tools integrated into their builds so they couldn’t tell.

So I first turned my attention to increasing unit tests in the projects and adding code coverage tools. Once we had the code coverage levels, we could work to increase the code coverage of the project codebases. If our code coverage level was high, it would give us more confidence about the quality of the system at a low level. But how many test cases should we create?

With this question in mind, I started thinking how I could use the Test Pyramid concept and apply it across our products.

What is the Test Pyramid?

The Test Pyramid is a concept that is used to determine what type of tests and how many of each type should be used at different levels within your product’s development. There are usually three different levels to the pyramid (although some depictions may contain between three to five). The image below is referenced from Martin Fowler’s blog post regarding the Test Pyramid.

As you go up in the levels there should be less tests. Also, as you progress up the pyramid, the tests become more complex because they involve connecting to more than one component or system. Therefore tests will take longer to run as you progress higher.

The Different Levels

The bottom level contains the majority of your tests. Unit tests are used for this level as run fast, have a small focus, are easy to add to and require little maintenance. Overall, they provide fast feedback on changes to the system at a low level.

The middle level contains your integration tests and the top level is for UI tests. The tests in the middle and top levels tend to be more susceptible to breaking because they connect with a number of components and systems.

UI tests are particularly brittle because they rely on web page elements which can be changed easily. Because the automation code is so tightly coupled to the page elements, if one is updated without the other, the tests will break. And, as these two codebases are usually maintained by different teams, sometimes the teams won’t remember to update one another. This could lead to one codebase being updated without the other one. These tests usually require more maintenance than unit and integration tests.

Fix issues as early as possible

The Test Pyramid supports the idea that a lot of thorough testing at the low levels and early phases of development helps to prevent bugs reaching the production environment.

Catching issues within your product during the later stages when the project is completed and ready for consumers is costly and can be more time consuming to fix. Attempting to apprehend issues, before they become defects i.e. production bugs, during the early phases of development is preferred because:

• It involves less people to fix so can be presumed to be less time consuming
• There is no impact to the customers so it’s less costly to the business

In your project, try and do the following when focusing on building your base level:

• You should try to do the majority of your testing during the earlier phases of development with tests that are the cheapest to run.
• It’s important to make sure that the amount of unit tests in your project is large enough to ensure that issues are caught and fixed before shipping.
• Make sure you monitor the level of code coverage you have in your project to make sure you have the right amount to reach your target level of quality that will provide you with confidence in your product.

Follow the Test Pyramid and see what results you get.

Integrating Code Coverage

In order to follow the Test Pyramid concept, we first needed to build up the amount of tests in the base level.

Now that the tests were being added to, we needed them to be integrated within the CI tools we were using. So for one team, I set about getting these running within TeamCity. Using the Unity command-line test runner I managed to get this working when we were using Unity 5.4. Unfortunately, we found that this set up broke when we upgraded to Unity 5.6. Luckily, this has now be rectified in version of Unity 2017.2 and we have unit tests running in TeamCity on every build once again.

Unfortunately, there’s no way to measure code coverage on Unity projects so we’re just going have to be content with increasing the number of tests and making sure that these tests are meaningful. I’ll continue to monitor this though.

I next looked at getting a code coverage tool installed for the other team. Because of the tech stack here and how simple it was to setup on a simple project, I chose to use Istanbul. This has now been set up into a CI tool that runs on every push so that all code is checked and a code coverage report is produced. This gives us an idea of where we need to focus our efforts to increase the amount of unit tests.

Our Next Steps

As the production of unit tests are continuing and the developers are actively doing this themselves, I turn my attention to the next level of the Test Pyramid, the Integration tests.

[wordable-io]