Building a Test Pyramid from the Bottom Up

I was brought into my current role at Immerse.io to uphold a high level of quality for the company’s awesome products. But where to start?

After I put together my test strategy I needed to know how much test coverage was currently in place.

Plenty of manual testing was taking place but it needed refining as not everything needed to be tested manually. Also, there were no test cases for the new system documented or being run so there was no way to know how much had been tested, what test was last run let alone what results those tests gained.

So with no test cases to look at, I turned my attention to the developers and their unit tests. The good news is that they had unit tests, the bad news is that they didn’t know how much. There were no code coverage tools integrated into their builds so they couldn’t tell.

So I first turned my attention to increasing unit tests in the projects and adding code coverage tools. Once we had the code coverage levels, we could work to increase the code coverage of the project codebases. If our code coverage level was high, it would give us more confidence about the quality of the system at a low level. But how many test cases should we create?

With this question in mind, I started thinking how I could use the Test Pyramid concept and apply it across our products.

What is the Test Pyramid?

The Test Pyramid is a concept that is used to determine what type of tests and how many of each type should be used at different levels within your product’s development. There are usually three different levels to the pyramid (although some depictions may contain between three to five). The image below is referenced from Martin Fowler’s blog post regarding the Test Pyramid.

As you go up in the levels there should be less tests. Also, as you progress up the pyramid, the tests become more complex because they involve connecting to more than one component or system. Therefore tests will take longer to run as you progress higher.

The Different Levels

The bottom level contains the majority of your tests. Unit tests are used for this level as run fast, have a small focus, are easy to add to and require little maintenance. Overall, they provide fast feedback on changes to the system at a low level.

The middle level contains your integration tests and the top level is for UI tests. The tests in the middle and top levels tend to be more susceptible to breaking because they connect with a number of components and systems.

UI tests are particularly brittle because they rely on web page elements which can be changed easily. Because the automation code is so tightly coupled to the page elements, if one is updated without the other, the tests will break. And, as these two codebases are usually maintained by different teams, sometimes the teams won’t remember to update one another. This could lead to one codebase being updated without the other one. These tests usually require more maintenance than unit and integration tests.

Fix issues as early as possible

The Test Pyramid supports the idea that a lot of thorough testing at the low levels and early phases of development helps to prevent bugs reaching the production environment.

Catching issues within your product during the later stages when the project is completed and ready for consumers is costly and can be more time consuming to fix. Attempting to apprehend issues, before they become defects i.e. production bugs, during the early phases of development is preferred because:

  • It involves less people to fix so can be presumed to be less time consuming
  • There is no impact to the customers so it’s less costly to the business

In your project, try and do the following when focusing on building your base level:

  • You should try to do the majority of your testing during the earlier phases of development with tests that are the cheapest to run.
  • It’s important to make sure that the amount of unit tests in your project is large enough to ensure that issues are caught and fixed before shipping.
  • Make sure you monitor the level of code coverage you have in your project to make sure you have the right amount to reach your target level of quality that will provide you with confidence in your product.

Follow the Test Pyramid and see what results you get.

Integrating Code Coverage

In order to follow the Test Pyramid concept, we first needed to build up the amount of tests in the base level.

Now that the tests were being added to, we needed them to be integrated within the CI tools we were using. So for one team, I set about getting these running within TeamCity. Using the Unity command-line test runner I managed to get this working when we were using Unity 5.4. Unfortunately, we found that this set up broke when we upgraded to Unity 5.6. Luckily, this has now be rectified in version of Unity 2017.2 and we have unit tests running in TeamCity on every build once again.

Unfortunately, there’s no way to measure code coverage on Unity projects so we’re just going have to be content with increasing the number of tests and making sure that these tests are meaningful. I’ll continue to monitor this though.

I next looked at getting a code coverage tool installed for the other team. Because of the tech stack here and how simple it was to setup on a simple project, I chose to use Istanbul. This has now been set up into a CI tool that runs on every push so that all code is checked and a code coverage report is produced. This gives us an idea of where we need to focus our efforts to increase the amount of unit tests.

Our Next Steps

As the production of unit tests are continuing and the developers are actively doing this themselves, I turn my attention to the next level of the Test Pyramid, the Integration tests.

This post was imported into WordPress in one click using Wordable.

What Do We Mean By Testing?

Testing is an area of development. A product or service is measured against a number of criteria at different stages of the development cycle. This is measured against the original spec, goal or intended use. Testing can be carried out manually by individuals or automated by computer systems. The automated scripts that run the systems are written and also maintained by individuals.

Why Choose Automation?

It may take longer to set up initially, but automated testing may save a lot of time and money in the long term. There are various arguments for and against automated versus manual testing. It will just depend on the task needed to be carried out and the industry as to which method is more suitable.

When I was at university not a lot of emphasis was placed on testing and the roles within this field. More attention was on the career path of the developer. But, the roles within testing can be as technical as that of the developer and just as important.

Depending on the company you work in, the job of a developer and tester may blur. You may need skills useful for either role in order to do your job effectively.

What Qualities Make You A Good Tester?

These aren’t all the qualities that you need to be a good tester but they’re certainly highly important.

  • The ability to communicate efficiently and effectively

This includes between technical (developers and other testers) and non-technical (business) members of your team. Both written and verbal communication skills are important to improve and perfect.

  • High attention to detail

You should be able to pick up minor issues as well as the glaringly obvious ones and be able to reproduce them correctly. The more issues you find and help fix, the higher level of quality the product you ship has.

  • Being able to pick up new software and languages quickly

Like most areas in IT, the tools that testers use are updated and change very quickly. As well as the tools you will use on a daily basis you may need to be able to use the tools that the developers in your team utilise.

Why Specialise In Testing?

The field of testing is extremely varied. As I mentioned, an automation role in testing can be just as technical as a role as a developer. But if you’re not into technical jobs you can still be a valuable manual tester. Manual testers frequently carry out exploratory testing which is extremely essential and something that automation can’t replicate.

As testing is the last stage in development before the product is released to the customer or to production, testers are responsible for identifying issues that could impact the reliability and use of the product. When these issues are found testers work with developers to get the issue fixed and retested to ensure the problem is solved and that nothing else is broken. Testers are the gatekeepers of quality.

Is Testing For You?

If you enjoy ensuring that something is of the highest level of quality, testing could be for you. You need to be willing to carry out some tasks that may be long and repetitive. However, others may give you a unique perspective into the development cycle and could expose you to different fields. For example, you could get more exposure to business analysis when you’re trying to determine the expected results for test cases. Or, exposure to software development when you’re writing automation or other technical tests if you’re utilising test-driven development or behaviour-driven development.

Ultimately, it’s a very satisfying role knowing that you’re the one upholding your brand’s image by ensuring a high level of quality.