Building a Test Pyramid from the Bottom Up

I was brought into my current role at Immerse.io to uphold a high level of quality for the company’s awesome products. But where to start?

After I put together my test strategy I needed to know how much test coverage was currently in place.

Plenty of manual testing was taking place but it needed refining as not everything needed to be tested manually. Also, there were no test cases for the new system documented or being run so there was no way to know how much had been tested, what test was last run let alone what results those tests gained.

So with no test cases to look at, I turned my attention to the developers and their unit tests. The good news is that they had unit tests, the bad news is that they didn’t know how much. There were no code coverage tools integrated into their builds so they couldn’t tell.

So I first turned my attention to increasing unit tests in the projects and adding code coverage tools. Once we had the code coverage levels, we could work to increase the code coverage of the project codebases. If our code coverage level was high, it would give us more confidence about the quality of the system at a low level. But how many test cases should we create?

With this question in mind, I started thinking how I could use the Test Pyramid concept and apply it across our products.

What is the Test Pyramid?

The Test Pyramid is a concept that is used to determine what type of tests and how many of each type should be used at different levels within your product’s development. There are usually three different levels to the pyramid (although some depictions may contain between three to five). The image below is referenced from Martin Fowler’s blog post regarding the Test Pyramid.

As you go up in the levels there should be less tests. Also, as you progress up the pyramid, the tests become more complex because they involve connecting to more than one component or system. Therefore tests will take longer to run as you progress higher.

The Different Levels

The bottom level contains the majority of your tests. Unit tests are used for this level as run fast, have a small focus, are easy to add to and require little maintenance. Overall, they provide fast feedback on changes to the system at a low level.

The middle level contains your integration tests and the top level is for UI tests. The tests in the middle and top levels tend to be more susceptible to breaking because they connect with a number of components and systems.

UI tests are particularly brittle because they rely on web page elements which can be changed easily. Because the automation code is so tightly coupled to the page elements, if one is updated without the other, the tests will break. And, as these two codebases are usually maintained by different teams, sometimes the teams won’t remember to update one another. This could lead to one codebase being updated without the other one. These tests usually require more maintenance than unit and integration tests.

Fix issues as early as possible

The Test Pyramid supports the idea that a lot of thorough testing at the low levels and early phases of development helps to prevent bugs reaching the production environment.

Catching issues within your product during the later stages when the project is completed and ready for consumers is costly and can be more time consuming to fix. Attempting to apprehend issues, before they become defects i.e. production bugs, during the early phases of development is preferred because:

  • It involves less people to fix so can be presumed to be less time consuming
  • There is no impact to the customers so it’s less costly to the business

In your project, try and do the following when focusing on building your base level:

  • You should try to do the majority of your testing during the earlier phases of development with tests that are the cheapest to run.
  • It’s important to make sure that the amount of unit tests in your project is large enough to ensure that issues are caught and fixed before shipping.
  • Make sure you monitor the level of code coverage you have in your project to make sure you have the right amount to reach your target level of quality that will provide you with confidence in your product.

Follow the Test Pyramid and see what results you get.

Integrating Code Coverage

In order to follow the Test Pyramid concept, we first needed to build up the amount of tests in the base level.

Now that the tests were being added to, we needed them to be integrated within the CI tools we were using. So for one team, I set about getting these running within TeamCity. Using the Unity command-line test runner I managed to get this working when we were using Unity 5.4. Unfortunately, we found that this set up broke when we upgraded to Unity 5.6. Luckily, this has now be rectified in version of Unity 2017.2 and we have unit tests running in TeamCity on every build once again.

Unfortunately, there’s no way to measure code coverage on Unity projects so we’re just going have to be content with increasing the number of tests and making sure that these tests are meaningful. I’ll continue to monitor this though.

I next looked at getting a code coverage tool installed for the other team. Because of the tech stack here and how simple it was to setup on a simple project, I chose to use Istanbul. This has now been set up into a CI tool that runs on every push so that all code is checked and a code coverage report is produced. This gives us an idea of where we need to focus our efforts to increase the amount of unit tests.

Our Next Steps

As the production of unit tests are continuing and the developers are actively doing this themselves, I turn my attention to the next level of the Test Pyramid, the Integration tests.

This post was exported in one click from Google Docs into WordPress with Wordable.

How Important Is CI and CD For An Appreneur?

Here’s my next post which appeared first on Simple Programmer..

During the day, I work as a test engineer. But I also consider myself a developer—more specifically, an appreneur.

What is an appreneur? Why, it’s any entrepreneur trying to build a business via mobile applications. You could be producing templates for others to use, creating apps to sell directly on the app stores, or providing a number of other services reliant on mobile applications.

I work by myself, so I’m running a business solo. Most appreneurs work in this way, though a few have a small team of trusted contractors handling specialized jobs.

It’s important to find tools and processes to help you continue to maintain the quality of your app once it is launched. Especially if it’s just you; you want to spend more of your time and resources creating and launching new products than maintaining older ones.

You want to reduce the time you spend on individual tasks, improve the quality of your end product, and reduce the cost of your overall efforts, and you need to carefully consider what might be the right tools to help you do all that.

Continuous integration and continuous development have been used by project development teams for many years to make the development process more efficient and get better end results. Can we use the same processes as appreneurs?

As an appreneur, I’m the one responsible for the design, development, testing, and business activities involved in creating and releasing a product to market. Make no mistake, completing one of these projects is no small feat.

If you are an appreneur, you are probably trying to increase or update your range of apps being sold in the App Store, or expand your catalog into a new App Store ecosystem, or create a new sustainable revenue stream related to app development. It all depends on where you are in your business.

Regardless of what specific paths you are pursuing, you are certainly hard at work, juggling multiple activities and milestones and products for your business. Even if you have a small team of contractors, you’re still overseeing the whole thing. Add to that all the pressures and distractions of life, and you’re managing a lot.

Continuous integration and continuous deployment were designed to let teams function better and create a smoother workflow. Generally, they’re used when you’ve got a lot of people working on a product or project to decrease confusion, potential bottlenecks, and the possibility of releasing products with issues. With appreneurs, it’s just you or a small team—will these processes actually help?

Let’s look at what exactly continuous integration and continuous development are, discuss some of the advantages and drawbacks, and then talk about if they are right for you as a solo or small team appreneur.

What is Continuous Integration?

Continuous integration (CI) is the practice of merging development branches that have been verified into a single, shared repository several times a day. CI is used by established and professional businesses across various sectors.

Developers use CI to avoid integration hell, the problem where conflict resolving battles are waged against merging multiple code branches into one shared location.

The concept of CI was adopted into the practice of extreme programming (XP) (which was designed to improve software quality and responsiveness) to be used in combination with automated unit tests that would automatically run in the developer’s local environment before commits to the mainline branch would occur.

A commit is the act of moving your development code from your local repository to the remote repository. Once the code has all passed these automated tests (which are run on build servers), commits to mainline (which is the shared repository) are accepted.

These automated tests are run on build servers (usually set up by people with experience with CI and CD tools like DevOps or test engineers) after a build is successful.

This process helps avoid one developer’s work-in-progress breaking another developer’s features. It also means that developers can be more confident that if the tests fail, then their work won’t make it into the production environment or, worse, into their customers’ hands with bugs.

XP is not usually adopted by solo developers or small teams; it is generally used by larger, professional, established product teams. CI isn’t a process specifically for appreneurs developing apps, but it is a general practice that can be adopted by anyone developing software.

Common Tools for CI

At the time of writing, the most common CI tools are:

  • Jenkins
  • TeamCity
  • Travis CI
  • Go CD
  • Bamboo
  • GitLab CI
  • CircleCI
  • Codeship

For a more detailed explanation about all of the above tools, please see this excellent article on Code Maze.

What Is Continuous Deployment?

Continuous deployment (CD) continually releases updates to the production environment after changes to the product are verified by automated tests at various stages through the development process. It is also used by established and professional businesses across various sectors.

CD tools are used in a chain of events, so when one step in the workflow is successful, the next is kicked off. When that is successful, the next begins. This practice means that no manual intervention is necessary to deploy to production.

The process works step by step until it gets to the production environment. If at any point a step fails, the appropriate people are alerted and fixes are made to ensure the process starts and runs again successfully.

Using CD in conjunction with CI means that development teams should have less chance of releasing products with issues to customers.

Continuous deployment should be the goal of most companies that are not constrained by regulatory or other requirements. Because it automates the process, it won’t be held up by errors or a forgotten step, and it also means that a failed step can’t be missed.

Again, CD isn’t specifically for appreneurs developing apps but can make you more self-sufficient in your deployment process to production.

Common Tools for CD

There are some useful tools that help with automated deployment. These tools are generally used by larger teams, but you can use them as a solo appreneur. Here are some of the most common CD tools:

  • Jenkins
  • ElectricFlow
  • Microsoft Visual Studio
  • Chef
  • Octopus Deploy
  • IBM UrbanCode
  • AWS CodeDeploy
  • DeployBot
  • Shippable
  • TeamCity
  • Codar
  • Distelli
  • XL Deploy
  • GoCD
  • Capistrano

For a more detailed explanation about these tools and others, please see this great list of CD and CI tools by DZone.

Fastlane

Automating other processes can be a big help, too. A tool that that I have found for mobile app developers which automates the app submission process is called Fastlane. Fastlane is open source, so it’s free to use. This tool can be used in conjunction with CI and CD to help push your apps through the submission process to production faster.

As the app submission process is so tiresome, even if you’re only submitting for one platform, it’s worth looking into Fastlane to save you time.

The Benefits and Drawbacks of CI and CD

Generally speaking, CI and CD are considered practices that most companies should use. Since that’s the case, should appreneurs try and implement these concepts, too?

Let’s look at some of the benefits and drawbacks of these practices and see whether it can work for us small timers.

The Benefits

Peace of mind
No need to worry about broken builds getting deployed because, providing you have the right tests in place, the build will fail and it won’t be released.

But if you have tests in place, how might the build still fail? Building software is always an iterative process. We are only human, so minor and even major bugs can slip through to the production environment if the majority of your code is not covered by tests.

The way to prevent the same bugs from reappearing is to write a new test to help identify when this situation will occur. So if the test fails, you will know that this bug is likely to occur again. You can then fix your code to make sure the test passes and the bug never again sees production.

New tests are automatically run
Any tests that you add during your development process are automatically picked up and run. You can also pass different parameters to run version tests in certain conditions.

Allows you to work with the minimum team size
You don’t need a full-time DevOps, release manager, and several testers to ensure your product’s quality level is maintained across environments. Once you put in place the relevant build chains, you can make sure your development and production releases are monitored by the minimum amount of people from different disciplines.

Flexible and trustworthy
If your team expands, you don’t need to worry that they won’t remember to run tests before they perform a commit to the master branch. If their commit fails, it doesn’t break everything for everyone else.

Easy to extend with templates
After you have come up with the perfect configuration for your project, you can create a template to reuse. As your project grows, you can use the template to create new build configurations.

The Drawbacks

Time to set CI/CD up
Once your CI and CD builds are done, you just have to maintain them. But until then, you’ll have to dedicate a bit of time to setting up the initial build configuration templates.

How much time? Well, that’ll all depend on how quickly you learn, how much experience you have with CI and CD tools, and what tools you select. So it’s pretty subjective, based on you.

There will be a learning curve
How steep your learning curve to overcome is will depend on how much experience you have with CI and CD tools.

You have to learn how to set up build configurations, link these to your code repository, write unit tests (if you haven’t been doing this already), integrate the tests into the builds, and possibly also add in a tool to output your level of code coverage after every build. Then once the build is created and passed, you have to learn to link the build’s output (probably the app executable) to the CD tool to push to your next step.

Maintenance
Those new shiny tools will get updated as well, so you’ll need to keep on top of the software updates and licenses. It may not be essential to upgrade to the latest version of everything, but ensure your tools are compatible with each other and that you keep an eye on the new features appearing in new versions.

Can be costly for the more popular tools
Let’s not forget that some of these tools aren’t free and some of them that do offer a free version are capped. So if you want the more costly option, make sure you have enough budgeted for these development costs.

Although the pricier options are sometimes preferred in larger companies because they come with a certain level of support that you don’t get when using free licenses, I find you can get quite far using a free license and having Google at your disposal. I would recommend that you try a free tool before spending money on a paid-for service. After all, using CI and CD tools may not be suitable for you or your project, so keep yourself and your budget in mind.

Builds can be fickle
During a previous role for my day job, when my team was new to TeamCity, we found that our builds were constantly failing. Again, this failure is part of the learning curve; it should also be considered in the time budget you need to put together a robust template. So if you find your builds are failing often, you’ll have to put more time aside to figure out why.

So: Should Appreneurs Use CI and CD?

There are some good points about using CI and CD—mainly that these processes enable an appreneur to remain self-sufficient for longer and makes you confident that your builds are stable and high quality all the way through to production.

But it does take a lot of work up front. And if you don’t have any previous experience with CI and CD tools, it will probably take you longer.

If you’re working with apps as a hobby and don’t have any previous experience with these tools, I wouldn’t recommend setting up a CI or CD process. The time you’ll take to learn the tools and maintain it all may not give you a great return.

But if this is the beginning of building a solid development base and practices for a long-term project, you really don’t have anything to lose.

Start with one process first, preferably CI. You can then build up your test suite to ensure a high level of code coverage (code covered by unit tests) and still control the release process to ensure nothing gets released accidentally while you’re still learning the tools. After you’re confident with your CI builds, pick a complimentary CD tool to help you build your solid foundation.

Anything you can do to make building your app business more efficient and your products more robust and higher quality can only be a good thing.

What Do We Mean By Testing?

Testing is an area of development. A product or service is measured against a number of criteria at different stages of the development cycle. This is measured against the original spec, goal or intended use. Testing can be carried out manually by individuals or automated by computer systems. The automated scripts that run the systems are written and also maintained by individuals.

Why Choose Automation?

It may take longer to set up initially, but automated testing may save a lot of time and money in the long term. There are various arguments for and against automated versus manual testing. It will just depend on the task needed to be carried out and the industry as to which method is more suitable.

When I was at university not a lot of emphasis was placed on testing and the roles within this field. More attention was on the career path of the developer. But, the roles within testing can be as technical as that of the developer and just as important.

Depending on the company you work in, the job of a developer and tester may blur. You may need skills useful for either role in order to do your job effectively.

What Qualities Make You A Good Tester?

These aren’t all the qualities that you need to be a good tester but they’re certainly highly important.

  • The ability to communicate efficiently and effectively

This includes between technical (developers and other testers) and non-technical (business) members of your team. Both written and verbal communication skills are important to improve and perfect.

  • High attention to detail

You should be able to pick up minor issues as well as the glaringly obvious ones and be able to reproduce them correctly. The more issues you find and help fix, the higher level of quality the product you ship has.

  • Being able to pick up new software and languages quickly

Like most areas in IT, the tools that testers use are updated and change very quickly. As well as the tools you will use on a daily basis you may need to be able to use the tools that the developers in your team utilise.

Why Specialise In Testing?

The field of testing is extremely varied. As I mentioned, an automation role in testing can be just as technical as a role as a developer. But if you’re not into technical jobs you can still be a valuable manual tester. Manual testers frequently carry out exploratory testing which is extremely essential and something that automation can’t replicate.

As testing is the last stage in development before the product is released to the customer or to production, testers are responsible for identifying issues that could impact the reliability and use of the product. When these issues are found testers work with developers to get the issue fixed and retested to ensure the problem is solved and that nothing else is broken. Testers are the gatekeepers of quality.

Is Testing For You?

If you enjoy ensuring that something is of the highest level of quality, testing could be for you. You need to be willing to carry out some tasks that may be long and repetitive. However, others may give you a unique perspective into the development cycle and could expose you to different fields. For example, you could get more exposure to business analysis when you’re trying to determine the expected results for test cases. Or, exposure to software development when you’re writing automation or other technical tests if you’re utilising test-driven development or behaviour-driven development.

Ultimately, it’s a very satisfying role knowing that you’re the one upholding your brand’s image by ensuring a high level of quality.