SCORING YOUR TEAM FOR QA QUALITY

Last week some colleagues and I attended the National Software Testing Conference held at the De Vere Grand Connaught Rooms in central London. I hadn’t attended this conference since 2019 when I first spoke here so I was keen to go back. The calibre of topics when I attended in 2019 were far from beginner level and I came away with a wealth of new knowledge and ideas from the topics these industry leaders were sharing. This year was equally beneficial.

One talk in particular by Simon Amey was called Measuring QA quality: What do your teams score? In this short talk, he outline a framework that could be used to measure the level of quality in your development teams.

At the end of this simple assessment, your team will have a picture of where you are in terms of the absolute highest level. Once you have your current level, you can then lay out steps to take to go from where you are to where you want to be.

What did I find useful?

What I liked about this framework and the delivery of this talk overall was that it:

• quickly explained key points in all levels
• the information for this talk is immediately implementable within teams
• results are fast and simple to obtain

What’s the point of all this anyway?

By following this framework, it will hopefully help you to:

1. Improve the quality of your code
2. Gain faster feedback (during development and from the end users)
3. Get the best product to your customers

Please remember, that while working with a framework like this is good, metrics are not the be all and end all. Don’t focus blindly on metrics. The biggest question to keep asking is “is this delivering value to the customer?” and to keep making moves in order to delight them.

Team Quality Dimensions Framework

There are five dimensions to measure your team against:

1. Requirement capture
2. Process
3. QA Effectiveness
4. Test platform and tools
5. Deployment

Note: There are no levels which cover observability or monitoring but these can be added in with similar levels to the other dimensions.

Under each dimension, there are 6 different levels of adoption that range from 1 to 6. 1 is the very basic level and 6 is the superior level. You score yourself at the number which best represents your team’s current position in that dimension.

• Requirement capture
  1. Ad hoc coding/no documented requirements
  2. Headline feature descriptions only
  3. Key requirements documented/Error cases and non-functional requirements not covered
  4. Detailed requirement document written and reviewed
  5. Requirements cover all changes in details and updated with ongoing feedback
  6. Requirements cover all changes in detail with regular, detailed feedback from internal stakeholders and customers
• Process
  1. Ad hoc coding/little planning or documentation
  2. Tasks documented with little forward planning or scoping/feedback is mainly reporting issues
  3. Sprint planning in place with timescales for dev work, but not testing/no versioned builds available
  4. All agile processes in place but actions not tracked/Time for dev work and testing/gets ad hoc proactive feedback from projects, support and operations/ versioned builds available
  5. Agile processes practiced and enforced including improvement feedback
  6. Full short- and long-term plans in place with regular detailed feedback from projects, test, support and operations
• QA Effectiveness
  1. New starter knowledge only/no QA resource
  2. User-level knowledge/test plan run not run yet
  3. Experienced tester/test plan run at least once
  4. Good system knowledge/knows system architecture, log locations and contents etc
  5. Expert tester/knows unit, integration and e2e tests, product history and problematic areas
  6. Dev crossover/can find and fix simple issues on their own
• Test platform and tools
  1. No test platform (dev or live only)
  2. Shared test platform/little tooling available/little automation
  3. Dedicated test platform/little tooling available/little automation
  4. Can generate most relevant data/some automation
  5. Comprehensive data generation and tools available
  6. Detailed automation at unit, integration and e2e levels with full tooling on a dedicated, representative system
• Deployment
  1. Builds and upgrades have multiple, complex, manual step/No official build environment/Configuration is not version controlled
  2. Builds and upgrades have multiple, complex, manual steps/Dedicated build environment/Configuration is not version controlled
  3. Automated builds (CI); deployment requires manual steps/Configuration is not version controlled
  4. Standard builds and deployments to test systems are automated (CI/CD)/Special cases and live releases require manual intervention/Configuration is version controlled
  5. CI/CD system covering build, test, SAST and deployments in place for standard releases to test and live environments/Configuration is version controlled
  6. Comprehensive and reliable CI/CD system with detailed feedback covering code and configuration deploys automatically to test and live environments

Recommendations after you take the assessment

1. Get your total. Simon recommends either just taking the value of the level and adding each up or multiplying them together (I guess there’s no hard and fast rule yet). After this, compare your score to the total and see where you are. I would suggest to do this exercise every 3 – 6 months to check your progression.
2. To measure all teams in your company, create a survey from these questions e.g. in a Google Form. Then create visualisations from the metrics gathered. Every time the team does a subsequent measurement, you can do a comparison against your previous score.
3. Have someone in the team responsible for making sure that you are moving forward in your steps (it’s no good having metrics if you’re not utilising them).

Lastly…

Thanks to Simon for a great talk. And as a further thank you, here’s a link to his book (it’s also below). I hope you find this framework useful!