The world of test automation capability

Imagine a traditional capability matrix. A spreadsheet where each column is skill, each row is the name of an individual, and then the map is populated by people who rate their competence. These spreadsheets are useful for seeing trends within a team, both strengths and opportunities for improvement, but there are limitations.

A matrix captures state, but not journeys. It shows skills, but not always how the skills are applied. It lists what is important in a specific domain, but not how other people are using different skills to solve the same problem. The limitations of the matrix can stifle thinking about capability where evolution, application, and diversity of skill are important.

We could try to capture this information in our spreadsheet. Telling these stories by colour-coding or cross-referencing multiple tabs. Or we could think about capability from another angle.

The world of test automation capability is a model that illustrates the skills and experience of a test team using layers of the earth: core, mantle and crust. At the core are the testers with the least knowledge about coding and test automation frameworks. Paths to the surface travel through the mantle of programming languages towards the crust of applying coding skills.

Divisions exist within the layers. The mantle of programming languages might include Java, JavaScript, Visual Basic, Ruby, and Swift. The crust of test automation frameworks might include suites for web-based user interfaces, iOS and Android mobile applications, desktop applications, and APIs. Splitting each layer of the model proportionally allows us to see relative strengths and weaknesses in capability and tooling.

Divisions within the mantle and crust can be shared by different teams. In my organisation testers are split across four different departments. There is some overlap in the programming languages and automation frameworks in use across these different areas. In forming a strategy for training and framework evolution, we want to consider the distribution of testers within this model.

Part of the test coaching function in my organisation is to formalise paths between layers of the world, to support testers to learn a programming language and apply this knowledge to implementation of a test automation framework. These paths are created through training courses, coding dojos and knowledge sharing opportunities. Informal paths will exist where testers are self-taught or prototyping their own frameworks.

To illustrate, our current world of test automation capability can be visualised as:

The model shows testers at the core who have no coding or framework experience. In the mantle, the strongest programming language is Java, though capability also exists in four other languages. Our division in programming language is usually by department, with different areas adopting different solutions. At the crust, the largest population are working on WebDriver Frameworks, which includes testers who work on web-based applications across all areas of the organisation. The crust also includes API frameworks as an emerging area, UFT as a legacy framework choice, and the mobile teams who have specialised tooling in xCode and Espresso.

The world of test automation capability is a snapshot at a given point in time. Divisions within the model will shift as programming languages and automation frameworks evolve. People move through the model as their knowledge and experience develops. Paths will change as learning needs transform.

It is quite a different snapshot to a matrix though. It is one that includes relationships between training and skill level, between skills and frameworks, and between people across different departments. The evolution, application, and diversity of skill is easier to see.

Our Test Automation Strategy is formed within the context of this world, describing the forces that the Test Coaches will apply to the model to facilitate and direct movement. Our strategy differs by department, applying a different lens on the overarching visual.

Though how we rate our skill relative to others is useful, there is rich narrative in mapping skills within an environment then talking about how that environment is changing. The world of test automation capability is a new perspective that creates an opportunity for a new story.