3 ways to define your role without a RACI matrix

I lead a team of Test Coaches, which is still a relatively unusual role in our industry. As a Test Coach I was part of a number of conversations about my role. As we scaled the team of coaches and I took a leadership position, these conversations increased in frequency and audience.

Uncertainty about roles can happen when there are:

• new roles created, or 
• multiple people performing the same role in different areas of an organisation, or 
• a number of roles that interact closely with a shared purpose.

In all three situations there are alternatives to drawing up a traditional RACI matrix, which can feel like marking territory when executed poorly. I prefer collaborative approaches to role definition. This article explains how to use an interactive activity with practical scenarios, a role expectation mapping canvas, and a rainbow roles overview.

Interactive activity with practical scenarios

When our test coaching function grew there were questions about how we would operate. Some people had preconceptions about our role being similar to test management. Others had concerns that we might overstep boundaries toward people leadership or setting technical direction.

To kick-off our expanded services we scheduled a session for all the testers in the organisation, along with their line managers and delivery leadership, to explain test coaching. Attended by over 120 people, part of the agenda was an interactive activity to clarify and define our role.

We asked the audience to pair up with someone that they wouldn't normally work with, preferably someone from another department. We gave each pair a questionnaire that listed 13 scenarios and asked them to consider whether or not they would involve a Test Coach in each situation.

The scenarios in the activity were descriptive and practical. They were framed in the first person to encourage people to imagine themselves in each situation. Not just "bugs are being ignored", but "my bugs are being ignored".

Questionnaire for Test Coach role

Where a pair disagreed, they were allowed to record a two responses, however we encouraged discussion of differing opinions in the hope that a single answer could be found.

Once this task was complete, we handed out sets of coloured paper squares to each pair. Each set included green, yellow, and pink sheets to represent yes, maybe, and no respectively.

We used a set of slides to work through the questionnaire. Each question was displayed on a slide with a blue background, a neutral colour. I read out the question and each pair in the audience had to hold up a coloured card that corresponded to their answer. The card was green if they would engage a Test Coach, pink if they would not, and yellow if they were unsure.

Asking each pair to vote kept the audience engaged, and helped us to understand where there was confusion about our role. Where the responses were mixed, or where they consistently differed to what we expected the response to be, we gained insight into situations where there might be overlap with other roles or where we might have to clarify the intent of our involvement.

After each vote, the next slide had the same question with a background colour that revealed the answer: green, yellow or pink. I provided a short explanation of our rationale for involvement in each situation and gave the audience the opportunity to ask any questions that they had.

Slides that explain when to ask a Test Coach

This approach to defining our role was well received and created a clear set of expectations about how to interact with the new Test Coaches.

I have since applied the same type of exercise in a delivery tribe with five leadership roles. Both the team and the leaders were uncertain of who should be driving the outcome in some instances. I adapted the activity so that the columns of the questionnaire represented each role.

Questionnaire for support roles in agile delivery

The audience split into pairs who discussed a set of scenarios, choosing which role they would expect an action from in each situation. We used five different coloured cards for the audience to share their thoughts, one colour per role. With more roles in scope and a larger audience, there was robust conversation about some of these scenarios and the activity was longer than when it was focused on a single role.

Role expectation mapping canvas

The test coaches in my organisation are part of a wider team that includes technical and practices coaching. As the team grew there was confusion, particularly from newer coaches, about our responsibilities. During a team offsite day, we split into our specialist coaching disciplines for an activity to discuss this.

My manager facilitated the activity using the role expectation mapping canvas developed by Tony O'Halloran based on ideas from Jimmy Janlen's description of role expectation mapping. Each subset of coaches in the team completed the activity together i.e. all of the Test Coaches worked on a single canvas.

The canvas is separated into five parts:

1. Roles that you work with
2. Externally visible signs of success
3. Things that you are responsible for
4. Expectations of the people that you interact with
5. Challenges that you face

Role expectation mapping canvas

As the manager of the test coaching function and the longest serving test coach, I decided to act as the scribe for this activity. I refrained from contributing to the conversation and instead focused on recording it. This was difficult, but my silence created space for the other test coaches to express their ideas, opinions and doubts.

As each test coach worked in a different neighbourhood the purpose of the activity was to understand what was common. Across each department in our organisation there are consistent stakeholder roles, indicators of success, responsibilities, expectations and challenges. The canvas was useful to focus conversation and capture these, to create clarity and alignment in our function.

This canvas was designed to be applied in a delivery team to clarify relationships between different roles. Though I am yet to use it in that context, I can see how it might be successful.

Rainbow roles

A tribe that I interact with as a Test Coach has other test leadership roles at a tribe and team level, along with delivery managers who have an interest in quality. This environment had created confusion in the test chapter of the tribe and conflict in direction for testing.

To resolve this, a meeting was scheduled for the people directly involved in test leadership. The facilitator drove the discussion towards a RACI matrix, which was distributed to the participants of the meeting after the session.

In this context the RACI matrix was usefully divisive for the leadership audience. It clearly showed which role was responsible for which task. However I felt that we needed to present something else to the rest of the tribe. Something that illustrated how we had agreed to work collaboratively towards a shared purpose, rather than defining our separate territories.

I converted the information from the RACI matrix into a one-page overview with five streams that showed how the test leadership team would work together on common testing goals: strategy, standards, day-to-day support, recruitment, and personal development.

Rainbow Roles

It isn't miles away from the original, but the information is presented in simple language and intentionally illustrates our different contributions towards a single result. The rainbow colours started as a joke to further emphasize our aspirations of harmony, but became core to the identity of this diagram: rainbow roles.

It is not unusual for roles to become murky as the organisation evolves. In each example we found clarity through collaboration using tools that emphasized how different people work together. I encourage you to treat confusion or conflict as an opportunity to positively engage with others about your role and theirs.

For another take on this topic, you might also want to read Riot's agile team leadership model: A story of challenging convention by Ahmed Sidky and Tricia Broderick.

The world of test automation capability

Imagine a traditional capability matrix. A spreadsheet where each column is skill, each row is the name of an individual, and then the map is populated by people who rate their competence. These spreadsheets are useful for seeing trends within a team, both strengths and opportunities for improvement, but there are limitations.

A matrix captures state, but not journeys. It shows skills, but not always how the skills are applied. It lists what is important in a specific domain, but not how other people are using different skills to solve the same problem. The limitations of the matrix can stifle thinking about capability where evolution, application, and diversity of skill are important.

We could try to capture this information in our spreadsheet. Telling these stories by colour-coding or cross-referencing multiple tabs. Or we could think about capability from another angle.

The world of test automation capability is a model that illustrates the skills and experience of a test team using layers of the earth: core, mantle and crust. At the core are the testers with the least knowledge about coding and test automation frameworks. Paths to the surface travel through the mantle of programming languages towards the crust of applying coding skills.

Divisions exist within the layers. The mantle of programming languages might include Java, JavaScript, Visual Basic, Ruby, and Swift. The crust of test automation frameworks might include suites for web-based user interfaces, iOS and Android mobile applications, desktop applications, and APIs. Splitting each layer of the model proportionally allows us to see relative strengths and weaknesses in capability and tooling.

Divisions within the mantle and crust can be shared by different teams. In my organisation testers are split across four different departments. There is some overlap in the programming languages and automation frameworks in use across these different areas. In forming a strategy for training and framework evolution, we want to consider the distribution of testers within this model.

Part of the test coaching function in my organisation is to formalise paths between layers of the world, to support testers to learn a programming language and apply this knowledge to implementation of a test automation framework. These paths are created through training courses, coding dojos and knowledge sharing opportunities. Informal paths will exist where testers are self-taught or prototyping their own frameworks.

To illustrate, our current world of test automation capability can be visualised as:

The model shows testers at the core who have no coding or framework experience. In the mantle, the strongest programming language is Java, though capability also exists in four other languages. Our division in programming language is usually by department, with different areas adopting different solutions. At the crust, the largest population are working on WebDriver Frameworks, which includes testers who work on web-based applications across all areas of the organisation. The crust also includes API frameworks as an emerging area, UFT as a legacy framework choice, and the mobile teams who have specialised tooling in xCode and Espresso.

The world of test automation capability is a snapshot at a given point in time. Divisions within the model will shift as programming languages and automation frameworks evolve. People move through the model as their knowledge and experience develops. Paths will change as learning needs transform.

It is quite a different snapshot to a matrix though. It is one that includes relationships between training and skill level, between skills and frameworks, and between people across different departments. The evolution, application, and diversity of skill is easier to see.

Our Test Automation Strategy is formed within the context of this world, describing the forces that the Test Coaches will apply to the model to facilitate and direct movement. Our strategy differs by department, applying a different lens on the overarching visual.

Though how we rate our skill relative to others is useful, there is rich narrative in mapping skills within an environment then talking about how that environment is changing. The world of test automation capability is a new perspective that creates an opportunity for a new story.

9 quick ideas for flexible testing

When you're a tester in an agile team, it can be easy to fall into a comfortable testing pattern. The transparency of daily stand-up and reflection of retrospectives can create an illusion of continuous improvement. These routines make us feel that we work in a flexible way but, if we dig a little deeper, we may not be as adaptable as we think.

If you think back to the last time that you felt uncomfortable at work, there's a strong probability that this feeling was associated with a change that you were experiencing. A flexible approach means that you are willing to accept and adopt change regularly, which means that you routinely experience discomfort.

When was the last time that you were surprised by the outcome of your retrospective or quizzed by a colleague in your stand-up? When was the last time that a stakeholder asked questions about your test artifacts? If you can't remember being challenged recently then you, and your team, might be stuck.

Being flexible is not just about activities or outcomes. Imagine that you used to plan your testing in an Excel spreadsheet and now you capture test ideas in a mind map. Does this make you a flexible tester? Not necessarily.

To be a versatile thinker you need to regularly inspect your own habits and create opportunities to collaborate with different people. If you cultivate flexibility as an attitude and make it part of the way that you work, you'll become more aware of how you think. You can change what you deliver, but a flexible tester will also challenge how they deliver.

How can you do that?

Here are nine quick, practical ideas that may help you develop your flexibility as a tester:

1. Change the order of your test approach to break a routine.
2. Ask for advice from a non-tester on how to diagnose a bug.
3. Actively seek test ideas from non-testers outside your agile team e.g. UX, Ops.
4. Copy the format of a colleague's test report to see your own results in a new light.
5. Pair with a tester in another team to see a different test approach first-hand.
6. Invite someone else to test your product, then debrief about what they found.
7. Experiment with a tool that you haven't tried before.
8. Take a second look at something that you thought was a bad suggestion.
9. Ask for constructive feedback about your testing.

Being in an agile team does not guarantee that you are behaving in an agile way. Try to develop the habits that cultivate flexibility, so that you continue to learn and your testing continues to evolve.

No unit tests? No problem!

A couple of weeks ago I created a Twitter poll about unit tests that asked:

"Is code without unit tests inherently bad code?" 

Is code without unit tests inherently bad code?

— Katrina Clokie (@katrina_tester) May 1, 2018

The conversations that emerged covered a number of interesting points, which challenged some of my assumptions about unit tests and how we evaluate code.

What is bad code?

When I framed my original question, I deliberately chose the phrase "inherently bad code". I was trying to emphasize that the code would be objectively bad. That the absence of unit tests would be a definitive sign, one of a set of impartial measures for assessing code.

In my organisation, most of our agile development teams include unit tests in their Definition of Done. Agile practitioners define a Definition of Done to understand what is required for a piece of work to be completed to an acceptable level of quality. In this context, the absence of unit tests is something that the agile development team have agreed would be bad.

A Definition of Done may seem like an unbiased measure, but it is still a list that is collectively agreed by a team of people. The code that they create isn't good or bad in isolation. It is labeled as good or bad based on the criteria that this group have agreed will define good or bad for them. The bad code of one team may be the good code of another, where the Definition of Done criteria differs between each.

Is the code inherently bad when it doesn't do what the end user wanted? Not necessarily. What if it the unexpected is still useful? There are a number of famous products that were originally intended for a completely different purpose e.g. bubble wrap was originally marketed as wallpaper [Ref].

I believe there is no such thing as inherently bad code. It is important to understand how the people who are interacting with your code will judge its value.

Why choose to unit test?

Many people include unit testing in their test strategy as a default, without thinking much about what type of information the tests provide, the practices used to create them, or risks that they mitigate.

Unit tests are usually written by the same developer who is writing the code. They may be written prior to the code, in a test driven development approach, or after the code. Unit tests define how the developer expects the code to behave by coding the "known knowns" or "things we are aware of and understand" [Ref].

By writing unit tests the developer has to think carefully about what their code should do. Unit tests catch obvious problems with an immediate feedback loop to the developer, by running the tests locally and through build pipelines. If the developer discovers issues and resolves them as the code is being created, this offers opportunities for other people to discover unexpected or interesting problems via other forms of testing.

Where there are different types of automated testing, across integration points or through the user interface, unit tests offer an opportunity to exercise a piece of functionality at the source. This is especially useful when testing a function that behaves differently as data varies. Rather than running all of these variations through the larger tests, you may be able to implement these checks at a unit level.

Unit tests require the developer to structure their code so that it is testable. These implementation patterns create code that is more robust and easier to maintain. Where a production problem requires refactoring of existing code, the presence of unit tests can make this a much quicker process by providing feedback that the code is still behaving as expected.

The existence of unit tests does not guarantee these benefits. It is entirely possible to have a lot of unit tests that add little value. The developer may have misunderstood how to implement the tests, worked in isolation, or designed their test coverage poorly. The merit of unit tests is often dependent on team culture and other collaborative development practices.

No unit tests? No problem!

Though there are some solid arguments for writing unit tests, their absence isn't always a red flag. In some situations we can realise the benefits of unit testing through other tools.

Clean implementation patterns that make code easier to maintain may be enforced by static analysis tools. These require code to follow a particular format and set of conventions, rejecting anything that deviates from the agreed norm before it is committed to the code base. These tools can even detect some of the same functional issues as unit tests.

Rather than writing unit tests to capture known behaviour, you may choose to push this testing up into an integration layer. Where the data between dependent systems includes a lot of variation, shifting the tests can help to examine that the relationship is correct rather than focusing on the individual components. There is a trade-off in complexity and time to execution, but simple integrated tests can still provide fast feedback to the developers in a similar fashion to unit testing.

When dealing with legacy code that doesn't include unit tests, trying to retrofit this type of testing may not be worth the effort. Similarly if the code is unlikely to change in the future, the effort to implement unit tests might not provide a return through easy maintainability, as maintenance will not be required.

There may be a correlation between unit tests and code quality, but one doesn't cause the other. "Just because two trends seem to fluctuate in tandem ... that doesn’t prove that they are meaningfully related to one another" [Ref].

How do you choose a test automation tool?

When you’ve agreed what you want to automate, the next thing you’ll need to do is choose a tool. As a tester most of the conversations I observed from a distance, between managers and people selecting a tool, focused on only one aspect.

Cost.

Managers do care about money, don’t get me wrong. But choosing a tool based on cost alone is foolish, and I believe that most managers will recognise this.

Instead of making cost the primary decision making attribute, where possible defer the question of cost until you’re asking for investment. Your role as the testing expert in a conversation about choosing a tool is to advocate for the other aspects of the tool, beyond cost, that are important to consider.

Support

You want to choose a tool that is well supported so that you know help is available if you encounter problems.

If you’re purchasing a tool from a vendor, is it formally supported? Will your organisation have a contractual arrangement with the tool vendor in the event that something goes wrong? What type of response time can you expect when you encounter issues, or have questions? Is the support offered within your time zone? In New Zealand, there is rarely support offered within our working hours, which makes every issue an overnight feedback loop. This has a big impact in a fast-paced delivery environment.

If it’s an open source tool, or a popular vendor tool, how is it supported by documentation and the user community? When you search for information about the tool online, do you see a lot of posts? Are they from people across the world? Are their posts in your language? Have questions posted in forums or discussion groups been responded to? Does the provided documentation look useful? Can you gauge whether there are people operating at an expert level within the toolset, or is everything you see online about people experimenting or encountering problems at an entry level?

The other aspect of support is to consider how often you’ll need it. If the tool is well designed, hopefully it’s relatively intuitive to use. A lack of online community may mean that the tool itself is of a quality where people don’t need to seek assistance beyond it. They know what to do. They can reach an expert level using the resources provided with the tool.

Longevity

How long has the tool been around? There’s no right or wrong with longevity. If you’re extending a legacy application you may require a similarly aged test tool. If you’re working in a new JavaScript framework you may require a test tool that’s still in beta. But it’s important to go into either situation with eyes open about the history of the tool and it’s possible future.

Longevity is not just about the first release, but how the tool has been maintained and updated. How often is the tool itself changing? When did the last update occur? Is it still under active development? As a general rule, you probably don’t want to pick a tool that isn’t evolving to test a technology that is.

Integration

Integration is a broad term and there are different things to consider here.

How does the tool integrate with the other parts of your technology? Depending on what you’re going to test, this may or may not be important. In my organisation, our web services automation usually sits in a separate code base to our web applications, but our selenium-based browser tests sit in the same code base as the product. This means that it doesn’t really matter what technology our web services automation uses, but the implementation of our selenium-based suite must remain compatible with the decisions and direction of developers and architects.

What skills are required to use the tool and do they align with the skills in your organisation? If you have existing frameworks for the same type of testing that use a different tool, consider whether a divergent solution really makes sense. Pause to evaluate how the tool might integrate with the people in your organisation, including training and the impact of shifting expectations, before you look further afield.

Integration is also about whether the test tool will integrate with the development environment or software development process that the team use. If you want the developers in your team to get involved in your test automation, then this is a really important factor to consider when choosing a tool. The smaller the context switch for them to contribute to test code, the better. If they can use the same code repository, the same IDE development environment on their machine, and their existing skills in a particular coding language, then you’re much more likely to succeed in getting them active in your solution. Similarly, if the tool can support multiple people working on tests at the same time, do clean merging from multiple authors, offer a mechanism for review or feedback, these are all useful points to consider related to integration of the tool to your team.

Analyse

In a conversation with management about choosing a tool, your success comes from your ability to articulate why a particular tool is better than another, not just in it’s technical solution. Demonstrate that you’ve thought broadly about the implications of your choice, and how it will impact on your organisation now and in the future.

It’s worth spending time to prepare for a conversation about tools. Look at all the options available in the bucket of possible tools and evaluate them broadly. Make cost the lesser concern, by speaking passionately about the benefits of your chosen solution in terms of support, longevity and integration, along with any other aspects that may be a consideration in your environment.

You may not always get the tool that you’d like, but a decision that has been made by a manager based on information you’ve shared, that has come from a well-thought through analysis of the options available, is a lot easier to accept than a decision made blindly or solely based on cost.

Setting strategy in a Test Practice

Part of my role is figuring out where to lead testing within my organisation. When thinking strategically about testing I consider:

• how testing is influenced by other activities and strategies in the organisation,
• where our competitors and the wider industry are heading, and
• what the testers believe to be important.

I prefer to seek answers to these questions collaboratively rather than independently and, having recently completed a round of consultation and reflection, I thought it was a good opportunity to share my approach.

Consultation

In late March I facilitated a series of sessions for the testers in my organisation. These were opt-in, small group discussions, each with a collection of testers from different parts of the organisation.

The sessions were promoted in early March with a clear agenda. I wanted people to understand exactly what I would ask so that they could prepare their thoughts prior to the session if they preferred. While primarily an information gathering exercise, I also listed the outcomes for the testers in attending the sessions and explained how their feedback would be used.

***

AGENDA

Introductions

Round table introduction to share your name, your team, whereabouts in the organisation you work

Transformation Talking Points

8 minute time-boxed discussion on each of the following questions:

• What is your biggest challenge as a tester right now?
• What has changed in your test approach over the past 12 months?
• How are the expectations of your team and your stakeholders changing?
• What worries you about how you are being asked to work differently in the future?
• What have you learned in the past 12 months to prepare for the future?
• How do you see our competitors and industry evolving?

  

If you attend, you have the opportunity to connect with testing colleagues outside your usual sphere, learn a little about the different delivery environments within <organisation>, discover how our testing challenges align or differ, understand what the future might look like in your team, and share your concerns about preparation for that future. The Test Coaches will be taking notes through the sessions to help shape the support that we provide over the next quarter and beyond.

***

The format worked well to keep the discussion flowing. The first three questions targeted current state and recent evolution, to focus testers on present and past. The second three questions targeted future thinking, to focus testers on their contribution to the continuous changes in our workplace and through the wider industry. Every session completed on time, though some groups had more of a challenge with the 8 minute limit than others!

Just over 50 testers opted to participate, which is roughly 50% of our Test Practice. There were volunteers from every delivery area that included testing, which meant that I could create the cross-team grouping within each session as I intended. There were ten sessions in total. Each ran with the same agenda, a maximum of six testers, and two Test Coaches.

Reflection

From ten hours of sessions with over 50 people, there were a lot of notes. The second phase of this exercise was turning this raw input into a set of themes. I used a series of questions to guide my actions and prompt targeted thinking.

What did I hear? I browsed all of the session notes and pulled out general themes. I needed to read through the notes several times to feel comfortable that I had included every piece of feedback in the summary.

What did I hear that I can contribute to? With open questions, you create an opportunity for open answers. I filtered the themes to those relevant to action from the Test Practice, and removed anything that I felt was beyond the boundaries of our responsibilities or that we were unable to influence.

What didn't I hear? The Test Coaches regularly seek feedback from the testers through one-on-one sessions or surveys. I was curious about what had come up in previous rounds of feedback that wasn't heard in this round. This reflected success in our past activities or indicated changes elsewhere in the organisation that should influence our strategy too.

How did the audience skew this? Because the sessions were opt-in, I used my map of the entire Test Practice to consider whose views were missing from the aggregated summary. There were particular teams who were represented by individuals and, in some instances, we may seek a broader set of opinions from that area. I'd also like to seek non-tester opinion, as in parts of the organisation there is shared ownership of quality and testing by non-testers that makes a wider view important.

How did the questions skew this? You only get answers to the questions that you ask. I attempted to consider what I didn't hear by asking only these six questions, but I found that the other Test Coaches, who didn't write the questions themselves, were much better at identifying the gaps.

From this reflection I ended up with a set of about ten themes that will influence our strategy. The themes will be present in the short-term outcomes that we seek to achieve, and the long-term vision that we are aiming towards. The volume of feedback against each theme, along with work currently in progress, will influence how our work is prioritised.

I found this whole process energising. It refreshed my perspective and reset my focus as a leader. I'm looking forward to developing clear actions with the Test Coach team and seeing more changes across the Test Practice as a result.

How do you decide what to automate?

When you start to think about a new automated test suite, the first conversations you have will focus on what you're going to automate. Whether it's your manager that is requesting automation or you're advocating for it yourself, you need to set a strategy for test coverage before you choose a tool.

There are a lot of factors that contribute to the decision of what to automate. If you try to decide your scope in isolation, you may get it wrong. Here are some prompts to help you think broadly and engage a wide audience.

Product Strategy

The strategy for the product under test can heavily influence the scope of the associated test automation solution.

Imagine that you're part of a team who are releasing a prototype for fast feedback from the market. The next iteration of the product is likely to be significantly different to the prototype that you're working on now. There is probably a strong emphasis on speed to release.

Imagine that you're part of a team who are working on the first release version of a product that is expected to become a flagship for your organisation over the next 5 - 10 years. The product will evolve, but in contrast to the first team it may be at a steadier pace and from a more solid foundation. There is probably a strong emphasis on technical design for future growth.

Imagine that you're part of a team who will deliver the first product in a line of new products that will all use the same technology and software development process. An example may be a program of work that switches legacy applications to a new infrastructure. The goal may be to shift each piece in a like-for-like manner. There is probably a strong emphasis on standard practices and common implementation.

What you automate and the way that you approach automation will vary in each case.

You need to think about, and ask questions about, what's happening at a high-level for your product so that you can align your test automation strategy appropriately. You probably won't invest heavily in framework design for a prototype. Your automation may be shallow, quick and dirty. The same approach would be inappropriate in other contexts.

At a slightly lower level of product strategy, which features in your backlog are most important to the business? Are they also the most important to your customers? There may be a list of features that drive sales and a different set that drive customer loyalty. Your automation may target both areas or focus only on the areas of the application that are most important to one set of stakeholders.

Can you map the upcoming features to your product to determine where change might happen in your code base? This can help you prepare appropriate regression coverage to focus on areas where there is a high amount of change.

Risk Assessment

When testers talk about the benefits of automation, we often focus on the number of defects it finds or the time that we’re saving in development. We describe the benefits in a way that matters to us. This may not be the right language to talk about the benefits of automation with a manager.

In the 2016 World Quality Report the number one reason that managers invest in testing is to protect the corporate image. Testing is a form of reputation insurance. It’s a way to mitigate risk.

When you’re deciding what to automate, you need to think about the management perspective of your underlying purpose and spend your effort in automation in a way that directly considers risk to the organisation. To do this, you need to engage people in a conversation about what those risks are.

There are a lot of resources that can help support you in a conversation about risk. James Bach has a paper titled Heuristic Risk-Based Testing that describes an approach to risk assessment and offers common questions to ask. I've used this as a resource to help target my own conversations about risk.

Both the strategy and the risk conversation are removed from the implementation detail of automation, but they set the scene for what you will do technically and your ongoing relationships with management. These conversations help you to establish credibility and build trust with your stakeholders before you consider the specifics of your test automation solution. Building a strong foundation in these working relationships will make future conversations easier.

How do you consider what to automate at a more technical level?

Objective Criteria

As a general rule, start by automating a set of objective statements about your application that address the highest risk areas.

Objective measures are those that can be assessed without being influenced by personal feelings or opinion. They’re factual. The kind of measure that you can imagine in a checklist, where each item is passed or failed. For example, on a construction site a hard hat must be worn. When you look at a construction worker it is easy to determine whether they are wearing a hat or not.

On the flip side, subjective measures are where we find it difficult to articulate the exact criteria that we’re using to make a decision. Imagine standing in front of a completed construction project and trying to determine whether it should win the “Home of the Year”. Some of the decision making in this process is subjective.

If you don’t know exactly how you’re assessing your product, then a tool is unlikely to know how to assess the product. Automation in a subjective area may be useful to assemble a set of information for a person to evaluate, but it can’t give you a definitive result like an objective check will do.

A common practice to support conversations about objective criteria is the gherkin syntax for creating examples using Given, When, Then. A team that develop these scenarios collaboratively through BDD, or Three Amigos, etc. are likely to end up with a more robust set of automation than a person who identifies scenarios independently.

Repetitive Tasks

Repetition can indicate an opportunity for automation, both repetition in our product code and repetition in our test activities.

In the product, you might have widgets that are reused throughout an application e.g. a paginated table. If you automate a check that assess that widget in one place, you can probably reuse parts of the test code to assess the same widget in other places. This creates confidence in the existing use of the widgets, and offers a quick way to check behaviour when that same widget is used in future.

In our testing, you might perform repetitive data setup, execute the same tests against many active versions of a product, or repeat tests to determine whether your web application can be used on multiple browsers. These situations offer an opportunity to develop automation that can be reused for many scenarios.

By contrast, if we have a product that is unique, or an activity that’s a one-off, then we may not want to automate an evaluation that will only happen once. There are always exceptions, but as a general rule we want to seek out repetition.

There is benefit to collaboration here too, particularly in identifying repetitive process. Your colleagues may perform a slightly different set of tasks to what you do, which could mean that you're unaware of opportunities to automate within their area. Start conversations to discover these differences.

Opportunity Cost

When you automate it’s at the expense of something else that you could be doing. In 1998 Brian Marick wrote a paper titled When should a test be automated?. In his summary at the conclusion of the paper he says:

“The cost of automating a test is best measured by the number of manual tests it prevents you from running and the bugs it will therefore cause you to miss.”

I agree this is an important consideration in deciding what to automate and believe it is often overlooked.

When you decide what to automate, it’s important to discuss what proportion of time you should spend in automating and the opportunity cost of what you're giving up. Agreeing on how long you can spend in developing, maintaining, and monitoring your automation will influence the scope of your solution.

Organisational Change

So far we've considered five factors when deciding what to automate: product strategy, risk, objectivity, repetition, and opportunity cost. All five of these will change.

Product strategy will evolve as the market changes. Risks will appear and disappear. As the features of the product change, what is objective and repetitive in its evaluation will evolve. The time you have available, and your opportunity costs, will shift.

When deciding what to automate, you need to consider the pace and nature of your change.

Is change in your organisation predictable, like the lifecycle of a fish? You know what steps the change will take, and the cadence of it. In 2 weeks, this will happen. In 6 months, that will happen.

Or is change in your organisation unpredictable, like the weather? You might think that rain is coming, but you don’t know exactly how heavy it will be or when it will happen, and actually it might not rain at all.

The pace and nature of change may influence the flexibility of your scope and the adaptability of your implementation. You can decide what to automate now with a view to how regularly you will need to revisit these conversations to keep your solution relevant. In environment with a lot of change, it may be less important to get it right the first time.

Collaboration

Deciding what to automate shouldn’t be a purely technical decision. Reach out for information from across your organisation. Establish credible relationships with management by starting conversations in their language. Talk to your team about objective measures and repetitive tasks. Discuss time and opportunity costs of automation with a wide audience. Consider the pace of change in your environment.

It it important to agree the scope of automation collaboratively.

Collaboration doesn’t mean that you call a meeting to tell people what you think. It isn't a one-way information flow. Allow yourself to be influenced alongside offering your opinion. Make it clear what you believe is suited to automation and what isn’t. Be confident in explaining your rationale, but be open to hearing different perspectives and discovering new information too.

You don't decide what to automate. The decision does not belong to you individually. Instead you lead conversations that create shared agreement. Your team decide what to automate together, and regularly review those choices.

A stability strategy for test automation

As part of the continuous integration strategy for one of our products, we run stability builds each night. The purpose is to detect changes in the product or the tests that cause intermittent issues, which can be obscured during the day. Stability builds give us test results against a consistent code base during a period of time that our test environments are not under heavy load.

The stability builds execute a suite of web-based user interface automation against mocked back-end test data. They run to a schedule and, on a good night, we see six successful builds:

The builds do not run sequentially. At 1am and 4am we trigger two builds in close succession. These execute in parallel so that we use more of our Selenium Grid, which can give early warning of problems caused by load and thread contention.

When things are not going well, we rarely see six failed builds. As problems emerge, the stability test result trend starts to look like this:

In a suite of over 250 tests, there might be a handful of failures. The number of failing tests, and the specific tests that fail, will often vary between builds. Sometimes there is an obvious pattern e.g. tests with an image picker dialog. Sometimes there appears to be no common link.

Why don't we catch these problems during the day?

These tests are part of a build pipeline that includes a large unit test suite. In the build that is run during the day, the test result trend is skewed by unit test failures. The developers are actively working on the code and using our continuous integration for fast feedback.

Once the unit tests are successful, intermittent issues in the user interface tests are often resolved in a subsequent build without code changes. This means that the development team are not blocked, once the build executes successfully they can merge their code.

The overnight stability build is a collective conscience for everyone who works on the product. When the build status changes state, a notification is pushed into the shared chat channel:

Each morning someone will look at the failed builds, then share a short comment about their investigation in a thread of conversation spawned from the original notification message. The team decide whether additional investigation is warranted and how the problem might be addressed.

It can be difficult to prioritise technical debt tasks in test automation. The stability build makes problems visible quickly, to a wide audience. It is rare that these failures are systematically neglected. We know from experience that ignoring the problems has a negative impact on cycle time of our development teams. When it becomes part of the culture to repeatedly trigger a build in order to get a clean set of test results, everything slows down and people become frustrated.

If your user interface tests are integrated into your pipeline, you may find value in adopting a similar approach to stability. We see benefits in early detection, raising awareness of automation across a broad audience, and creating shared ownership of issue resolution.

Three types of coding dojo for test automation

The Test Coaches in my organisation provide support for our test automation frameworks. We create tailored training material, investigate new tools in the market, agree our automation strategy, monitor the stability of our suites, and establish practices that keep our code clean.

A lot of this work is achieved in partnership with the testers. We create opportunities for shared learning experiences. We facilitate agreement of standards and strategy that emerge through conversation. I like using coding dojos to establish these collaborative environments for test automation.

When I run a coding dojo, all the testers of a product gather with a single laptop that is connected to a projector. The group set a clear objective for a test automation task that they would like to complete together. Then everyone participates in the code being written, by contributing verbally and taking a turn at the keyboard. Though we do not adopt the strict principles of a coding dojo as they were originally defined, we operate almost exactly as described in this short video titled 'How to run a coding dojo'.

When using this format for three different types of test automation task - training, refactoring, and discovery - I've observed some interesting patterns in the mechanics of each dojo.

Three types of coding dojo for test automation

Training

Where there are many testers working on a product with multiple types of test automation, individuals may specialize e.g. user interface testing or API testing. It isn't feasible for every person to be an expert in every tool.

Often those who have specialized in one area are curious about others. A coding dojo in this context is about transfer of knowledge to satisfy this curiosity. It also allows those who don't regularly see the code to ask questions about it.

The participants vary in skill from beginner to expert. There may be a cluster of people at each end of the spectrum based on whether they are active in the suite day-to-day.

Training dojo

Communication in this dojo can feel like it has a single direction, from expert to learner. Though everyone participates, it is comfortable for the expert and challenging for the beginner. In supporting the people who are unfamiliar, the experts need to provide a lot of explanation and direction.

This can create quite different individual experiences within the same shared environment. A beginner may feel flooded by new information while an expert may become bored by the slow pace. Even with active participation and rotation of duties, it can be difficult to facilitate this session so that everyone stays engaged.

 

Refactoring

When many people contribute to the same automation suite, variation can emerge in coding practices through time. Occasionally refactoring is required, particularly where older code has become unstable or unreadable.

A coding dojo in this context is useful to agree the patterns for change. Rather than the scope and nature of refactoring being set by the first individual to tackle a particular type of problem, or dictated by a Test Coach, a group of testers collectively agree on how they would like to shape the code.

Though the skill of the participants will vary, they skew towards expert level. The audience for refactoring work is usually those who are regularly active in the code - perhaps people across different teams who all work with the same product.

Refactoring dojo

Communication in this dojo is convergent. There are usually competing ideas and the purpose of the session is to reach agreement on a single solution. As everyone participates in the conversation, the outcome will often include ideas from many different people.

In this example I've included one beginner tester, who might be someone new to the team or unfamiliar with the code. Where the context is refactoring, these people can become observers. Though they take their turn at the keyboard, the other people in the room become their strong-style pair, which means that "for ideas to reach the computer they must go through someone else's hands".

Discovery

As our existing suites become out-dated, or as people hear of new tools that they would like to experiment with, there is opportunity for discovery. An individual might explore a little on their own, then a coding dojo is an invitation to others to join the journey.

The participants of this dojo will skew to beginner. The nature of prototyping and experimentation is that nobody knows the answer!

Discovery dojo

Communication in this dojo is divergent, but directed towards a goal. People have different ideas and want to try different things, but all suggestions are in the spirit of learning more about the tool.

The outcome of this dojo is likely to be greater understanding rather than shared understanding. Though we probably won't agree, but we'll all know a little bit more.

For training, refactoring, and discovery, I enjoy the dynamics of a coding dojo format. I would be curious to know how these experiences match your own, or where you've participated in a dojo for test automation work in a different context.

30 articles for tech leaders written by women

When I was first promoted to a leadership role in tech, I looked for leadership resources that were written by women with advice targeted to a tech environment.

It took some time to discover these articles, which resonated with me and have each contributed to my leadership style in some way. They are written by a variety of women in the US, UK, Europe and New Zealand, many have ties to the software testing community.

This list includes several themes: leadership, communication, learning, inclusion, and recruitment. I would love your recommendations for other articles that could be added.

Why we should care about doing better - Lynne Cazaly
Follow the leader - Marlena Compton
Dealing with surprising human emotions: desk moves - Lara Hogan
You follow the leader because you want to - Kinga Witko

Entering Groups - Esther Derby
Agile Managers: The Essence of Leadership - Johanna Rothman
Recovering from a toxic job - Nat Dudley
Yes, and... - Liz Keogh
Ask vs. Guess cultures - Katherine Wu
"I just can't get her to engage!" - Gnarly Retrospective Problems - Corinna Baldauf
Eight reasons why no one's listening to you - Amy Phillips
Don't argue with sleepwalkers - Fiona Charles

Change Artist Super Powers - Esther Derby
How I work with someone who is learning - Emily McAfee

What learning to knit has reminded me about learning - Emily Webber
Effective learning strategies for programmers - Allison Kaptur
Five models for making sense of complex systems - Christina Wodtke
The comfort zone - Christina Ohanian
WTF are you doing? Tell your teams! - Cassandra Leung
We don't do that here - Aja Hammerly
Here's how to wield empathy and data to build an inclusive team - Ciara Trinidad
Tracking compensation and promotion inequity - Lara Hogan
The other side of diversity - Erica Joy
Hiring isn't enough - Catt Small
'Ladies' is gender neutral - Alice Goldfuss
Where does white privilege show up? - Kirstin Hull

Recruiting diverse engineering candidates: What tech companies still get wrong - Angie Jones

Better hiring with less bias - Trish Khoo
1000 different people, the same words - Kieran Snyder
Why do women try to get ahead by pulling men down? - Missy Titus

Pairing for skill vs. Pairing for confidence

I went to a WeTest leadership breakfast this morning. We run in a Lean Coffee format and today we had a conversation about how to build confidence in people who have learned basic automation skills but seem fearful of applying those skills in their work.

I was fortunate to be sitting in a group with Vicki Hann, a Test Automation Coach, who had a lot of practical suggestions. To build confidence she suggested asking people to:

• Explain a coding concept to a non-technical team mate
• Be involved in regular code reviews
• Practice the same type of coding challenge repeatedly

Then she talked about how she buddies these people within her testing team.

Traditionally when you have someone who is learning you would buddy them with someone who is experienced. You create an environment where the experienced person can transfer their knowledge or skill to the other.

In a situation where the person who is learning has established some basic knowledge and skills, their requirements for a buddy diversify. The types of activities that build confidence can be different to those that teach the material.

Confidence comes from repetition and experimentation in a safe environment. The experienced buddy might not be able to create that space, or the person who is learning may have their own inhibitions about making mistakes in front of their teacher.

Vicki talked about two people in her organisation who are both learning to code. Rather than pairing each person with someone experienced, she paired them with each other. Not day-to-day in the same delivery team, but they regularly work together to build confidence in their newly acquired automation skills.

In their buddy session, each person explains a piece of code that they’ve written to the other. Without an experienced person in the pair, both operate on a level footing. Each person has strengths and weaknesses in their knowledge and skills. They feel safe to make mistakes, correct each other, and explore together when neither know the answer.

I hadn’t considered that there would be a difference in pairing for skill vs. pairing for confidence. In the past, I have attempted to address both learning opportunities in a single pairing by putting the cautious learner with an exuberant mentor. I thought that confidence might be contagious. Sometimes this approach has worked well and others not.

Vicki gave me a new approach to this problem, switching my thinking about confidence from something that is contagious to something that is constructed. I can imagine situations where I’ll want to pair two people who are learning, so that they can build their confidence together. Each person developing a belief in their ability alongside a peer who is going through the same process.

Conference Budgets

There has been conversation on Twitter recently about conferences who do not offer speaker compensation. If you haven't been part of this discussion I would encourage you to read Why I Don't Pay to Speak by Cassandra Leung, which provides a detailed summary.

I take an interest in these conversations from two perspectives: I regularly speak at international conferences and I co-organise the annual WeTest conferences for the New Zealand testing community.

As an organiser, the events that I help to run cover all speaker travel and accommodation. We make a special effort to care for our conference speakers and have built a reputation in the international testing community as being an event that is worth presenting at.

WeTest is a not-for-profit company that is entirely driven by volunteers. How do we afford to pay all of our speakers?

Humble Beginnings

Our 2014 WeTest conference was a half-day event in a single city.

We had 80 participants who paid $20 per person. They received a conference t-shirt along with a catered dinner of pizza and drinks.

All of our speakers were local to the city, so there were no travel or accommodation expenses. Our budget was balanced by the support of our primary sponsor, Assurity.

Our total budget for this event was approximately $3,000 where our income and expenses were:

WeTest Budget 2014

Stepping Up

By 2016 we felt that we had built an audience for a more ambitious event. We embarked on a full-day conference tour with the same program running in two cities within New Zealand.

We had 150 participants in each city who paid $150 per person. This was a significant jump in scale from our previous events, so we had to establish a formal scaffold for our organisation. WeTest was registered as a company, we created a dedicated bank account, and launched our own website.

This was also the first year that we invited international speakers. 25% of our speaker line-up, or three out of twelve speakers, traveled to New Zealand from overseas. Covering their travel and accommodation costs significantly altered the dynamics of our budget. Running the conference in two different cities meant that there were travel and accommodation costs for our New Zealand based speakers and organisers too.

Our total budget for this event was approximately $50,000 where our income and expenses were:

WeTest Budget 2016

The Big League

Our 2016 events sold out quickly and we had long waiting lists. To accommodate a larger audience, we grew again in 2017. This meant securing commercial venues, signing contracts, paying deposits, registering for a new level of tax liability and formalising our not-for-profit status.

In 2017 we had around 230 participants in each city. We introduced an earlybird ticket at $150 per person, so that our loyal supporters would not experience a price-hike and we could collect some early revenue to cover upfront costs. Our standard ticket was $250 per person.

40% of our speaker line-up, or four out of ten speakers, traveled to New Zealand from overseas. We incurred similar speaker travel and accommodation expenses to the previous year.

Our total budget for this event was approximately $100,000 where our income and expenses were:

WeTest Budget 2017

To re-iterate, WeTest is a not-for-profit organisation that is volunteer-led. The profit of our 2017 events will be reinvested into the testing community and help us to launch further events in the New Year.

In the discussion about speaker reimbursement we often discuss in the abstract. I hope that these examples provide specific evidence of how a conference might approach speaker reimbursement, whether they are a small community event or a larger endeavour.

At WeTest we have consistently balanced our budget without asking speakers to pay their own way. We are proud of the diverse speaker programs that have been supported by this approach. In 2018 we look forward to continuing to provide a free platform for our speakers to deliver great content.

Strategies for automated visual regression

In my organisation we have adopted automated visual regression in the test strategy for three of our products. We have made different choices in implementing our frameworks, as we use automated visual regression testing for a slightly different purpose in each team. In this post I introduce the concept of automated visual regression then give some specific examples of how we use it.

What is visual regression?

The appearance of a web application is usually defined by a cascading style sheet (CSS) file. Your product might use a different flavour of CSS like SCSS, SASS, or LESS. They all describe the format and layout of your web-based user interface.

When you make a change to your product, you are likely change how it looks. You might intentionally be working on a design task e.g. fixing the display of a modal dialog. Or you might be working on a piece of functionality that is driven through the user interface, which means that you need to edit the content of a screen e.g. adding a nickname field to a bank account. In both cases you probably need to edit the underlying style sheet.

A problem can arise when the piece of the style sheet that you are editing is used in more than one place within the product, which is often the case. The change that you make will look great in the particular screen that you're working in, but cause a problem in another screen in another area of the application. We call these types of problems visual regression.

It is not always easy to determine where these regression issues might appear because style sheets are applied in a cascade. An element on your page may inherit a number of display properties from parent elements. Imagine a blue button with a label in Arial font where the colour of the button is defined for that element but the font of the button label is defined for the page as a whole. Changing the font of that button by editing the parent definition could have far-reaching consequences.

We use automated visual regression to quickly identify differences in the appearance of our product. We compare a snapshot taken prior to our change with a snapshot taken after our change, then highlight the differences between the two. A person can look through the results of these image comparisons to determine what is expected and what is a problem.

Manufactured example to illustrate image comparison

Team One Strategy

The first team to adopt automated visual regression in my organisation was our public website, a product with a constantly evolving user interface.

The test automation strategy for this product includes a number of targeted suites. There are functional tests written in Selenium that examine the application forms, calculators, and other tools that require user interaction. There are API tests that check the integration of our website to other systems. We have a good level of coverage for the behaviour of the product.

Historically, none of our suites specifically tested the appearance of the product. The testers in the team found it frustrating to repetitively tour the site, in different browsers, to try to detect unexpected changes in how the website looked. Inattentional blindness meant that problems were missed.

The team created a list of the most popular pages in the site based on our analytics. This list was extended, so that it included at least one page within each major section of the website, to define an application tour for the automated suite to capture screenshots for comparison.

The automated visual regression framework was implemented to complete this tour of the application against a configurable list of browsers. It launches BrowserStack, which means that it is able to capture images against desktop, tablet, and mobile browsers. The automated checks replace a large proportion of the cross-browser regression testing that the testers were performing themselves.

The team primarily use the suite at release, though occasionally make use of it during the development process. The tool captures a set of baseline images from the existing production version of the product and compares these to images from the release candidate. The image comparison is made at a page level: a pixel-by-pixel comparison with a fuzz tolerance for small changes.

Team Two Strategy

The second team to adopt automated visual regression was our UI toolkit team. This team develop a set of reusable user interface components so that all of our products have a consistent appearance. The nature of their product means that display problems are important. Even a difference of a single pixel can be significant.

The tester in the this team made automated visual regression the primary focus of their test strategy. They explored the solution that the first team had created, but decided to implement their own framework in a different way.

In our toolkit product, we have pages that display a component in different states e.g. the button page has examples of a normal button, a disabled button, a button that is being hovered on, etc. Rather than comparing the page as a whole with a fuzz tolerance, this tester implemented an exact comparison at a component level. This meant that the tests were targeted and would fail with a specific message e.g. the appearance of the disabled button has changed.

The initial focus for this framework was getting component level coverage across the entire toolkit with execution in a single browser. This suite was intended to run after every change, not just at release. The tester also spent some time refining the reporting for the suite, to usefully abstract the volume of image comparisons being undertaken.

Once the tests were reliable and the reporting succinct, the tester extended the framework to run against different browsers. Cross-browser capability was a lower priority than in the Team One.

Team Three Strategy

A third team are starting to integrate automated visual regression into their test strategy. They work on one of our authenticated banking channels, a relatively large application with a lot of different features.

This product has mature functional test automation. There are two suites that execute through the user interface: a large suite with mocked back-end functionality and a small suite that exercises the entire application stack.

For this product, implementing automated visual regression for a simple application tour is not enough. We want to examine the appearance of the application through different workflows, not just check the display of static content. Rather than repeating the coverage provided by the existing large test suite, the team extended the framework to add an automated visual regression test.

This suite is still under development and, of the three solutions, it is the largest, the slowest, and requires the most intervention by people to execute. The team created a configuration option to switch on screenshot collection as part of the existing functional tests. This generates a set of images that will either represent the 'before' or the 'after' state, depending on which version of the application is under test.

Separate to the collection of images is a comparison program that takes the two sets of screenshots and determines where there are differences. The large suite of functional tests means that there are many images to compare, so the developers came up with an innovative approach to perform these comparisons quickly. They first compare a hash string of the image then, in the event that these differ, they perform the pixel-by-pixel comparison to determine what has changed.

In this team the automated visual regression has a fractured implementation. The collection and comparison happen separately. The focus remains on a single browser and the team continue to iterate their solution, particularly by improving the accuracy and aesthetics of their reporting.

Conclusion

We use automated visual regression to quickly detect changes in the appearance of our product. Different products will require different strategies, because we are looking to address different types of risk with this tool.

The three examples that I've provided, from real teams in my organisation, illustrate this variety in approach. We use visual regression to target:

• cross-browser testing, 
• specific user interface components, and
• consistent display of functional workflows. 

As with any test automation, if you're looking to implement automated visual regression consider the problem that you're trying to solve and target your framework to that risk.

Identifying and influencing how people in your team contribute to test automation

This is a written version of my keynote at The Official 2017 European Selenium Conference in Berlin, Germany. 

If you'd prefer to watch the talk, it is available on the Selenium YouTube channel.

How do your colleagues contribute to test automation?

Who is involved in the design, development and maintenance of your test suites?

What would happen if people in your team changed how they participate in test automation?

How could you influence this change?

This article will encourage you to consider these four questions.

Introduction

When I was 13 years old I played field hockey. I have a lot of fond memories of my high school hockey team. It was a really fun team to be part of and I felt that I really belonged.

When I was 10 years old I really wanted to play hockey. I have clear memories of asking my parents about it, which is probably because the conversation happened more than once. I can remember how much I wanted it.

What happened in those three years, between being a 10 year old who wanted to play hockey and a 13 year old creating fond memories in a high school hockey team? Three things.

The first barrier to playing hockey was that I had none of the gear. I lived in a small town in New Zealand, both my parents were teachers, and hockey gear was a relatively large investment for my family. To participate in the sport I needed a stick, a mouth guard, shin guards, socks. Buying all this equipment gave me access to the sport.

Once I had the gear, I needed to learn how to use it. It turned out that my enthusiasm for getting onto the field did not translate into a natural ability. In fact, initially I was quite scared of participating. I had to learn to hit the ball and trap it, the different positions on the field, and what to do in a short corner. Learning these skills gave me the confidence to play the game, which meant that I started to enjoy it.

The third reason that I ended up in a hockey team when I was 13 years old was because that is where my friends were. As a teenager, spending time with my friends after school was excellent motivation to be part of a hockey team.

Access, skills, and motivation. These separated me at 10 years of age from me at 13 years of age. These separated a kid who really wanted to participate in a sport from someone who felt like they were part of a team.

This type of division is relatable. Team sports are an experience that many of us share, both the feelings of belonging and those of exclusion. Access, skills, and motivation also underpin other types of division in our lives.

Division in test automation

If you look at a software development team at a stand-up meeting, they are all standing together. People are physically close to each other, not on opposite sides of a chasm. But within that group are divisions, and different divisions depending on the lens that you use.

If we think about division in test automation for a software development team then, given what I’ve written about so far, you imagine something like this:

A linear diagram of division

People are divided into categories and progress through these pens from left to right. To be successful in test automation I need access to the source code, I need the skills to be able to write code, and I need to be enthusiastic about it. Boom!

Except, perhaps it’s not that simple or linear.

What if I’m a new tester to a team, and I have the coding skills but not permissions to contribute to a repository? What if I’m enthusiastic, but have no idea what I’m doing? It’s not always one, then the other, then the other. I am not necessarily going to acquire each attribute in turn.

Instead, I think the division looks something like this.

A Venn diagram of division

A Venn diagram of division by access, skills and motivation. An individual could have any three of these, or any combination of the three, or none at all.

To make sense of this, I’d like to talk in real-life examples from teams that I have been part of, which feature five main characters:

Five characters of an example team

The gray goose represents the manager. The burgundy red characters represent the business: the dog is the product owner, the horse is the business analyst. Orange chickens are the developers, and the yellow deer are the testers.

Team One

Team One

This was an agile development team in a large financial institution. I was one of two testers. We were the only two members of the team who were committing code to the test automation suite. We are the two yellow deer right in the middle of the Venn diagram with access, skills and motivation.

The developers in this team could have helped us, they had all the skills. They didn’t show any interest in helping us, but also we didn’t give them access to our code. The three orange chickens at the top of the Venn diagram show that the developers had skills, but no motivation or access.

The business analyst didn’t even know that we had test automation, and there was no product owner in this team. However there was a software development manager and they were a vocal advocate for the test automation to exist, though they didn’t understand it. The burgundy red horse at the top right is outside of the diagram, the grey goose is in motivation alone.

The test automation that this farmyard created was low-level, it executed queries against a database. As we were testing well below the user interface where the business felt comfortable, they were happy to have little involvement. The code in the suite was okay. It wasn’t as good as it could have been if the developers were more involved, but it worked and the tests were stable.

Team Two

Team Two

This was a weird team for me. You can see that as a tester, the yellow deer, I had the skills and motivation to contribute to test automation, but no access. I was bought in as a consultant to help the existing team of developers and business analysts to create test automation.

The developers and business analysts had varying skills. There were a couple of developers who were the main contributors to the suites. The business analysts had access and were enthusiastic, but they didn’t know how to write or read code. Then there were a couple of developers who had the access and skills, but firmly believed that test automation was not their job, they’re the chickens on the left without motivation.

This team built a massive backlog of technical debt in their test automation because the developers who were the main contributors preferred to spend their time doing development. The test code was elegant, but the test coverage was sparse.

Team Three

Team Three

In this team everyone had access to the code except the project manager, but skills and motivation created division.

I ended up working on this test suite with one of the business analysts. He bought all the domain and business knowledge, helped to locate test data, and made sure that the suites had strong test coverage across all of the peculiar scenarios. I bought the coding skills to implement the test automation.

In this team I couldn’t get any of the developers interested in automation. Half had the skills, half didn’t, but none of them really wanted to dive in. The product owner and the other BA who had access to the code were not interested either. They would say that they trusted what the two of us in the middle were producing, so they felt that they didn’t need to be involved.

I believe that the automation we created was pretty good. We might have improved with the opportunity to do peer review within a discipline. The business analyst reviewed my work, and I reviewed his, but we didn’t have deep cross-domain understanding.

Team Four

Team Four

This was a small team where we had no test automation. We had some unit tests, but there wasn’t anything beyond that. This meant that we did a lot of repetitive testing that, in retrospect, seems a little silly.

I was working with two developers. We had a business analyst and a product owner, but no other management alongside us. The technical side of the delivery team all had access to the code base and the skills to write test automation, but we didn’t have time or motivation to do so. The business weren’t pushing for it as an option.

You may have heard similarities to your situation in these experiences. Take a moment to consider your current team. Where would you put your colleagues in a test automation farmyard?

Contributors to test automation

Next, think about how people participate in test automation dependent on where they fall into this model. Originally I labelled the parts of the diagram as access, skills, and motivation:

If I switch these labels to roles they might become:

A person who only has access to test automation is an observer. They're probably a passive observer, as they don't have the skills or the motivation to be more involved.

A person who only has skills is a teacher. They don't have access or motivation, but they can contribute their knowledge.

A person who only has motivation is an advocate. They're a source of energy for everyone within the team.

Where these boundaries overlap:

A problem solver is someone with access and skills who is not motivated to get involved with test automation day-to-day. These people are great for helping to debug specific issues, reviewing pull requests, or asking specific questions about test coverage. Developers often sit in this role.

Coaches have skills and motivation, but no access. They’re an outside influence to offer positive and hopefully useful guidance. If you consider a wider set of colleagues, you might treat a tester from another development team as a coach.

Inventors are those who have access and motivation, but no skills. These are the people who can see what is happening and get super excited about it, but don’t have skills to directly participate. In my experience they’ll throw out ideas, some are crazy and some are genius. These people can be a source of innovation.

And in the middle are the committers. These are generally the people who keep the test suite going. They have access, skills, and motivation.

How people contribute to test automation

Changing roles

Now that you have labels for the way that your colleagues contribute to your test automation, consider whether people are in the "right" place.

I’m not advocating for everyone in your team to be in the middle of the model. Being a committer is not necessarily a goal state, there is value in having people in different roles. However there might be specific people who you can shift within the model that would create a big impact for your team.

Consider how people were contributing to test automation in the four teams that I shared earlier.

Team One

In team one, all the developers were teachers. They had skills, but nothing else. In retrospect if I was choosing one thing to change here, we should have given at least one of the developers access to the code so that they could step into a problem solving role and provide more hands-on help.

Team Two

In team two, I found it frustrating to coach the team without being able to directly influence the code. In retrospect, I could have fought harder for access to the code base. I think that as a committer I would have had greater impact on the prioritisation of testing work and the depth of test coverage provided by automation.

Team Three

In team three, it would have been good to have a peer reviewer from the same discipline. Bringing in a developer to look at the implementation of tests, and/or another business analyst to look at business coverage of the tests, could have made our test automation more robust.

Team Four

In team four, we needed someone to advocate for automation. Without a manager, I think the product owner was the logical choice for this. If they’d been excited about test automation and created an environment where it had priority, I think it would have influenced the technical team members towards automation.

Think about your own team. Who would you like to move within this model? Why? What impact do you think that would have?

Scope of change

To influence, you first need to think about what specifically you are trying to change. Let's step back out to the underlying model of skills, access, motivation. These three attributes are not binary themselves. If you are trying to influence change for a person in one of these dimensions, then you need to understand what exactly you are targeting, and why.

Access

What does it mean to have access to code? Am I granted read-only permissions to a repository, or can I edit existing files, or even create new ones? Does access include having licenses to tools, along with permission to install and setup a development environment locally.

In some cases, perhaps access just means being able to see a report of test results in a continuous integration server like Jenkins. That level of access may be enough to involves a business analyst or a product owner in the scope of automated test coverage.

When considering access, ask:

• What are your observers able to see?
• What types of problems can your problem solvers respond to?
• How does access limit the ideas of your inventors?

Skill

Skill is not just coding skill. Ash Winter has developed a wheel of testing which I think is a useful prompt for thinking more broadly about skill:

Wheel of Testing by Ash Winter

Coding is one skill that helps someone contribute to test automation. Skill also includes test design, the ability to retrieve different types of test data, creating a strategy for test automation, or even generating readable test reports.

How do the skills of your teachers, coaches, and problem solvers differ? Where do you have expertise, and where is it lacking? What training should your team seek?

Motivation

Motivation is not simply "I want test automation" or "I don’t want test automation". There's a spectrum - how much does a person want it? You might have a manager who advocates for 100% automated test coverage, or a developer who considers anything more than a single UI test to be a waste of time.

How invested are your advocates? Should they be pushing for more or backing off a little?

How engaged are your coaches and inventors?

Wider Perspective

The other thing to consider is who isn’t inside the fences at all. The examples that I shared above featured geese, dogs, horses, chickens, deer. Who is not in this list? Are there other animals around your organisation who should be part of your test automation farmyard?

Test automation may be helpful for your operations team to understand the behaviour of a product prior to its release. If you develop executable specifications using BDD, or something similar, could they be shared as a user manual for call centre and support staff?

A wider perspective can also provide opportunities for new information to influence the design of your test automation. Operations and support staff may think of test scenarios that the development team did not consider.

Conclusion

Considering division helps us to feel empathy for others and to more consciously split ourselves in a way that is "right" for our team. Ask whether there are any problems with the test automation in your team. If there are no problems, do you see any opportunities?

Next, think about what you can do to change the situation. Raise awareness of the people around you who don't have access that would be useful to them. Support someone who is asking for training or time to contribute to automation. Ask or persuade a colleague to move themselves within the model.

Testers have a key skill required to be an agent of change, we ask questions daily.

How do your colleagues contribute to test automation? 

Who is involved in the design, development and maintenance of your test suites?

What would happen if people in your team changed how they participate in test automation?

How could you influence this change?