The ratio myth

Recently I've been hearing a lot about the developer to tester ratio for agile teams. This is usually evangelised as an ideal of 4:1; four developers to one tester.

Having worked in many different agile teams, this ratio doesn't align with my experience. Here are a few examples of successful teams I have worked in with structures outside the norm.

More Developers [6:1]

In one of my earliest roles within agile development, I worked as the sole tester in a team with six developers. We were tasked with developing a system from scratch that would replace a legacy application.

The business had a clear vision, were engaged in what the team were doing, and sought to provide timely feedback on early versions of new features. Their involvement bought the users' expectations to any new features as soon as practical.

The developers had a lot of infrastructure oriented work to do, as they were setting up an entirely new application stack. In addition to technical stories, the team always carried at least one with a customer focus, to provide continuous delivery of business value. Features were small and well-defined from a business perspective, but usually involved the interaction of multiple systems behind-the-scenes, which meant they took time to develop.

As a tester, my primary focus was developing and maintaining a tidy suite of automated regression checks. The testing I did was either quick and sympathetic, so that simple issues were discovered and resolved before early engagement of the business, or aggressively sought out interesting problems.

In this team I often felt bored. Though I had six developers, it took a lot of programming to produce small pieces of front-end function. Fast feedback from the business reduced the amount of testing I had to do. I didn't feel challenged.

Less Developers [2:1]

In a different organisation, I worked in a team tasked with re-designing an existing purchasing workflow in an online marketplace. The team included two designers, two developers and myself as the sole tester.

In this team, the pace of development was determined by the rate at which the designers could agree on the updated user interface and workflow. Programming changes were to reflect paper prototypes; updating stylesheets, JavaScript and links between pages.

Though development was relatively simplistic, the testing required was significant. The existing workflow included a number of different business rules, which were recorded in varying detail across multiple documents. There were a vast number of variables to consider in purchasing an item, both obvious variables and those that were indirectly accessible. There was no automated checking available in this domain.

In this team I was busy. Due to the financial impact of failure in this workflow, testing had to be comprehensive and meticulous. Proportionally, much more time was spent in testing, so the output of two developers was enough to keep me occupied.

More Testers [4:2]

Finally, in a third organisation, I worked in a team asked to fix a production issue in a complex shared web service. This team included four developers from four specialist areas; mainframe, database, integration and front-end. I was one of two testers.

In this environment, the complexity of development varied dependent on the system. Most changes were isolated to a specific aspect of functionality, which could only be observed when users followed a specific workflow.

From a test perspective there was a lot of work to do. As in the previous team, the code change was in an important area of the application with complex business rules. Unlike the previous team, there were automated regression checks available at multiple layers of the architecture.

In this team I was challenged. Testing had to occur on each piece of development in isolation, then via multiple front-end consumers of the web service. In addition, due to their specialist focus, a large component of testing in this team was to facilitate communication among developers. With a tight deadline, and proportionally more testing than development, we needed two testers to get through everything.

*****

There are a lot of variables in determining the best ratio of developers to testers in an agile team. A ratio of 4:1 may be a good place to start, but inflexibility in resourcing is folly. There is no hard and fast rule.

If you encounter a scrum master that attempts to enforce a ratio at the expense of common sense, perhaps you can remind them of the first principle of the agile manifesto. Individuals and interactions over processes and tools. A ratio is just a tool, so be sure to listen to your people if they tell you it's not working.

Personal Brand

Recently a colleague in Marketing referred to my "strong personal brand". I found it quite a confrontational statement as it was something I hadn't consciously considered before.

That same week, I had someone approach me to speak at a public event. Usually if I want to speak in a public forum I have to slave over a proposal, then wait to find out if it has been accepted. To receive an invitation was an unexpected honour.

These two experiences prompted me to consider my profile, both in my local community and internationally. Upon reflection I realised how far I have come. I thought I would share some key points in my journey, in the hope that it offers ideas or inspiration to others.

Join your community

The first role I had with testing in my job title was not my first role doing testing as an activity, which is a fairly common path into the profession. My journey starts here because it was in this role that I was lucky enough to have Aaron Hodder as a colleague.

It was Aaron who, fairly relentlessly, pestered me to join the Software Testers New Zealand mailing list on Google, and to start asking questions in that forum. It was Aaron who told me I had to go to the Rapid Software Testing course that James Bach was delivering in Wellington. It was Aaron who wrangled my invite to the second Kiwi Workshop on Software Testing (KWST). As a result of Aaron's interfering, I found myself part of a small community of testers spread across New Zealand.

Speak up

Being a passive community member is a good, but being vocal is better. I entered these forums with reluctance and a selfish agenda. I wanted to use these people for help in solving my own problems. However, once I was engaged, I was surprised to realise that I had ideas and opinions to share too.

Participating in a mailing list is one thing, participating face-to-face is an entirely different proposition. Though I didn't plan on saying much at my first KWST peer conference, I found myself in a situation where I was unable to stay silent for fear it would be construed as complicit agreement. I spoke up.

It is only by speaking up that people can gain a sense of who you are, what you believe, and how you behave as a professional. Starting to contribute, even in a small way, is the first step in making yourself known.

@katrina_tester @kengel100 That's when I knew Katrina was a heavyweight.
— James Marcus Bach (@jamesmarcusbach) June 15, 2014

See how far you've come

Around this point, Brian Osman invited me to do a brown bag lunch talk at his workplace. I remember his pitch for this being very understated. "It will be relaxed, a few people, eating food, talking testing". I agreed, then panicked when Brian asked for a copy of my slides in advance!

I think I spoke about test planning; nothing that seemed exciting to me. Yet the small audience viewed my ideas as a radical new way of doing things. I realised that in challenging and being challenged within the community I had joined, my ideas had developed beyond those who were yet to question their work. I realised that I had grown into a position to give knowledge back.

Invite others
I was lucky to have Aaron pester me to take the first step on my journey. When I realised how the community had helped me grow, I wanted others to experience the same.

Though I saw a national testing community in New Zealand, I didn't see a local community in Wellington. I wanted to invite people to something that was ours, so Aaron and I decided to create regular peer workshops in our city. WeTest Workshops was born.

It took weeks for our first event to fill. We asked everyone we knew. Then we asked them to ask people. Then we asked on mailing lists and twitter. I remember how excited we were to finally find 20 people who wanted to talk about testing. [Community Question]

It was, and still is, such a buzz to see other testers discover a community of their peers, find a friendly forum for their problems, and challenge their thinking.

Look further afield

At the third KWST, Erin Donnell inspired me to join Twitter and start a blog. These were two steps that I was incredibly reluctant to take.

In retrospect, I can identify that the primary reason I didn't want to do this was fear. I was comfortable in my own community. I didn't trust who or what was out there in the wider world. People kept talking about "the international testing community". I thought it sounded ominous.

One of my first Twitter followers was James Bach. Though I felt this was a huge honour, I also felt a crushing responsibility to only tweet things that were interesting, unique, and insightful. Many of my first tweets were typed, edited multiple times, then abandoned.

Similarly on seeing people read my blog, I felt paralysed when considering a new post. What could I say that hadn't been said by others before me? How could I show my own thoughts and skills?

Someone told me that the first step was to start writing, the second step was to make it good. So the way that I got past these fears was via self imposed measurements.

On my blog, I decided that I wanted to write at least three blogs per month. I embraced a quantity over quality mentality to grow my confidence and find my writing voice.

On Twitter, I decided that I should write a tweet per follower; for over six months I kept this ratio at 1:1. This allowed me to be measured in what I contributed, but stopped me from adopting an entirely silent presence.

Present a consistent image
I like things to look consistent. When I started to create a professional presence online, I chose to present myself the same way in every platform. Same photo. Same colour schemes. Same choice of background images.

Though these choices were driven by my peculiar personality, they had the unintended consequence of giving me a very clear "personal brand" (to retrofit the phrase from my marketing colleague). It's something I would recommend to others who want to develop a recognisable identity in the community.

Take opportunities to participate
Having found my feet on Twitter, I discovered a world of opportunities for contributing. The lessons I had learned in my local community applied to an international context; to be known you must speak up.

I wrote for Testing Circus [Is That Testing?] and Testing Planet [Popping the Bubble]. I realised that there was an opportunity for the New Zealand and Australian test community to have a collective voice on the international stage and launched Testing Trapeze, a new software testing magazine, with help from a lot of awesome folk.

I started to advertise my writing. I listed my blog on Ministry of Testing. I asked people I knew to link back to my blog from their own. Over time, people I didn't know started to link back to my blog too.

I wrote conference proposals for EuroSTAR, Let's Test, CAST2014, Agile 2014, and Let's Test Oz. I found writing a proposal a great experience for clarifying my ideas, even when I was ultimately rejected. Of the five proposals I've written, I'm excited to be heading to New York and Sydney this year.

There are a lot of people out there who want to hear what you have to say; start taking these opportunities.

Read widely and share selectively
The people that I see as leaders in the international community have a particular approach. They create good content themselves, and they share the best content of others. This is something that I try to emulate.

Twitter can be a noisy platform, so being known as a reliable filter of information is another way to grow your following. Where a retweet is sparingly given, I believe it holds greater credibility. Complement your voice with others of value; have people trust you as a gateway.

Additionally there's an individual benefit in being widely read. You see themes in the types of problems that testers are facing. You see communities of thinking appear within the whole. You see a variety of opinions and thoughts that help to shape your own.

Be generous
Community is reciprocal. Be generous in promoting the achievements and events of your local and international peers. Demonstrate your pride. Show you care about the problems that people have overcome. If you're generous to others, then your work will be shared with the same enthusiasm.

The occupation question

I have been away on holiday. As I left New Zealand I had to complete immigration paperwork. One of the prompts on immigration forms is for occupation. Before you can enter or leave a country, their government would like to know what you do.

I have done a fair bit of international travel and used to diligently write 'software tester' in this space. I would hand my paperwork to a bored official for them to scan their eyes across, only to have them pause in their process and look at me through narrowed eyes. "Software tester" they would question, "What is that?".

Without fail, I would end up in a long conversation about what my occupation actually was. A conversation that usually concluded with the official saying dismissively "Oh, you work in IT", stamping my passport, and sending me on my way.

Because of this, I've changed my strategy. Now when I fill in the form, I just write 'IT'. This annoys me as it seems so general and sweeping. But the officials are satisfied and there are no questions asked.

Why doesn't everyone know about software testing?

A job exists to fulfill a need. In most cases, the need is obvious when you think of the alternative. For example, a bus driver is employed to drive a bus. Without the driver, the bus would not go. The need for a person to fill this role is very clear.

Without doctors we would stay sick or injured. Without plumbers our toilets would not work. Without electricians we couldn't use our lights. Without dairy farmers we wouldn't have milk, cheese or butter.

Without software testers, then what? My first instinct is to claim that software quality would degrade. To people outside of the software industry that means very little. Computer users are conditioned to expect their software to be imperfect. Though the degree of imperfection would change in the absence of testing, without seeing the better solution many would just assume that what they were given was how it had to be.

How do others within software development avoid this problem? Programmers write code, so it's easy to understand that without them the software would not exist at all. Architects and Project Managers have strong parallels to similar roles outside of our industry, which makes them easier to grasp.

Those with an understanding of software development will, generally, see the need for and value of testing. They have expectations of software and can identify when these expectations are not met. I think it would be nice to see this knowledge spread to a wider audience.

I'd like to be specific about my occupation without having to explain what it is. Wouldn't you?

How to get feedback on a strategy

In my experience, writing then distributing a strategy document is a fairly limited process to receive input and feedback. The type of critique that you get is generally narrow rather than far-reaching. The number of people who choose to respond is often very low.

As part of my role, I'm responsible for developing strategy. It's my belief that the best way to get people engaged in a specific direction is to get them involved in setting the course. Recently, as part of defining strategy, I decided to run a workshop activity to seek feedback in an interactive way.

The structure of the exercise I used was based on one demonstrated by Adele Graham in a training course that I attended. Adele split the class into groups of four or five people, then gave each group a set of three coloured cards that read:

Adele then made a statement and gave the class five minutes to discuss the statement in their groups. At the end of this time each group had to pick the card that reflected their collective opinion of the statement. There was then discussion among the class where differences existed between groups.

I decided to use this same exercise to get input on my strategic thinking, focusing on areas in which I was uncomfortable about making a decision. By opening these aspects to critique, I hoped that the group would bring forward ideas that I hadn't considered.

I had seen that the best discussion in Adele's session came where the statements expressed a polarity; my language had to be clear and absolute. I worked to frame my statements so that the tester favourite of "it depends" would be abandoned in favour of clear choices, yes or no.

I also decided to intersperse the strategy discussions with those that would give me insight into the existing opinions of those present, to give me confidence in areas that I felt my thinking was sound.

Ultimately, I had a set of four statements for discussion:

• A mind map is an easy way to present my test ideas to other people
• [PLATFORM] is the best way to create a [PRACTICE] community at [ORGANISATION]
• I could do [PRACTICE] in my current role at my current client
• One hour of training, once per month, is the best way for me to learn more about [PRACTICE]

I had planned to spend 20 minutes on this piece of the workshop. I estimated that I would need to allow three minutes for each individual group to decide, then two additional minutes for wider discussion in the case that there was disagreement between groups.

On the day, I had 23 people at the workshop. For this exercise, I split the attendees into five separate groups based on their current client engagement, in order to support discussions against the third statement.

We began, and agreement was reached on the first statement almost instantly:

Today I had 23 testers agree, no concerns, with the statement "A mind map is an easy way to present my test ideas to other people" #testing
— Katrina Clokie (@katrina_tester) May 15, 2014

This made me a little concerned that the exercise wouldn't achieve what I had hoped!

Fortunately, the remaining three statements created the type of conflict I was expecting to emerge and provided a good catalyst for opinionated group discussion. As a result, I felt that the group understood the types of compromise inherent in these areas.

My expected timing didn't hold true for each piece of the process. In every case the groups had decided well within the three minutes. However the resulting workshop-wide discussion was much more detailed than I had planned, as different opinions were aired. Overall, we hit the 20 minute estimate for the entire exercise.

I was happy with the breadth of thinking that appeared, and the conclusions reached where opinions were varied. I felt that this workshop gave us a strong foundation to begin from, as the strategy should now reflect the agreement reached here.

How do you seek input on strategic direction? Any other innovative ideas?

Washing the blanket

I use Microsoft Paint to save screenshots while I'm testing. When I see something worth keeping, I hit SHIFT+PrntScrn, then paste it into Paint. I can crop out what I want, annotate it, and save it to my computer. This is how I've saved screenshots for over a decade; a comfortable, familiar practice.

I know that there are other ways to do this task. No, not just other ways, better ways. My colleague uses Greenshot. I've seen her capture and save screenshots without breaking her testing rhythm; seamless. It looks so much quicker and easier than how I go about things, yet I am still doing it my way.

I like to think that everyone has some irrational practice that they can't quite shake. The professional equivalent of the ratty security blanket that a child constantly carries with them. Though I know that it doesn't make sense, though I know that there are better ways, I find that I am incredibly stubborn; I don't want to change how I take screenshots. It's a nonsensical resistance, much the same as the tantrum thrown by that same child who doesn't want to see their blanket put through the washing machine.

It is exceptionally frustrating to see people reject a better way of working. As someone who is often advocating for change, I've encountered many people who are determined to continue in their ways. What I recently realised is that these people are just like me. They may already know that what I'm suggesting may be worthwhile. They understand that a different way might be a better way. They just don't want to give up the familiarity of their practices.

When advocating for change, I tend to structure my reasons around advantages and benefits. I like to offer others the opportunity to identify problems then illustrate how they might be solved by adopting a new approach. I try to appeal to the logical mind; my tactics are focused on critical reasoning and common sense.

These attributes are often absent when trying to separate a child and their blanket. A child knows when their blanket is dirty. They know that putting it through the washing machine will make it clean. They don't care! It's uncomfortable to be without their blanket, and they will not hear reason.

If a parent wants to succeed in separating child and blanket, they have to adopt different methods of persuasion. We can use similar tactics in software development teams to create change where reasoning alone does not work.

First, what not to do. Never tease people about their attachment to a beloved practice, and don't insist that they give it up. [1] Instead try to:

• Agree a set of conditions for when a practice is appropriate, and when it is not. Mandate a different way of working for specific situations only.

• Schedule time for a new practice. Get people used to working in a different way by having them try it on a regular basis for a set period of time. Only encourage adoption once it is known.

• Keep the comfort; don't try and change everything at once. Retain plenty of familiar and comfortable practices alongside adoption of new ones.

• Enlist their help. Make people feel responsible for some aspect of the change, to give them shared ownership of its success.

• Keep people busy, so that they have less time to wistfully ponder the way things used to be.

I'm going to try and remember that change is rarely received rationally. My persuasive tactics need to go beyond critical reasoning and draw on techniques that allow people time to gradually adjust their emotional response to change. Next time I encounter resistance that I see as irrational, I'll remember that we were all children once.

Test Strategy Retrospective

Once an agile team is established and has started delivering working software, a test strategy retrospective can help to determine whether everyone in the team knows what the test strategy is, and agrees on which aspects of the strategy have been implemented. 

Why do a test strategy retrospective?

When people talk about testing in agile they often refer to cross-functional teams. By definition a cross-functional team is a group of people with different functional expertise working toward a common goal [1].

Many agile practitioners include in their understanding of the term the idea that any individual within the team is capable of completing any task. They speak of resources becoming T-shaped, with a depth of skill in their specialist area and a breadth across disciplines other than their own [2][3]. As a simplified example, a tester may have depth of skill in testing with a breadth of skill across business analysis and development.

Although there is usually a specialist tester in a cross-functional team, they are not the only person doing testing. Instead testing can be performed by anyone, which means that the quality of testing will vary depending on who is performing it.

Those from a development background may test that something works, often by creating an automated check, and consider testing complete. Those from the business are requirements driven and may test only to confirm that their needs are met. Those who are not testers are generally less interested in thinking about the ways in which a function doesn't work or could be exploited, so testing becomes more confirmatory and less investigative.

It's apparent to a tester that a shift towards confirmation of requirements comes at the expense of other types of thinking. When faced with this eroding test coverage the specialist tester has two options; alliance or surrender.

By alliance, the specialist tester implements practices that ensure critical thinking and interrogation of the application retain their place. They may institute peer review of the testing performed by non-specialist testers. They may adopt pair testing as a means of complementing the thinking of their colleagues.

By surrender, the specialist tester adopts the belief that testing is confirming that the requirements have been met. They may support automated checks as the primary means of testing an application. They may advocate for a minimum viable product, where the quality of the application is "good enough" for market and nothing more [4].

In either scenario, alliance or surrender, the specialist tester is making a conscious decision to alter the test strategy of the team. They are actively thinking about the trade-off in adopting one practice over another, the implications to test coverage and the impact on the overall quality of the product. But they are often thinking and deciding as an individual.

In a cross-functional team the performance of testing is considered open to all, yet strategic thought about testing is often not. This means that testers, in the loosest application of the word, may be adopting a practice without understanding why.

You may argue that the specialist tester is the only person in a cross-functional team with the ability to create a test strategy, given that testing is the area in which they have a depth of skill. I don't disagree, but counter that the method by which a strategy is decided and shared is important. A tester who fails to make their strategic decisions visible is adopting a high level of risk; taking ownership of choices that may not be theirs to make. And the benefits of a strategy are limited when the tester fails to communicate it to the team so that it is understood and widely adopted.

So, how can a tester determine whether their cross-functional team understands the test strategy that is in place and the decisions that underpin it? By leading a test strategy retrospective.

Creating a visualisation

A test strategy retrospective is designed to be engaging and interactive; to get people who are not testers to think about what types of testing are happening and why. It should take approximately one hour.

The specialist tester should lead this retrospective but not participate in creating the visualisation. This prevents the team from being lead by the opinion of the tester, and ensures that others engage their brains.

To run a test strategy retrospective you will need post-it notes in four different colours and a large surface to apply them to. A large boardroom table is ideal, as it allows the team to gather around all four sides. A large, empty wall is a good second choice.

Start the retrospective by clearly stating the purpose to those gathered; to visualise your test strategy and check that there is shared understanding in the team about what testing is happening.

Take two post-it notes, one labelled IDEA and the other labelled PRODUCTION. These are placed at the top left corner and top right corner of the surface, creating a timeline that reflects the process of software development from an idea to a deployed application.

Within this timeline, different types of test activities can occur. Some of these activities will be part of the test strategy, and some will not. Ask each team member to think about the test activities that are happening in the project, and those that should be.

Allocate five minutes for each person to write a set of post-it notes that each name one test activity, where the colour of the post-it note shows whether or not the activity is part of the test strategy and, if so, whether it is being implemented.

In this example, purple, pink and yellow post-it notes are used to mean:

Each individual should stick their post-it notes on to the timeline at the point they think the test activity will occur. At the end of the five minutes there should be a haphazard display of a large number of post-it notes.

Ask the team to collaboratively group activities with the same name, and agree on the placement of activities within the timeline. Where different names have been used to refer to the same concept, keep these items separate. Once the team are happy with their visualisation, or the conversation starts to circle, call a halt.

An example of a test strategy retrospective visualisation is below.

Leading a discussion on strategy

If you've reached this point of the retrospective by killing a circular thread of conversation then that may be the first place to start a discussion. But there are a number of other questions to ask of this visualisation.

Are there groupings that include different coloured post-it notes? Why?

Have people used different terminology to refer to the same type of test activity? Why?

Why are there activities that are in the test strategy that aren't being implemented?

What are the activities that aren't in the strategy and should be? Do we want to include these?

Are there any activities that are missing from the visualisation altogether? What are they?

These questions not only uncover misunderstanding about the current state of testing, but they also surface the decisions that have been made in determining the strategy that is in place. The visualisation is a product of the whole team and they are invested in it, creating a catalyst for a deep discussion.

For example, the team above are in a surrender state; the test activities that are shown in purple are largely for automated checking. This illustrates that testing is primarily confirmatory, with tools verifying that the requirements have been met. Yet, judging by the number of yellow post-it notes on the right hand side of the timeline, a number of people in the team feel there should be more investigative testing. Who made the decision to focus on automation? It appears that this choice that has not been widely publicised and agreed by the team as a whole. The retrospective offers an opportunity to discuss.

In a cross-functional team where anyone can perform testing, it is important for there to be a shared understanding of both the practical approach to testing tasks and the underlying test strategy. By creating agreement about the type of test activities being performed, any person who picks up a testing task understands the wider context in which they operate. This helps them to make decisions about the boundaries of their task; what they should cover and what sits within another activity.

Evolution of a model

I've been meaning to share some examples of visual test coverage models using mind maps and how these may evolve over time. This example is from a group of testers working on a training project that I've blogged about previously. It is a tidy demonstration of the rapid learning that is common in software testing and clearly illustrates how a model changes over time.

The first version of the model was directed by what the Test Manager told the team was important to test. Though the team had access to the application, there was a limited testing time frame and they did not stray far from the functionality mentioned in their initial briefing. Test ideas within the model were grouped into sessions with a specific charter, which are represented as pale gray boxes.

The team working on this piece of function included three testers who each executed a session. Every tester found at least one bug in their first session. They decided to reflect this against their model by colouring the completed sessions in red.

Before executing their second session, the group re-evaluated. Their first set of sessions had exposed them to a wider variety of function than the Test Manager had mentioned. They questioned whether other aspects of the application under test were also important, and whether the focus of testing should change accordingly. As a result, the left hand side of the model started to expand and, after discussion with the Test Manager, the priorities for their second set of sessions changed. To be clear about where their focus was, the team updated their model to show priority by adding numbering.

At the same time the wider project team adopted a common colour scheme for reporting, so that the Test Manager could expect models from different teams to adopt the same conventions. As a result, completed sessions were marked in green regardless of whether bugs had been discovered.

After the third set of sessions things changed again. The testers felt comfortable with the application, which had been new to them at the start of the process, and started questioning the Test Manager on the applicability of their originally proposed scope. As a result, the right hand side of the diagram became significantly simpler.

In addition, scope added during exploration was refined into succinct and understood coverage. At the end of these sessions testing against the left hand side of the model was complete.

Once all of the scoped sessions had been completed, the model looked quite different to the initial version.

Though this project was slightly contrived for training purposes, I have observed the same rapid evolution of visual models in many client projects. By representing test ideas in a format that is easy to adapt, we make our testing more flexible and responsive to change.

See also:

• Using mind mapping as a visual management tool - Aaron Hodder
• Mind Mapping 101 - Darren McMillan
• To mind map or not to mind map - Adam Knight