The Testing Pendulum: Finding balance in exploration

How detailed should exploratory testing be?

I spotted this question in the Cambridge Lean Coffee topics that James Thomas collated in his blog. It's a question that I'm often asked, and I consistently use the same analogy in my response: the testing pendulum.

The Testing Pendulum

Imagine a pendulum at rest. The space in which the pendulum can swing is our test approach. At the left apex we are going too deep in our testing and at the right apex we are staying too shallow. Initially, the testing pendulum sits directly in the middle of these two extremes:

The Testing Pendulum

When a tester starts in a new organisation or a new project, we apply the first movement to our testing pendulum. Think of it as lifting the pendulum up to the highest point and letting go. The swing starts from the shallow side of the spectrum to reflect the limited knowledge that we have as we enter a new situation. As our experience deepens, so too does our testing.

Starting the pendulum

"When given an initial push, [the pendulum] will swing back and forth at a constant amplitude. Real pendulums are subject to friction and air drag, so the amplitude of their swings declines." [Ref]

Similarly in testing, the pendulum will swing backwards and forwards. When our testing has become too intensive, we switch direction. When our testing has become ineffectual, we switch again.

Pendulum changing direction at the peak of its swing

The skill in knowing how detailed testing should be is to recognise the indicators that tell you when the pendulum is at the top of its swing. You need to be able to identify when you've gone too deep or stayed too shallow, so that you can adjust your approach appropriately.

Indicators

Indicators help us determine the position of our pendulum in the testing spectrum. I see three categories of indicator: bug count, team feedback and management feedback.

Bug count

If you are not finding many bugs in your testing, but your customers are reporting a lot of problems in production, then your testing may be too shallow. On the flip side, if you raise a lot of bugs but not many are being fixed, or your customers are reporting zero problems in production, then your testing may be too deep.

As a caveat, the zero problems in production measure may not apply in some industries. A web-based application may allow some user interface problems to be released where the economics of fixing these does not make sense, but a company that produces medical hardware may seek to release a product that they believe is perfect no matter how long it takes to test. Apply the lens of your own organisation.

Team feedback

Whether you're working in waterfall testing team or an agile delivery team, you are likely to receive feedback from those around you. Be open to those opinions and use them to adjust your behaviour.

If your colleagues are frequently adding scope to your testing, questioning whether you've spent enough time doing your testing, or perform testing themselves that you think is unnecessary, then your testing may be too shallow. On the flip side, if your colleagues are frequently removing scope from your testing, questioning what else you could be doing with the time that you spend testing, or do not perform any testing themselves, then your testing may be too deep.

On the point of colleagues doing testing, this is a particularly useful indicator in agile teams. In the extreme case, if no unit tests are being written and your developers are outsourcing their testing to you, or if the business trust you to do their user acceptance testing, then it's likely that you're testing too deeply. If you want to cultivate an environment with shared ownership of quality then you have to allow room for that to happen.

Management feedback

If your testing pendulum is sitting at a point in the spectrum where your team are unhappy, it's likely that your manager will have a direct conversation with you about your test approach.

If you're testing too much your manager will probably feel comfortable about telling you this directly. If your testing is too shallow, you might be asked for more detail about what you're testing, be questioned about bugs reported by users, or have to explain where you're spending your time.

Indicators are heuristics, not rules. They include subjective language, i.e. "many", "not many", "frequently" or "a lot", which will mean different things in different situations. As always, apply the context of your organisation to your decision making.

The indicators that I've described can be summarised by opposing statements that represent the extremes of the pendulum swing:

Finding equilibrium

Eventually, most testers will end up at an equilibrium where the pendulum hangs at rest, halfway between the two extremes. The comfort of this can be problematic. Once we have confidence in our approach we can become blind to the need to adjust it.

I believe the state that we want to strive for lies slightly towards the left of the spectrum, or too deep. In order to keep a pendulum positioned off-center, we have to regularly apply small amounts of pressure: a bump!

Pushing the boundaries

If you've been testing a product for a long time and wish to avoid becoming stale, give your testing a bump towards greater depth. Apply some different test heuristics. Explore using different customer personas. Alter your test data. Any variation that could provide you with a fresh crop of problems to explore the validity of with your team.

You can use the outcome of a bump to calibrate your test approach within a smaller range of pendulum movement. Which changes are too much? Which are welcome and should be permanent added to your testing repertoire? Continued small experiments help to determine that you are in the right place.

Conclusion

The testing pendulum is a useful analogy for describing how to find balance in a test approach. When entering a new team or project, it can be used to illustrate how we experience large initial swings in the depth of our testing as we learn the indicators of our environment. For those who have been testing the same product for a while, it can remind us to continuously verify our approach through bumps.

There is no one right answer to "How detailed should exploratory testing be?", but I hope the testing pendulum will help you to determine and describe the right level of detail for you.

Take control of your test environment

As CAST 2015, Ioana Serban delivered a talk titled "Take control of your test environment". It's an entertaining and animated tale of her experiences with test environments. I'd encourage you to watch the talk if you haven't seen it previously.

I chose Ioana's presentation as the basis for a recent online testing knowledge sharing session in my organisation. As we are still working as a dispersed team, it currently isn't practical to bring all the testers together in one physical location to share ideas. Instead I asked them to watch Ioana's talk in their own time prior to a discussion session where we met for an hour in an online group chat to talk about the ideas that she raised.

Though our discussion was a little difficult to follow at times, on account of the many people typing in parallel, there were some very interesting points raised. We covered four topics: our build stability, techniques for test automation triage, the keys to our environments, and the people we work with in our own test environment quests.

Build Stability

Ioana spoke about techniques for encouraging stability in automated builds. She contrasted an approach where punishment is a deterrent e.g. wear a funny hat or pay a fine if you are the person responsible for a broken build, to a more positive outlook e.g. a counter on the wall that tracks the number of days since the last time that the build failed. Rather than amplifying mistakes, celebrate success.

We started by talking about how we currently encourage stability in our teams and tribes.

The initial response was that we don't focus on the stability of our master code base but rather the stability of what we are merging to it. The testers are careful to only push change from their branches when it has been peer reviewed and executes reliably. If bugs do end up in master, the tester will take ownership of investigating the automation failure, talk to the developers in their team, and collaboratively create a fix. The testers commented that "most of the time our build is stable compared to other places I have worked" and "in other organisations, it takes days to resolve problems, but in [our product] it takes hours".

But we also had one delivery area where things weren't so rosy. One tester said that their builds are usually broken "to the point where most people don’t have a reaction to a failed or unstable build". There was interesting discussion around the approach to resolving this that touched on the willingness of capable individuals to take ownership of the problems, allocation of time to build-related activities, incorrect configuration in our continuous integration server, failure fatigue and aging infrastructure.

We talked about using the time since last failure that is recorded in our continuous integration server to track stability of our builds. We haven't historically put much focus on this metric, but this may need to change if we move to a model of celebrating success.

The current method to keep attention on stability is to push build failure notifications to a team chat channel. Repeated messages make it difficult to ignore instability. Some of the testers also run their automation every time that a developer deploys into their test environment, which keeps both the automation and the environments stable.

Finally we talked a little about how instability can be a positive thing too. If the tests pass all the time, is there any point to them? There was a comment that "a test that never fails never gives you good information". However, we want our failures to be on branches and not on master. It also shouldn't be the test code itself that fails but rather a problem in the application or configuration that is highlighted by the test.

Test Automation Triage

Ioana spoke about separating her test automation into categories and treating each set differently: a suite of known bugs, a suite of flaky tests, and a suite that works reliably. We don't use an approach like this to triage our automation and I was curious about what the testers thought of it.

It's fair to say that opinions were mixed.

In one of our products we have a relatively old automation suite that has created some stability problems in the past. Testers in this tribe were eager to try Ioana's approach, though some investigation will be required to determine whether it's possible with the tool set that we use.

In our other products instability is either extremely rare or completely pervasive. Where instability is rare, we generally have the luxury of focusing on and resolving a single problem. There is little benefit to segregating this work from our master code repository. Where instability is pervasive, we would effectively be tipping the entire suite into the flaky bucket, which negates the benefit of having a separate category.

Though we are familiar with flaky tests, the idea of having an automation suite of known bugs was not something that we'd considered before. Some people thought that it might be useful to highlight known problems and give the developers a resource to target their code changes. But there was also a viewpoint that it would be an expensive waste of time to code a suite that always failed.

Finally, we diverged slightly into discussion about failures beyond our automation. We realised that we don't have visibility of which bugs are part of our product backlogs or a centralised register of known issues with our release test environments. Both were taken as areas for action.

Keys to our environments

Ioana talked about keys as a metaphor for access to aspects of her test environments. The four keys that she mentioned were access to code, access to the database, access to monitoring, and permission to deploy. Our next discussion topic was about the keys that the testers have for our test environments and the keys that they are missing.

Through this conversation I learned that many of the testers have limited access to our release testing environments. Some can examine server logs and query the release environment database, but others cannot. In theory, every new tester who joins the organisation is granted identical permissions. In practice, it seems that there are inconsistencies in the access that has been configured for individuals. Now that I am aware of this, I've started asking questions.

The other keys that the testers identified weren't specifically about test environments. Instead they were keys to information that will help focus their testing. One tester asked for improvements in device usage analytics to better target their native mobile application testing. Another requested access to customer feedback to get a better understanding of how our customers are using our products.

The idea of keys made people think broadly about areas that were locked to them and who they would need to speak with to open these doors.

People

Ioana finished by talking about the relationships that she'd built with people across her organisation who helped her to take control of her test environments. The last discussion topic focused on sharing the go-to people that the testers would call on for help when facing their own test environment challenges.

I was expecting people to name individuals but the most common response was that the testers direct their questions to a group. One commented that asking a group opens the opportunity for someone who you perhaps wouldn't ask directly to respond. I think the order in which the groups were named in our discussion reflects the order in which people would ask them for help.

First, developers. This was overwhelmingly the immediate response. I like that the testers have built relationships with their developers where they feel completely comfortable asking them questions, even those that aren't directly related to the code. I think developers as a first response is a symptom of healthy delivery teams.

The problems that the developers cannot solve are usually resolved by a solution coming from outside the expertise of the delivery team. The next group mentioned in the discussion were "all of the testers". As a collective that spans across multiple teams and tribes, we may not always know the answer but we are well-connected to people who might. Someone else said that they would choose an appropriate online chat channel to ask their question to.

Finally, one tester commented that testers should try to build our capability to identify the problem with our environments instead of simply saying that one exists. As with raising a bug, it's easier to resolve if more information is provided. Particularly when we turn to a group for assistance, asking the question with appropriate details is important.

Conclusion

Ioana's talk is interesting in its own right, but I would also recommend it as the foundation of a team discussion about test environments. We found a lot of scope for deep conversation about applying Ioana's ideas in our organisation.

I finished our session by asking the team to share one thing that they would think more about or take action on. The points that came back were indicative of a presentation that covered attitude, approach and technical techniques, including: 

• Keep asking questions, be polite, and build relationships with your colleagues.
• "We are the processes, they are living agreements."
• Learn what keys are available for your test environments and work to get them. 
• Do your research and send good information to the support team when asking for help.
• Create a failing test to make a bug fix easier for a developer.

I hope that this session will encourage the testers that I work with to be more mindful of build stability, techniques for test automation triage, the keys to our environments, and the people we work with in our own test environment quests. Ultimately, we'll take control of our test environments.

Stop fighting. Start participating.

In her editorial for the October edition of Women Testers, Karen Johnson talks about change in the testing profession, particularly the increasing demand for test engineers and automation. She concludes her thoughts by saying: 

"...we had to fight for our profession in the beginning and, as it turns out, we might need to fight the good fight yet again."

I find the rhetoric of fight relatively common in the testing community. It's strong language to use in a professional context. To encourage people to fight is to encourage them to "take part in a violent struggle to overcome, eliminate, or prevent". It seems at odds with the type of environment that many of us work within or would choose to create with our colleagues.

In my experience, the quickest way to be removed or excluded from a collaborative conversation in the workplace is to focus solely on fighting for your viewpoint. If you are seen as aggressive or stubborn, people will probably choose not to talk to you. Decisions start to happen around you, rather than with you. Change still occurs, but by fighting you've removed your ability to influence it.

I don't believe that we should approach change with an attitude to fight it. 

Instead, I would like testers to focus on being part of the conversations that create evolution in our roles. We need to invite ourselves along, contribute with enthusiasm and pragmatism, and shape change that includes our perspective. Our mindset is not to fight, but to participate.

How can we do that?

I believe that any tester, regardless of their experience or position in an organisation hierarchy, can develop the skills and relationships required to effect decision making. But it's not easy and the scope of what you can influence will vary.

Start by developing your understanding of where change comes from in your organisation. As testers our roles may be shaped by test managers, or software development managers, or managers in the wider organisation. How far away from you are the decision makers? 

What forum are decisions being made in? Could you attend? If you're not sure, don't assume that the meeting is closed. Speak to the organiser. If you're unable to attend as a contributor you may be permitted to observe. 

Regardless of whether you can be present, try to find your closest advocate in that forum. This is probably the person whose views most closely match your own. It could also be the person who is most receptive to alternate perspectives. If your manager is in the forum and you believe your best advocate is someone else, use extra care and diplomacy as you navigate future conversations!

Try to develop a connection with that person. Take time to understand their viewpoint and how they've come to develop it. What problems do they believe will be solved with proposed change? What do they understand to be the potential benefits and drawbacks of their choice? 

Think beyond information to the context that it originated from. How much do you know about the environment that the decision maker is in? What pressure is being applied to them from other areas of the organisation? What is their personal history in the organisation? Empathy will help you feel the weight of their context in their choices.

Only after all this groundwork do you start thinking about your own contribution to the conversation. Can you broaden understanding of the existing problems? Can you offer additional information about the options being considered that may alter the final choice? What new alternatives can you suggest that still resolve the specific concerns of the decision maker?

Then, how can you present your viewpoint persuasively? Successful participation is not just about what you say, but also in how you deliver it. Can you control the pitch and pace of your voice? Can you demonstrate positive body language and illustrate your message with appropriate gestures? If you need to, practice first. You should appear confident in your ideas.

Finally, be comfortable with the outcome, whatever it is. Even when you feel some disappointment in a decision, participating in the discussion will have improved your understanding of how it was made. And remember, the goal is not to "overcome, eliminate, or prevent". The goal is to be involved. 

I've described a high-level approach to contributing to change that illustrates a collaborative mindset rather than a combative one. Behind each step is a depth of knowledge to uncover and new skills to develop. It may be harder to be a diplomat than a warrior, but I believe this path will focus your energy in a constructive way.

There is no doubt that the testing profession is evolving. If you want to help shape the changes that affect you, then contribute in your own organisation. Stop fighting and start participating.