Benefits of session-based test management

When selling session-based test management to those who are unfamiliar with the concept, you may be asked for evidence of the benefits of the approach. It makes sense that those responsible for implementing change would first want to know what they might stand to gain from it.

I agree with James Bach when he says:

There are no reasonable numbers; no valid studies of testing productivity that compares ET with scripted testing. All we have are anecdotes.

Bach, J. (2003). Exploratory Testing Explained.

However, there are many anecdotes available in our industry that, when assembled in one place,  form a chorus of common opinion on the benefits that an organisation may observe by adopting session-based test management. These are people who speak not to the what, but to the why. Once you have read the theory, why might you be compelled to try it yourself?

Here are a series of referenced quotes detailing the benefits observed in session-based test management.

Better communication and increased collaboration

The methods above gave us the tools to communicate within and outside the team. By improving communication, we felt that we reduced the number of misunderstandings. Communication also helped to increase trust, which both improved personal relations, but also helped facilitate solutions

Lyndsay, J. & van Eeden, N. (2003). Adventures in Session-Based Testing.

In this case, the testing team changed how they documented their testing - cutting most of the documentation - increased collaboration with those outside of the testing team, and continued to deliver high-quality defects and metrics

Kelly, M. (2011). Session-Based Test Management. In M. Heusser & G. Kulkarni (Eds.), How to Reduce the Cost of Software Testing.

The act of removing this documentation task from my team allowed them to become more involved in the design and requirements clarifying meetings with the developers and the Product Manager and Software Architect.  This enhanced their understanding of what was required and what could be validated.  This led to a reduction in the number of bugs that we entered and had to disqualify as “performs as designed” from nearly seventy on the previous release to just three on this last release.  That is a metric that clearly demonstrates how much better the design and behavior of the application was understood by the developers and the testers as a shared vision.  This allowed us to recoup the time that would have been spent discussing and documenting these bugs and apply that to more testing.

Petty, K. (2005). Reflections on the Use of Session-Based Exploratory Testing As the Primary Test Methodology for Software in an Agile Environment. Presented at the Indianapolis Workshops on Software Testing (IWST).

SBTM allows us to evaluate how the software actually behaves.  It allows us to then compare the documented requirements with the behavior.  Sometimes we find bugs in the software, sometimes we find gaps in the requirement that were presumed or, usually, not recognized as requirements - and that forces the conversation where the testers can help facilitate.  So, the developers and designers can talk about what their vision was and the business reps/experts can talk about theirs.

Walen, P. (2014).

Higher proportion of testing spent in execution rather than documentation

Test sessions aren’t a new way of testing, but a different way to plan and track testing. One that provides a stronger connection between testing and test objectives. They increase the time available for testing, by improving flexibility and responsiveness, whilst reducing the time spent on test planning and documentation. That makes them valuable for both traditional and Agile testing.

Prentice, A. (2011). Testing’s In Session.

The amount of time spent on testing vs. how much time you spend on planning is a lot different from the regular test scripts approach. This is the core of Exploratory testing and this is where SBTM shines.

Jansson, M. (2010). An analysis of Session-based test management.

Test execution can start immediately 

The test team had a lot of knowledge of the system to be migrated so a mind map on functionality and test areas could be easily determined. We used the mind maps to generate test ideas, so we could almost directly start with the test execution.

Schrijver, S. (2014). Where to Use The Session Based Test Approach - A Single Project (different in size).

Information for decision making is available earlier

Rather than wait until the end of testing to find out how good the system was, the decision could be assessed earlier, allowing warning of problems and re-prioritisation of effort.

Lyndsay, J. & van Eeden, N. (2003). Adventures in Session-Based Testing.

More tests are executed

The indication that I see is that you will do a whole lot more tests using session-based testing than when you are focused on running specific test cases with predefined steps you should take.

Jansson, M. (2010). An analysis of Session-based test management.

When we started to work with SBTM we noticed how much more we could do. We could dig a lot deeper into the system

Forsberg, J. (2014).

Higher number of bugs found

I helped one large company start an exploratory test team in a department surrounded by scripted testers. They found that the ET people, unencumbered by the maintenance associated with test artifacts, were able to pursue more leads and rumors about risk. They found more bugs, and more important bugs. Three years later, the team was still in place.

Bach, J. (2003). Exploratory Testing Explained.

… the rate at which significant bugs were found stayed the same on the introduction of these methods, and increased for the next five months …

Lyndsay, J. & van Eeden, N. (2003). Adventures in Session-Based Testing.

Further, session-based testing provides a cost-effective means of identifying defects. The method is simple to understand, easy to implement, and has demonstrated effectiveness in ensuring the viability of medical device software.

Wood, B. & James, D. (2003). Applying Session-Based Testing to Medical Software.

Faster response when retesting bug fixes

Test sessions allowed a faster response to the arrival of a fix, and served as effective proof that the fix had been well implemented and tested. The team found that looking at the session for the test when the problem had been found helped plan the retest.

Lyndsay, J. & van Eeden, N. (2003). Adventures in Session-Based Testing.

Empower testers to make decisions in response to changing risk

with each test executed the tester learns more about the product and uses that information to design the next best test to be executed. It's a process that allows the tester to respond to each test in a way that should maximize focus on the most relevant risks.

Kelly, M. (2009). The benefits of exploratory testing in agile environments.

The test team felt in control of their work. They could see the size of it, see how much they had done, and what was left. They could decide what to do next, and back up those decisions.

Lyndsay, J. & van Eeden, N. (2003). Adventures in Session-Based Testing.

With a test script you are following a predetermined path on your way to an expected outcome. You don´t write down any thing unless you see something unexpected. ... With Session Based Exploratory Testing, when you are busy with a test session, you observing the behavior of the application under test. You write down what your experiences are, which steps you have executed, the test data you have used. When something unexpected occurs, you can investigate further. 

Schrijver, S. (2013). My "elevator pitch" for Session Based Exploratory Testing.

Exploratory testing has multiple benefits – one of the greatest being that it doesn't lock down your testing into the forms of tests you can imagine before you've even had “first contact” with a prototype version of software.

Talks, M. (2014). Learning to use exploratory testing in your organisation.

Improve tester morale and job satisfaction

[Testers] were encouraged to measure their own progress and their estimates were trusted. Morale improved, and the test team was seen as an interesting and valuable place to work.

Lyndsay, J. & van Eeden, N. (2003). Adventures in Session-Based Testing.

I believe that you become more satisfied in as a tester when you use the session-based test management instead of running hardcore test scripts.

Jansson, M. (2010). An analysis of Session-based test management.

Experience Reports

What‘s really great about it? 

• It gets everyone involved! 

• It focuses everyone involved! 

• It allows testers variation in their work! 

• It gets the job done quickly! QA is not the bottleneck! 

• It‟s fun, and creates team spirit! (Important for embedded QA!) 

• It allows multiple perspectives simultaneously on the task at hand! 

• It shows everyone what QA does!

Womack, M. (2011). Meeting Agile Deadlines with Session Based Testing. Presented at Swiss Testing Night Zurich.

What Happened

• Morale improved significantly

• Defect rates for NEW development were kept on par with maintenance projects.

• More “Critical” bugs found faster.

• Major bugs found after code freeze reduced.

• Almost steady defect ratios of 50% Critical, 28% Moderate, and 22% Minor across all releases

• 90% reduction in “Non-Bug” Bugs.

• 300% increase in number of test cases performed

• Fewer late nights for SQA.

Petty, K. (2009). Transitioning from Standard V&V to Rapid Testing Practices in a Chaotic Project Environment. Presented at the Conference of the Association for Software Testing 2009.


If you would like to add to this list, please leave a comment below.