Visual Test Models & State Transition Diagrams

This article was originally printed in the November edition of Testing Circus

Visual test models and state transition diagrams are two means of visualising information. Though they may appear similar at a glance, the structure and purpose of each is unique. When used in software testing, they act as tools to guide entirely different test techniques.

This article compares these two types of visualisation in the context of a real-world example based on purchasing a book from e-commerce retailer Amazon.

Visual Test Models

Testing often begins by focusing on function first. Sometimes this is because testing is being driven by a requirements document or the acceptance criteria of a user story. Sometimes it’s because not all of the functionality is present in an iterative delivery model, so the testers have to concentrate on one piece at a time. Sometimes it is just due to personal preference.

When focusing on function, we are testing to see what the software can do.  James Bach lists four key points of function testing in the Heuristic Test Strategy Model:

1. Identify things that the product can do (functions and sub-functions). 
2. Determine how you’d know if a function was capable of working. 
3. Test each function, one at a time. 
4. See that each function does what it’s supposed to do and not what it isn’t supposed to do.

A visual test model begins as a useful way to quickly capture the things that a product can do, acting as a note taking tool during investigation of a function.

Imagine that we have received the Amazon Shopping Cart function to test. A screenshot of a cart with one item awaiting purchase is shown below:

A visual test model that identifies things that the product can do might look like this:

Once the visual test model shows what the product can do, it can then be used to capture how the tester could know whether each function was capable of working. Test ideas are added to each branch and might include negative tests – those ideas that determine that the function doesn’t do things that it shouldn’t.

When switching between documenting function and capturing test ideas it may be useful to switch colour, to clearly delineate between different types of information. A continued example for the Amazon Shopping Cart is shown below:

From this point the continued use of the model will vary dependent on the test approach. It may be extended to show exploratory test sessions and progress of testing. It may drive discussion around which ideas are suited to automated checking. It may be used as a reference to create pre-scripted test cases in a test case management tool.

A visual test model is a good way to capture quick thinking about what a product can do. It is a simple mechanism to capture test ideas that drive function testing.

State Transition Diagrams

Once the pieces of the product have been functionally verified, the tester will want to examine how they interact together. The shopping cart, the checkout and the order review page may all work in isolation, but unless they also integrate correctly with each other a customer will not be able to purchase a book.

Flow testing is where a tester does one thing after another. The Heuristic Test Strategy Model lists three key points about the flow testing technique:

1. Perform multiple activities connected end-to-end; for instance, conduct tours through a state model.
2. Don’t reset the system between actions.
3. Vary timing and sequencing, and try parallel threads.

State transition diagrams offer a way to visualise paths through a product. Determining how activities are connected end-to-end will help the tester to identify varied flow through the application that may otherwise have been missed.

To create a state transition diagram start by identifiying the scenario for a successful interaction. When purchasing a book using Amazon, the simplest flow through the product may look like this:

Then investigate the other scenarios that a user could encounter at each point in this simple flow. An expanded diagram may look like this:

Because the state transition diagram is focused on interaction, it is a good visualisation for flow testing. Traversing the diagram through varied paths, while adopting different timing and number of users, is a good way to explore a piece of software through a different lens.

Visual Test Ideas

There are some circumstances in which a mind map is not the best format to visualise your testing. In particular, where there are complex relationships between data in the application under test.

Visual test ideas can be useful when it feels clumsy to explain what you plan to test in words. If you need the assistance of a whiteboard to outline your thinking to others, this might indicate that you should record your test ideas pictorially instead of in writing.

As with other visual approaches, visual test ideas are a great way to engage people in the activities of testing and encourage collaboration that improves test coverage.

Here are three examples of different ways to record your test ideas visually.

Timelines

Imagine the Transaction History for a bank account. When you load the page, you see the last 30 days of transactions by default. You can change the date range for transactions returned by updating the start and end dates:

Transactions are only returned for the first 90 days of the specified range. If the specified date range is greater than 90 days then the oldest transactions are discarded. Data is only available up to 180 days ago, no transaction information will return prior to that date. And of course, there are no transactions available in the future.

Given these requirements, a number of different test ideas spring to mind. As these ideas are discussed with other people, they might be captured in a timeline format:

Using a timeline to explain these test ideas clearly shows how they relate to one another within the time relationship, where a mind map format would not.

Buckets

Imagine that you have taken out a loan where your repayment obligations are tied to your income. When your earn beyond specific thresholds, then your repayment amount will change. These thresholds are dependent on the type of earning you receive, whether salary, wages or adjusted income.

The example below is from Aaron Hodder. Using a buckets marked with different thresholds, Aaron clearly shows how his test ideas relate to one another:

Venn Diagrams

Imagine a school grading system. Within a given subject, you can receive credits across one or many topics. For example, in Mathematics you can gain credits that apply to Calculus, Algebra, or Calculus & Algebra combined. The number of credits you have in each topic, and in total, will determine whether you are awarded a given qualification for a subject.

Nigel Charman has written about a team operating in this domain:

"In this case, the team are visualising examples of these complex business rules using Venn diagrams. The richness of the visualisation helps us wrap our brains around the complexity, acting as a shorthand form for discussion." [1]

An accompanying picture shows a team working around a whiteboard that is filled with various Venn diagrams:

Source: Visual Specification by Example

Instead of then attempting to reflect these scenarios in words, these same diagrams become the documented test ideas:

Source: Visual Specification by Example

Using Venn diagrams to capture these test ideas clearly shows the relationships between topics within subjects that a mind map format would not.

*****

There are many options for representing test ideas and the relationships between them by making use of engaging illustrations. Think laterally. Though a mind map is an obvious option to present visual information, it isn't always the best way for a tester to share their thoughts with their team.

Three examples of context-driven testing using visual test coverage modelling

I talk a lot about visual test coverage modelling. Sometimes I worry that people will think I am advocating it as a testing practice that can be used in the same way across any scenario; a "best practice". This is not the case.

A tester may apply the same practice in different contexts, yet still be context-driven. The crux is whether they have the ability to decompose and recompose the practice to fit the situation at hand [1].

To help illustrate this, here are three examples of how I have adapted visual test coverage modelling to my context, and one example where I chose not to adopt this practice at all.

Integrated with Automation

I was the only tester in an agile team within a large organisation. There were two agile teams operating in this organisation, everybody else followed a waterfall process.

The other agile team had a strong focus on test automation. When I joined, there was an expectation that I would follow the process pioneered in that team. I was asked to create an automated test suite on a shared continuous integration server.

Having previously worked in an organisation that put a high value on continuous integration, I wanted to be sure that its worth was properly understood. I decided to integrate visual test coverage models into the reports generated by the automated suites to illustrate the difference between testing and checking.

At this client I used FreeMind to create mind maps that showed the scope of testing. Icons were used against branches in the map to show where automated checks had been implemented, and where they had not. They also showed problems and where testing had been descoped.

Source: Mind Maps and Automation

The primary purpose of the visual test coverage model in this context was to provide a high-level electronic testing dashboard for the Product Owner that was integrated into the reporting from our automation.

Paper-based Thinking

I was a test consultant in an agile team where my role was to teach the developers and business analysts how to test. This team had no specialist testers. The existing team members wanted to become cross-functional to cover testing tasks.

The team had been working together for a while, with an attitude that testing and checking were synonymous. They were attempting "100% automation" and had started to require test sprints in their scrum process to achieve this.

At this client I wanted to separate the thinking about testing from the limits of what is possible with an automation tool. I wanted to encourage people in the team to think broadly about test ideas before they identified which were candidates for automation.

I asked the team to create their visual test coverage models on large pieces of paper; one per story. They would use one colour pen to note down the functionality, then another colour for test ideas. There was a process for review at several stages within the construction of the model.

Source: How to create a visual test coverage model

The primary purpose of the visual test coverage model in this context was to encourage collaborative lateral thinking about testing. By using paper-based models, each person was forced to think without first considering the capabilities of a tool.

Reporting Session Based Testing

I was one of two testers in an agile team, working in a department of many agile teams. I was involved in the project part time, with the other tester working full time.

The team was very small as most of the work in this particular project was testing. We had a lot of freedom in determining our test approach and the other tester was interested in implementing session based testing.

We used xMind to create a mind map that captured our test ideas and showed how the scope of our testing was separated into sessions. We updated the model to reflect progress of testing and issues discovered. It was also clear where there were still sessions to be executed, so the model also helped us to divide up the work.

Source: Evolution of a Model

The primary purpose of the visual test coverage model in this context was for quick communication of session based test progress between two testers in a team. As someone who was only involved part time, the model served as a simple way to rapidly identify how the status of testing had changed while I was away from client site.

A Quiet Team

I was a test consultant in an organisation where my role was to coach the existing testers to improve their confidence. The testers worked together in a single agile team. One of their doubts was that they thought production issues were due to tests that they had missed. They weren't sure about their test coverage.

The testers were very quiet. The team in general conversed at a low volume and with reluctance. There were only a couple of people who were confident communicators and, as a result, they tended to dominate conversation, both informal and in meetings.

I wanted to shift ownership of testing from the testers to the team. I felt that scoping of test ideas and decisions about priority should be shared. It seemed that the lack of confidence was due, in part, to the testers often working in isolation.

Though I wanted to implement a collaborative practice for brainstorming, I felt that visual test coverage models wouldn't work in this team dynamic. I could imagine the dominant team members would have disproportionate input, while the testers may have ideas that were never voiced.

Instead I thought that the team could adopt a time boxed, silent brainstorm where test ideas were written out on post-it notes. This allowed every person to share their ideas. Decisions about priority of tasks, and candidates for automated checks, could be discussed once everyone had contributed by using the collective output of the group to guide conversation.

*****

Before anything else, ask yourself why the practice you'd like to implement is a good fit for your situation. Being able to articulate the underlying purpose is important, both for communicating to the wider team and for knowing what aspects of the practice must remain to meet that need.

I have found visual test coverage modelling useful in many scenarios, but have yet to implement it in exactly the same way twice. I hope that this illustrates how a tester who aims to be context-driven might adapt a practice they know to suit the specific environment in which they are operating.

How to create a visual test coverage model

Creating a visual test coverage model to show test ideas in a mind map format is not a new idea. However it can be a challenging change in paradigm for people who are used to writing test cases that contain linear test steps. Through teaching visual modelling I have had the opportunity to observe how some people struggle when attempting to use a mind map for the first time.

Though there is no single right way to create a visual test coverage model, I teach a simple approach to help those testers who want to try using mind maps but aren't sure where to begin. I hope that from this seed, as people become confident in using a mind map to visualise their test ideas, they would then adapt this process to suit their own project environment.

Function First

The first step when considering what to test for a given function is to try and understand what it does. A good place to start a mind map is with the written requirements or acceptance criteria.

Imagine a story that includes developing the add, edit, view, and delete operations for a simple database table. The first iteration of the visual test coverage model might look like this:

Collaborate

Next consider whether all the behaviour of this function is captured in the written requirements. There are likely to be items that have not been explicitly listed. The UI may provide a richer experience than was originally requested. The business analyst may think that "some things just go without saying". There may be application level requirements that apply to this particular function.

Collaboration is the key to discovering what else this function can do. Ask a business analyst and a developer to review the mind map to be sure that every behaviour is captured. This review generally doesn't take long, and a quick conversation early in the process can prevent a lot of frustration later on.

Imagine that the developer tells us that the default design for view includes sort, filter, and pagination. Imagine that the business analyst mentions that we always ask our users to confirm before we delete data. The second iteration of the visual test coverage model might look like this:

Think Testing

With a rounded understanding of what the function does the next thing to consider is how to test it.

For people that are brand new to using a mind map, my suggestion is to start by thinking of the names of the test cases that they would traditionally scope. Instead of writing down the whole test case name, just note the key word or phrase that differentiates that case from others. This is a test idea.

Test ideas are written against the behaviour to which they apply. This means that tests and requirements are directly associated, which supports audit requirements.

Imagine that the tester scopes a basic set of test ideas. The third iteration of the visual test coverage model might look like this:

Expand your horizons

When inspiration evaporates, the next challenge is to consider whether the test ideas captured in the model are sufficient. There are some excellent resources to help answer this question.

The Test Heuristics Cheat Sheet by Elisabeth Hendrickson is a quick document to scan through, and there is almost always a Data Type Attack that I want to add to my model. The Heuristic Test Strategy Model by James Bach is longer, but I particularly like the Quality Criteria Categories that prompt me to think of non-functional test ideas that may apply. Considering common test heuristics can help achieve better test coverage than when we think alone.

Similarly, if there are other testers working in the project team ask them to review the model. A group of testers with shared domain knowledge and varied thinking are an incredibly valuable resource.

Imagine that referring to test heuristic resources and completing a peer review provides plenty of new test ideas. The fourth iteration of the visual test coverage model would have a lot of extra branches!

Lift Off!

From this point the visual test coverage model can be used in a number of ways; a base for structured exploratory testing using session based testing, a visual representation of a test case suite, a tool to demonstrate whether test ideas are covered by automated checks or testing, or as a radar to report progress and status of test execution. Regardless of use, the model is likely to evolve over time.

I hope that this process encourages those who are new to visual test coverage modelling to try it out.

Evolution of a model

I've been meaning to share some examples of visual test coverage models using mind maps and how these may evolve over time. This example is from a group of testers working on a training project that I've blogged about previously. It is a tidy demonstration of the rapid learning that is common in software testing and clearly illustrates how a model changes over time.

The first version of the model was directed by what the Test Manager told the team was important to test. Though the team had access to the application, there was a limited testing time frame and they did not stray far from the functionality mentioned in their initial briefing. Test ideas within the model were grouped into sessions with a specific charter, which are represented as pale gray boxes.

The team working on this piece of function included three testers who each executed a session. Every tester found at least one bug in their first session. They decided to reflect this against their model by colouring the completed sessions in red.

Before executing their second session, the group re-evaluated. Their first set of sessions had exposed them to a wider variety of function than the Test Manager had mentioned. They questioned whether other aspects of the application under test were also important, and whether the focus of testing should change accordingly. As a result, the left hand side of the model started to expand and, after discussion with the Test Manager, the priorities for their second set of sessions changed. To be clear about where their focus was, the team updated their model to show priority by adding numbering.

At the same time the wider project team adopted a common colour scheme for reporting, so that the Test Manager could expect models from different teams to adopt the same conventions. As a result, completed sessions were marked in green regardless of whether bugs had been discovered.

After the third set of sessions things changed again. The testers felt comfortable with the application, which had been new to them at the start of the process, and started questioning the Test Manager on the applicability of their originally proposed scope. As a result, the right hand side of the diagram became significantly simpler.

In addition, scope added during exploration was refined into succinct and understood coverage. At the end of these sessions testing against the left hand side of the model was complete.

Once all of the scoped sessions had been completed, the model looked quite different to the initial version.

Though this project was slightly contrived for training purposes, I have observed the same rapid evolution of visual models in many client projects. By representing test ideas in a format that is easy to adapt, we make our testing more flexible and responsive to change.

See also:

• Using mind mapping as a visual management tool - Aaron Hodder
• Mind Mapping 101 - Darren McMillan
• To mind map or not to mind map - Adam Knight

Mind Maps and Automation

I've written in the past about the risk of relying solely on automated checking in release decisions. On a recent project I had great success in changing how the automated test results were used by including mind maps in the reports generated.

Technical background

We were developing a web application using Java, which was built using maven. I created a separate module in the same source repository as the application code for the automated checks. It used Concordion to execute JUnit tests against the application in a browser using Selenium WebDriver. The suite executed from a maven target, or tests could run individually via the IDE. I had a test job in our Jenkins continuous integration server that executed the tests regularly and produced an HTML report of the results using the HTML Publisher Plugin.

What does it look like

Below are the results for a Sample Application*. The job in Jenkins is green and more information about this success can be found by hovering on the job and clicking the HTML report.

My past experience was that management engage with the simple green / red feedback from Jenkins more than any other type of test reporting. Rather than continuing to fight this, I decided to change what it could tell them. There will always be aspects of functionality where it does not make sense to add an automated check, bugs that fall beyond the reach of automation, and decisions about scope that cannot be easily captured by automation alone. I wanted to communicate that information in the only place that management were listening.

The report is designed to be accessible to a non-technical audience and includes a pre-amble to explain the report structure. The entry point is designed to provide a high level visual overview of what testing has occurred, not just the results of automation. I found that the scrum master, product owner and project manager didn't drill further in to the report than this. This is essentially a living test status report that does not contain any metrics.

Each feature was highlighted to reflect a high level status of the checks executed within that space (that's why Sample Feature above is green). It linked to living documentation, as we used Specification by Example to define a set of examples in the gherkin format for each story. 

Although written in plain English with a business focus, these specifications were rarely accessed by management. Rather they were used extensively by the business analysts and developers to review the behaviour of the application and the automated checks in place to verify it. The business analysts in particular would regularly provide unsolicited feedback on these examples, which is indicative of their engagement.

Having different levels of detail accessible in a single location worked extraordinarily well. The entire team were active in test design and interested in the test results. I realised only recently that this approach creates System 1 reporting for mind maps without losing the richer content. It's an approach that I would use again.

What do you think?

__

* This post was a while in coming because I cannot share my actual test reports. This is a doctored example to illustrate what was in place. Obviously...