Tuesday, 4 October 2016

Observation in Testing

At the WeTest Wellington Quick Lunch Talk today, Donal Christie of Powershop spoke on the topic "Do you see what I see?". Donal has been fascinated by observation from an early age - his favourite childhood toys included a magnifying glass, microscope and telescope. His talk focused on what we see as testers when we examine software.

Donal shared a variety of things to be mindful of, but there were three particularly interesting stories that resonated for me: Rubin's vase, Monet's cataracts, and Walmgate Bar.

Rubin's vase

Donal shared a picture and story about the vase created for the Silver Jubilee of Queen Elizabeth:

Most people are familiar with the Ambiguous Vase illusion. Devised by the Danish psychologist Edgar Rubin, we are not sure if we are looking at a vase, or at two faces, staring at each other.

In 1977, a wonderful 3 dimensional version of this illusion was made, to commemorate the Silver Jubilee of Queen Elizabeth. It was a porcelain vase, but one with a wonderful twist. The profile on one side of the vase was of Her Majesty, but on the other side of the vase, the profile was of Prince Philip. [Ref: The Queen's Speech]

Credit: The Queen's Speech

If you were asked to test this vase, what would be important? Is it the vase itself? Or the silhouette of the vase that shows the royal profiles? Or both?

How does this relate back to software? It's important to have a conversation with your business stakeholders about what the customer wants from your product, then learn what part of your architecture delivers that. What you see may not actually be what you need to test.

One example is feature development that introduces a new screen to the user interface and requires a new web service. It may be easy to test the user interface changes at face value. But we could see an entirely different perspective by testing in the web services layer.

Think of the web service change as the silhouette of the user interface changes. Perhaps it holds a lot of the business logic that the customer desires. Make sure you're testing what you see, but also think about what's around it.

Monet's cataracts

Donal shared a picture and a story about Monet's cataracts:

From 1900 onwards Monet had problems with his vision and complained to his friends that everything he saw was a fog. Although cataract operations had been performed for thousands of years they were still a risky business at the time. He agreed to surgery to totally remove the lense in his left eye in 1923 at the age of 82 and the operation was a success. There were no replacement implant lenses at the time and he had to wear thick glasses but his vision was transformed.

However, the operation had an unexpected side effect; as mentioned before it’s claimed that he began seeing the world with UV vision. His palette which before the operation had been red, brown and earthy took on a more bluish hue. [Ref: Claude Monet and Ultraviolet Light]

Credit: Claude Monet and Ultraviolet Light

People perceive colour differently. Though Monet's example is an extreme one, there are many people with impaired vision and colour blindness. For these people, what they see is not what you see.

Donal made the point that in these cases there can be more than one truth. To one person, the house as seen from the rose garden is red. To another, it's blue. To another, it's grey. None of these people are wrong. The way that they see the house will depend on how they see.

When we test software, you might hear people say "Did you see that bug?". In some cases, perhaps they didn't! Two people observing the same piece of software will form two separate truths. What you see and perceive will be different from your colleagues and your customers.

Donal advocated for pair testing and accessibility testing, approaches that try to incorporate multiple perspectives during the development process. I hadn't boiled down the benefits of these practices to a basic need for many people to observe a system. This is an argument I will be adding to my repertoire.

Walmgate Bar

Donal shared a picture and a story about Walmgate Bar, a historic location in York that he visited with his wife. They saw a plaque that described the site:

Credit: Donal Christie

Take a moment to read the inscription.

It may not be particularly striking. You probably know a little bit more about Walmgate Bar. Did you spot the two small errors? The first is that the word siege is spelled incorrectly. The second is in a sentence that has a duplicate word: "erected in the the reign of".

Occasionally we need to consciously shift our thinking to find different types of problems in software. We need to think about the system as a whole and determine whether it is fit for purpose. In this case, the sign is successfully communicating the intended information. We also need to examine the parts that make up the system and determine whether they are behaving correctly. This is where the problems crept in with the sign above.

I've found it particularly difficult to switch between these levels of thinking as a tester in an agile team. It can be easy to focus on testing each individual story and forget about testing the whole. I have a tendency to get bogged down in detail. The Walmgate Bar sign is a good reminder to think about both perspectives.

Interestingly it looks like this particular sign has now been replaced with one that is correct:

Credit: Yorkshire Walks

Donal's talk was a great reminder about observation and interpretation. He reminded me to consider:

  1. whether the product is what I can see, 
  2. that what I see may not be what others see, and 
  3. that the problems I find will change based on where I look.


  1. #2 problem resonated with me - well, all 3 do, but I wrote a blog post on the perspective issue a few months ago. http://janetgregory.ca/im-right-im-wrong-all-the-time/

    I seem to spend much of my time trying to work with people to see the other side. At least two perspectives to just about everything. One way to get testers/ team members to consider these differences is using personas to do exploratory testing.