Non-functional testing in continuous delivery

Last year I worked in an organisation that did twice-daily production releases with very little non-functional testing. Often the litmus test of whether a release met non-functional requirements was to push it out to users and closely monitor the results. There was an amazing operations team who had a rapid automated production rollback procedure, which could clean things up in a flash, if required.

On joining this organisation as a tester I was astounded by their approach. On my first day they had a bad release go to production and roll back off the live platform. I felt that I had entered the Wild West of software development; it appeared reckless.

As time went on I saw the process repeated, there were more good releases than bad, and things started to make a certain amount of sense. In a twice-daily release cycle there isn't time to spend on detailed performance testing, usability sessions, or security audits. Though I became more comfortable, I had a nagging doubt that this was not the way that others were solving this problem.

At CITCON this weekend I had the opportunity to find out, by facilitating a session titled "How do you incorporate non-functional testing in continuous delivery?".

The first thing that struck me, after sharing what I just described, was that no-one jumped to volunteer a better solution. People started to talk around the topic, but not to it. I had to repeat my question several times before one attendee said "We focus on functionality and kind of ignore non-functional testing". I felt this statement was reflective for many in the group.

Someone proposed that the first problem in incorporating non-functional testing is a lack of written non-functional requirements. People can quickly determine whether something is not working by means of it being too slow, or difficult to use, or succumbing to malicious infiltration. Defining what is expected from the application for performance, usability, and security, is much more difficult. The rapid pace of continuous delivery, coupled with a relatively robust process for testing in production, creates a compelling excuse not to stop and think about non-functional requirements.

In the case that requirements are present, how do testers find time to test them? General consensus was that the requirements would form the basis for a suite of discrete automated checks designed to alert the tester; a prompt to hold the release while the tester investigated the problem. Pre-release non-functional testing would be driven by a failing check.

In the case of performance, the check may fire when a threshold is exceeded, or highlight a marked degradation that still falls within the threshold e.g. if the page load time jumps from 0.3s to 2s, and the threshold is set at 4s, we would still want to know about this change. Some in the audience had already implemented lightweight, targeted, automated performance checks that were running in their continuous integration environment.

As the conversation turned to security there was doubt that the same principle could be applied. However one tester in the audience was doing just this by using the results of security audits to create scripted security checks. Though vigilance is required to keep up with evolving security threats, he felt that the maintenance overhead was no different to any other automated test suite.

Finally we spoke about usability. The first thought from the audience was that perhaps A/B testing is how most companies achieve this in a continuous delivery environment. Those assembled were familiar with the concept as New Zealand is often used as the trial region for new Facebook features. Some used this approach, though others argued that if your focus is user loyalty or sales you may not want to risk alienating a proportion of your clientele by giving them a weaker design.

Interestingly, there were those who thought that the same principle of checks may even work for usability. In particular, the accessibility aspects that often require that the application can be used by a machine. Tools to check for tab order, alternate text in images, appropriate colour and contrast, and valid HTML were all mentioned.

The session finished with a conversation about whether this would really work. The arguments against seem to be invalidated by the type of organisations that choose continuous delivery. Organisations that make frequent releases a priority and pride themselves on responsiveness must acknowledge that this comes at the expense of quality. It's fine if a user sees something that isn't quite right, so long as its only briefly. I found it interesting that those with real-world experience in continuous delivery often worked in an iconic or monopolistic organisation where the user has strong brand loyalty and little choice.

Are you using continuous delivery? How do you incorporate non-functional testing?

A culture challenge

A comment on my previous post reads:

I'm currently in one of those places where finger pointing and politics are the norm (at higher levels - my team itself is great). While proactively accepting blame might sound like a great & noble thing to do, that's like voluntarily putting your head on the chopping block when no one is even asking that of you. Not gonna happen! We just talk amongst ourselves about what we can do better next time.

I felt that I needed a whole post to respond to this one, because it really made me wonder.

Where do you think culture comes from?

If you're in an organisation where finger pointing and politics are the norm, then ask yourself why that is. Try walking in the shoes of the person who is behaving in this way. Imagine being in a management position with responsibilities that straddle a number of teams; you try to manage risk, ensure that mistakes aren't repeated, and report on your department to higher levels of the organisation.

Now imagine that the teams you are responsible for are insular. That they talk amongst themselves, but they won't tell you anything. You know that anything you do hear is only part of the story. What do you do? Without accurate and complete information you cannot do your job.

Finger pointing happens when your manager is frustrated. When there is an endemic lack of ownership, finger pointing feels like the only way to assign and action improvement. I believe that finger pointing is not a reflection on your manager, it's a reflection of your behaviour as a team.

Who do you think changes your culture?

A manager is not going to change their approach when it feels as though a witch-hunt is the only way to find out what is going on. If you want to stop being persecuted, start taking responsibility for failures and communicating the things that you plan to improve as a result. Own it. Take away the reason that they behave the way that they do.

It's scary to be the first person to stick your neck out. I don't deny that. But if you want to build a relationship of trust, then you have to act like someone who can be trusted. The culture of an organisation is the result of the behaviour of every individual in it. If you want to see change, you have to make it.

Own it

It feels like testing suffers as a profession because we fail to own our failures. We are quick to point out the plethora of reasons that something is not our fault. Where a product is released with problems we didn't have enough time, or we weren't listened to, and anyway, we didn't write the bugs into the code in the first place so why are you blaming us?

Testing, perhaps more than any other discipline in software development, includes a number of pretenders. These people may be called fake testers, possums, or zombies;  there are no shortage of names for a problem that is widely acknowledged. Yet they remain sheltered in software development teams throughout the world, pervasive in an industry that allows them to survive and thrive. Why?

We don't take the blame.

Think of a retrospective or post-project review where a tester took ownership of a problem, identified their behaviour as a cause, and actively worked to prevent recurrence in their work. Now think of the problems identified in testing that would be gone if the developer had delivered the code earlier, or if the project manager had allowed more time for defect fixing, or if the business analyst had identified the requirement properly. It seems that more often we attribute our problems elsewhere; fingers are pointed.

It is a brave thing to claim a failure. In doing so we acknowledge our imperfection and expose our flaws. I think that testers do not show this bravery enough. Instead, criticism of a poor product, or a failed project, is water off a testers back. We escape the review unscathed. We cheer our unblemished test process. We see this as a victory to be celebrated with other testers.

This is what allows bad testers to hide. Where a tester leaves a review without ownership of any problems they are warranted in considering their contribution successful. A tester may then consider themselves associated with any number of "successful" projects, by definition that none of its failures were attributed to them.

How do we fix this? By considering how everything could be our fault.

Imagine a project that goes live in production with a high number of defects. In the review meeting, one tester claims that the project manager did not allow enough time in her schedule for defect fixing. An action is taken by the project manager to allow more time for this activity in the next project.

Another tester on the project thinks about how this same problem could be their fault using the test of reasonable opposites, the idea that for every proposition you come up with there are contrasting explanations that are just as plausible. In this example the proposition is that the project goes live with a high number of defects because the project manager did not allow enough time in her schedule for defect fixing. A reasonable opposite may be that the project goes live with a high number of defects because the testers raised many minor problems that the business did not need to see resolved before release.

From a reasonable opposite we now have an action to consider; should the testers treat these minor problems differently in future? The tester is prompted to think about how their behaviour could have contributed to a perceived failure. Once you start imagining ownership, it becomes easier take it, where appropriate.

As good testers start to claim problems and action change we erode the position of bad testers who consider themselves above reproach. When we stop finger pointing, we stop enabling others to do so. To change the culture of our industry and expose those who hide among us, we need to be comfortable in accepting that sometimes the things that go wrong on a project are because of things that we did badly.

The law of reasonable opposites; a good tool for testers in a review meeting.

Learner-Learner Interaction

I went on Adele Graham's Train the Trainers course last week, which started with an activity called "fears in the box". We were prompted to list any fears and worries we had about being in a training role and to circle our biggest concern. Each person wrote their name on the back of their list and posted it in a box at the front of the room. At the end of our training our fears were returned to us. We were asked to strike from the list any that had been addressed, then pick one lesson to share with the class.

One of the things I had written was "I have to keep remembering not to just tell them". What I meant by this is that I struggle to not respond to direct questions with an answer. In a training environment the trainer must encourage participants to think for themselves. There is no impetus for critical thinking where the answers are given readily. As a trainer I have to keep reminding myself that a question is not a loop to close, but rather an opportunity to encourage further learning.

Adele had addressed this fear by teaching us activities that facilitate learning rather than dictate it. But she went further by introducing me to a simple teaching principle that fundamentally altered my view of my responsibilities as a trainer. Being prudent with direct answers is not enough. I should also strive not to be the sole point of expertise in my classroom.

There are three types of interaction necessary for effective learning; learner-instructor, learner-content and learner-learner [1]. The first two were already part of my training room; the communication between teacher and student is present in any classroom and our exercise-driven training material is designed so that students engage intellectually with our content. But learner-learner interaction was a revelation.

Learner-learner interaction is the "communication between and among peers with or without the teacher present" [1]. By adding this layer of interaction you change the interactive topography of your classroom. Traditionally the teacher is considered the sole source of truth, which creates a centralised network of communication where dialog flows between the teacher and each individual learner. The learner-learner environment acknowledges the presence of other experts in the room, changing the dynamic to a fully connected network in which students interact with and learn from both the teacher and one another.

Credit: http://en.wikipedia.org/wiki/Network_topology

Learning about learner-learner interaction was a freeing experience. I had previously felt that all teaching in my classroom was my responsibility. It was nerve-wracking to stand in front of adult students with years of experience and shoulder the burden of expertise alone. Now I understand how I can recognise knowledge in the room and facilitate learner-learner interaction to supplement and enhance my teaching. I have strategies to identify the strengths of my students and activities that control the delivery of student contributions.

Fortunately I was in the classroom this week, putting what I learned into immediate practice. I was teaching to a class of ten graduates from a wide variety of subject disciplines. The material included a practical technical exercise to introduce performance testing. Limited computer hardware meant that the students had to work in pairs.

To start the session I asked the students to stand in a line up. I labeled one side of the room "I know nothing about performance testing" and the opposite "I can confidently explain performance testing and have experience in it". The students were instructed to identify their place in the line between these points and stand in it. The majority of the group stood in the middle of the room, reflecting a basic understanding of performance testing but no practical experience. However, there were two confident Computer Science graduates and two uncertain Arts graduates at the outer edges.

I used this activity to pair the group for the exercise. I took one person from each end of the line to form a pair. This meant that those who identified themselves as experts were paired with those who felt they had no prior knowledge. As the students took their seats I instructed that the member of the pair who was closer to the end of the line that knew "nothing" should use the mouse and keyboard, with the other person assisting.

During the exercise I was really pleased to observe how well the pairs worked. The exercise provided a basic starting point then a set of goals to drive independent learning. By pairing the outliers the ability of each couple was roughly equal; the pairs progressed through the independent portion at a similar pace.

By recognising the knowledge in the room, I was able to utilise it. My time as an instructor was mostly spent with those pairings from the middle of the line up. The two students who self-identified as requiring the most assistance had personalised instruction from their peers. Facilitating learner-learner interaction made my job as a trainer much easier, without compromising learning. I'm looking forward to applying more of Adele's material in future.

[1] Moore, M. (1989). Three types of interaction. American Journal of Distance Education, 3(2), 1-6. via wikipedia

Through the looking glass

I created some training material late last year that has since been adopted by my colleagues and tailored to their style. Today was the first time I heard it presented back to me; the same and yet significantly improved. It was a really interesting experience.

As a presenter, you're often working in an overloaded state. You try to maintain a confident persona, while attempting to remember what comes next in your presentation slide deck, and monitoring your listeners for signs of boredom. You adopt selective attention to filter aspects of the challenging environment.

When you sit in a classroom as an observer, your perspective is entirely different. Without the pressure of public speaking, your mind has space to see and to think.

After watching my colleague today, I could see that the underlying structure of the material I had given him was not as good as it could be. The independent exercise was failing because the presentation assumed a higher level of prior knowledge than was warranted. We were drowning our students rather than leading them slowly in to deep water.

This insight was not at all clear to me when I was the person presenting the material. I had felt that something wasn't right; the material wasn't flowing smoothly and it was difficult to generate enthusiasm in the class. But I had no opportunity to step back and work out the root cause of the problem; my thoughts were instinctive not conscious.

And, importantly, I hadn't had this feedback from observers of the original presentation. Though I had valuable pointers on how and what I was presenting, it was mostly trivial or cosmetic. Giving blunt and far-reaching feedback without ownership of the material is a daunting task. It was great to go through the looking glass to offer feedback on my own work.

This is likely to have been an exceptional case; I can't imagine I will often find another person who can or will want to present my material back to me. But it was an incredibly enlightening experience that felt like something between pair testing and rubber duck debugging. I had all the benefits of watching a version of my presentation on video without the horrific awkwardness of hearing myself speak (nothing worse than listening to your own voice).

Have other people tried this?

Bold Goals

I was asked some very good questions at work today. They were the kind of questions that really got my brain going. I thought I would share them here in case they activate thinking for anyone else.

What is your bold goal for 2014? 

Not one of the goals you trot out with ease when prompted, but rather the one that feels like a stretch. Something daring and brave; the thing that you don't really want to voice because there's a risk you won't achieve it. Share it with someone you trust to change it from a dream to a possibility.

What is your biggest challenge for 2014? 

When I think about my challenges, I often focus on my frustrations. Paying so much attention to noise can be counter-productive. What's your biggest challenge? Identify it, then focus your fight on the general and not his soldiers.

What's your first action to make 2014 what you want it to be?

Trite but true, a journey starting with a single step. You know where you want to go and you've identified the biggest thing standing in your way. What's the first thing you need to do? Make it a priority. Steadfastly completing the pieces can turn our boldest and biggest in to everyday.

Three simple questions that gave shape and direction to my year.

Training Witches

It's holiday season in New Zealand. In the past week I have been reading the Modern Witch novels. There are seven books in the series that each tell a story of a new witch, from their discovery to independent practice. The process that is used to train each witch is consistent across the novels and has interesting parallels to software testing.

A practical evaluation of skill

The first step when encountering a new witch is to evaluate their skills. A healer scans the witch to determine the type of power they possess, then a witch with the same type of power conducts a practical evaluation of their skill in a controlled environment. There are parallels to a job interview where a person from human resources scans a CV to determine which role an applicant is most suited to then calls on someone from an appropriate department to conduct an interview.

I think the witches are ahead of us in what the interview contains. In every case the new witch is asked to demonstrate their skill. In my experience a software testing interview is a conversation to assess personality. A skilled interviewer may be able to extrapolate testing skill based on the responses of the applicant, but rarely is a practical demonstration requested that would confirm their suspicions. In many cases, the first time a new tester will actually test a piece of software is in paid employment.

When we skip a practical demonstration we are guessing what the practical skills of our new testers are based on the way they present themselves.

Apprenticeship training

After evaluation the new witch is assigned a trainer with the same type of power. The trainer teaches by demonstration; knowledge is transferred verbally and not by written word. The new witch sees the spell executed successfully before attempting to replicate it. They question what they observe and have a safe environment for failure in their own efforts. Witchcraft is taught via apprenticeship.

Apprenticeships give a new generation of practitioners competency in a set of basic skills. They are prevalent in manual trades but seem largely ignored in white collar work. This type of mentored learning would allow us to teach software testing in the workplace within a formal and acknowledged relationship. It gives the apprentice permission to seek, question and learn. It gives the trainer permission to guide, review and suggest.

When the practical education of a new tester is not prioritised and owned by an individual trainer the education path becomes varied. Though this is not universally bad, those who are impressionable may take a wrong turn. In the absence of personal assistance the learner may seek out textbooks and certifications to support their journey, which will have variable quality. Those who are trained by a team may miss core competencies as knowledge arrives from multiple sources. The potential of a new tester may not be realised.

A training circle

New witches work in a training circle to protect the environment from the power of the witch. The circle allows the new witch to experiment freely, failure is contained within its boundaries. 

A test environment is intended to be a place for the tester to experiment freely, but how much freedom do you feel? Often testers within these environments are still constrained; by limitations of our test data, by vulnerable links to third party systems, or by a single environment supporting a test team. To exercise our testing skills without fear we need a dedicated bubble in which to operate.

I think these fictional witches have a training path that we should strive for in software testing. What do you think?