#KWST4

I made it along to Day Two of the fourth Kiwi Workshop for Software Testing (#KWST4) this weekend. The theme was "How to speed up testing, and why we shouldn't". It was great to see many new faces, and hear some very interesting experiences. I'd like to share three topics of conversation that resonated with me.

Fast go, throw away stuff

The morning opened with Viktoriia Kuznetcova who spoke on her three strategies for speeding up testing in an environment with time pressure; cluster, prioritise, and parallelise.

Viktoriia spoke of an environment where release dates were often determined in advance, then extensively advertised to the existing user base. This meant that there was little opportunity to amend project timelines when understanding of the solution changed. She said that she felt, at times, the release dates were "not possible". Yet because the sales department had set an expectation in the market that the feature would be delivered on a certain date, the team would occasionally be asked to release on deadline without completing testing.

For me, this experience illustrated the perception that stakeholders outside of testing often have; to speed testing up, we just cut down the timeline and stop testing earlier. I feel that this view is born from observing that testers are naturally cautious, and experience with successful releases where testing was incomplete.

On the topic of "fast go, throw away stuff", people spoke about:

• Are other teams are dependent on a certain level of testing being completed? It's important to consider this when prioritising your own test activities with a view to eliminating the least important. 

• If testing a release for a specific client, and testing is being targeted to meet their specific needs, will the cost of regression testing for later releases with a general audience then be inflated?

• If you can't quantify the change in quality associated with a change in test scope, then it's hard to offer evidence to support a longer testing time frame. Without this, managers may not see any difference in the delivery to the client.

• For management, testing is often about building confidence. Once an expectation has been set that testing can be completed in X number of weeks, they may then take some confidence in future purely from the same period of time elapsing. "You've had four weeks of testing. We released after four weeks last time and nothing went wrong."

• Where permitted, testers may choose to continue testing beyond a release date. By stating that testing will continue, doubt is introduced in the stakeholder decision making about releases. If the test team are planning to continue regardless, this sends a very clear message that testing is not done.

Reducing your testing due to time constraints is gambling, sometimes you win, sometimes you lose, and you can become addicted #KWST4
— Sean (@seancresswell) July 11, 2014

Bug reports are not the output of testing

The second experience report of the day was from Rachel Carson. She spoke about her organisation shifting from a waterfall to an agile development methodology, and how this had resulted in a faster test process.

Rachel talked about how her bug reporting style had changed. She used to raise every bug she found in the bug tracking system of her organisation. With the shift to agile, she found that her approach became a lot more pragmatic. With frequent conversation between developers and testers, Rachel didn't always need to use the tool. When she did have to raise a bug, she though pragmatically about what would realistically be fixed.

For me, this experience illustrated how we can speed up testing when we stop thinking that bug reports are our output. The measure of a good tester is not the number of tickets against their name in a bug tracking system.

On this topic, people spoke about:

• It is a big mindset shift to step out of a blame culture, where the tester wants a written record of every issue they have observed in the system, to one in which problems are discovered and resolved without a paper trail.

• As testers, our job is not to generate information, it's to provide useful information. More is not always better, especially for written bug reports.

• In an agile environment, bug triage is owned by the team. Where there is a decision not to fix a problem, this doesn't necessarily need to be documented.

Primary product of testing is "informed people" says @AR_Tester - sometimes that's a bug report and sometimes it isn't. #KWST4
— Adam Howard (@adammhoward) July 12, 2014

Testers becoming BAs

The third and final experience report of the day was by Adam Howard. He spoke of his experiences in implementing visual modelling and session based testing in a challenging project environment.

Adam spoke about working on a defect-fix release. The defects were focused in a specific area of the application, but were so high in number that they essentially represented a complete re-write of that piece of the system. Adam used visual test coverage models to build a holistic view of the collection of defects, as developing and testing each in defect isolation would have resulted in a fragmented end product.

For me, this experience illustrated how we can speed up testing by taking ownership of some business analysis activities. The tester should not be the first person to visualise a model of the solution, yet often we are. By leading a collaborative approach to create a visual document, the team develops a shared understanding of what is being built, which can the job of the tester much easier.

Check out

My final and most practical take-home was a tip from Sean Cresswell. He spoke of a useful method for determining whether there was shared understanding of a technical solution across a team. Place a developer and a tester on opposite sides of a free standing whiteboard and ask them to draw a diagram of how they think the system works. I thought this was a quick, easy and brilliant way to spot any discrepancies in thinking.

I enjoyed my day at KWST4. Special thanks to Oliver Erlewein for organising and Richard Robinson for facilitating.

Thoughts shared here are as a result of group discussion between all KWST4 attendees; (pictured below from left to right) Katrina Clokie, Chris Rolls, Aaron Hodder, Parvathy Muraleedharan, Joshua Raine, Adam Howard, Richard Robinson, Andrew Robins, Oliver Erlewein, Viktoriia Kuznetcova, Thomas Recker, Rachel Carson, James Hailstone, Ben Cooney, Till Neunast, Nigel Charman & Sean Cresswell.

#KWST4 THE TROUPE! Big thanks to everyone for having some awesome Testing time. pic.twitter.com/7oESHuMJTT
— Oliver Erlewein (@Oliver_NZ) July 12, 2014

Open to Feedback

I spoke at AgileWelly last night in order to practice my talk for CAST2014. It was the first time that I'd presented in an auditorium, to a very large audience, from behind a podium, under a spotlight, with a microphone. The environment was certainly intimidating.

Before beginning, I emphasised to the audience that I was keen to receive feedback on my presentation, so that I could improve it before repeating myself on an international stage. I was nervous about asking people to critique my work because I was worried about what they would say, but I was also worried that they might not say anything! Indifference is the worst reaction.

As I finished speaking, I had already self-identified a couple of areas that I wanted to improve.

I thought that my introduction and conclusion were weak. These were the areas in which I was least prepared, and it wasn't as easy as I had imagined to improvise the content.

Additionally, when I checked the clock at the end of my first section, I realised that I was speaking far too quickly. I was so nervous that I flown through my slides, and I had to consciously collect myself in order to continue at my planned, and more sedate, pace.

Given that it was so easy for me to identify these two changes, even in overwhelming-post-presentation-brain-overload, my feedback fears intensified. No longer did I think that people wouldn't have anything to say. Instead I thought that they'd have so much constructive criticism that I would be overwhelmed!

In the last 24 hours I have been privileged to receive a great deal of feedback from a number of different people. Thankfully, it's been largely positive. The suggestions offered have been constructive, and I've seen some consistent themes emerge.

Many people have endorsed my self assessment. But in addition to identifying these same issues, they've also offered helpful and specific suggestions as to how I might change my approach. These have included both general presentation skills, and ways to expand particular pieces of my content.

I've also had feedback that I never would have thought of myself. Great ideas for how I might add content to the presentation based on the questions I received at the end, tips to promote my associated blog posts, and terminology for concepts that I was describing.

The feedback I've been given is a reflection on the strong IT community in Wellington. Thank you Aaron, Sarah, Craig, Ben, Stu, Larrie, Nigel, William, Adrian, Shaun, Yvonne, and others.

Though the whole experience was really challenging, both presenting in a difficult environment and opening myself to critique, I really have learned a lot from it. I would have felt annoyed to be leaving the stage in New York thinking that I could have done it better. I want to deliver the best talk that I am capable of.

If you're yet to open your next presentation to feedback, I would encourage you to do so.

How to make a workshop hum

On Monday I co-facilitated a two hour workshop. It was good, but not great. I felt that the room was a bit flat, and wasn't entirely sure that my message had been conveyed effectively.

Today I had the opportunity to run the same content again in a different city. Thirty minutes before the first student arrived, I decided to tweak the material. I wanted this version of the workshop to hum.

And hum it did. Though I can't be sure whether it was the people in the room or the material, I thought I'd share the three things I changed. I certainly think they contributed to the shift in outcome.

Give and Take

The first change was near the beginning of the session. We had asked the participants to do a series of activities in fairly quick succession, which came about as a result of attempting to condense some old material into a smaller time frame. We had valued our interactive exercises above our static slides, which meant that there was no longer any significant presenter content between some of our activities.

On Monday, it felt like we were taking a lot from our students and giving very little in return. We wanted them to think about this, and then think about that, and then think about something else, with very little time in between for them to pause, digest and absorb. Though we ultimately wove all the pieces together, the balance felt wrong.

Today I re-instated one slide. Just one. Doing this created a few minutes of space between two activities that asked people to think pretty hard. By adding this piece of content, I gave something back as the presenter. I gave people enough time to feel that the output of the first activity was acknowledged, and gave their brains time to recover!

This change altered the attitude of participants. The second activity in this series was tackled with enthusiasm instead of reluctance.

Grow the Numbers

The second change was in the classroom dynamics for our exercises. In the Monday session we jumped between asking people to work as groups, as individuals, as groups, then as individuals again. Working individually makes people introspective, somber and comparatively withdrawn. Working in groups is collaborative, dynamic and engaging. By mixing the numbers in each of our activities, classroom participation was see-sawing.

Today I changed the activities specifically to create an increasing momentum through the module of material. I started with an informal group conversation, which worked as an icebreaker to have people comfortable with those around them. Then I ran an individual exercise, an exercise in pairs, then finally an exercise in larger groups.

Creating this progression significantly altered communication through the workshop. Student engagement evolved in a much more cohesive fashion; I had attention and participation to the very end.

Set up for Success

The third change was to our final exercise. It was migrated from another area of our training, but on Monday we found that it was a much harder problem in its new context than in its original one. In addition, we asked students to complete this last exercise alone. Having a final exercise that was both challenging and silent meant that the class finished on quite a flat, serious note.

Today I re-designed this exercise so that the students had a better chance of success. I left the answers from the previous problem on the whiteboard, as a prompt for their thinking. I provided an expanded mnemonic as a reference. I also switched to a group format so that they could use one another as resources, and actively encouraged them to collaborate.

These changes meant that the answers provided by each group at the end of the session reflected a real understanding of the concepts that I was trying to teach. Further, the students themselves recognised that they had grasped the material, and the room was buzzing with their shared success.


In the coming weeks I plan to revisit the rest of my training material and apply these same three principles across each module; give and take, grow the numbers, set up for success. Today makes me believe that this is how to make a workshop hum.

The ratio myth

Recently I've been hearing a lot about the developer to tester ratio for agile teams. This is usually evangelised as an ideal of 4:1; four developers to one tester.

Having worked in many different agile teams, this ratio doesn't align with my experience. Here are a few examples of successful teams I have worked in with structures outside the norm.

More Developers [6:1]

In one of my earliest roles within agile development, I worked as the sole tester in a team with six developers. We were tasked with developing a system from scratch that would replace a legacy application.

The business had a clear vision, were engaged in what the team were doing, and sought to provide timely feedback on early versions of new features. Their involvement bought the users' expectations to any new features as soon as practical.

The developers had a lot of infrastructure oriented work to do, as they were setting up an entirely new application stack. In addition to technical stories, the team always carried at least one with a customer focus, to provide continuous delivery of business value. Features were small and well-defined from a business perspective, but usually involved the interaction of multiple systems behind-the-scenes, which meant they took time to develop.

As a tester, my primary focus was developing and maintaining a tidy suite of automated regression checks. The testing I did was either quick and sympathetic, so that simple issues were discovered and resolved before early engagement of the business, or aggressively sought out interesting problems.

In this team I often felt bored. Though I had six developers, it took a lot of programming to produce small pieces of front-end function. Fast feedback from the business reduced the amount of testing I had to do. I didn't feel challenged.

Less Developers [2:1]

In a different organisation, I worked in a team tasked with re-designing an existing purchasing workflow in an online marketplace. The team included two designers, two developers and myself as the sole tester.

In this team, the pace of development was determined by the rate at which the designers could agree on the updated user interface and workflow. Programming changes were to reflect paper prototypes; updating stylesheets, JavaScript and links between pages.

Though development was relatively simplistic, the testing required was significant. The existing workflow included a number of different business rules, which were recorded in varying detail across multiple documents. There were a vast number of variables to consider in purchasing an item, both obvious variables and those that were indirectly accessible. There was no automated checking available in this domain.

In this team I was busy. Due to the financial impact of failure in this workflow, testing had to be comprehensive and meticulous. Proportionally, much more time was spent in testing, so the output of two developers was enough to keep me occupied.

More Testers [4:2]

Finally, in a third organisation, I worked in a team asked to fix a production issue in a complex shared web service. This team included four developers from four specialist areas; mainframe, database, integration and front-end. I was one of two testers.

In this environment, the complexity of development varied dependent on the system. Most changes were isolated to a specific aspect of functionality, which could only be observed when users followed a specific workflow.

From a test perspective there was a lot of work to do. As in the previous team, the code change was in an important area of the application with complex business rules. Unlike the previous team, there were automated regression checks available at multiple layers of the architecture.

In this team I was challenged. Testing had to occur on each piece of development in isolation, then via multiple front-end consumers of the web service. In addition, due to their specialist focus, a large component of testing in this team was to facilitate communication among developers. With a tight deadline, and proportionally more testing than development, we needed two testers to get through everything.

*****

There are a lot of variables in determining the best ratio of developers to testers in an agile team. A ratio of 4:1 may be a good place to start, but inflexibility in resourcing is folly. There is no hard and fast rule.

If you encounter a scrum master that attempts to enforce a ratio at the expense of common sense, perhaps you can remind them of the first principle of the agile manifesto. Individuals and interactions over processes and tools. A ratio is just a tool, so be sure to listen to your people if they tell you it's not working.

Personal Brand

Recently a colleague in Marketing referred to my "strong personal brand". I found it quite a confrontational statement as it was something I hadn't consciously considered before.

That same week, I had someone approach me to speak at a public event. Usually if I want to speak in a public forum I have to slave over a proposal, then wait to find out if it has been accepted. To receive an invitation was an unexpected honour.

These two experiences prompted me to consider my profile, both in my local community and internationally. Upon reflection I realised how far I have come. I thought I would share some key points in my journey, in the hope that it offers ideas or inspiration to others.

Join your community

The first role I had with testing in my job title was not my first role doing testing as an activity, which is a fairly common path into the profession. My journey starts here because it was in this role that I was lucky enough to have Aaron Hodder as a colleague.

It was Aaron who, fairly relentlessly, pestered me to join the Software Testers New Zealand mailing list on Google, and to start asking questions in that forum. It was Aaron who told me I had to go to the Rapid Software Testing course that James Bach was delivering in Wellington. It was Aaron who wrangled my invite to the second Kiwi Workshop on Software Testing (KWST). As a result of Aaron's interfering, I found myself part of a small community of testers spread across New Zealand.

Speak up

Being a passive community member is a good, but being vocal is better. I entered these forums with reluctance and a selfish agenda. I wanted to use these people for help in solving my own problems. However, once I was engaged, I was surprised to realise that I had ideas and opinions to share too.

Participating in a mailing list is one thing, participating face-to-face is an entirely different proposition. Though I didn't plan on saying much at my first KWST peer conference, I found myself in a situation where I was unable to stay silent for fear it would be construed as complicit agreement. I spoke up.

It is only by speaking up that people can gain a sense of who you are, what you believe, and how you behave as a professional. Starting to contribute, even in a small way, is the first step in making yourself known.

@katrina_tester @kengel100 That's when I knew Katrina was a heavyweight.
— James Marcus Bach (@jamesmarcusbach) June 15, 2014

See how far you've come

Around this point, Brian Osman invited me to do a brown bag lunch talk at his workplace. I remember his pitch for this being very understated. "It will be relaxed, a few people, eating food, talking testing". I agreed, then panicked when Brian asked for a copy of my slides in advance!

I think I spoke about test planning; nothing that seemed exciting to me. Yet the small audience viewed my ideas as a radical new way of doing things. I realised that in challenging and being challenged within the community I had joined, my ideas had developed beyond those who were yet to question their work. I realised that I had grown into a position to give knowledge back.

Invite others
I was lucky to have Aaron pester me to take the first step on my journey. When I realised how the community had helped me grow, I wanted others to experience the same.

Though I saw a national testing community in New Zealand, I didn't see a local community in Wellington. I wanted to invite people to something that was ours, so Aaron and I decided to create regular peer workshops in our city. WeTest Workshops was born.

It took weeks for our first event to fill. We asked everyone we knew. Then we asked them to ask people. Then we asked on mailing lists and twitter. I remember how excited we were to finally find 20 people who wanted to talk about testing. [Community Question]

It was, and still is, such a buzz to see other testers discover a community of their peers, find a friendly forum for their problems, and challenge their thinking.

Look further afield

At the third KWST, Erin Donnell inspired me to join Twitter and start a blog. These were two steps that I was incredibly reluctant to take.

In retrospect, I can identify that the primary reason I didn't want to do this was fear. I was comfortable in my own community. I didn't trust who or what was out there in the wider world. People kept talking about "the international testing community". I thought it sounded ominous.

One of my first Twitter followers was James Bach. Though I felt this was a huge honour, I also felt a crushing responsibility to only tweet things that were interesting, unique, and insightful. Many of my first tweets were typed, edited multiple times, then abandoned.

Similarly on seeing people read my blog, I felt paralysed when considering a new post. What could I say that hadn't been said by others before me? How could I show my own thoughts and skills?

Someone told me that the first step was to start writing, the second step was to make it good. So the way that I got past these fears was via self imposed measurements.

On my blog, I decided that I wanted to write at least three blogs per month. I embraced a quantity over quality mentality to grow my confidence and find my writing voice.

On Twitter, I decided that I should write a tweet per follower; for over six months I kept this ratio at 1:1. This allowed me to be measured in what I contributed, but stopped me from adopting an entirely silent presence.

Present a consistent image
I like things to look consistent. When I started to create a professional presence online, I chose to present myself the same way in every platform. Same photo. Same colour schemes. Same choice of background images.

Though these choices were driven by my peculiar personality, they had the unintended consequence of giving me a very clear "personal brand" (to retrofit the phrase from my marketing colleague). It's something I would recommend to others who want to develop a recognisable identity in the community.

Take opportunities to participate
Having found my feet on Twitter, I discovered a world of opportunities for contributing. The lessons I had learned in my local community applied to an international context; to be known you must speak up.

I wrote for Testing Circus [Is That Testing?] and Testing Planet [Popping the Bubble]. I realised that there was an opportunity for the New Zealand and Australian test community to have a collective voice on the international stage and launched Testing Trapeze, a new software testing magazine, with help from a lot of awesome folk.

I started to advertise my writing. I listed my blog on Ministry of Testing. I asked people I knew to link back to my blog from their own. Over time, people I didn't know started to link back to my blog too.

I wrote conference proposals for EuroSTAR, Let's Test, CAST2014, Agile 2014, and Let's Test Oz. I found writing a proposal a great experience for clarifying my ideas, even when I was ultimately rejected. Of the five proposals I've written, I'm excited to be heading to New York and Sydney this year.

There are a lot of people out there who want to hear what you have to say; start taking these opportunities.

Read widely and share selectively
The people that I see as leaders in the international community have a particular approach. They create good content themselves, and they share the best content of others. This is something that I try to emulate.

Twitter can be a noisy platform, so being known as a reliable filter of information is another way to grow your following. Where a retweet is sparingly given, I believe it holds greater credibility. Complement your voice with others of value; have people trust you as a gateway.

Additionally there's an individual benefit in being widely read. You see themes in the types of problems that testers are facing. You see communities of thinking appear within the whole. You see a variety of opinions and thoughts that help to shape your own.

Be generous
Community is reciprocal. Be generous in promoting the achievements and events of your local and international peers. Demonstrate your pride. Show you care about the problems that people have overcome. If you're generous to others, then your work will be shared with the same enthusiasm.

The occupation question

I have been away on holiday. As I left New Zealand I had to complete immigration paperwork. One of the prompts on immigration forms is for occupation. Before you can enter or leave a country, their government would like to know what you do.

I have done a fair bit of international travel and used to diligently write 'software tester' in this space. I would hand my paperwork to a bored official for them to scan their eyes across, only to have them pause in their process and look at me through narrowed eyes. "Software tester" they would question, "What is that?".

Without fail, I would end up in a long conversation about what my occupation actually was. A conversation that usually concluded with the official saying dismissively "Oh, you work in IT", stamping my passport, and sending me on my way.

Because of this, I've changed my strategy. Now when I fill in the form, I just write 'IT'. This annoys me as it seems so general and sweeping. But the officials are satisfied and there are no questions asked.

Why doesn't everyone know about software testing?

A job exists to fulfill a need. In most cases, the need is obvious when you think of the alternative. For example, a bus driver is employed to drive a bus. Without the driver, the bus would not go. The need for a person to fill this role is very clear.

Without doctors we would stay sick or injured. Without plumbers our toilets would not work. Without electricians we couldn't use our lights. Without dairy farmers we wouldn't have milk, cheese or butter.

Without software testers, then what? My first instinct is to claim that software quality would degrade. To people outside of the software industry that means very little. Computer users are conditioned to expect their software to be imperfect. Though the degree of imperfection would change in the absence of testing, without seeing the better solution many would just assume that what they were given was how it had to be.

How do others within software development avoid this problem? Programmers write code, so it's easy to understand that without them the software would not exist at all. Architects and Project Managers have strong parallels to similar roles outside of our industry, which makes them easier to grasp.

Those with an understanding of software development will, generally, see the need for and value of testing. They have expectations of software and can identify when these expectations are not met. I think it would be nice to see this knowledge spread to a wider audience.

I'd like to be specific about my occupation without having to explain what it is. Wouldn't you?

How to get feedback on a strategy

In my experience, writing then distributing a strategy document is a fairly limited process to receive input and feedback. The type of critique that you get is generally narrow rather than far-reaching. The number of people who choose to respond is often very low.

As part of my role, I'm responsible for developing strategy. It's my belief that the best way to get people engaged in a specific direction is to get them involved in setting the course. Recently, as part of defining strategy, I decided to run a workshop activity to seek feedback in an interactive way.

The structure of the exercise I used was based on one demonstrated by Adele Graham in a training course that I attended. Adele split the class into groups of four or five people, then gave each group a set of three coloured cards that read:

Adele then made a statement and gave the class five minutes to discuss the statement in their groups. At the end of this time each group had to pick the card that reflected their collective opinion of the statement. There was then discussion among the class where differences existed between groups.

I decided to use this same exercise to get input on my strategic thinking, focusing on areas in which I was uncomfortable about making a decision. By opening these aspects to critique, I hoped that the group would bring forward ideas that I hadn't considered.

I had seen that the best discussion in Adele's session came where the statements expressed a polarity; my language had to be clear and absolute. I worked to frame my statements so that the tester favourite of "it depends" would be abandoned in favour of clear choices, yes or no.

I also decided to intersperse the strategy discussions with those that would give me insight into the existing opinions of those present, to give me confidence in areas that I felt my thinking was sound.

Ultimately, I had a set of four statements for discussion:

• A mind map is an easy way to present my test ideas to other people
• [PLATFORM] is the best way to create a [PRACTICE] community at [ORGANISATION]
• I could do [PRACTICE] in my current role at my current client
• One hour of training, once per month, is the best way for me to learn more about [PRACTICE]

I had planned to spend 20 minutes on this piece of the workshop. I estimated that I would need to allow three minutes for each individual group to decide, then two additional minutes for wider discussion in the case that there was disagreement between groups.

On the day, I had 23 people at the workshop. For this exercise, I split the attendees into five separate groups based on their current client engagement, in order to support discussions against the third statement.

We began, and agreement was reached on the first statement almost instantly:

Today I had 23 testers agree, no concerns, with the statement "A mind map is an easy way to present my test ideas to other people" #testing
— Katrina Clokie (@katrina_tester) May 15, 2014

This made me a little concerned that the exercise wouldn't achieve what I had hoped!

Fortunately, the remaining three statements created the type of conflict I was expecting to emerge and provided a good catalyst for opinionated group discussion. As a result, I felt that the group understood the types of compromise inherent in these areas.

My expected timing didn't hold true for each piece of the process. In every case the groups had decided well within the three minutes. However the resulting workshop-wide discussion was much more detailed than I had planned, as different opinions were aired. Overall, we hit the 20 minute estimate for the entire exercise.

I was happy with the breadth of thinking that appeared, and the conclusions reached where opinions were varied. I felt that this workshop gave us a strong foundation to begin from, as the strategy should now reflect the agreement reached here.

How do you seek input on strategic direction? Any other innovative ideas?