A Practical Guide to Testing in DevOps is now available for purchase on LeanPub

Here is a sample chapter of the book:



DevOps is the latest evolution in a history of changes to software development. Waterfall to Agile, Agile to DevOps, each step has created corresponding change in the role and activities of the tester. The steps may not be distinct in every organisation, but for simplicity let’s consider them separately.

In a waterfall environment, testers have sole ownership of test strategy and execution. We revel in thinking up our own test ideas, independently exploring the software, locating and then reporting problems that we discover. Though we work with developers, business analysts, designers, test managers, and others, there is comfort in knowing that testing as an activity belongs to us. Testing is our role and our identity in the team.

In an agile environment, testing becomes an activity that the development team own together. The team think of test ideas. The team explore the software. With a variety of perspectives, the team find a variety of problems to discuss and resolve together.

The relationship between testing and the business changes too. Discussions about requirements became more collaborative through Behaviour Driven Development, the importance of acceptance testing is diluted by day-to-day business involvement in the development process, and fast feedback on iterative releases tempers the need for exhaustive pre-release checks. All these activities influence testing.

Agile challenges our definition of testing and ownership of testing as an activity. It changes the scope and pace of testing, engulfing the whole development team.

DevOps takes this challenge further.

DevOps pushes the development team to actively engage more widely within their organisation, beyond the roles that are involved in development to those that support it: operations infrastructure and monitoring, customer support, and analytics. This change broadens the network of people who collaborate, which will further extend the boundaries and nature of testing.

Understanding the skills, practices and tools available in areas that have historically been peripheral to the development team creates opportunities to push test activities outwards. Examples include on-demand cloud-based infrastructure that enables testing in a production-like environment, feedback from A/B test experiments provided by customer metrics, or beta testing groups that offer rapid customer feedback.

On the flipside, information into the development team also changes. Problems that are discovered in real-time monitoring dashboards, customer support tickets, or unusual analytics reports are received directly for prioritisation and resolution.

The other facet of DevOps is speed. The aim of fostering a more collaborative culture is to achieve rapid and reliable releases. The pace of DevOps can be particularly challenging to testers who may struggle to find the right balance of investigating new features without impeding the release.

Automation is usually touted as the solution to speeding up testing, particularly by those who are driving change. Smart use of tools is usually the best way to go faster, though this need not be through automating testing within the development team. Improving production monitoring and alerting, coupled with a rapid automated deploy and rollback, could be just as effective.

Several definitions of DevOps exist in the software community. Some are tool-focused definitions that have strayed from the original intentions of the movement. To help you read this book, here are the definitions we will use. This section also explains DevOps, its relationship to continuous delivery and agile, and where testing fits.

What is DevOps? 

The DevOps movement was born in 2009. The focus of the movement is to bridge the gap between development and operations, to reduce the risk in deployment of software.

"... there's still a gaping hole in what Chris Read calls 'the last mile'. The thing is, 'dev complete' is a long long way from 'live, in production, stable, making money'.

The problem is that it's typically the sysadmins who are responsible for getting the software out live - but often they don't know, trust, like, or even work in the same city as the developers. How on earth are we expected to deliver good software that way?"

Stephen Nelson-Smith

The original DevOps material focused on empowering people to work across disciplines to create a more reliable release process. The emphasis was on creating trust, both between people and in their processes and tools.

Releasing without fear encourages people to release more often. John Allspaw and Paul Hammond highlighted this benefit at VelocityConf in 2009 where they spoke of their 10 deploys per day through Dev and Ops cooperation at Flickr. At the time, this was an impressive release cadence.

DevOps hit the mainstream in 2013 when The Phoenix Project was published. Written by Gene Kim, Kevin Behr, and George Stafford, the book is a novel that illustrates DevOps principles through fictional characters. Widely read, and with largely positive reviews, this book is how many people have become familiar with the term DevOps.

I read widely for a definition of DevOps that resonated. The one that I liked best came from the collaborative input of many people and captured the broad scope of DevOps. Wikipedia defines DevOps as:

DevOps (a clipped compound of development and operations) is a term used to refer to a set of practices that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably. 

DevOps and continuous delivery

DevOps and continuous delivery are movements that developed in parallel, with closely aligned timelines and similar goals. This can create some confusion about the difference between the two.

In 2009, the same year that DevOps originated, Timothy Fitz coined the term continuous deployment.

"Continuous Deployment is simple: just ship your code to customers as often as possible."
Timothy Fritz

This was followed a little over a year later by the publication of a book titled Continuous Delivery. This term from Jez Humble and David Farley held a slightly different intent and became more prevalent.

"Implementing continuous delivery means making sure your software is always production ready throughout its entire lifecycle - that any build could potentially be released to users at the touch of a button using a fully automated process in a matter of seconds or minutes."
Jez Humble

Continuous delivery has a smaller scope than DevOps. It focuses on the technical practices that allow a development team to move quickly. It includes coding practices, source control management, and concepts such as feature flagging, which allows code to be deployed to production without being available to users. Jez Humble summarises what is necessary to achieve continuous delivery as:

"... ensuring our code is always in a deployable state, even in the face of teams of thousands of developers making changes on a daily basis. We thus completely eliminate the integration, testing and hardening phases that traditionally followed "dev complete", as well as code freezes."
Jez Humble

It is possible for the development teams in an organisation to adopt continuous delivery without any change in their operations team. This limited scope may be a sensible first step in improving the release process, but eventually it will create strain due to dependencies between teams that operate with a different rhythm. DevOps is the broader culture change that addresses this tension.

DevOps and Agile

Agile, at its genesis, was a manifesto with 12 accompanying principles. Its goal was to challenge the mindset of people involved in software development. There was no implementation detail, though adopting an ‘agile’ way of thinking would lead to implementation change within an organisation.

DevOps is more directive. It focuses on the relationship between development and operations. DevOps has a single desired outcome - to improve the reliability and frequency of releases.

Without agile, DevOps may never have emerged. When people within the development team start to think differently and work collaboratively under agile, they are able to regularly create smaller pieces of useful software. This causes tension with other areas of the organisation who have been accustomed to a slower process. DevOps addresses this tension by encouraging those who develop the software to work closely with those who support it.

But can you do DevOps without doing agile?

Yes. DevOps can strengthen relationships between people who are using any methodology. Improving reliability of releases could mean reducing the number of support patches for a product or decreasing the outage window that users experience. More frequent releases could be a target. In a non-agile environment, a release might be monthly or yearly. DevOps doesn't necessarily mean releasing every minute or hour.

However, DevOps is a harder path to follow without agile. People from different disciplines may not be comfortable working closely together. Agile provides the groundwork, so this book targets people who are looking to apply DevOps in an agile context.

DevOps and testing

Testing is conspicuous by its absence from most literature about DevOps. By nature, the focus is on developers and operations. Understandably this tends to make testers nervous about their role.

The people involved in shaping these communities fail to highlight testing because they are not from specialist testing backgrounds. This does not mean that testing isn’t happening, but rather that it is a pervasive activity throughout development, that isn't considered notable.

In reference to the DevOps tool chain, Dan Ashby writes:

"… you can see why people struggle to understand where testing fits in a model that doesn’t mention it at all. For me, testing fits at each and every single point in this model."
Dan Ashby

Dan visualises this as:

Image credit: Dan Ashby, Continuous Testing in DevOps

In their book on continuous delivery, Jez Humble and Dave Farley summarise the place of testing in a similar vein:

"Testing is a cross functional activity that involves the whole team, and should be done continuously from the beginning of the project." 
Jez Humble & Dave Farley

The difficulty for testers is identifying what their role is, when testing is expected to be a part of everything. What exactly is your responsibility? How should you try to contribute?

Beyond the testing role, there are questions for anyone involved in testing as an activity. Is there benefit in distinguishing testing tasks from other types of work in the development team? What weighting should you give to testing?


Want to read more?

A Practical Guide to Testing in DevOps is now available for purchase on LeanPub