▼

Tuesday, 15 September 2015

Continuous Delivery Testing Pathway

This pathway is a tool to help guide your self development in continuous delivery testing. It includes a variety of steps that you may approach linearly or by hopping about to those that interest you most.

Each step includes:
  • links to a few resources as a starting point, but you are likely to need to do your own additional research as you explore each topic.
  • a suggested exercise or two, which focus on reflection, practical application and discussion, as a tool to connect the resources with your reality.

Take your time. Dig deep into areas that interest you. Apply what you learn as you go.


STEP - Removing release testing

Why does this pathway exist? Understand the key reasons to significantly shorten a release process, the arguments against release testing and why organisations aim to avoid batched releases in agile environments:
EXERCISE
[2 hours] Research your existing release process and talk to people within your organisation to find out whether there are any current initiatives to improve it.


STEP - Introduction to continuous delivery

What is the end goal? Discover the basics of continuous delivery and the theory of how it can be implemented in organisations.
EXERCISE
[1 hour] Based on what you've read, try to explain the theory of continuous delivery in your own words to someone in your team. Describe what appeals to you about continuous delivery, what you disagree with, and things that you think will be difficult to implement in your organisation. Afterwards, if you have any remaining questions, raise these with a technical lead or coach for further discussion.


STEP - Experiences in continuous delivery

How are other organisations doing continuous delivery? There is a lot of variance in implementation and differing opinions about how to approach the theory. Understand the realities of the people, processes and tools of teams doing continuous delivery:
EXERCISE
[2 hours] Compare the experiences shared in the links above and the theory of continuous delivery. Identify common themes, and areas where ideas or implementation details differ. Discuss your analysis with a technical lead or coach.


STEP - Starting with continuous integration

What is the first step? Understand the concept of continuous integration:
EXERCISE
[3 hours] At the start of this talk transcript, Jez Humble points out that most people aren't doing continuous integration. How does the approach to continuous integration in your team differ to the theory? Talk to a developer to confirm your understanding of your branching strategy, the way you use source control management tools, and how you manage merging to master. If you use a continuous integration tool, create a list of the jobs that are used by your team during development, and be sure that you understand what each one does. Reflect on how quickly your team respond to build failures in these jobs, and who takes ownership for resolving these. Discuss this exercise with a technical lead or coach to collaboratively identify opportunities for improvement, then raise these ideas at your next team retrospective.


STEP - Theory of test automation

Continuous delivery puts a lot of focus on test automation. In order to support development of an effective pipeline it's important to understand common strategies for automation, and the distinction between checking and testing:
EXERCISE
[1 hour] Read through the automation strategy for your product. How well does your existing strategy for automation support your delivery pipeline? What opportunities exist to improve this strategy? Discuss your thoughts with a technical lead or coach.


STEP - A delivery pipeline

Understand how to construct delivery pipeline and the role of automation:
EXERCISE
[3 hours] Create a visual representation of the current delivery pipeline for your product. Use a timeline format that shows the build jobs in your continuous integration tool at every stage from development through to production deploy, any test jobs that execute automated suites, and points where the tester is hands-on, exploring the product. Compare your pipeline to the simplified images by Yassal Sundman for continuous delivery and continuous deployment, then reflect on the following questions:
  1. How would your approach to testing change, or not, if we were able to deploy to production 10 times a day? How about 100 times a day?
  2. Does the coverage provided by your automation give you a degree of comfort or confidence? If not, what needs to change?
  3. Does your automation execute fast enough? How fast do you think it should be? How can you achieve this?
  4. Where in the pipeline would you want to retain hands-on testing? How would you justify this?
Discuss your ideas with a technical lead or coach. Work together to identify actions from your thinking and determine how to proceed in implementing change.


STEP - Non-functional testing in continuous delivery

Learn more about integrating security, performance, and other non-functional testing in a continuous delivery pipeline:
EXERCISE
[2 hours] Does your organisation have a non-functional testing "sandwich"? Having read more about organisations who integrate these activities earlier in the process, what opportunities do you see to improve the way that you work? What would the first steps be? Talk to a technical lead or coach about what you'd like to see change.


STEP - Cross-browser testing

For continuous delivery of a web application, it's important to include cross-browser testing in the delivery pipeline. Discover strategies for cross-browser testing and the tools available to support it:
EXERCISE
[8 hours] Learn more about the common cross-browser tools that are available, understand the advantages and disadvantages of each option, then select a tool to trial. Create a prototype to execute existing browser-based automation for your product across multiple browsers. If successful on your local environment, attempt to create a prototype job in your continuous integration tool to verify that your chosen solution works as part of your continuous integration. Discuss what you learned about the tool and the results of your experiment with a technical lead or coach.


STEP - Test data & databases

Discover the additional considerations around test data in continuous delivery:
EXERCISES
[1 hour] Data is a constant headache for testers. Consider the limitations of the test data in use by your automation. How could you improve the data within your delivery pipelines? How could you improve the way that you locate data for testing? Talk through your ideas with a technical lead or coach.


STEP - Configuration management & environments

An effective delivery pipeline is supported by multiple test environments. Learn more about configuration management, environments and infrastructure services in continuous delivery:
EXERCISES
[1 hour] Talk to your operations or support team about how they provide test environments for continuous integration, the infrastructure required to support a delivery pipeline, and what their plans are for future changes in this space.


STEP - Testing in production

Understand A/B testing and feature toggles:
EXERCISE
[1 hour] Talk to people in your organisation to find out how you currently use feature toggles and how you make decisions about what to keep based on user analytics. Could your approach be more responsive through targeted use of a monitoring tool like splunk? Share your thoughts with a technical lead or coach.

at

2 comments:

  1. Maaret Pyhäjärvi15 September 2015 at 20:29

    Great pathway once again, Katrina. I was missing one step though: monitoring / testing in production. A lot of the work in this area end up sharing the testing activities to stuff we do before we release, and then to stuff we do when we have released. The concepts such as A/B testing, releasing to a percentage of users, finding out "the metric" that will tell you things have gone wrong badly and general ways to see what your users actually do would seem to be things that this pathway should include.

    ReplyDelete
    Replies
    1. Katrina Clokie15 September 2015 at 20:34

      Excellent point. I'll do a bit more research tomorrow and see if I can add another step. Thanks.

      Delete