Tuesday 29 September 2015

Performance Testing in Continuous Delivery

I went to Test Professionals Network in Wellington tonight and heard Diana Omuoyo speaking about Performance Testing in Continuous Delivery. Diana shared a really interesting story of her time working at Expedia in the US where she was a member of a small performance testing team. She focused on the changes made in performance testing to help support a shift from monthly product releases towards continuous delivery.

The boundaries of performance testing

Before beginning the journey towards continuous delivery, the Expedia performance testing team had two conversations with the wider organisation to decide:

  1. When is the application ready for performance testing?
  2. What performance testing results mean the application is ready for production?

These questions generated healthy discussion as there was a lot of diversity in opinion. The agreed answers established the boundaries of focus for Diana's team to implement change in their approach to performance testing.

Looking at their existing approach, the performance testers felt that most of their time was being lost in the slow feedback loop between development and performance testing. These two activities are so far removed from one another in a traditional development lifecycle, with several phases of testing happening between them, that raising and remedying performance problems takes a long time. They felt that where performance testing delayed the release to production, it was often due to the time being spent in these interactions.

The performance team decided to introduce performance testing of individual components in a continuous integration pipeline to help reduce the number of problems being discovered during integrated performance testing later in the lifecycle. Diana observed that as the team established this continuous integration pipeline they started to work "more with developers than testers".

Creating a performance pipeline

The "functional folk" had already built a pipeline that compiled the web application, ran unit tests, deployed to a functional test environment, ran basic regression tests and created a release build. Diana's team decided to create a separate performance pipeline that logically followed on from the functional pipeline.

The performance pipeline took the release build from the functional pipeline and deployed it to an environment, then discovered the version of the same application that was currently in production and deployed it to a different environment with the same hardware specification. A two hour performance test was then run in parallel against both versions of the application and a comparative report generated.

The team was extremely fortunate to have access to a large pool of production-like servers on which to run their performance tests, so being able to run tests in parallel and at scale wasn't an issue. When asked whether the same approach would work on smaller test environments, Diana felt that it would so long as the production traffic profile was appropriate scaled to the test environment hardware specification.

The two hourly builds generated a lot of information, in fact too much to be useful. The performance team decided to save the data from the two hour performance tests and then run automated analysis that detected trending degradation across the combined result of three builds.

The thresholds at which to report performance decay were set in consultation with the business and were high enough to alleviate the risk of false positives in the report caused by developers simply adding functionality to the application. Diana noted that it was ultimately a business decision whether to release, and that where a new piece of functionality caused a performance degradation that resulted in failing performance tests the business could still opt to release it.

Even with trend based analysis, there was some balancing to get the email notifications from these two hourly builds correct. They were initially being sent to people within the development team who subscribed. As the performance analysis was improved and refined, the notifications became increasingly relevant and they started to be delivered to more people.

Performance testing the pieces

The performance pipeline was designed to test the deployed web application in isolation rather than in an integrated environment. It made use of a number of stubs so that any degradation detected would likely relate to changes in the application rather than instability in third party systems.

In addition to testing the web application, the performance team created continuous integration pipelines for the database and services layer underneath the UI. Many of these lower level performance tests used self-service Jenkins jobs that the developers could use to spin up a cloud instance suited to the size of the component, deploy the component in isolation, run a performance test, tear down the environment and provide a report.

Diana also mentioned A/B performance testing where the team would deploy a build with a feature flag switched on, and the same build with the same feature flag switched off, then run a parallel performance test against each to determine whether the flag caused any significant performance problems.

Integrated performance

The performance testing team retained their traditional integration performance tests as part of the release to production, but with the presence of earlier performance testing in the development process this became more of a formality. Fewer problems were discovered late in the release process.

Diana estimated that these changes were about 2 years of work for 2 - 3 people. She commented that it was relatively easy to set up the tests and pipelines, but difficult to automate analysis of the results.

Ultimately Diana was part of taking the Expedia release process from monthly to twice per week. I imagine that their journey towards continuous delivery continues!

Sunday 27 September 2015

Security Testing Pathway

This pathway is a tool to help guide your self development in security testing. It includes a variety of steps that you may approach linearly or by hopping about to those that interest you most.

Each step includes:
  • links to a few resources as a starting point, but you are likely to need to do your own additional research as you explore each topic.
  • a suggested exercise or two, which focus on reflection, practical application and discussion, as a tool to connect the resources with your reality.

Take your time. Dig deep into areas that interest you. Apply what you learn as you go.

This pathway was developed in conjunction with Daniel Billing & Sarah Burgess

STEP - Introduction to oWASP

The Open Web Application Security Project (OWASP) is a worldwide not-for-profit charitable organisation focused on improving the security of software. Understand the breadth of information and resources available on the oWASP site:
[1 hour] The volume of information available on the oWASP site can be overwhelming. The resources on the site are a product of thousands of active wiki users, however the aspects of security that your organisation prioritises will depend on the views of individuals. To bring your focus back to what is relevant for your context, talk to someone in your team about security. You may like to find out:
  • What security testing does your organisation currently prioritise? Why? 
  • Have you been attacked in the past? In what way? 
  • How are developers preventing vulnerabilities in your applications?

STEP - Threat Modelling

Understand how threat modeling can help clarify risks to the organisation:
[1 hour] Use the STRIDE model to think about threats to your application. Try to get specific about the ways in which your organisation is vulnerable to each threat. Share your thinking with a security specialist or technical lead, see if you can add anything extra to your threat model with their help.

STEP - Approach to security testing

Learn about how others plan security testing and integrate it into their development process:
[1 hour] Talk to your team about who is currently responsible for security testing and how it is integrated into your existing development process. Talk to a technical lead or coach about the opportunities for improving what you do now.

WARNING: In many countries it is illegal to use the following attacks. Please make sure that you practice your security testing skills in the demonstration environments specified in the exercises only.

STEP - SQL injection attack

Injection flaws occur when untrusted data is sent to an interpreter as part of a command or query. The attacker’s hostile data can trick the interpreter into executing unintended commands or accessing data without proper authorization. Understand more about SQL injection:
[1 hour] Work through the exercises in how to exploit a SQL injection attack. Using this intentionally vulnerable demonstration site, you should learn how to gain unauthorised access to an application, find user account and password details, and discover details of the underlying database.

[1 hour] The Altoro Mutual website is published by Watchfire, Inc. for the sole purpose of demonstrating the effectiveness of Watchfire products in detecting web application vulnerabilities and website defects. Based on what you’ve learned, how many ways can you gain access to the application using SQL injection? If you’re unsure where to begin, you may wish to try the Online Banking Login form as a starting point for your attack.

STEP - Cross site scripting attack

Cross site scripting, or XSS, flaws occur whenever an application takes untrusted data and sends it to a web browser without proper validation or escaping. XSS allows attackers to execute scripts in the victim’s browser which can hijack user sessions, deface web sites, or redirect the user to malicious sites. Understand more about XSS:
[2 hours] Gruyere is an application with a number of security vulnerabilities for use as a learning tool. Using a Firefox browser, start up your own instance of Gruyere and try to complete the XSS Challenges. Each challenge includes hints to help you expose the vulnerabilities independently, then explains one way to exploit and fix the problems in the application.

STEP - Exploiting authentication and session management

Application functions related to authentication and session management are often not implemented correctly, allowing attackers to compromise passwords, keys, or session tokens, or to exploit other implementation flaws to assume other users’ identities.
[3 hours] For this exercise you’ll need a proxy application that allows you to capture, edit and resubmit network traffic. You may wish to download Fiddler, ZAP, Burp Suite, or use another tool of your choice.

Use the deliberately insecure Supercar Showdown website, complete the following challenges:
  1. Spoof another user’s session and perform actions against their user account
  2. Elevate your access privileges from a standard user to an administrative user
  3. Attempt to reset the password of another user
Some assistance for these challenges may be found in the Hack Yourself First course materials in the section on Account Management.

STEP - Practice makes perfect

The previous steps give information and exercises for the Top 3 attacks in the oWASP Top 10. All of the vulnerable learning environments provided so far are susceptible to these attacks in some way.

[8 hours] Take the opportunity to practise the three types of exploit that have been introduced in this pathway: SQL injection attacks, XSS attacks, and authentication exploitation. Re-visit each learning environment and see how many additional vulnerabilities you can discover:

STEP - Serious security

Security testing is a rich, specialised discipline. Beyond this taster of what’s possible, there are a number of other aspects of application security to consider:
[8 hours] If you’d like to get deeper into security testing, the next logical step is to run your own vulnerable learning environment. oWASP WebGoat aims to provide a de-facto interactive teaching environment for web application security. The application must be downloaded and installed on a local web server, the user guide includes instructions. When running the application, your machine will be very susceptible to attack and should disconnected from the internet. WebGoat is for educational purposes and it includes a number of lesson plans to teach the different aspects of application security.

Tuesday 15 September 2015

Continuous Delivery Testing Pathway

This pathway is a tool to help guide your self development in continuous delivery testing. It includes a variety of steps that you may approach linearly or by hopping about to those that interest you most.

Each step includes:
  • links to a few resources as a starting point, but you are likely to need to do your own additional research as you explore each topic.
  • a suggested exercise or two, which focus on reflection, practical application and discussion, as a tool to connect the resources with your reality.

Take your time. Dig deep into areas that interest you. Apply what you learn as you go.

STEP - Removing release testing

Why does this pathway exist? Understand the key reasons to significantly shorten a release process, the arguments against release testing and why organisations aim to avoid batched releases in agile environments:
[2 hours] Research your existing release process and talk to people within your organisation to find out whether there are any current initiatives to improve it.

STEP - Introduction to continuous delivery

What is the end goal? Discover the basics of continuous delivery and the theory of how it can be implemented in organisations.
[1 hour] Based on what you've read, try to explain the theory of continuous delivery in your own words to someone in your team. Describe what appeals to you about continuous delivery, what you disagree with, and things that you think will be difficult to implement in your organisation. Afterwards, if you have any remaining questions, raise these with a technical lead or coach for further discussion.

STEP - Experiences in continuous delivery

How are other organisations doing continuous delivery? There is a lot of variance in implementation and differing opinions about how to approach the theory. Understand the realities of the people, processes and tools of teams doing continuous delivery:
[2 hours] Compare the experiences shared in the links above and the theory of continuous delivery. Identify common themes, and areas where ideas or implementation details differ. Discuss your analysis with a technical lead or coach.

STEP - Starting with continuous integration

What is the first step? Understand the concept of continuous integration:
[3 hours] At the start of this talk transcript, Jez Humble points out that most people aren't doing continuous integration. How does the approach to continuous integration in your team differ to the theory? Talk to a developer to confirm your understanding of your branching strategy, the way you use source control management tools, and how you manage merging to master. If you use a continuous integration tool, create a list of the jobs that are used by your team during development, and be sure that you understand what each one does. Reflect on how quickly your team respond to build failures in these jobs, and who takes ownership for resolving these. Discuss this exercise with a technical lead or coach to collaboratively identify opportunities for improvement, then raise these ideas at your next team retrospective.

STEP - Theory of test automation

Continuous delivery puts a lot of focus on test automation. In order to support development of an effective pipeline it's important to understand common strategies for automation, and the distinction between checking and testing:
[1 hour] Read through the automation strategy for your product. How well does your existing strategy for automation support your delivery pipeline? What opportunities exist to improve this strategy? Discuss your thoughts with a technical lead or coach.

STEP - A delivery pipeline

Understand how to construct delivery pipeline and the role of automation:
[3 hours] Create a visual representation of the current delivery pipeline for your product. Use a timeline format that shows the build jobs in your continuous integration tool at every stage from development through to production deploy, any test jobs that execute automated suites, and points where the tester is hands-on, exploring the product. Compare your pipeline to the simplified images by Yassal Sundman for continuous delivery and continuous deployment, then reflect on the following questions:
  1. How would your approach to testing change, or not, if we were able to deploy to production 10 times a day? How about 100 times a day?
  2. Does the coverage provided by your automation give you a degree of comfort or confidence? If not, what needs to change?
  3. Does your automation execute fast enough? How fast do you think it should be? How can you achieve this?
  4. Where in the pipeline would you want to retain hands-on testing? How would you justify this?
Discuss your ideas with a technical lead or coach. Work together to identify actions from your thinking and determine how to proceed in implementing change.

STEP - Non-functional testing in continuous delivery

Learn more about integrating security, performance, and other non-functional testing in a continuous delivery pipeline:
[2 hours] Does your organisation have a non-functional testing "sandwich"? Having read more about organisations who integrate these activities earlier in the process, what opportunities do you see to improve the way that you work? What would the first steps be? Talk to a technical lead or coach about what you'd like to see change.

STEP - Cross-browser testing

For continuous delivery of a web application, it's important to include cross-browser testing in the delivery pipeline. Discover strategies for cross-browser testing and the tools available to support it:
[8 hours] Learn more about the common cross-browser tools that are available, understand the advantages and disadvantages of each option, then select a tool to trial. Create a prototype to execute existing browser-based automation for your product across multiple browsers. If successful on your local environment, attempt to create a prototype job in your continuous integration tool to verify that your chosen solution works as part of your continuous integration. Discuss what you learned about the tool and the results of your experiment with a technical lead or coach.

STEP - Test data & databases

Discover the additional considerations around test data in continuous delivery:
[1 hour] Data is a constant headache for testers. Consider the limitations of the test data in use by your automation. How could you improve the data within your delivery pipelines? How could you improve the way that you locate data for testing? Talk through your ideas with a technical lead or coach.

STEP - Configuration management & environments

An effective delivery pipeline is supported by multiple test environments. Learn more about configuration management, environments and infrastructure services in continuous delivery:
[1 hour] Talk to your operations or support team about how they provide test environments for continuous integration, the infrastructure required to support a delivery pipeline, and what their plans are for future changes in this space.

STEP - Testing in production

Understand A/B testing and feature toggles:
[1 hour] Talk to people in your organisation to find out how you currently use feature toggles and how you make decisions about what to keep based on user analytics. Could your approach be more responsive through targeted use of a monitoring tool like splunk? Share your thoughts with a technical lead or coach.

Tuesday 8 September 2015

API, Web Services & Microservices Testing Pathway

This pathway is a tool to help guide your self development in API, web services and microservices testing. It includes a variety of steps that you may approach linearly or by hopping about to those that interest you most.

Each step includes:
  • links to a few resources as a starting point, but you are likely to need to do your own additional research as you explore each topic.
  • a suggested exercise or two, which focus on reflection, practical application and discussion, as a tool to connect the resources with your reality.

Take your time. Dig deep into areas that interest you. Apply what you learn as you go.

STEP - Distinguishing APIs and web services

An API (Application Programming Interface) is the means by which third parties can write code that interfaces with other code. A Web Service is a type of API that:
  • is used to exchange data between applications,
  • uses a standard defined by W3C, 
  • has an interface that is depicted in a machine-processable format usually specified as a WSDL (Web Service Description Language), and 
  • almost always operates over HTTP.
Example web service protocols include SOAP, REST, and XML-RPC. An example of an API that is not a web service is the Linux Kernel API, which is written in C for use on a local machine.

References: API vs Web Service, Difference between web API and web service, Difference between API and web service

[1 hour] Once you feel that you understand the difference between APIs and web services, talk to a developer. Ask what APIs and web services exist within the application that you're working on. Work with your developer to draw a simple architecture diagram that shows whereabouts in your application these interfaces are located. Be sure you can distinguish which are APIs and which are web services, and that you know which protocols each interface uses.

STEP - Understanding SOAP and REST

Learn more about two common implementations of web services and the differences between them:
[1 hour] Find out whether you have any services with both a SOAP and a REST implementation. This means that the same business operation can be served in two different formats through two different APIs. Talk to a developer or technical lead and ask them to demonstrate a request in each implementation. Discuss the differences between these two interfaces and some of the reasons that both exist.

STEP - API and web service testing

Discover the tools available and some common mnemonics to approach web service testing:
[3 hours] Repeat the 53rd Weekend Testing Europe session by running some comparative tests on the SongKick API and associated website. SongKick is a service that matches users to live music events taking place near them. Use your web browser to make API requests as you would a website URL. Alongside the links from Amy Phillips and Alan Richardson above, you can refer to the SongKick API and the full transcript of the weekend testing Europe session for guidance. Experiment with locating different test data and using different API requests until you understand how the API functions. Please abide by all terms of use and do not experiment with load or security testing on this API.

[3 hours] Install Postman and use it to test the TradeMe Sandbox API. TradeMe is the leading online marketplace and classified advertising platform in New Zealand. Public, unregistered, access to their developer API is restricted to catalogue methods. Experiment with retrieving information and compare your results against the TradeMe Sandbox site. Please abide by all terms of use and do not experiment with load or security testing on this API.

[3 hours] Explore the Predic8 online REST web services demo using the advanced REST client Chrome extension or PAW - the ultimate REST client for Mac. You will need to install your chosen software and read the supporting documentation for the demonstration REST service. Explore the different functions provided. In addition to retrieving information you should be able to modify data using POST, PUT and DELETE requests. Please abide by all terms of use and do not experiment with load or security testing on this API.

[3 hours] Select an API or web service within your application. Seek out the reference material to discover what requests are allowed. Apply what you've learned through testing the third party APIs to compare the behaviour of your internal interfaces and application. Use the tools you've tried before, or select a different tool to explore. Afterwards, discuss your testing with a developer or another tester within your team, share what you found and ask how this interface is tested now.

STEP - Technical implementation of REST API

Get a deeper understanding of REST APIs by understanding how they are designed and implemented:
[3 hours] Create a set of requests using a REST API within your organisation. Investigate how resources are modeled, e.g. resource URL, HTTP verbs (GET PUT DELETE POST). Talk to a developer or technical lead to check your understanding and ask questions about your REST implementation.

STEP - Security testing APIs

Explore the basics of security testing APIs:

[8 hours] Repeat the 56th Weekend Testing Europe session by investigating the deliberately insecure API for the Supercar Showdown website, which forms the basis of Troy Hunt’s Pluralsight Course Hack Your API First. Alongside the write-up from Dan Billing above, you can refer to the Hack Your API First course materials and the full transcript of the Weekend Testing Europe session for guidance. Alongside the course materials, conduct your own experiments with the different facets of API security.

[3 hours] Apply what you've learned to assess the security of one of your APIs in a development or test environment, not production. Document any vulnerabilities that you discover to discuss with your development team. Talk to an architect about additional protection that is in place in your production environments to prevent attacks.

STEP - Service virtualization

Discover service virtualization and how it can be used in testing:
[2 hours] Determine whether any of your test suites use service virtualization. Draw a detailed architecture diagram that reflects your understanding of where services are virtualized and how this has been implemented. Check your understanding with a developer or another tester and make sure you understand the reasons that the tests use service virtualization.

STEP - Introduction to microservices

These articles give an introduction to microservices and share practical experiences from organisations who use them:
[1 hour] Talk to a developer or technical lead to check your understanding of microservices, then discuss the benefits and drawbacks of switching to a microservices architecture.

STEP - Microservices testing

Discover how to test in a microservices world:
[1 hour] Demonstrate your understanding of microservices testing by describing to another tester or test lead, in your own words, the types of testing that are possible in a microservices architecture.

STEP - A broader look at APIs

A brief introduction to API management and APIs within IoT, hypermedia, machine learning, etc.
[1 hour] Talk to a developer or technical lead about the future direction for our API implementation. Discuss how your organisation might be impacted by these ideas, or other innovations.