Essential Software Testing Tools Blog

In this post we look at some of the benefits of implementing the right software testing tools. In particular we’re going to take test automation tools as an example and list reasons why it’s worthwhile investing in automation. We’ll also look at the reasons why many companies shy away from implementing test automation. Many test teams look at test automation tools and then back off once the reality of what is involved becomes clear. With the right approach though the implementation of an automation process need not be as daunting as it may first appear.

If you’ve looked at automation solutions and then decided not to proceed, it’s probable that you hit one of these issues:

1. During the trials of your selected products you realised that the learning curve for becoming productive with a tool is far higher than expected. As a result just for the trial you find yourselves investing significant amounts of time trying to get just a basic automated test working.

2. After starting your trial, your team were overtaken by day to day events and never had the time to invest in a comprehensive software testing tool evaluation. As a result you keep promising yourself that you’ll pick up the trial again in one or two months time.

3. When you added up the costs of the software licenses, new hardware required, training and the investment in time required you quickly decided that the investment wasn’t worth it. Or at least the financial figures you came up with were unlikely to be signed off.

4. When you realised that test automation isn’t something that can implemented by one tester in his or her spare time you figured that it’s likely to require significant business process changes. This would mean setting up a dedicated automation team and trying to work out a process for evaluating good tests to automate. It may even have meant working with the development team to get changes made to the code for the automation tool to work. Either way making changes to the existing process seemed quite daunting.

5. When you looked at the recommended best practice for test automation projects you realised that you had something that looked more like a development project than a test project. Source code control, scripting languages and the certainty that your automated code would have defects, suddenly made you aware that this isn’t going to be like any other test project you’ve undertaken.

Whilst there are any number of points to put a test manager off starting an automation project, for many companies the effort, time and money is all outweighed by the benefits. When software testing tools are implemented well the following benefits can far outweigh the issues listed above.

1. Once a test is automated, unless the feature being tested changes, then it can be run consistently day in day out with little overhead.

2. It’s usually far quicker to run an automated test than to run a test manually (exceptions to the rule on this are when you have a poorly written automated test and you spend a significant amount of time maintaining and setting up a test run).

3. Automated tests can be scaled up quickly to provide load and stress testing capabilities

4. Software tests can be run continuously and out of hours. Usually the biggest benefits are gained when tied in to an automated build environment. In this kind of setup the testers don’t even need to evaluate the results. The results from an automated build can be pushed directly back to the development team.

5. the same test can be run on combinations of platform and operating system quickly and easily. Whilst it’s never quite that simple with automation tools, when you do get this right the benefits can be huge if you have a complex test matrix to cover.

There are many other benefits and also many other pitfalls. Nevertheless, as many successful test teams have found, the cost benefit ratio usually falls down a long way on the side of delivering benefits. The benefits can be huge yet despite this many never get past the first hour of trialing a software testing tool. The key, before starting a trial, is to consider these points and set realistic expectations.

1. Effective test automation requires significant changes to existing test practices. For example most successful test automation projects have dedicated test automation engineers who’s sole purpose is to focus on automation. You will need to bring in new skills or train existing staff.

2. As well as writing the automated tests the automation team needs to be responsible for putting new business processes in place. That process includes taking and evaluating requests from others as to which tests to automate. You can’t automate everything so a process that evaluates where your time is best spent is essential.

3. Supporting the development of automated software testing tools is a development project. You will need to consider requirements definition, source code control and defect tracking. The development team follow these best practices when they write code. Your automation team should be following these best practices too.

4. Monitoring the development and execution progress is essential. Without stats to monitor your progress you’ll never know if you are getting the benefits you expected in the first place. Think through and define the management information you need at the outset of the project.

5. Don’t forget to consider training and maintenance. The hidden costs of software testing tools can make or break an automation project. If you’re selecting a new automation tool chances are you’ll need training. Once the automated tests are in place you can guarantee that the code will need to be maintained. This all takes time, effort and money.

In short don’t just think that you can succeed with software testing automation by expecting one tester in your team to spend a bit of his or her spare time learning and implementing. This approach may work for some but experience shows it usually results in failure. Far better to tackle this as a development project in it’s own right. In doing so you approach the implementation of new software testing tools with the right mindset and increase your chances of success.

If test automation is on your list of things to do for 2010 then this might be of interest. We have written 3 new guides on software testing automation.

“Implementing Software Testing Automation Tools” describes an essential Plan-Do-Check-Act process to help ensure the success of your software test automation project.

“The Software Testing Automation Business Case” is a guide to help you get sign off for your test automation project within your business.

“The Software Testing Automation Check List” provides a list of checks that should be covered when implementing, monitoring and rolling out a test automation project.

If you are interested in any of these software testing guides then we’d be happy to send you free copies. You can request them here (opens in new window):

Software Testing Automation Guides

Overview of PDCA

1. Plan a small-scale project, e.g. learn and use a test automation tool.

2. Do. Execute the plan, e.g. assign a resource to learn the tool and automate a suite of regression tests.

3. Check. Review the project, analyze the results and determine what you have learned.

4. Act. Take action based on what you have learned in the project:

i.    If the project did not work, go through the steps again with a different plan, e.g. maybe use a different tool if the first one did not work out.

ii.    If the project was successful, incorporate what you learned from the project into wider changes, e.g. automate all of your regression tests, or use the tool on other projects.

iii.    Use what you have learned to plan new improvements, beginning the PDCA cycle again, e.g. make use of advanced automation techniques such as data driven testing.

PDCA Project Example

PLAN

Project: Develop a project plan to incorporate a test automation tool into your current testing process on a trial basis and automate a series of regression tests. Keep in mind that most tool vendors have a 30 to 60 day free trial period for their tools.

Process Definition: Define your current process. Incorporating a tool will fundamentally change your process, so benchmark your testing current process.

Data collection: To justify the investment in the tool you will need to determine the value it has added to your process, i.e. this is a cost-benefit analysis. Existing data to collect for the cost- benefit analysis will be:

i.    Number of manual regression tests.
ii.    Number of times regression tests were executed in previous releases.
iii.    Number of people required to execute the tests.
iv.    Time taken to execute the tests.

DO

Implement the plan. Collect data for the cost-benefit analysis and record observations during the project. The data to collect for the cost-benefit analysis will be:

i.    Number of regression tests automated.
ii.    Number of times regression tests were executed.
iii.    Number of people required to automate and execute tests.
iv.    Time to automate the tests.
v.    Time to learn how to use the tool.

CHECK

Review the project, analyze the results and identify what you’ve learned. You can now determine the value using tool has added to your process, with:

Value = Benefits – Costs

Where:

Benefits, represent the time and resources saved in executing automated tests instead of manual test.

Costs, represent the time and resources invested in automating the tests.

In addition to the Value added, you now have a basis for training others to use the tool.

ACT

Let’s assume the project was a success, i.e. the tool added value to your process, now you can act on what you have learned in this project, therefore;

i.    Update the current test process to incorporate automation.

ii.    Start collecting cost and benefit data during other projects that incorporate automation.

iii.    Develop a training regime for others to learn the tool.

At this point you are in a strong position to plan your next PDCA cycle for test automation. For example, you could automate the remaining regression tests for your next project, and follow that project by incorporating automation into other projects, or incorporating advanced automation techniques, etc.

Continuous Improvement

You now have a basis for continuous process improvement using test automation; however, as you continue to use automation to improve your test processes, and as test automation is adopted throughout your organization, the following overall pattern will emerge;

• There will be an increase in the people and resource investments required for things like training, user support and documentation.
• As such, there will need to be greater understanding, attention and commitment from management towards automation.
• Consequently, you will need to have further controls, measurements, and reporting in your processes to provide the feedback that will tell you if your improvements are adding value.

Why Implement Test Automation Tools?

If you currently perform manual software regression testing for each product release, have a large number of tests that need to be executed for each release, and you have frequent product releases then this is an ideal situation to make use of a test automation tool.

Manual software regression testing is an inefficient use of a test analyst’s time, especially if they are merely repeating documented tests.  You can reduce your test cycle times and improve the accuracy of your testing by letting a software test automation tool execute these tests instead.

How to Start – The “Ad Hoc” Approach

So let’s assume you purchase a software automation tool for the test analysts to use but you find things do not go as you expected:

• Some analysts do not want to use the tool to automate their tests preferring to keep on testing manually.
• Some analysts give up using the tool, but some users are using the tool successfully.
• After the several releases the application has changed and a significant number of automated test scripts need to be updated.
• The tool was expensive to purchase and so much time and resources are being used to maintain the automated test scripts it seems the tool is of marginal benefit.

This is a small sample of the problems that are likely to occur when you attempt to adopt a software test automation tool in to your organization in an ad hoc manner.

A Better Approach – “Plan – Do – Check – Act”

When you adopt any tool into your organization you intend to improve your current process but remember you are also changing the current process and this can be problematic.

Quality assurance and quality control people have developed a system that can make changes like these less problematic. It is an ideal way to adopt test automation into your organization.

This system consists of four steps, plan-do-check-act (PDCA).