AutomatedQA’s webinar on the 1st of July will teach you how to use a blend of both automated and manual software testing. Getting the blend right so that your software testing ensures that releases do not break existing features.  In the webinar, you will learn how to:

1. Identify test cases worth automating
2. Organize your automated software testing
3. Create effective manual regression suites
4. Develop manual test cases to cover new features for your release
5. Map test cases to requirements for improved traceability

You can sign up for this here:

* Webinar Date:  1st July 2010
* Webinar Time:  0900-1030 ET-US, 0630-0800 IST, 1200-1330 EST-AU
* Signup Page:   http://www.softwareplanner.com/webinaruniting.asp

If you are not available to attend at that time, register anyway and you’ll get a link to the recorded version of this software testing webinar.

The webinar will focus on TestComplete and SoftwarePlanner. However, it will be full of tips that apply regardless of the automation or management solution you use.

Well worth a look in my opinion.

6.   Key Performance Measures

Performance measures evaluate the success of the project. They indicate how the project will meet the objectives listed at the beginning of the business case. The business case will:

•     list the plan objectives

•     state the evaluation criteria for each objective

•     outline how or by whom each will be evaluated

7.   Needs Assessment

The needs assessment analyses the problem and explains why the problem needs to be corrected. It provides the information as to whether the project should be undertaken at all. The needs assessment should be broken down into two sections:

i. The Problem

•     what is it?

•     why does it exist?

•     who is affected?

•     what is the extent of the problem?

•     what is the damage (liability) if the problem is not fixed?

ii. Benefits from Correcting the Problem?

•     financial: Added Returns

•     financial: Reduced costs

•     increased quality

•     time saved

8.   Technical Analysis

The technical analysis outlines the technical information used to make the decision and tells why the proposal represents the best or most cost-effective solution. It describes:

•     technical problems encountered with any existing solutions

•     what alternative solutions were considered

•     why this is the best course of action to choose

•     why this is the most cost-effective solution, or if not, why

•     what innovative technologies are being used

9.   The Project Plan

The work plan spells out the terms that will form the basis of any contracts, including the jobs to be done, the time frames and milestones. It will help the project manager by including the evaluation criteria for each step or milestone here. Name those responsible for managing the project and contracts as soon as they are known.

•     describe key activities and locations

•     outline milestones and timelines for completion

•     identify risks to project completion and contingencies

•     list project staff and consultants if known, and their responsibilities

Although you will create a detailed work plan listing tasks, detailed processes and reporting mechanisms to manage the project, it is not necessary to include this level of detail in the high-level project plan forming part of the business case.

10.   The Financial Plan

The financial plan shows how the software testing project will be financed and how returns will be achieved.

Give an explanation of why this software testing projects’ funding is necessary and how funds will be used in the introductory paragraph. Elements of the financial plan should include:

•     detailed budget

•     sources of funding

•     returns from project performance (with time)

•     operating and administrative costs

•     cash flow statement.

In the last post in this series we’ll look at critical factors for success when writing your business case for software testing tools.

1. Title Page
The title page is the first impression a reader gets of a business case. Make sure it is neat and orderly, simple, balanced and easy to read. It should contain as a minimum, the following:

• title of project

• project designation (number, location, etc.)

• name of the organization

• date of approval.

2. Table of Contents
The table of contents lists the major headings in the business case, and the page on which each is found. Do not forget to number the pages in the document.

The table of contents can usually be generated automatically by your word processing software: if not, then do not forget to bring it up to date once you have finished the whole document and carefully check over the headings and page numbers to ensure it is 100% correct.

3. Management Summary
This is the most important selling tool. It is where you create the critical first impression of the project so it is important to summarise the most important elements of the project in a concise, compelling manner using non-technical terms that are easy to read and easy to understand.

Guidelines for writing the executive summary are listed below.

• describe the project precisely and concisely, avoiding excess descriptive words

• avoid any technical terms, jargon and abbreviations (especially TLAs: three-letter-acronyms) that will lose the attention of your audience

• explain why the project is necessary and why it is the best solution

• outline the most important benefits of the project to the business

• outline the costs and major disadvantages, if any

• summarize the most important reasons for recommending the project

• limit to one page in length only

• write after the business case is completed

Keep the executive summary to one page or less. A busy senior manager should be able to read through the management summary and have all the critical information needed (cost, benefit, timescales) to make a decision there and then.

4. Mission Statement
This is a concise and general statement of what the team intends to achieve by completing the test automation project. It explains what is to be done, for whom, and why. If possible do not exceed one sentence.

For example: the test team will implement a test automation framework in order to reduce the time taken to regression test applications thereby increasing the consistency of our regression testing efforts and releasing testers to concentrate on more critical testing tasks.

5. Project Objectives
What, precisely, will be achieved by completing the project? State the objectives clearly – one short statement for each, without accompanying arguments or documentation. These appear in the body of the report and in the project summary. It should be clear to the reviewer how these objectives relate to the mission statement of the test automation project.

Objectives are Specific, Measurable, Achievable, Realistic and Timely (S.M.A.R.T.). These define the results expected as a direct consequence of the project’s completion. Such hard data verifies the value of the project and will justify the project in business terms to senior management.

They can include such goals as:

• increasing product release quality by

• running more regression tests

• performing tests which are impossible to run manually (e.g. load tests)

• improve the consistency and repeatability of tests

• reuse of tests over different releases and products

• improving use of resources by automating boring and menial tests

• speeding up time to market by completing tests quicker

• increasing confidence in release quality when automated tests complete successfully

• earlier identification of problems and issues

• increasing tester motivation through learning new skills

• freeing up test resources to work earlier in the software development life cycle

• removing barriers and bottle necks to release

Some test automation projects have long- and short-term objectives. Identify these as such if it adds to the understanding of the project. For example, the short-term objective may be to increase the amount of regression tests run before a release. In the long term, the objective may be to attract better quality test team staff due to the advanced status of the test environment.

Sample objectives:

• to automate 60% of regression tests by (date)

• to reduce time required to complete a full regression test by 3 days

• to execute load testing on every version of the product delivered to the test team

• to identify 30% of all high priority defects raised earlier in the test cycle

• to attract 7 new, highly qualified, testers by (date)

The discussion following each objective should clarify the analysis or rationale for arriving at the objective.

For example: Currently our full regression test suite takes 2 weeks to run. A fully automated regression test suite would take 1 day to run and 2 days to analyse the results. This will reduce our regression testing resource requirements by 7 man days. On a typical project this will increase the product release quality and reduce our time to test by 1 month.

In the next post we’ll cover the other 5 essential elements that should be included in your business case.

To help you achieve this we’re giving you a step-by-step guide that will enable you to successfully understand, research, plan and present a solid business case. One that will hold management attention and justify your project.

After reading this step-by-step guide you will:

• know why you need to write a business case
• learn how to research, plan and present a convincing business case to your management
• discover the 8 critical success factors that make a good business case into a great one

The Opportunity for Software Testing Tools

Software testing is a relatively low-key part and sometimes invisible part of the overall software development lifecycle. However, in many of today’s organisations, testing activities can account for up to 60%-75% of the total cost of software development.

That’s a staggering amount of time and money.

By implementing an enterprise-wide software testing strategy with the right tools, an organisation can benefit from reduced costs, accelerated delivery and improved software quality. Examples of this are:

• use of live environment hardware for ‘out of hours’ testing without impacting the business

• significantly reduced cost of running repeat tests

• the ability to run tests that are not physically possible without automation

• freeing testers to become involved earlier in the software development lifecycle, thus encouraging earlier detection of bugs in the development cycle and reduced costs to fix

• reduced test resources required – human, hardware and infrastructure

• increased test coverage

• increased quality of test metrics through consistent test execution and results collection

• greater motivation, higher morale and lower turnover of test team members: staff are relieved of monotonous, repetitive tasks and can concentrate on activities that add value such as test planning and earlier involvement in the software development lifecycle earlier

• greater bandwidth of testing capability with the same resources (i.e. the ability to test more in a shorter time), frequently a bottleneck in the overall software development lifecycle

• Automated software testing is also certainly the only option for certain types of testing – in particular stress, volume, load and performance testing of large systems where it would be unfeasible to provide the staffing and hardware resources.

All of these factors should feed into your business case to support the argument for implementing software testing tools.

The Need for a Business Case

In today’s financially tough climate business leaders are keeping an even closer eye on the ‘bottom line’ than ever before: every IT project must be clearly justified in business terms or it does not get approval.

This particularly applies to a project with no immediately evident business benefits. For example replacing a manual software testing strategy with a new one based on automated software testing.

The answer is a well-written business case.

In general most software testing tools have higher upfront costs (including software licences, training, working practice changes, test environment setup, test script development and subsequent maintenance) but then significantly lower ongoing costs for each repeat cycle of testing. This is particularly true for lowering resource costs and increasing productivity.

Clearly documenting and presenting a well-researched understanding of the goals, scope, risks and resources of your project will help justify these higher upfront costs. Especially when justified with measurable financial, procedural and quality assurance (QA) benefits. Present the results clearly in a non-technical document that the business team can easily understand and you will have created the perfect argument to support and justify your software testing tool project.

As a result you will significantly increase the probability of gaining the management commitment that you need to get underway. And, as a basic planning document, you will have an invaluable tool to help you manage the project until successful completion from the outset.

Why Write a Business Case?

The business case provides evidence that the software test tool project is a good investment for both the test team and the business. Preparing a business case is an integral part of any planning process and crucial for software test automation project for example. It’s safe to say that this also becomes more important as the cost and complexity of the project increases.

The trick is to prepare an effective business case for securing the resources — both within the test team and with senior management.

A business case is similar to a business plan prepared for private business. Its purpose is to outline the business rationale for undertaking the project and to define the parameters and management factors involved in the project itself. It will also provide the project manager with a tool to manage the software test tool implementation.

The business case serves four purposes. It will:

1. help the testers think through the project in a systematic, step-by-step manner

2. explain to parties external to the test team why the project should be undertaken

3. help the business to understand the financial value of the project

4. provide a framework for completion of the project on time and on budget

An effective business case is one that matches the overall goals of the business as well as the goal of the test team.

In short, an effective business case justifies:

• why the test automation project should be undertaken
• why the business should invest resources and time in the project
• why the project makes good financial sense for the business

While the business case may be presented in various formats, there are certain elements to include. The guidelines that will follow over next few days will allow you to build a logical, well-structured business case.

I used to get very angry about not having any testing tools. I’d compare the software testing teams use of tools to the software development teams use of tools. I’d always see a huge disparity. The development team would have tools coming out of their ears. IDE’s, emulators, modelling, static analysis. You name it, they had it. The reason they got them and we didn’t? Well…

When a software developer approaches the budget holder to purchase software tools the conversation goes something like: “So you’d like to buy a software tool? How much? … Okay, what will it give us?”. And to that the developer says something along the lines of “We’ll deliver the product 2 weeks sooner and we’ll be able to deliver it with all these additional features”. At which point the guy signing off the tool purchase sees the return on investment and waves it through.

Now see it from the test team angle. When a software tester approaches someone for the budget to purchase testing tools the conversation goes something like “So you’d like to buy a tool? How much? … Okay, what will it give us?”. And to that the tester says something along the lines of “Well the quality will improve and it will make our life easier”. To which the reply comes back with “by how much will it improve?” “Err well I can’t quantify that exactly”. At which point the guy signing off the tool purchase fails to see the benefit and tells you to get lost.

After a few years of constant rejection I wised up. I realised it all comes down to selling the benefits. And the best way to sell those benefits for a test team is to write a good business case.

And for that reason we’ll spend the next few weeks looking at how to construct a well written business case for software testing tools.

You captured the test tool requirements accurately.

You selected the right product from your list of software suppliers.

You trained the users to use your new software testing tool.

You’ve even planned for the ongoing effort and costs associated with this new testing tool.

And now that the test tool is live the team is using it consistently. In fact you’ve actually succeeded with the implementation of a new software testing tool. This is bringing big productivity gains to your team. You suspect that even the software development team is impressed with the testing improvements you’ve bought about. You’ve achieved a great result.

Although, let’s be honest here…….IT WASN’T JUST YOU WAS IT?

In an old edition of Fortune magazine, an article on teamwork noted that ‘Neil Armstrong didn’t get to the moon through rugged individualism; there is no such thing as a self-made astronaut’.

Successful software test tool implementation involves the whole team.

“Implementing Software Testing Automation Tools” describes an essential Plan-Do-Check-Act process to help ensure the success of your software test automation project.

“The Software Testing Automation Business Case” is a guide to help you get sign off for your test automation project within your business.

“The Software Testing Automation Check List” provides a list of checks that should be covered when implementing, monitoring and rolling out a test automation project.

If you are interested in any of these software testing guides then we’d be happy to send you free copies. You can request them here (opens in new window):

Software Testing Automation Guides

As testers we’ve enough experience to get past the big implementation hurdles. Installation, setup and education are usually a breeze for us. We spend our day jobs working and learning about new software. So we should be good at this!

Yet we’re just as prone to one ‘gotcha’ as everyone else……..”the failure to treat new software systems and processes as ONGOING projects”.

Like everyone else we think implementing a new software testing tool is a one off project. Just switch it on, use it a few times and away you go. Treat your projects like this and you may as well be putting up a new shelf for the software to sit on.

Value from new software testing tools and processes grows OVER TIME not OVER NIGHT!

We must concentrate on the ongoing effort just as much as the initial effort.

When you buy a new car you don’t pay for it and then expect it to run at no cost for the next 5 years. Once you’ve made that initial outlay you expect to purchase petrol each week. You expect to pay for a service once a year. It’s the same for implementing new software testing tools.

New software testing tools need:

• administering
• training for new testers
• monitoring for consistent usage across the team
• tracking of user adoption
• adjustment to match changing business process
• implementation of new features
• completion of upgrades

Don’t overlook any of these points! There’s a cost associated with each of them. They all take TIME,
EFFORT and MONEY.

The only option that incurs little cost is placing the software testing tools on a new shelf you’ve been carefully constructing. Even the conscious decision (or more commonly subconscious decision) to place new software testing tools on the shelf comes with a cost though.

That cost is the ‘opportunity cost’ of missing out on the benefits you were so close to realising.

Ongoing costs may be less that the up front costs. Yet you must budget money and time to address. If you do then you stand a far higher chance of realising the benefits you were hoping for.

If you’re considering implementing new software testing tools don’t forget the on going costs. They may be small but they are no less critical.

1. Plan a small-scale project, e.g. learn and use a test automation tool.

2. Do. Execute the plan, e.g. assign a resource to learn the tool and automate a suite of regression tests.

3. Check. Review the project, analyze the results and determine what you have learned.

4. Act. Take action based on what you have learned in the project:

i.    If the project did not work, go through the steps again with a different plan, e.g. maybe use a different tool if the first one did not work out.

ii.    If the project was successful, incorporate what you learned from the project into wider changes, e.g. automate all of your regression tests, or use the tool on other projects.

iii.    Use what you have learned to plan new improvements, beginning the PDCA cycle again, e.g. make use of advanced automation techniques such as data driven testing.

PDCA Project Example

PLAN

Project: Develop a project plan to incorporate a test automation tool into your current testing process on a trial basis and automate a series of regression tests. Keep in mind that most tool vendors have a 30 to 60 day free trial period for their tools.

Process Definition: Define your current process. Incorporating a tool will fundamentally change your process, so benchmark your testing current process.

Data collection: To justify the investment in the tool you will need to determine the value it has added to your process, i.e. this is a cost-benefit analysis. Existing data to collect for the cost- benefit analysis will be:

i.    Number of manual regression tests.
ii.    Number of times regression tests were executed in previous releases.
iii.    Number of people required to execute the tests.
iv.    Time taken to execute the tests.

DO

Implement the plan. Collect data for the cost-benefit analysis and record observations during the project. The data to collect for the cost-benefit analysis will be:

i.    Number of regression tests automated.
ii.    Number of times regression tests were executed.
iii.    Number of people required to automate and execute tests.
iv.    Time to automate the tests.
v.    Time to learn how to use the tool.

CHECK

Review the project, analyze the results and identify what you’ve learned. You can now determine the value using tool has added to your process, with:

Value = Benefits – Costs

Where:

Benefits, represent the time and resources saved in executing automated tests instead of manual test.

Costs, represent the time and resources invested in automating the tests.

In addition to the Value added, you now have a basis for training others to use the tool.

ACT

Let’s assume the project was a success, i.e. the tool added value to your process, now you can act on what you have learned in this project, therefore;

i.    Update the current test process to incorporate automation.

ii.    Start collecting cost and benefit data during other projects that incorporate automation.

iii.    Develop a training regime for others to learn the tool.

At this point you are in a strong position to plan your next PDCA cycle for test automation. For example, you could automate the remaining regression tests for your next project, and follow that project by incorporating automation into other projects, or incorporating advanced automation techniques, etc.

Continuous Improvement

You now have a basis for continuous process improvement using test automation; however, as you continue to use automation to improve your test processes, and as test automation is adopted throughout your organization, the following overall pattern will emerge;

• There will be an increase in the people and resource investments required for things like training, user support and documentation.
• As such, there will need to be greater understanding, attention and commitment from management towards automation.
• Consequently, you will need to have further controls, measurements, and reporting in your processes to provide the feedback that will tell you if your improvements are adding value.

If you currently perform manual software regression testing for each product release, have a large number of tests that need to be executed for each release, and you have frequent product releases then this is an ideal situation to make use of a test automation tool.

Manual software regression testing is an inefficient use of a test analyst’s time, especially if they are merely repeating documented tests.  You can reduce your test cycle times and improve the accuracy of your testing by letting a software test automation tool execute these tests instead.

How to Start – The “Ad Hoc” Approach

So let’s assume you purchase a software automation tool for the test analysts to use but you find things do not go as you expected:

• Some analysts do not want to use the tool to automate their tests preferring to keep on testing manually.
• Some analysts give up using the tool, but some users are using the tool successfully.
• After the several releases the application has changed and a significant number of automated test scripts need to be updated.
• The tool was expensive to purchase and so much time and resources are being used to maintain the automated test scripts it seems the tool is of marginal benefit.

This is a small sample of the problems that are likely to occur when you attempt to adopt a software test automation tool in to your organization in an ad hoc manner.

A Better Approach – “Plan – Do – Check – Act”

When you adopt any tool into your organization you intend to improve your current process but remember you are also changing the current process and this can be problematic.

Quality assurance and quality control people have developed a system that can make changes like these less problematic. It is an ideal way to adopt test automation into your organization.

This system consists of four steps, plan-do-check-act (PDCA).