Essential Software Testing Tools Blog

As a consultant on software testing tools I’m always on the look out for tips on managing both manual and automated software test cases. Webinars are a great source of these tips.

AutomatedQA’s webinar on the 1st of July will teach you how to use a blend of both automated and manual software testing. Getting the blend right so that your software testing ensures that releases do not break existing features.  In the webinar, you will learn how to:

1. Identify test cases worth automating
2. Organize your automated software testing
3. Create effective manual regression suites
4. Develop manual test cases to cover new features for your release
5. Map test cases to requirements for improved traceability

You can sign up for this here:

* Webinar Date:  1st July 2010
* Webinar Time:  0900-1030 ET-US, 0630-0800 IST, 1200-1330 EST-AU
* Signup Page:   http://www.softwareplanner.com/webinaruniting.asp

If you are not available to attend at that time, register anyway and you’ll get a link to the recorded version of this software testing webinar.

The webinar will focus on TestComplete and SoftwarePlanner. However, it will be full of tips that apply regardless of the automation or management solution you use.

Well worth a look in my opinion.

The core components of any great business case for securing software testing tools should include 10 essential elements. The second 5 of these are:

6.   Key Performance Measures

Performance measures evaluate the success of the project. They indicate how the project will meet the objectives listed at the beginning of the business case. The business case will:

•     list the plan objectives

•     state the evaluation criteria for each objective

•     outline how or by whom each will be evaluated

7.   Needs Assessment

The needs assessment analyses the problem and explains why the problem needs to be corrected. It provides the information as to whether the project should be undertaken at all. The needs assessment should be broken down into two sections:

i. The Problem

•     what is it?

•     why does it exist?

•     who is affected?

•     what is the extent of the problem?

•     what is the damage (liability) if the problem is not fixed?

ii. Benefits from Correcting the Problem?

•     financial: Added Returns

•     financial: Reduced costs

•     increased quality

•     time saved

8.   Technical Analysis

The technical analysis outlines the technical information used to make the decision and tells why the proposal represents the best or most cost-effective solution. It describes:

•     technical problems encountered with any existing solutions

•     what alternative solutions were considered

•     why this is the best course of action to choose

•     why this is the most cost-effective solution, or if not, why

•     what innovative technologies are being used

9.   The Project Plan

The work plan spells out the terms that will form the basis of any contracts, including the jobs to be done, the time frames and milestones. It will help the project manager by including the evaluation criteria for each step or milestone here. Name those responsible for managing the project and contracts as soon as they are known.

•     describe key activities and locations

•     outline milestones and timelines for completion

•     identify risks to project completion and contingencies

•     list project staff and consultants if known, and their responsibilities

Although you will create a detailed work plan listing tasks, detailed processes and reporting mechanisms to manage the project, it is not necessary to include this level of detail in the high-level project plan forming part of the business case.

10.   The Financial Plan

The financial plan shows how the software testing project will be financed and how returns will be achieved.

Give an explanation of why this software testing projects’ funding is necessary and how funds will be used in the introductory paragraph. Elements of the financial plan should include:

•     detailed budget

•     sources of funding

•     returns from project performance (with time)

•     operating and administrative costs

•     cash flow statement.

In the last post in this series we’ll look at critical factors for success when writing your business case for software testing tools.

The core components of any great business case for securing software testing tools should include 10 essential elements. The first 5 of these are:

1. Title Page
The title page is the first impression a reader gets of a business case. Make sure it is neat and orderly, simple, balanced and easy to read. It should contain as a minimum, the following:

• title of project

• project designation (number, location, etc.)

• name of the organization

• date of approval.

2. Table of Contents
The table of contents lists the major headings in the business case, and the page on which each is found. Do not forget to number the pages in the document.

The table of contents can usually be generated automatically by your word processing software: if not, then do not forget to bring it up to date once you have finished the whole document and carefully check over the headings and page numbers to ensure it is 100% correct.

3. Management Summary
This is the most important selling tool. It is where you create the critical first impression of the project so it is important to summarise the most important elements of the project in a concise, compelling manner using non-technical terms that are easy to read and easy to understand.

Guidelines for writing the executive summary are listed below.

• describe the project precisely and concisely, avoiding excess descriptive words

• avoid any technical terms, jargon and abbreviations (especially TLAs: three-letter-acronyms) that will lose the attention of your audience

• explain why the project is necessary and why it is the best solution

• outline the most important benefits of the project to the business

• outline the costs and major disadvantages, if any

• summarize the most important reasons for recommending the project

• limit to one page in length only

• write after the business case is completed

Keep the executive summary to one page or less. A busy senior manager should be able to read through the management summary and have all the critical information needed (cost, benefit, timescales) to make a decision there and then.

4. Mission Statement
This is a concise and general statement of what the team intends to achieve by completing the test automation project. It explains what is to be done, for whom, and why. If possible do not exceed one sentence.

For example: the test team will implement a test automation framework in order to reduce the time taken to regression test applications thereby increasing the consistency of our regression testing efforts and releasing testers to concentrate on more critical testing tasks.

5. Project Objectives
What, precisely, will be achieved by completing the project? State the objectives clearly – one short statement for each, without accompanying arguments or documentation. These appear in the body of the report and in the project summary. It should be clear to the reviewer how these objectives relate to the mission statement of the test automation project.

Objectives are Specific, Measurable, Achievable, Realistic and Timely (S.M.A.R.T.). These define the results expected as a direct consequence of the project’s completion. Such hard data verifies the value of the project and will justify the project in business terms to senior management.

They can include such goals as:

• increasing product release quality by

• running more regression tests

• performing tests which are impossible to run manually (e.g. load tests)

• improve the consistency and repeatability of tests

• reuse of tests over different releases and products

• improving use of resources by automating boring and menial tests

• speeding up time to market by completing tests quicker

• increasing confidence in release quality when automated tests complete successfully

• earlier identification of problems and issues

• increasing tester motivation through learning new skills

• freeing up test resources to work earlier in the software development life cycle

• removing barriers and bottle necks to release

Some test automation projects have long- and short-term objectives. Identify these as such if it adds to the understanding of the project. For example, the short-term objective may be to increase the amount of regression tests run before a release. In the long term, the objective may be to attract better quality test team staff due to the advanced status of the test environment.

Sample objectives:

• to automate 60% of regression tests by (date)

• to reduce time required to complete a full regression test by 3 days

• to execute load testing on every version of the product delivered to the test team

• to identify 30% of all high priority defects raised earlier in the test cycle

• to attract 7 new, highly qualified, testers by (date)

The discussion following each objective should clarify the analysis or rationale for arriving at the objective.

For example: Currently our full regression test suite takes 2 weeks to run. A fully automated regression test suite would take 1 day to run and 2 days to analyse the results. This will reduce our regression testing resource requirements by 7 man days. On a typical project this will increase the product release quality and reduce our time to test by 1 month.

In the next post we’ll cover the other 5 essential elements that should be included in your business case.

Obtaining management support for any new software testing tools can be a daunting and difficult task. But by clearly presenting a well-researched understanding of the goals, scope, risks and resources required by your software test tool project and then justifying those with tangible financial, procedural and quality benefits, you will significantly increase the probability of gaining the management commitment you need to get your project underway.

To help you achieve this we’re giving you a step-by-step guide that will enable you to successfully understand, research, plan and present a solid business case. One that will hold management attention and justify your project.

After reading this step-by-step guide you will:

• know why you need to write a business case
• learn how to research, plan and present a convincing business case to your management
• discover the 8 critical success factors that make a good business case into a great one

The Opportunity for Software Testing Tools

Software testing is a relatively low-key part and sometimes invisible part of the overall software development lifecycle. However, in many of today’s organisations, testing activities can account for up to 60%-75% of the total cost of software development.

That’s a staggering amount of time and money.

By implementing an enterprise-wide software testing strategy with the right tools, an organisation can benefit from reduced costs, accelerated delivery and improved software quality. Examples of this are:

• use of live environment hardware for ‘out of hours’ testing without impacting the business

• significantly reduced cost of running repeat tests

• the ability to run tests that are not physically possible without automation

• freeing testers to become involved earlier in the software development lifecycle, thus encouraging earlier detection of bugs in the development cycle and reduced costs to fix

• reduced test resources required – human, hardware and infrastructure

• increased test coverage

• increased quality of test metrics through consistent test execution and results collection

• greater motivation, higher morale and lower turnover of test team members: staff are relieved of monotonous, repetitive tasks and can concentrate on activities that add value such as test planning and earlier involvement in the software development lifecycle earlier

• greater bandwidth of testing capability with the same resources (i.e. the ability to test more in a shorter time), frequently a bottleneck in the overall software development lifecycle

• Automated software testing is also certainly the only option for certain types of testing – in particular stress, volume, load and performance testing of large systems where it would be unfeasible to provide the staffing and hardware resources.

All of these factors should feed into your business case to support the argument for implementing software testing tools.

The Need for a Business Case

In today’s financially tough climate business leaders are keeping an even closer eye on the ‘bottom line’ than ever before: every IT project must be clearly justified in business terms or it does not get approval.

This particularly applies to a project with no immediately evident business benefits. For example replacing a manual software testing strategy with a new one based on automated software testing.

The answer is a well-written business case.

In general most software testing tools have higher upfront costs (including software licences, training, working practice changes, test environment setup, test script development and subsequent maintenance) but then significantly lower ongoing costs for each repeat cycle of testing. This is particularly true for lowering resource costs and increasing productivity.

Clearly documenting and presenting a well-researched understanding of the goals, scope, risks and resources of your project will help justify these higher upfront costs. Especially when justified with measurable financial, procedural and quality assurance (QA) benefits. Present the results clearly in a non-technical document that the business team can easily understand and you will have created the perfect argument to support and justify your software testing tool project.

As a result you will significantly increase the probability of gaining the management commitment that you need to get underway. And, as a basic planning document, you will have an invaluable tool to help you manage the project until successful completion from the outset.

Why Write a Business Case?

The business case provides evidence that the software test tool project is a good investment for both the test team and the business. Preparing a business case is an integral part of any planning process and crucial for software test automation project for example. It’s safe to say that this also becomes more important as the cost and complexity of the project increases.

The trick is to prepare an effective business case for securing the resources — both within the test team and with senior management.

A business case is similar to a business plan prepared for private business. Its purpose is to outline the business rationale for undertaking the project and to define the parameters and management factors involved in the project itself. It will also provide the project manager with a tool to manage the software test tool implementation.

The business case serves four purposes. It will:

1. help the testers think through the project in a systematic, step-by-step manner

2. explain to parties external to the test team why the project should be undertaken

3. help the business to understand the financial value of the project

4. provide a framework for completion of the project on time and on budget

An effective business case is one that matches the overall goals of the business as well as the goal of the test team.

In short, an effective business case justifies:

• why the test automation project should be undertaken
• why the business should invest resources and time in the project
• why the project makes good financial sense for the business

While the business case may be presented in various formats, there are certain elements to include. The guidelines that will follow over next few days will allow you to build a logical, well-structured business case.

If you don’t ask you don’t get. At least that’s how the saying goes. In my early days in testing I’d ask but still didn’t get. That is, I’d ask for software testing tools to support the test process but never get any.

I used to get very angry about not having any testing tools. I’d compare the software testing teams use of tools to the software development teams use of tools. I’d always see a huge disparity. The development team would have tools coming out of their ears. IDE’s, emulators, modelling, static analysis. You name it, they had it. The reason they got them and we didn’t? Well…

When a software developer approaches the budget holder to purchase software tools the conversation goes something like: “So you’d like to buy a software tool? How much? … Okay, what will it give us?”. And to that the developer says something along the lines of “We’ll deliver the product 2 weeks sooner and we’ll be able to deliver it with all these additional features”. At which point the guy signing off the tool purchase sees the return on investment and waves it through.

Now see it from the test team angle. When a software tester approaches someone for the budget to purchase testing tools the conversation goes something like “So you’d like to buy a tool? How much? … Okay, what will it give us?”. And to that the tester says something along the lines of “Well the quality will improve and it will make our life easier”. To which the reply comes back with “by how much will it improve?” “Err well I can’t quantify that exactly”. At which point the guy signing off the tool purchase fails to see the benefit and tells you to get lost.

After a few years of constant rejection I wised up. I realised it all comes down to selling the benefits. And the best way to sell those benefits for a test team is to write a good business case.

And for that reason we’ll spend the next few weeks looking at how to construct a well written business case for software testing tools.