Essential Software Testing Tools Blog

There seems to be little agreement in the software testing industry about how much of our work we should document. Stop and consider this for a moment. It’s not just the software testing industry that disagrees on this topic. It’s not just organisations, sectors in industry or individuals that disagree. Socrates and Plato, two great classical Greek philosophers, both disagreed about what they should write down.

Socrates had a complete aversion to writing any of his philosophical musings down. In fact he actively objected to writing anything down and encouraged others to avoid writing anything down too. His argument going along the lines of; text without teaching gives the illusion of learning and little else. In short he disliked the idea that you could know lots but understand very little.

Plato on the other hand argued that if we write nothing down, how do we impart our knowledge and wisdom on future generations? And with hindsight, if it wasn’t for Plato writing down a lot of what he was taught by Socrates, we’d have lost an immense body of work and understanding.

So these arguments for and against writing have been raging for centuries. And clearly these arguments go well beyond the software testing industry. For us though, there are some unique considerations. Some of those unique considerations include:

1. How much time do we have to document and write things down?
2. What legal requirements are imposed on us to document our work?
3. To what extent does writing help us as individuals with what we do?
4. What do our customers expect from us?
5. How else are we going to convey our thoughts and knowledge if we don’t write things down?

In some software testing environments I’ve worked in there just hasn’t been time to write much down. In some situations no amount of pushing back is going to get you the time you need either. So, as long as the team you’re working with understands that you can’t write much down, and they’re not demanding much in the way of documented tests anyway, then fine. In some environments though (e.g. air traffic control, medical, nuclear, etc) you have legal obligations to meet. So the requirement to document just about everything is built in from the start.

Another perspective to look at this from is what we demand from our counter parts in the development team. If I knew a coder was never writing comments in the code then it would suggest to me that the code may become difficult to maintain in the future. As a software tester I wouldn’t be happy about that. It’s likely to result in me finding bugs at a later stage that could have been avoided. Similarly if a development team has more than one developer I’d expect them to be using a source code control tool to help them collaborate and work effectively together. I’m a customer of the developers who write code. I expect them to write comments and use source code control tools. I expect this because I’ve seen the chaos the ensues when code isn’t controlled with source code control tools. Equally I assume that my customers expect me to write and manage my test cases with a test management tool. I may or may not think it’s necessary. If I’ve got a customer to deliver to, then I think it’s fair for them to demand this from me.

On a personal level I also know that when I write down test cases, and document software test plans, it helps me clarify my thinking. I come up with more test cases as a result, I’m clearer in my own head about what I need to do and I waste less time doing unnecessary tests. It might not work that way for everybody but I know writing things down helps me.

Finally there’s the ‘what happens if you get knocked down by a bus tomorrow’ scenario. If you don’t write anything down (or what your write is poorly written) then how do you expect another software tester to pick from where you left off. Maybe you convey that information regularly by talking to people; which is fine. If you don’t then you ought to be writing it down. That is unless you think what you do is so un-important that nobody else needs know about it.

In short I’m not advocating one approach over the other. I do think we need to stop every now and again and consider the questions above though. Every job, organisation and situation is different. It’s in our interest as professionals in the software testing industry to consider the best approach for what we are doing right now.

The following has probably happened to most software testers more than they would care to remember. You are testing a new application and something does not look right with the test results. They meet the criteria you have established but something about the results puts you off and you can’t pinpoint what it is.  You ask for a second opinion but other testers and the developers tell you the results look ok.

Not satisfied you scrutinize the test results further, possibly rerunning tests or creating new tests and executing them.  Eventually you do find a problem, the developer confirms the defect in the code, and you are left shaking your head thinking, “Is this anyway to test a software application?”

The answer is, “Yes!”

Call it “a gut feeling”, “a hunch”, or “intuition” but experts use this in their work to make decisions and so can you.  So before you scrap your existing QA department or abandon your testing processes to hire more “intuitive” employees let’s see what is actually happening with “intuition”.

The mental processes that we use to acquire, store, transform, and use knowledge or information are known as cognitive processes. Our mind’s cognitive processes work to be as efficient as possible to reduce the workload on our brain. For example, as we gain expertise in a subject area we rely more on our subject matter recall and less on explicit rules; and we do this automatically without awareness.

Consider the example above; a novice test analyst may accept the test results as is because they meet the test criteria (explicit rules), however an experienced analyst has a feeling (recall) that something is wrong even though they do not know exactly what the problem is, for example the test results may look “too good” for a first pass.

For the “expert” test analyst years of testing, verifying results, investigating problems, and education or training are unconsciously linked in memory to form patterns that can be easily and quickly accessed without thought. Scientists describe this as skilled memory or long term working memory.

Another advantage of this “expertise” is automatic processing, i.e. we may know what is required to do in situations we are familiar with automatically without considering alternatives. In the example above the experienced analyst knew not to accept the results immediately and to keep investigating.

Decision researcher, Gary Klein, describes our use of intuition as a process:

1. A situation arises in our area of expertise.
2. Cues allow us to recognize patterns.
3. Patterns activate action scripts. Action scripts are the actions we do automatically in response to a situation, these scripts are developed over time through our knowledge and experience in our area of expertise.

When Klein studied how experts made decisions in their areas of expertise he found they used the intuitive process as opposed to considering alternative solutions. However Klein notes that if expertise is not available then considering alternatives, or some other analysis, would be a preferred method to decision making.

Along with the benefits of expertise and intuition, there are limitations to be aware of.

Expertise is domain specific. Just because you can quickly design tests and analyze test results when testing software does not mean you can quickly analyze a company’s financial reports and pick stocks correctly.

• Expertise allows you to make quicker decisions but not necessarily optimum decisions.
• Analysis of a situation will inhibit the intuition. Intuition is an automatic, unconscious process.
• Confirmation bias. An intuitive judgment is actually a hypothesis we develop based on our knowledge, experience and beliefs and we have tendency to look for evidence that confirms our own beliefs.
• Use of heuristics, or rules of thumb, as mental shortcuts when making decisions. Our use of heuristics may lead to biases and errors in judgment when coupled with intuition, therefore we need to be aware of these heuristics.

In his book Intuition at Work, Gary Klein describes a three step approach to developing your own intuition skills.

1. Identify and understand the decision requirements of your job.
2. Practice the difficult decisions in context.
3. Review your decision-making experiences.

By doing this we are, in effect, acquiring and storing highly specific, relevant and organized information into our long term memory, thus making it easily accessible when a situation arises in our area of expertise. And that can only be a good thing for us as software testers.

As a consultant on software testing tools I’m always on the look out for tips on managing both manual and automated software test cases. Webinars are a great source of these tips.

AutomatedQA’s webinar on the 1st of July will teach you how to use a blend of both automated and manual software testing. Getting the blend right so that your software testing ensures that releases do not break existing features.  In the webinar, you will learn how to:

1. Identify test cases worth automating
2. Organize your automated software testing
3. Create effective manual regression suites
4. Develop manual test cases to cover new features for your release
5. Map test cases to requirements for improved traceability

You can sign up for this here:

* Webinar Date:  1st July 2010
* Webinar Time:  0900-1030 ET-US, 0630-0800 IST, 1200-1330 EST-AU
* Signup Page:   http://www.softwareplanner.com/webinaruniting.asp

If you are not available to attend at that time, register anyway and you’ll get a link to the recorded version of this software testing webinar.

The webinar will focus on TestComplete and SoftwarePlanner. However, it will be full of tips that apply regardless of the automation or management solution you use.

Well worth a look in my opinion.

The core components of any great business case for securing software testing tools should include 10 essential elements. The second 5 of these are:

6.   Key Performance Measures

Performance measures evaluate the success of the project. They indicate how the project will meet the objectives listed at the beginning of the business case. The business case will:

•     list the plan objectives

•     state the evaluation criteria for each objective

•     outline how or by whom each will be evaluated

7.   Needs Assessment

The needs assessment analyses the problem and explains why the problem needs to be corrected. It provides the information as to whether the project should be undertaken at all. The needs assessment should be broken down into two sections:

i. The Problem

•     what is it?

•     why does it exist?

•     who is affected?

•     what is the extent of the problem?

•     what is the damage (liability) if the problem is not fixed?

ii. Benefits from Correcting the Problem?

•     financial: Added Returns

•     financial: Reduced costs

•     increased quality

•     time saved

8.   Technical Analysis

The technical analysis outlines the technical information used to make the decision and tells why the proposal represents the best or most cost-effective solution. It describes:

•     technical problems encountered with any existing solutions

•     what alternative solutions were considered

•     why this is the best course of action to choose

•     why this is the most cost-effective solution, or if not, why

•     what innovative technologies are being used

9.   The Project Plan

The work plan spells out the terms that will form the basis of any contracts, including the jobs to be done, the time frames and milestones. It will help the project manager by including the evaluation criteria for each step or milestone here. Name those responsible for managing the project and contracts as soon as they are known.

•     describe key activities and locations

•     outline milestones and timelines for completion

•     identify risks to project completion and contingencies

•     list project staff and consultants if known, and their responsibilities

Although you will create a detailed work plan listing tasks, detailed processes and reporting mechanisms to manage the project, it is not necessary to include this level of detail in the high-level project plan forming part of the business case.

10.   The Financial Plan

The financial plan shows how the software testing project will be financed and how returns will be achieved.

Give an explanation of why this software testing projects’ funding is necessary and how funds will be used in the introductory paragraph. Elements of the financial plan should include:

•     detailed budget

•     sources of funding

•     returns from project performance (with time)

•     operating and administrative costs

•     cash flow statement.

In the last post in this series we’ll look at critical factors for success when writing your business case for software testing tools.

The core components of any great business case for securing software testing tools should include 10 essential elements. The first 5 of these are:

1. Title Page
The title page is the first impression a reader gets of a business case. Make sure it is neat and orderly, simple, balanced and easy to read. It should contain as a minimum, the following:

• title of project

• project designation (number, location, etc.)

• name of the organization

• date of approval.

2. Table of Contents
The table of contents lists the major headings in the business case, and the page on which each is found. Do not forget to number the pages in the document.

The table of contents can usually be generated automatically by your word processing software: if not, then do not forget to bring it up to date once you have finished the whole document and carefully check over the headings and page numbers to ensure it is 100% correct.

3. Management Summary
This is the most important selling tool. It is where you create the critical first impression of the project so it is important to summarise the most important elements of the project in a concise, compelling manner using non-technical terms that are easy to read and easy to understand.

Guidelines for writing the executive summary are listed below.

• describe the project precisely and concisely, avoiding excess descriptive words

• avoid any technical terms, jargon and abbreviations (especially TLAs: three-letter-acronyms) that will lose the attention of your audience

• explain why the project is necessary and why it is the best solution

• outline the most important benefits of the project to the business

• outline the costs and major disadvantages, if any

• summarize the most important reasons for recommending the project

• limit to one page in length only

• write after the business case is completed

Keep the executive summary to one page or less. A busy senior manager should be able to read through the management summary and have all the critical information needed (cost, benefit, timescales) to make a decision there and then.

4. Mission Statement
This is a concise and general statement of what the team intends to achieve by completing the test automation project. It explains what is to be done, for whom, and why. If possible do not exceed one sentence.

For example: the test team will implement a test automation framework in order to reduce the time taken to regression test applications thereby increasing the consistency of our regression testing efforts and releasing testers to concentrate on more critical testing tasks.

5. Project Objectives
What, precisely, will be achieved by completing the project? State the objectives clearly – one short statement for each, without accompanying arguments or documentation. These appear in the body of the report and in the project summary. It should be clear to the reviewer how these objectives relate to the mission statement of the test automation project.

Objectives are Specific, Measurable, Achievable, Realistic and Timely (S.M.A.R.T.). These define the results expected as a direct consequence of the project’s completion. Such hard data verifies the value of the project and will justify the project in business terms to senior management.

They can include such goals as:

• increasing product release quality by

• running more regression tests

• performing tests which are impossible to run manually (e.g. load tests)

• improve the consistency and repeatability of tests

• reuse of tests over different releases and products

• improving use of resources by automating boring and menial tests

• speeding up time to market by completing tests quicker

• increasing confidence in release quality when automated tests complete successfully

• earlier identification of problems and issues

• increasing tester motivation through learning new skills

• freeing up test resources to work earlier in the software development life cycle

• removing barriers and bottle necks to release

Some test automation projects have long- and short-term objectives. Identify these as such if it adds to the understanding of the project. For example, the short-term objective may be to increase the amount of regression tests run before a release. In the long term, the objective may be to attract better quality test team staff due to the advanced status of the test environment.

Sample objectives:

• to automate 60% of regression tests by (date)

• to reduce time required to complete a full regression test by 3 days

• to execute load testing on every version of the product delivered to the test team

• to identify 30% of all high priority defects raised earlier in the test cycle

• to attract 7 new, highly qualified, testers by (date)

The discussion following each objective should clarify the analysis or rationale for arriving at the objective.

For example: Currently our full regression test suite takes 2 weeks to run. A fully automated regression test suite would take 1 day to run and 2 days to analyse the results. This will reduce our regression testing resource requirements by 7 man days. On a typical project this will increase the product release quality and reduce our time to test by 1 month.

In the next post we’ll cover the other 5 essential elements that should be included in your business case.