Essential Software Testing Tools Blog

Overview of PDCA

1. Plan a small-scale project, e.g. learn and use a test automation tool.

2. Do. Execute the plan, e.g. assign a resource to learn the tool and automate a suite of regression tests.

3. Check. Review the project, analyze the results and determine what you have learned.

4. Act. Take action based on what you have learned in the project:

i.    If the project did not work, go through the steps again with a different plan, e.g. maybe use a different tool if the first one did not work out.

ii.    If the project was successful, incorporate what you learned from the project into wider changes, e.g. automate all of your regression tests, or use the tool on other projects.

iii.    Use what you have learned to plan new improvements, beginning the PDCA cycle again, e.g. make use of advanced automation techniques such as data driven testing.

PDCA Project Example

PLAN

Project: Develop a project plan to incorporate a test automation tool into your current testing process on a trial basis and automate a series of regression tests. Keep in mind that most tool vendors have a 30 to 60 day free trial period for their tools.

Process Definition: Define your current process. Incorporating a tool will fundamentally change your process, so benchmark your testing current process.

Data collection: To justify the investment in the tool you will need to determine the value it has added to your process, i.e. this is a cost-benefit analysis. Existing data to collect for the cost- benefit analysis will be:

i.    Number of manual regression tests.
ii.    Number of times regression tests were executed in previous releases.
iii.    Number of people required to execute the tests.
iv.    Time taken to execute the tests.

DO

Implement the plan. Collect data for the cost-benefit analysis and record observations during the project. The data to collect for the cost-benefit analysis will be:

i.    Number of regression tests automated.
ii.    Number of times regression tests were executed.
iii.    Number of people required to automate and execute tests.
iv.    Time to automate the tests.
v.    Time to learn how to use the tool.

CHECK

Review the project, analyze the results and identify what you’ve learned. You can now determine the value using tool has added to your process, with:

Value = Benefits – Costs

Where:

Benefits, represent the time and resources saved in executing automated tests instead of manual test.

Costs, represent the time and resources invested in automating the tests.

In addition to the Value added, you now have a basis for training others to use the tool.

ACT

Let’s assume the project was a success, i.e. the tool added value to your process, now you can act on what you have learned in this project, therefore;

i.    Update the current test process to incorporate automation.

ii.    Start collecting cost and benefit data during other projects that incorporate automation.

iii.    Develop a training regime for others to learn the tool.

At this point you are in a strong position to plan your next PDCA cycle for test automation. For example, you could automate the remaining regression tests for your next project, and follow that project by incorporating automation into other projects, or incorporating advanced automation techniques, etc.

Continuous Improvement

You now have a basis for continuous process improvement using test automation; however, as you continue to use automation to improve your test processes, and as test automation is adopted throughout your organization, the following overall pattern will emerge;

• There will be an increase in the people and resource investments required for things like training, user support and documentation.
• As such, there will need to be greater understanding, attention and commitment from management towards automation.
• Consequently, you will need to have further controls, measurements, and reporting in your processes to provide the feedback that will tell you if your improvements are adding value.

Why Implement Test Automation Tools?

If you currently perform manual software regression testing for each product release, have a large number of tests that need to be executed for each release, and you have frequent product releases then this is an ideal situation to make use of a test automation tool.

Manual software regression testing is an inefficient use of a test analyst’s time, especially if they are merely repeating documented tests.  You can reduce your test cycle times and improve the accuracy of your testing by letting a software test automation tool execute these tests instead.

How to Start – The “Ad Hoc” Approach

So let’s assume you purchase a software automation tool for the test analysts to use but you find things do not go as you expected:

• Some analysts do not want to use the tool to automate their tests preferring to keep on testing manually.
• Some analysts give up using the tool, but some users are using the tool successfully.
• After the several releases the application has changed and a significant number of automated test scripts need to be updated.
• The tool was expensive to purchase and so much time and resources are being used to maintain the automated test scripts it seems the tool is of marginal benefit.

This is a small sample of the problems that are likely to occur when you attempt to adopt a software test automation tool in to your organization in an ad hoc manner.

A Better Approach – “Plan – Do – Check – Act”

When you adopt any tool into your organization you intend to improve your current process but remember you are also changing the current process and this can be problematic.

Quality assurance and quality control people have developed a system that can make changes like these less problematic. It is an ideal way to adopt test automation into your organization.

This system consists of four steps, plan-do-check-act (PDCA).

Have you ever spent any time defining and documenting the requirements for your software testing process?

No? I Didn’t think so.

I rarely see software test teams that have stepped back and worked out where it is they want to get to. We’re all  so busy working to the next software release deadline that we forget to invest any time looking to the future.

We’re all guilty of looking to software solutions in the hope that they’ll free up time once we’ve implemented  them successfully.

It NEVER works this way!

I can’t give you quick solutions to finding the time you need. I will stress that you’ll waste even more time if you pick the first quick fix for your software testing process. But you knew that anyway.

So why not find the time to implement the right software testing system, the right way, the first time round?

Consider thought that if you don’t know what you want then you won’t find what you need.

You see it’s impossible to select the right software if you don’t know what you are trying to achieve.

Lewis Carroll said – ‘if you don’t know where you are going, any road will take you there’.

Which is why selecting the first good looking piece of software to help your software testing process will  usually take you somewhere you weren’t quite expecting.

In the long run it will save you time and money if you get your requirements right at the start. Then work  towards implementing your selected tools to improve your test process. If you do it the other way round you can  guarantee you’ll end up with at least some rework. Or you’ll start from scratch again.

The clearer your requirements, the less time you’ll spend on re-work finding the right solution.

I think it’s fair to say that we spend a lot of time slinging mud at our development teams and customers over poor requirement specifications. Then, when we come to implement software solutions to help our software testing process, we make exactly the same mistakes.

In any project, effective requirements capture separates the successful projects from the failures.

And that goes for us in the software testing arena too!

A few days back I touched on the fact that I thought UAT was a game of negotiation and compromise. Negotiation and compromise will always be required when discussing UAT issues with a customer.

If you want to remain in control of the UAT process then it’s important that ‘you’ track the issues and write up the issues list. Why?

Well there’s a negotiating gambit known as “write the contract”. In short this just means that you should try to be the one to write any contract during the negotiations. The reason for this is that there are often many details that are not discussed in the verbal negotiations. The person who writes the contract can write the details in his or her favour. It’s then up to the other person to get them changed.

How does this apply to the way you track issues during UAT?

It’s not so much the way in which “you” track them but the way “you and the client” track the issues between you. Establish at the start that you are happy to manage the tracking of the issues.

It may sound like extra work to manage the list yourself, and in the short term it may well be. In the long term you get an easier ride as you write up the details of any issues in your favour.

This approach isn’t going to allow you to remove or make large changes to issues! Your client will quickly lose faith in you if you start doing that.

If, however, you can handle the peripheral issues or small detail, and write these aspects up in your favour then it will make your life easier. I’m not saying you write it up such that the client ends up not getting what they wanted. What I’m saying is that you write it up in your favour. Then during your regular review sessions you cover these points and listen for any objections. If there are strong objections you may well have to change the detail. Often though there won’t be objections and you get the outcome that is best for you.

At the other end of the scale if you let the client control the list then you’ll be the one on the back foot. They get to add issues as and when they feel like it. The customer can even go as far as committing you to a host of embellishments they thought it would be nice to add. Remember, once they’re on the list, they’re on. They don’t come off until a resolution is agreed between all parties.

If you control the list then they have to email you or call you to get it updated. This way you get to vet the issues before they go on the list.

This isn’t about fleecing the customer. It’s just about playing things to your advantage.

It won’t always be possible for you to take personal ownership of the issues list. When the opportunity arises to own it though, take it.

The effort expended managing the list will be repaid many times during the UAT phase. It will be repaid in terms of less rework, less code changes and less testing.

If the development team knew what was going on they might even thank you for it. Then again they might not!

UAT (User Acceptance Testing) is the last hurdle before delivery.

If UAT goes badly it’s fair to say that much of the good work prior to UAT is wasted. Doesn’t matter if you were winning. Didn’t matter that this girl was winning:

Ouch. Bet that hurt!

You can clear every hurdle (requirements capture, project management, development, system test, etc) but get UAT wrong and it will hurt.

Unlike hurdling, when you reach UAT you are entitled to pick the best person for the job.  Select the best tester, test lead or even test manager to help you clear that last hurdle. He or she doesn’t have to be the quickest or even the fittest. They just need to be able to clear the last hurdle.

Always put the best person forward to make sure you clear that last UAT hurdle!

What I’m saying is get the most suitable person in your team to act as the UAT lead and the interface to the customer. This may not be the same test lead that’s been running the testing so far. It may not be the best tester on the team. It may not even be the test manager.

From your side you need to pick someone to lead the UAT who is good at building relationships, good at negotiating and good at reaching compromise.

Yes, sure you want someone who’s good technically, understands the product and understands the UAT test process.  Just as important though is the ability to:

• gently talk the customer round from unnecessary changes.
• admit you need to change a feature that will never work in the real world.
• sense which requests the customer is serious about and which they really could live without.

During UAT it’s all too easy for the customer to start saying “when we said we wanted X we didn’t really mean X”. “We meant X but with bells and whistles”. “And we can’t go live if it doesn’t whistle”.

I agree we’re trying to meet, and exceed, the customer’s expectations. However, in reality UAT is all about reaching compromise. If you’re going to stand any chance of delivering to a reasonable time scale you need compromise.

That is, compromise unless your business is in the habit of losing money. If you want to give the customer free reign to change and modify every part of what you’ve delivered, continually tweaking, then fine. Just don’t lose sight of the fact that delivering the product without making a profit is going to impact on you just as much as the company.

If things get real bad, as a last resort get your sales team involved.

It is, after all, a team effort here. Sales can be much better at negotiating with the customer. They are far less bashful when it comes to saying ‘that will cost you’! The ‘that will cost you’ approach very quickly stops customers asking for too much.

Sometimes it’s the customer that needs to compromise. Sometimes it’s us that needs to compromise. Just make sure you’ve got the right person in place to make those calls.

Whoever you choose, whoever jumps that last hurdle, you need to be certain that they are capable. Capable of building a good relationship as well as being prepared to face up to the customer.

If you fail you can be certain that the customer will be raising the height of that last hurdle whilst you’re not paying attention. And, that will hurt!