Essential Software Testing Tools Blog

A large part of software testing is about working with other teams and people. This is evident more so in the User Acceptance Testing (UAT) phase than any other phase. QA engineers involved in UAT are become the negotiators. On the one hand we’re working with a development team that have it in their heads that they are going to deliver X. On the other hand we have the customer who wants us to deliver Y. Our job is to make sure that both parties meet in the middle. We’re the customers advocate to the development team. Equally we’re the developers advocate to the customer. A developer may say that a particular feature is risky or technically not possible. Quite often it’s the tester that has to put this case forward to the customer.

Personally as a tester I found UAT more interesting, more demanding and more exciting than all the other stages. Everything comes together during this phase. Managing this dictates that you become a peace maker and negotiator between the two sides. A UAT test fails and the customer tells you that the feature doesn’t do what they need it to. You talk to the development team and they say it has to work that way when you consider other aspects of the application. Your is to understand all of the technical arguments supporting both sides. More than this your role is to understand the people on both sides so that you can bring them together with the right solution or even possibly a compromise on both sides.

If you’re involved with UAT then it’s not only these technical and contractual issues that present a challenge. Surrounding this are a unique set of software testing process issues that always crop up. Usually you’re out of your normal testing environment. You are forced to work outside of your usual process and sometimes even without your usual tool set. For example 3 challenges that consistently present themselves during this phase are…

1. how to track defects effectively
2. how to record requirement changes
3. how and where do the test cases get written

Yes we know these are areas that most software testing teams have solved a million times over. The trouble is when you’re on a customers site as part of a User Acceptance Testing program. You don’t usually have access to your defect tracking tool of choice. The client forces you to go back to the dark ages and start tracking defects with Excel. A stream of emails are used to explain some subtle changes to a requirement and the original requirements document doesn’t get updated. The client doesn’t have access to your test management tool so you have to write the test cases in word and then record results in Excel again.

You may have the perfect process when working in your own environment. However, the moment you get pulled into UAT you’re forced to compromise and start reinventing your process. You need a process that can work for the customer in the customers environment. Following is an example of how we can address some of these software testing issues with a test management tool called ALMComplete. A lot of other tools out there have similar features so we’re trying to focus on the principals here rather than the tool itself.

In particular this software testing video looks at:

1. Logging Defects – keeping everything in sync and visible.
2. Tracking Changes to Requirements – maintaining traceability and the approval process.
3. Writing the Test Cases – tracing approvals from your client.

UAT presents a unique set of challenges to the QA team. Don’t approach it thinking your usual processes and tools will deliver everything you need. Your customers requirements (both practical and process) will mean you have to bend towards meeting their needs. Typically your clients processes and practices will be completely different and probably far less mature than yours. If you can help guide them with some gentle persuasion you’ll reap many benefits from keeping close to your usual process and utilising existing tools you have. UAT is not just an extension of your usual software testing process. Quite often it demands a different approach altogether. However, if you have a clear picture of how you want to run your UAT from the start you can make sure this part of your software testing runs smoothly with your tools and your process.

A few days back I touched on the fact that I thought UAT was a game of negotiation and compromise. Negotiation and compromise will always be required when discussing UAT issues with a customer.

If you want to remain in control of the UAT process then it’s important that ‘you’ track the issues and write up the issues list. Why?

Well there’s a negotiating gambit known as “write the contract”. In short this just means that you should try to be the one to write any contract during the negotiations. The reason for this is that there are often many details that are not discussed in the verbal negotiations. The person who writes the contract can write the details in his or her favour. It’s then up to the other person to get them changed.

How does this apply to the way you track issues during UAT?

It’s not so much the way in which “you” track them but the way “you and the client” track the issues between you. Establish at the start that you are happy to manage the tracking of the issues.

It may sound like extra work to manage the list yourself, and in the short term it may well be. In the long term you get an easier ride as you write up the details of any issues in your favour.

This approach isn’t going to allow you to remove or make large changes to issues! Your client will quickly lose faith in you if you start doing that.

If, however, you can handle the peripheral issues or small detail, and write these aspects up in your favour then it will make your life easier. I’m not saying you write it up such that the client ends up not getting what they wanted. What I’m saying is that you write it up in your favour. Then during your regular review sessions you cover these points and listen for any objections. If there are strong objections you may well have to change the detail. Often though there won’t be objections and you get the outcome that is best for you.

At the other end of the scale if you let the client control the list then you’ll be the one on the back foot. They get to add issues as and when they feel like it. The customer can even go as far as committing you to a host of embellishments they thought it would be nice to add. Remember, once they’re on the list, they’re on. They don’t come off until a resolution is agreed between all parties.

If you control the list then they have to email you or call you to get it updated. This way you get to vet the issues before they go on the list.

This isn’t about fleecing the customer. It’s just about playing things to your advantage.

It won’t always be possible for you to take personal ownership of the issues list. When the opportunity arises to own it though, take it.

The effort expended managing the list will be repaid many times during the UAT phase. It will be repaid in terms of less rework, less code changes and less testing.

If the development team knew what was going on they might even thank you for it. Then again they might not!

UAT (User Acceptance Testing) is the last hurdle before delivery.

If UAT goes badly it’s fair to say that much of the good work prior to UAT is wasted. Doesn’t matter if you were winning. Didn’t matter that this girl was winning:

Ouch. Bet that hurt!

You can clear every hurdle (requirements capture, project management, development, system test, etc) but get UAT wrong and it will hurt.

Unlike hurdling, when you reach UAT you are entitled to pick the best person for the job.  Select the best tester, test lead or even test manager to help you clear that last hurdle. He or she doesn’t have to be the quickest or even the fittest. They just need to be able to clear the last hurdle.

Always put the best person forward to make sure you clear that last UAT hurdle!

What I’m saying is get the most suitable person in your team to act as the UAT lead and the interface to the customer. This may not be the same test lead that’s been running the testing so far. It may not be the best tester on the team. It may not even be the test manager.

From your side you need to pick someone to lead the UAT who is good at building relationships, good at negotiating and good at reaching compromise.

Yes, sure you want someone who’s good technically, understands the product and understands the UAT test process.  Just as important though is the ability to:

• gently talk the customer round from unnecessary changes.
• admit you need to change a feature that will never work in the real world.
• sense which requests the customer is serious about and which they really could live without.

During UAT it’s all too easy for the customer to start saying “when we said we wanted X we didn’t really mean X”. “We meant X but with bells and whistles”. “And we can’t go live if it doesn’t whistle”.

I agree we’re trying to meet, and exceed, the customer’s expectations. However, in reality UAT is all about reaching compromise. If you’re going to stand any chance of delivering to a reasonable time scale you need compromise.

That is, compromise unless your business is in the habit of losing money. If you want to give the customer free reign to change and modify every part of what you’ve delivered, continually tweaking, then fine. Just don’t lose sight of the fact that delivering the product without making a profit is going to impact on you just as much as the company.

If things get real bad, as a last resort get your sales team involved.

It is, after all, a team effort here. Sales can be much better at negotiating with the customer. They are far less bashful when it comes to saying ‘that will cost you’! The ‘that will cost you’ approach very quickly stops customers asking for too much.

Sometimes it’s the customer that needs to compromise. Sometimes it’s us that needs to compromise. Just make sure you’ve got the right person in place to make those calls.

Whoever you choose, whoever jumps that last hurdle, you need to be certain that they are capable. Capable of building a good relationship as well as being prepared to face up to the customer.

If you fail you can be certain that the customer will be raising the height of that last hurdle whilst you’re not paying attention. And, that will hurt!

A well respected friend of mine compares User Acceptance Testing (UAT) to putting up wallpaper while the plasterer is still plastering the walls.

As a specialist in implementing large CRM (customer relationship management) systems Richard has been through his fair share of user acceptance testing. He says the UAT testing phase usually gets dispensed with when the supplier finds themselves under time or budget pressure. As a result the customer effectively gets landed with the suppliers testing responsibilities, or at worse, ends up testing something that’s effectively still in development. Simply put the test team are wallpapering whilst the developers are still putting the plaster on the walls.

Adding to the pressure is the obvious fact that UAT is the final step before go live. So in many projects everything gets lined up for the live date with little scope to move dates. And guess who bears the brunt of this. The test team yet again who generally end up dispensing with life’s luxuries such as sleep.

All the books say that what is supposed to happen is that development is completed and a rigorously tested system is delivered to UAT. UAT is then just a question of putting a series of ticks in boxes in front of the customer.

What happens though is that a clam tick the box activity is turned into the most stressful phase of the whole project. Deadlines loom. Contract commitments get stretched. Testers and developers work all hours.

The onus is on the test team to act as the thermometer. Is the evidence from your early testing looking bad? The sooner you demonstrate this, the sooner people can consider the implications on the UAT phase. You’re not trying to depress people here. You just need to tell the rest of the project team how it is.

If you don’t think the software is ready for UAT you need evidence to show that it it won’t be. A track record of high priority bugs raised over the duration of the project is a good start. Explaining how embarrassing it would be for the customer to see a bug during UAT tends to hit home harder.

You could just say we’ve got 3 blocker bugs stopping us from going into UAT. Alternatively you could paint a picture. If we go into UAT, and we test feature X in front of the customer, then he’s going to see Y. And that’s going to make us look pretty stupid. Perhaps we should put back the UAT? We need time to resolve this. Then when we run through these UAT tests we’ll look like the professionals we really are. Okay, I’m stretching it a bit here but you get the idea.

Putting back UAT and delivery deadlines is never a comfortable discussion to have with the customer. In my experience you are better off having one difficult discussion before the UAT starts, rather than several very difficult discussions with the customer during UAT.

Maybe we’ll never reach the utopia of completing development and delivering a rigorously tested system to UAT. As testers, part of our remit is to let everyone know how likely we are to have a successful UAT phase. This process of telling it like it is, keeps the whole team focused on what really matters. And what really matters is the customer.

For the customer it’s about finding that balance between delivering the features, to an acceptable level of quality, within the time frame. A sort of ‘the plaster still needs a bit of drying time but we should be okay hanging the wallpaper anyway’ approach.