Essential Software Testing Tools Blog

In this post we look at some of the benefits of implementing the right software testing tools. In particular we’re going to take test automation tools as an example and list reasons why it’s worthwhile investing in automation. We’ll also look at the reasons why many companies shy away from implementing test automation. Many test teams look at test automation tools and then back off once the reality of what is involved becomes clear. With the right approach though the implementation of an automation process need not be as daunting as it may first appear.

If you’ve looked at automation solutions and then decided not to proceed, it’s probable that you hit one of these issues:

1. During the trials of your selected products you realised that the learning curve for becoming productive with a tool is far higher than expected. As a result just for the trial you find yourselves investing significant amounts of time trying to get just a basic automated test working.

2. After starting your trial, your team were overtaken by day to day events and never had the time to invest in a comprehensive software testing tool evaluation. As a result you keep promising yourself that you’ll pick up the trial again in one or two months time.

3. When you added up the costs of the software licenses, new hardware required, training and the investment in time required you quickly decided that the investment wasn’t worth it. Or at least the financial figures you came up with were unlikely to be signed off.

4. When you realised that test automation isn’t something that can implemented by one tester in his or her spare time you figured that it’s likely to require significant business process changes. This would mean setting up a dedicated automation team and trying to work out a process for evaluating good tests to automate. It may even have meant working with the development team to get changes made to the code for the automation tool to work. Either way making changes to the existing process seemed quite daunting.

5. When you looked at the recommended best practice for test automation projects you realised that you had something that looked more like a development project than a test project. Source code control, scripting languages and the certainty that your automated code would have defects, suddenly made you aware that this isn’t going to be like any other test project you’ve undertaken.

Whilst there are any number of points to put a test manager off starting an automation project, for many companies the effort, time and money is all outweighed by the benefits. When software testing tools are implemented well the following benefits can far outweigh the issues listed above.

1. Once a test is automated, unless the feature being tested changes, then it can be run consistently day in day out with little overhead.

2. It’s usually far quicker to run an automated test than to run a test manually (exceptions to the rule on this are when you have a poorly written automated test and you spend a significant amount of time maintaining and setting up a test run).

3. Automated tests can be scaled up quickly to provide load and stress testing capabilities

4. Software tests can be run continuously and out of hours. Usually the biggest benefits are gained when tied in to an automated build environment. In this kind of setup the testers don’t even need to evaluate the results. The results from an automated build can be pushed directly back to the development team.

5. the same test can be run on combinations of platform and operating system quickly and easily. Whilst it’s never quite that simple with automation tools, when you do get this right the benefits can be huge if you have a complex test matrix to cover.

There are many other benefits and also many other pitfalls. Nevertheless, as many successful test teams have found, the cost benefit ratio usually falls down a long way on the side of delivering benefits. The benefits can be huge yet despite this many never get past the first hour of trialing a software testing tool. The key, before starting a trial, is to consider these points and set realistic expectations.

1. Effective test automation requires significant changes to existing test practices. For example most successful test automation projects have dedicated test automation engineers who’s sole purpose is to focus on automation. You will need to bring in new skills or train existing staff.

2. As well as writing the automated tests the automation team needs to be responsible for putting new business processes in place. That process includes taking and evaluating requests from others as to which tests to automate. You can’t automate everything so a process that evaluates where your time is best spent is essential.

3. Supporting the development of automated software testing tools is a development project. You will need to consider requirements definition, source code control and defect tracking. The development team follow these best practices when they write code. Your automation team should be following these best practices too.

4. Monitoring the development and execution progress is essential. Without stats to monitor your progress you’ll never know if you are getting the benefits you expected in the first place. Think through and define the management information you need at the outset of the project.

5. Don’t forget to consider training and maintenance. The hidden costs of software testing tools can make or break an automation project. If you’re selecting a new automation tool chances are you’ll need training. Once the automated tests are in place you can guarantee that the code will need to be maintained. This all takes time, effort and money.

In short don’t just think that you can succeed with software testing automation by expecting one tester in your team to spend a bit of his or her spare time learning and implementing. This approach may work for some but experience shows it usually results in failure. Far better to tackle this as a development project in it’s own right. In doing so you approach the implementation of new software testing tools with the right mindset and increase your chances of success.

I joined a very interesting webinar on Proactive Software Testing on Thursday. It was run by a guy called Robin Goldsmith who specialise in improving the software development process and software quality. In particular he has his own take on Agile development he refers to as Proactive Testing.

Now I can’t get too excited about all the arguments that surround “this” agile method or “that” agile method. I figure no one methodology is the right methodology. What’s right is your team picking the best methodology for their specific circumstances and, where necessary, modifying it to extract the biggest benefit. Having said that, there were a number of points in this webinar that really struck home. Points that I think can be applied regardless of the particular methodology you use. So Robin’s methodology may or may not interest you but I think these points may be of interest.

There are limits to the benefits of embedding software testing early in your development life cycle. Lets take the Extreme Programming approach, where developers write testcases, write the code and then decide if the code works based on the test results. The main limitation to this approach is that the developer is viewing a small piece of the code. He or she maybe unaware of the bigger pieces and may not fully understand the interaction of these pieces.

In addition there’s a chance that the developer, when writing the testcases, is envisioning how they are going to write the program. This in turn influences the tests written. It also limits the objectivity of the tests. So testcases written by developers are not going to catch things that they don’t understand or that they’ve overlooked.

The agile movement is a reaction against documentation.
This struck a cord with me. I’m not a huge fan of lots of documentation (especially when it means writing pages and pages for a plan which no one else is likely to read). Yet from a testers perspective I really appreciate a well written design document from which I can base my tests. Some argue that agile is an approach that helps developers (and lets face it testers too) avoid writing is a fair point. We’re all guilty of wanting to develop code or run testcases rather than focus on writing stuff down. This is an interesting way of approaching this…

“Documentation is not about generating paper work but rather writing down ideas so that you and others don’t forget them.”

With that in mind the purpose of writing documents seems to take on a less onerous meaning. It’s certainly an approach to documentation that I can subscribe to. It also fits with my thoughts on software testing documentation.

Aim to catch requirements issues before they get into design.
Yes we all know this one. We all know that most bugs are introduced at the beginning of the life cycle. The practicalities of software testing early in the life cycle, especially right up at the requirements capture stage, is damn difficult though. In the real world, with time, money and delivery pressures, pushing the testing into the requirements phase can be almost impossible.

It is, however, a point worth driving home. For example in the world of Agile methodology the development team has a degree of interaction with user representatives. Yet just because a user says something doesn’t guarantee that it’s a solid business requirement. Firstly the end user might be wrong. Secondly the end user isn’t the only stake holder. Is the requirement the end user is describing right? Possibly not!

Question and test the requirements up front and you prevent mistakes making their way into the design. Prevent them getting into the design and you end up with better code. With better code the tester spends less time bogged down with trivial defects. It’s an easy argument to construct.

Pulling testers off physical software testing tasks, to work on testing requirements, is a tougher argument to defend. It doesn’t always have to be testers that get involved with this though. Yes a tester will bring that critical, left field, approach to the review process. But any review and analysis of the requirements has to be better than none at all. Persuading the project team and business analyst to spend a bit more time up front can deliver a big pay back to the testers (and the project) in the long run.

In summary, a provocative webinar with many concepts and ideas that have practical application. Certainly some interesting ideas for software testing in an Agile environment that are worth experimenting with.

It’s very easy to miss the hidden costs for software testing tools. Yes there’s a price on the license for the product you want to purchase but have you thought carefully about all of the hidden costs?

The most obvious hidden cost is training. Yes the vendor you purchase from will point you in the direction of training courses and clearly state the price of these training courses. Just because you’ve been on a training course though doesn’t mean you know what you are doing. It’s like passing your driving test and then spending years building experience. The training course will teach you how to use the tool but only time and effort will provide you with the experience you need to become really productive.

There’s also the temptation to skip any training. How many of us are guilty of just ignoring what’s on offer and assume we can learn the package in our own time without the need for any training? Many of us do this because we know the cost of training will add significantly to the purchase price. If the purchase price goes up then the chance of the purchase being signed off is less likely. So we forgo the training thinking that we’ll work it out ourselves anyway. More often than not this is false economy. Four or five days of training usually results in a saving of 4 times that in wasted time working things out for ourselves. Productivity goes up significantly after training and the likely-hood of a successful test tool implementation goes up significantly too.

It’s not difficult to quantify the benefits of purchasing training. It is time consuming to add that information to a well constructed software testing tools business case. It’s always worth investing that time.

One of the other important hidden costs is infrastructure. Quite often tool vendors won’t quote the cost of supporting applications in the headline price for the product. For example many software testing tools require a back end database and the tool vendor conveniently forgets to mention the cost of this in the headline price of the product. Granted for many companies they’ll already have site wide licenses for the databases back end but for many companies they won’t. The associated purchase price can be significant.

Integration is the other area that is forgotten about all together. Rarely these days does a software application exist in a vacuum all on it’s own. With many other business applications already implemented it’s more common to want, or even need, some degree of integration with existing systems that the software test team uses. Whilst you might have the in house knowledge and resources relating to your existing systems it’s likely you’ll need to bring in expertise for any new tool your purchasing. Specify your exact requirements for integration at the start to avoid any unpleasant surprises after you’ve completed the implementation of a new software testing tool.

The other game the software test tool vendors tend to play is putting many of the features you need in a higher priced edition of the product. So whilst the headline purchase price might look within your budget the price you really need to pay, for the feature set you need, is significantly higher. Be sure to question the vendor in detail about specifics during a product demo.

Lastly, the other charge that many teams forget about is the cost of peoples time to implement the tool successfully. The cost of software testing tools are usually insignificant compared to the staff costs. When you factor in the cost of having many members of your team implementing, learning and getting started with a new tool the cost can take on a whole new dimension.

Even if you can’t estimate all of these points accurately, a rough estimate will give you a completely different perspective on the purchase price. In short make sure you list and estimate all of these hidden software testing tool costs right from the start.

The implementation of a software testing process is usually one of the most difficult tasks facing an organisation. Consistency, continuity and repeatability of your procedures deliver a framework from which a growing team can work effectively. Without consistency chaos usually ensues.

Many testers thrive off chaos but ultimately the chaos approach doesn’t scale and usually falls apart under tight deadlines and with complex products. Having said that smaller teams can deliver an effective test function with little structure in place. Again though this depends largely on the competence of a small number of testers and tends not to scale as a team grows.
At some point a team usual begins to feel the need for a formal process.

That point usually comes as the amount of software testing needed escalates. There’s nothing like a poor quality product release to prompt a demand for more testing. Also, as the requirement to test surfaces in multiple teams within an organisation the demand for formal process can become apparent too. The other point at which formal process can be useful is when your outsource. The idea of outsourcing is usually to avoid any of the overheads of managing the testing in house. If you have few procedures is in place then it is difficult to convey your expectations to an outsourced team.

Ultimately the purpose of implementing procedures is to save your team effort, money and time. If the cost of implementation outweighs the cost savings then you have to question the need to implement. Although if you see increases in quality then the trade off, of more effort, money and time may be a trade off you’re willing to take.

With the need for a standard process identified the strategy for implementation needs to be considered. This is likely to involve appointing someone to drive the implementation, identify the scope, define an implementation plan and monitor the roll out. All of this is usually best covered with a small software testing pilot project before any organisation wide roll outs are considered. The goal being to test your approach on a small scale, make adjustments where necessary and then roll out further within the organisation.

With a procedures implemented the work doesn’t stop there. Ongoing maintenance is an important part of making sure the benefits are maintained. New procedures can be out of date pretty much as soon as it’s implemented if changes elsewhere within the organisation impact the team. So the implementation must be reviewed at regular intervals to see if areas of the process need to be tweaked or changed. Various review points can be chosen to evaluate the effectiveness. These review points would include the end of project, changes to the development procedures, implementation of new software testing tools, issues identified through monitoring metrics and changes to the organisational structure.

Implementing a standard test process should not be taken as a lightly. Having said that the implementation of an effective software testing process can bring big benefits in terms of quality, time, effort and costs to an organisation.

8MVFPTX38K9W