Essential Software Testing Tools Blog

In this post we look at some of the benefits of implementing the right software testing tools. In particular we’re going to take test automation tools as an example and list reasons why it’s worthwhile investing in automation. We’ll also look at the reasons why many companies shy away from implementing test automation. Many test teams look at test automation tools and then back off once the reality of what is involved becomes clear. With the right approach though the implementation of an automation process need not be as daunting as it may first appear.

If you’ve looked at automation solutions and then decided not to proceed, it’s probable that you hit one of these issues:

1. During the trials of your selected products you realised that the learning curve for becoming productive with a tool is far higher than expected. As a result just for the trial you find yourselves investing significant amounts of time trying to get just a basic automated test working.

2. After starting your trial, your team were overtaken by day to day events and never had the time to invest in a comprehensive software testing tool evaluation. As a result you keep promising yourself that you’ll pick up the trial again in one or two months time.

3. When you added up the costs of the software licenses, new hardware required, training and the investment in time required you quickly decided that the investment wasn’t worth it. Or at least the financial figures you came up with were unlikely to be signed off.

4. When you realised that test automation isn’t something that can implemented by one tester in his or her spare time you figured that it’s likely to require significant business process changes. This would mean setting up a dedicated automation team and trying to work out a process for evaluating good tests to automate. It may even have meant working with the development team to get changes made to the code for the automation tool to work. Either way making changes to the existing process seemed quite daunting.

5. When you looked at the recommended best practice for test automation projects you realised that you had something that looked more like a development project than a test project. Source code control, scripting languages and the certainty that your automated code would have defects, suddenly made you aware that this isn’t going to be like any other test project you’ve undertaken.

Whilst there are any number of points to put a test manager off starting an automation project, for many companies the effort, time and money is all outweighed by the benefits. When software testing tools are implemented well the following benefits can far outweigh the issues listed above.

1. Once a test is automated, unless the feature being tested changes, then it can be run consistently day in day out with little overhead.

2. It’s usually far quicker to run an automated test than to run a test manually (exceptions to the rule on this are when you have a poorly written automated test and you spend a significant amount of time maintaining and setting up a test run).

3. Automated tests can be scaled up quickly to provide load and stress testing capabilities

4. Software tests can be run continuously and out of hours. Usually the biggest benefits are gained when tied in to an automated build environment. In this kind of setup the testers don’t even need to evaluate the results. The results from an automated build can be pushed directly back to the development team.

5. the same test can be run on combinations of platform and operating system quickly and easily. Whilst it’s never quite that simple with automation tools, when you do get this right the benefits can be huge if you have a complex test matrix to cover.

There are many other benefits and also many other pitfalls. Nevertheless, as many successful test teams have found, the cost benefit ratio usually falls down a long way on the side of delivering benefits. The benefits can be huge yet despite this many never get past the first hour of trialing a software testing tool. The key, before starting a trial, is to consider these points and set realistic expectations.

1. Effective test automation requires significant changes to existing test practices. For example most successful test automation projects have dedicated test automation engineers who’s sole purpose is to focus on automation. You will need to bring in new skills or train existing staff.

2. As well as writing the automated tests the automation team needs to be responsible for putting new business processes in place. That process includes taking and evaluating requests from others as to which tests to automate. You can’t automate everything so a process that evaluates where your time is best spent is essential.

3. Supporting the development of automated software testing tools is a development project. You will need to consider requirements definition, source code control and defect tracking. The development team follow these best practices when they write code. Your automation team should be following these best practices too.

4. Monitoring the development and execution progress is essential. Without stats to monitor your progress you’ll never know if you are getting the benefits you expected in the first place. Think through and define the management information you need at the outset of the project.

5. Don’t forget to consider training and maintenance. The hidden costs of software testing tools can make or break an automation project. If you’re selecting a new automation tool chances are you’ll need training. Once the automated tests are in place you can guarantee that the code will need to be maintained. This all takes time, effort and money.

In short don’t just think that you can succeed with software testing automation by expecting one tester in your team to spend a bit of his or her spare time learning and implementing. This approach may work for some but experience shows it usually results in failure. Far better to tackle this as a development project in it’s own right. In doing so you approach the implementation of new software testing tools with the right mindset and increase your chances of success.

I joined a very interesting webinar on Proactive Software Testing on Thursday. It was run by a guy called Robin Goldsmith who specialise in improving the software development process and software quality. In particular he has his own take on Agile development he refers to as Proactive Testing.

Now I can’t get too excited about all the arguments that surround “this” agile method or “that” agile method. I figure no one methodology is the right methodology. What’s right is your team picking the best methodology for their specific circumstances and, where necessary, modifying it to extract the biggest benefit. Having said that, there were a number of points in this webinar that really struck home. Points that I think can be applied regardless of the particular methodology you use. So Robin’s methodology may or may not interest you but I think these points may be of interest.

There are limits to the benefits of embedding software testing early in your development life cycle. Lets take the Extreme Programming approach, where developers write testcases, write the code and then decide if the code works based on the test results. The main limitation to this approach is that the developer is viewing a small piece of the code. He or she maybe unaware of the bigger pieces and may not fully understand the interaction of these pieces.

In addition there’s a chance that the developer, when writing the testcases, is envisioning how they are going to write the program. This in turn influences the tests written. It also limits the objectivity of the tests. So testcases written by developers are not going to catch things that they don’t understand or that they’ve overlooked.

The agile movement is a reaction against documentation.
This struck a cord with me. I’m not a huge fan of lots of documentation (especially when it means writing pages and pages for a plan which no one else is likely to read). Yet from a testers perspective I really appreciate a well written design document from which I can base my tests. Some argue that agile is an approach that helps developers (and lets face it testers too) avoid writing is a fair point. We’re all guilty of wanting to develop code or run testcases rather than focus on writing stuff down. This is an interesting way of approaching this…

“Documentation is not about generating paper work but rather writing down ideas so that you and others don’t forget them.”

With that in mind the purpose of writing documents seems to take on a less onerous meaning. It’s certainly an approach to documentation that I can subscribe to. It also fits with my thoughts on software testing documentation.

Aim to catch requirements issues before they get into design.
Yes we all know this one. We all know that most bugs are introduced at the beginning of the life cycle. The practicalities of software testing early in the life cycle, especially right up at the requirements capture stage, is damn difficult though. In the real world, with time, money and delivery pressures, pushing the testing into the requirements phase can be almost impossible.

It is, however, a point worth driving home. For example in the world of Agile methodology the development team has a degree of interaction with user representatives. Yet just because a user says something doesn’t guarantee that it’s a solid business requirement. Firstly the end user might be wrong. Secondly the end user isn’t the only stake holder. Is the requirement the end user is describing right? Possibly not!

Question and test the requirements up front and you prevent mistakes making their way into the design. Prevent them getting into the design and you end up with better code. With better code the tester spends less time bogged down with trivial defects. It’s an easy argument to construct.

Pulling testers off physical software testing tasks, to work on testing requirements, is a tougher argument to defend. It doesn’t always have to be testers that get involved with this though. Yes a tester will bring that critical, left field, approach to the review process. But any review and analysis of the requirements has to be better than none at all. Persuading the project team and business analyst to spend a bit more time up front can deliver a big pay back to the testers (and the project) in the long run.

In summary, a provocative webinar with many concepts and ideas that have practical application. Certainly some interesting ideas for software testing in an Agile environment that are worth experimenting with.

The implementation of a software testing process is usually one of the most difficult tasks facing an organisation. Consistency, continuity and repeatability of your procedures deliver a framework from which a growing team can work effectively. Without consistency chaos usually ensues.

Many testers thrive off chaos but ultimately the chaos approach doesn’t scale and usually falls apart under tight deadlines and with complex products. Having said that smaller teams can deliver an effective test function with little structure in place. Again though this depends largely on the competence of a small number of testers and tends not to scale as a team grows.
At some point a team usual begins to feel the need for a formal process.

That point usually comes as the amount of software testing needed escalates. There’s nothing like a poor quality product release to prompt a demand for more testing. Also, as the requirement to test surfaces in multiple teams within an organisation the demand for formal process can become apparent too. The other point at which formal process can be useful is when your outsource. The idea of outsourcing is usually to avoid any of the overheads of managing the testing in house. If you have few procedures is in place then it is difficult to convey your expectations to an outsourced team.

Ultimately the purpose of implementing procedures is to save your team effort, money and time. If the cost of implementation outweighs the cost savings then you have to question the need to implement. Although if you see increases in quality then the trade off, of more effort, money and time may be a trade off you’re willing to take.

With the need for a standard process identified the strategy for implementation needs to be considered. This is likely to involve appointing someone to drive the implementation, identify the scope, define an implementation plan and monitor the roll out. All of this is usually best covered with a small software testing pilot project before any organisation wide roll outs are considered. The goal being to test your approach on a small scale, make adjustments where necessary and then roll out further within the organisation.

With a procedures implemented the work doesn’t stop there. Ongoing maintenance is an important part of making sure the benefits are maintained. New procedures can be out of date pretty much as soon as it’s implemented if changes elsewhere within the organisation impact the team. So the implementation must be reviewed at regular intervals to see if areas of the process need to be tweaked or changed. Various review points can be chosen to evaluate the effectiveness. These review points would include the end of project, changes to the development procedures, implementation of new software testing tools, issues identified through monitoring metrics and changes to the organisational structure.

Implementing a standard test process should not be taken as a lightly. Having said that the implementation of an effective software testing process can bring big benefits in terms of quality, time, effort and costs to an organisation.

8MVFPTX38K9W

There seems to be little agreement in the software testing industry about how much of our work we should document. Stop and consider this for a moment. It’s not just the software testing industry that disagrees on this topic. It’s not just organisations, sectors in industry or individuals that disagree. Socrates and Plato, two great classical Greek philosophers, both disagreed about what they should write down.

Socrates had a complete aversion to writing any of his philosophical musings down. In fact he actively objected to writing anything down and encouraged others to avoid writing anything down too. His argument going along the lines of; text without teaching gives the illusion of learning and little else. In short he disliked the idea that you could know lots but understand very little.

Plato on the other hand argued that if we write nothing down, how do we impart our knowledge and wisdom on future generations? And with hindsight, if it wasn’t for Plato writing down a lot of what he was taught by Socrates, we’d have lost an immense body of work and understanding.

So these arguments for and against writing have been raging for centuries. And clearly these arguments go well beyond the software testing industry. For us though, there are some unique considerations. Some of those unique considerations include:

1. How much time do we have to document and write things down?
2. What legal requirements are imposed on us to document our work?
3. To what extent does writing help us as individuals with what we do?
4. What do our customers expect from us?
5. How else are we going to convey our thoughts and knowledge if we don’t write things down?

In some software testing environments I’ve worked in there just hasn’t been time to write much down. In some situations no amount of pushing back is going to get you the time you need either. So, as long as the team you’re working with understands that you can’t write much down, and they’re not demanding much in the way of documented tests anyway, then fine. In some environments though (e.g. air traffic control, medical, nuclear, etc) you have legal obligations to meet. So the requirement to document just about everything is built in from the start.

Another perspective to look at this from is what we demand from our counter parts in the development team. If I knew a coder was never writing comments in the code then it would suggest to me that the code may become difficult to maintain in the future. As a software tester I wouldn’t be happy about that. It’s likely to result in me finding bugs at a later stage that could have been avoided. Similarly if a development team has more than one developer I’d expect them to be using a source code control tool to help them collaborate and work effectively together. I’m a customer of the developers who write code. I expect them to write comments and use source code control tools. I expect this because I’ve seen the chaos the ensues when code isn’t controlled with source code control tools. Equally I assume that my customers expect me to write and manage my test cases with a test management tool. I may or may not think it’s necessary. If I’ve got a customer to deliver to, then I think it’s fair for them to demand this from me.

On a personal level I also know that when I write down test cases, and document software test plans, it helps me clarify my thinking. I come up with more test cases as a result, I’m clearer in my own head about what I need to do and I waste less time doing unnecessary tests. It might not work that way for everybody but I know writing things down helps me.

Finally there’s the ‘what happens if you get knocked down by a bus tomorrow’ scenario. If you don’t write anything down (or what your write is poorly written) then how do you expect another software tester to pick from where you left off. Maybe you convey that information regularly by talking to people; which is fine. If you don’t then you ought to be writing it down. That is unless you think what you do is so un-important that nobody else needs know about it.

In short I’m not advocating one approach over the other. I do think we need to stop every now and again and consider the questions above though. Every job, organisation and situation is different. It’s in our interest as professionals in the software testing industry to consider the best approach for what we are doing right now.