The most obvious hidden cost is training. Yes the vendor you purchase from will point you in the direction of training courses and clearly state the price of these training courses. Just because you’ve been on a training course though doesn’t mean you know what you are doing. It’s like passing your driving test and then spending years building experience. The training course will teach you how to use the tool but only time and effort will provide you with the experience you need to become really productive.

There’s also the temptation to skip any training. How many of us are guilty of just ignoring what’s on offer and assume we can learn the package in our own time without the need for any training? Many of us do this because we know the cost of training will add significantly to the purchase price. If the purchase price goes up then the chance of the purchase being signed off is less likely. So we forgo the training thinking that we’ll work it out ourselves anyway. More often than not this is false economy. Four or five days of training usually results in a saving of 4 times that in wasted time working things out for ourselves. Productivity goes up significantly after training and the likely-hood of a successful test tool implementation goes up significantly too.

It’s not difficult to quantify the benefits of purchasing training. It is time consuming to add that information to a well constructed software testing tools business case. It’s always worth investing that time.

One of the other important hidden costs is infrastructure. Quite often tool vendors won’t quote the cost of supporting applications in the headline price for the product. For example many software testing tools require a back end database and the tool vendor conveniently forgets to mention the cost of this in the headline price of the product. Granted for many companies they’ll already have site wide licenses for the databases back end but for many companies they won’t. The associated purchase price can be significant.

Integration is the other area that is forgotten about all together. Rarely these days does a software application exist in a vacuum all on it’s own. With many other business applications already implemented it’s more common to want, or even need, some degree of integration with existing systems that the software test team uses. Whilst you might have the in house knowledge and resources relating to your existing systems it’s likely you’ll need to bring in expertise for any new tool your purchasing. Specify your exact requirements for integration at the start to avoid any unpleasant surprises after you’ve completed the implementation of a new software testing tool.

The other game the software test tool vendors tend to play is putting many of the features you need in a higher priced edition of the product. So whilst the headline purchase price might look within your budget the price you really need to pay, for the feature set you need, is significantly higher. Be sure to question the vendor in detail about specifics during a product demo.

Lastly, the other charge that many teams forget about is the cost of peoples time to implement the tool successfully. The cost of software testing tools are usually insignificant compared to the staff costs. When you factor in the cost of having many members of your team implementing, learning and getting started with a new tool the cost can take on a whole new dimension.

Even if you can’t estimate all of these points accurately, a rough estimate will give you a completely different perspective on the purchase price. In short make sure you list and estimate all of these hidden software testing tool costs right from the start.

AutomatedQA’s webinar on the 1st of July will teach you how to use a blend of both automated and manual software testing. Getting the blend right so that your software testing ensures that releases do not break existing features.  In the webinar, you will learn how to:

1. Identify test cases worth automating
2. Organize your automated software testing
3. Create effective manual regression suites
4. Develop manual test cases to cover new features for your release
5. Map test cases to requirements for improved traceability

You can sign up for this here:

* Webinar Date:  1st July 2010
* Webinar Time:  0900-1030 ET-US, 0630-0800 IST, 1200-1330 EST-AU
* Signup Page:   http://www.softwareplanner.com/webinaruniting.asp

If you are not available to attend at that time, register anyway and you’ll get a link to the recorded version of this software testing webinar.

The webinar will focus on TestComplete and SoftwarePlanner. However, it will be full of tips that apply regardless of the automation or management solution you use.

Well worth a look in my opinion.

You captured the test tool requirements accurately.

You selected the right product from your list of software suppliers.

You trained the users to use your new software testing tool.

You’ve even planned for the ongoing effort and costs associated with this new testing tool.

And now that the test tool is live the team is using it consistently. In fact you’ve actually succeeded with the implementation of a new software testing tool. This is bringing big productivity gains to your team. You suspect that even the software development team is impressed with the testing improvements you’ve bought about. You’ve achieved a great result.

Although, let’s be honest here…….IT WASN’T JUST YOU WAS IT?

In an old edition of Fortune magazine, an article on teamwork noted that ‘Neil Armstrong didn’t get to the moon through rugged individualism; there is no such thing as a self-made astronaut’.

Successful software test tool implementation involves the whole team.

No? I Didn’t think so.

I rarely see software test teams that have stepped back and worked out where it is they want to get to. We’re all  so busy working to the next software release deadline that we forget to invest any time looking to the future.

We’re all guilty of looking to software solutions in the hope that they’ll free up time once we’ve implemented  them successfully.

It NEVER works this way!

I can’t give you quick solutions to finding the time you need. I will stress that you’ll waste even more time if you pick the first quick fix for your software testing process. But you knew that anyway.

So why not find the time to implement the right software testing system, the right way, the first time round?

Consider thought that if you don’t know what you want then you won’t find what you need.

You see it’s impossible to select the right software if you don’t know what you are trying to achieve.

Lewis Carroll said – ‘if you don’t know where you are going, any road will take you there’.

Which is why selecting the first good looking piece of software to help your software testing process will  usually take you somewhere you weren’t quite expecting.

In the long run it will save you time and money if you get your requirements right at the start. Then work  towards implementing your selected tools to improve your test process. If you do it the other way round you can  guarantee you’ll end up with at least some rework. Or you’ll start from scratch again.

The clearer your requirements, the less time you’ll spend on re-work finding the right solution.

I think it’s fair to say that we spend a lot of time slinging mud at our development teams and customers over poor requirement specifications. Then, when we come to implement software solutions to help our software testing process, we make exactly the same mistakes.

In any project, effective requirements capture separates the successful projects from the failures.

And that goes for us in the software testing arena too!

Currently we tend to think up a test case, run the test case and document the process (whether that is with Word, Excel or a test case management tool). The very fact that as software testers we are writing the test case down, demonstrates the fact that we expect to re-run the test case. We may also expect someone else in the software test team to re-run the test case at a later date (usually as part of a regression test suite). If, as  software testers, we have the tools at our disposal to capture what we doing as we are executing a test case why don’t we just record it as a video rather than waste pressious time documenting and writing about the test case?

Yes it is useful to document, in writing, the purpose and aims of a test, but why spend ages writing in minute detail about the exact steps you follow to run the test case? Why not just record a video of these steps as you run the test case and then use that video as a documented record of the test steps and expected results?

I suspect that there are two reasons for this test case recording approach not catching on just yet. Firstly the concept of using video and screen capture in the software testing environment hasn’t quite taken hold, even as a simple tool to capture defects. Secondly there is the usual resistance to ‘change’ that affects all of us.  We’ve all been writing test cases down using tools like Word for years, so why change now?

Yet the moment you experience the time and effort savings associated with capturing and recording a software test case with video, over the laborious effort involved with typing a test up, you will be quickly convinced of the benefits. Mainly the benefit that you can spend more time testing than you might do typing up documents. There are big efficiency savings to be made here with such tools. More than that, such an approach leaves the software tester to do what he or she is best at, and that is running software test cases.

To this extent the software tester is going to be responsible for testing that product upgrade. Now you can approach this with some black box testing and completely ignore what is going on behind the user interface with the database. Or you can supplement this black box testing, get your hands dirty, and carry out some white box testing. For this you are going to need a little understanding of SQL and you may also need to know a little bit about the architecture of the database your application uses.

I can guarantee you that if you put some effort into this white box testing approach you’ll find more bugs. And with a failure during a database upgrade likely to cause some form of data corruption for the end users, finding more bugs has to be a good thing here. So as testers we’re faced with a difficult and technically demanding area of testing yet the consequences of getting this testing right are likely to prevent significant issues impacting the users (that is the potential for the corruption of their data that they store in your application).

The basic approach to running a comparison test like this consists of…

1. Start with the old version of the product running with the old database structure and data
2. Back up or copy the old database structure and data *
3. Upgrade your application
4. Back up or copy the upgraded database structure and data *
5. Install a clean version of the application
6. Back up or copy the clean database structure and data *

* – It’s usually easier to take two independent backups here. One backup with just the database structure and a second backup that covers the database structure and the database data.

Once you’ve completed this you should have the test artefacts listed below. If you’ve used a tool like mysqldump then these artefacts will essentially just be text files that can be compared with a standard text difference tool (like windiff):

1. Old database structure
2. Old database data
3. Upgraded database structure
4. Upgraded database data
5. New database structure
6. New database data

Once you’ve completed this there are a number of comparison tests you can employ now. If you’ve used a tool (like mysqldump) that creates the database backup/dump in text format then you can now compare these artefacts with a text difference tool, or you could employ specific database comparison tools (see below).

The easiest, and probably most useful, comparison test is the database structure comparison. However, all of the following will give you an important insight in to the success (or failure) of the upgrade process…

Upgraded database structure comparison
Take upgraded database structure and compare this structure against the new database structure. In 99% of cases these should be identical. If an upgraded database structure is different to the structure of a database from a cleanly installed new database then you’ve got questions to ask your development team

New database data comparison
With the data from the old database compare against the data from the upgraded database. You will expect to see differences here but you may find some that you wouldn’t expect to see. This can be a difficult comparison to make, depending on the complexity of the database, so it might be worth working with the development team whilst running this test.

New database structure comparison
This comparison should compare old database structure with the new database structure. This will show up the expected differences that have been implemented for the new database. This list of differences should be checked with the development team to ensure only expected changes are evident in the new database structure

The above tests can be executed with freely available text based comparison tools (like windiff). However, if your database structure has any degree of complexity to it, it is usually advisable to employ a dedicated SQL difference tool like SQL Delta, SQL Data Examiner, dbForge Data Compare, AdeptSQL Diff or SQL Data Compare . Products that have the capability to compare two different database (including both data and database structure) give the tester an easy way of checking changes between databases.

The other big advantage of employing SQL comparison tools is that they are very helpful when it comes to keeping test environment mirroring a live environment. That’s a discussion for another day though.