Essential Software Testing Tools Blog

I’m currently working on a product with a customer that is more than just software. The product has a significant hardware aspect to the delivery in addition to the software component. The product in question is a self service kiosk which run as a .Net application, linked to a Java framework which drives a number of hardware peripherals (printers, card dispensers, touch screens, etc). Whilst I’ve been responsible for overseeing the software quality for this product I learnt a valuable lesson in ‘holistic’ testing. Well to say I learnt a lesson is probably an understatement. Rather I was completely humiliated whilst attending the commissioning of this product in front of our customer.

To put this in context we’ve spent 6 months working on the software component of this product and I have to say I’m pleased with the quality of the completed software. We’ve completed system testing, integration testing and user acceptance testing. We’ve run in to the usual issues associated with completing such a process and managed issue lists in conjunction with the customer to good effect. Yes we have a number of minor issues outstanding following the completion of our software testing. These issues have been prioritised and a plan put together to resolve them. So from the software perspective we feel like we are in control and have a stable product that meets the requirements set out by the customers.

So why was the commissioning such a humiliating experience. Well with the client standing over my shoulder as we commissioned these kiosks, I found myself coming up with excuse after excuses for deficiencies in the hardware aspects of the product. Embarrassing issues like the power supply tripping out and paper receipts not dispensing correctly. Really fundamental hardware flaws that should have been picked up well before the units left the factory floor.

It did not matter one bit how well the software worked. The whole products quality came into question because of a few stupid hardware design flaws that should have been resolved in the factory. It didn’t matter that we’d got a great development/test team, followed well defined software testing processes and managed the software issues effectively. The fact was that the customer was only interested in the quality of the overall product. Rightly so!

So if I learnt one lesson in this it was that even if my remit only encompassed the software aspects there is a responsibility on everyone in the whole team to work towards improving the quality of the whole product. There is a responsibility on the whole team to ensure the quality assurance process for the whole product delivers the quality the customer demands for the whole product.

How are we going to improve this in for future releases? Well we’ll apply the same rigorous process and resources to the hardware factory acceptance test process that we apply to the software testing. The goal being to catch hardware issues as well as software issues before the product even leaves the factory floor. A worthy goal but still a significant amount of effort to implement before we start to avoid other humiliating experiences during the commissioning processes.

The success of any development project isn’t dependent on just the skill of the testers. It isn’t dependent on just the skill of the programmers. The success of most projects is dependent to a large extent on the relationship that is built between the test team and the development team.

Years ago I worked for a test manager who thought it was healthy to have an abrasive relationship between the test and development teams. He believed that this friction resulted in a higher quality of the delivered product. He assumed it was acceptable to have loud shouting matches with the development manager. He frequently stormed out of meetings with the development manager.

I would sit there watching the relationship between the development manager and the test manger evolve, or rather deteriorate. I would think; if I had that sort of relationship with my counterpart in the development team the developer would never feel like fixing any of the bugs I described to him.

“If there is conflict between test and development then you’ve lost the ability to work constructively together for the benefit of the whole team.”

I’m not saying you shouldn’t disagree with developers (you should if you think there is a serious bug that they are not paying enough attention to). What I’m saying is that if you have a constitutive, positive relationship with your counterparts in the development team then you stand a much better chance of working together to do what is right to deliver the right product, to the required quality levels, within the specified time scales.

Software testing and development may be distinctly different aspects of any project but the ability to work together usually dictates the end result.

With a couple of developers out on holiday last week I had a less senior developer come to me and ask if it is okay to release a build to the software test team without going though the formal build process. So the release would be a build compiled on his local machine with no formal release identifier and no release notes. His argument was that this would enable him to release to the test team earlier.

On the face of it this was quite a compelling argument because we were behind schedule on this project and the test team needed this release urgently for regression testing. I was tempted to take the bait and speed things up.

Experience has taught me though, that you’ve got to be slightly deluded if you think taking an informal release is going to speed things up over the long term.

Firstly, you open yourself up to lots of little issue to manage as you start to work with the release. What build number should you use when you raise a bug? Where should another developer get this build from when trying to recreate a bug? Whilst these issues may seem small on the surface they can take up an inordinate amount of accumulated time for your team to resolve. Each one small in isolation but over a period of days and weeks it soon builds up to a significant amount of time. Usually more time than it would have taken for the developer to create a formal release in the first place.

Secondly, and more importantly, once you’ve started down this road its very difficult to get back to the days where you had a well defined and adhered to build delivery process. Once you’ve started down this road every time things are slightly behind schedule the development team will assume that it is acceptable to repeat the informal release process. One informal release in the process can cause a problem but 2, 3 or more informal release is certainly going to get you unstuck.

So the first piece of advice is, what ever you do don’t accept an informal release from the development to test. No arguments, no discussion or any consideration. Just don’t take an informal release. The second piece of advice is that working with teams where the build process is automated makes even the idea of informal releases redundant.

I learnt a very important lesson very early in my software testing career.

Every now and again I do something which, with hindsight, seems like a really stupid thing to do. Take this example. I was purchasing an e-book on line where the payment page had an orange square box with the text “Click here to purchase” written on it.

I spent just over a minute trying to figure out where to click to make the payment. And yes that does seem stupid with hindsight considering there was an orange box with the text “Click here to purchase” displayed. So why did I find this so difficult?

Well firstly, like most people on the Internet, I scan hundreds of pages a day of web text, so I do just that …. i scan rather than read in detail. I wasn’t looking for the text “click here”, I was looking for a button to click.

Secondly, the orange square box with text in it didn’t react to the cursor changing shape when you put the mouse cursor over it. The designer had missed a fundamental principal of the Internet; the cursor shape should change to indicate the item can be clicked.

Thirdly, the orange box was embedded with lots of other images surrounding it, making it difficult to pick out from the crowd. In short this item, that they wanted me to click in order to make the purchase, was so far removed from any other ‘click here’ standard on the Internet that I completely missed it.

So how does this relate to one of the first software testing lessons I learnt. Well my reasoning is as follows.

Sometimes a software test I do may seem stupid on the face of it. However there are a lot of other ’stupid’ people out there with a similar level of intellect to me. So if I’m making the mistake you can guarantee that a lot more people are making exactly the same mistake.

This would tend to suggest that it doesn’t matter how stupid you think you’ve been when testing a piece of software, if you made the mistake you can guarantee that there will be hundreds if not thousands of people out there that will make that exact same mistake. So don’t let it go. Get someone to fix it!

Ever wondered why choosing a development framework is like shopping for clothes with your wife or girlfriend? “No”? Oh, well I’m going to tell you anyway!

A couple of months back, when the ball started rolling for work on version 7 of QaTraq we started looking at the various development frameworks available for Php. The options were quite simple. Only two real contenders were available; Symfony and Zend. Choosing between the two was, however, somewhat more complicated than just identifying them.

Right at the start, Adrian, our lead developer, suggested a combination between Symfony and Zend. The argument behind this was that we would get the flexibility of the Symfony framework and access to the great range of Zend modules and components. The fact that Symfony could utilize the Zend components meant that this was technically possible and would give us the best of both worlds. These reasons had not been thought through in detail at this point, but were probably based quite legitimately on instinct, intuition and years of experience.

In the name of best practice though, we went through a four week evaluation and prototyping period. We were attempting to select just one of the two options in the belief that choosing just one would simplify things for us. One day we’d be thinking Zend would best suit our needs, the next day we’d be extolling the benefits of having the flexibility of running with the Symfony solution. As our prototyping with both solutions went on for week after week so did our constantly changing opinion on which of the solutions was the best option to go with.

After about four weeks of constantly changing minds we had our final meeting to discuss which framework we were going to use to construct version 7 of QaTraq. We spent an hour changing our minds again as both solutions had very real pros and cons. That was until Adrian pointed out again that perhaps his initial suggestion combining both solutions, the approach suggested at the very start, was the way to go. One final prototype later and more web research, and we had made our decision.

So why did we spend 4 weeks changing our minds only to go back to the original suggestion? And how does this compare with buying clothes for your wife or girlfriend?

After 4 weeks we were back to square one with the original idea that should we go for Symfony and then use the Zend components we need where necessary. This is when it occurred to me that this evaluation is a lot like taking your wife or girlfriend shopping. First you take her in shop number 1, and she finds a really nice pair of trousers. She doesn’t buy them because she needs to see all the other shops before making a decision. So you visit what feels like hundreds more shops, see what feels like thousands more garments but 8 hours later you get taken back to buy the trousers in shop number one. So you’re left wondering why on earth we didn’t buy the trousers in the first shop in the first place.

Well I mentioned this analogy to the guys at which point Adrian spoke up and pointed out that this analogy wasn’t quite accurate. Adrian quickly pointed out that we hadn’t bought the best trousers from shop number one but rather we’d bought ALL the trousers just to be sure we’d got the best pair of trousers. I could see his point!

Regardless of how many pairs of trousers we’d decided to purchase the moral would seem to be that there is a lot to be said for trusting your first instincts. So we settled with a combination of Zend and Syfony, and we haven’t looked back since.