Essential Software Testing Tools Blog

Have you read Steve Jobs’ biography yet? If you haven’t you should. Why do I say this in a software testing blog when the term ‘software testing’ isn’t even mentioned once in 600 pages?

Well what struck me more than anything throughout this book was this guys relentless drive for quality. Not specifically software quality, hardware testing or any other form of product testing. Just the relentless quest to make sure he created brilliantly high quality products. This obsession for quality rippled right down through Apple. It was clearly endemic. It was enforced with actions like this…

1. Pausing the release of the first iPhone
Nearing completion of the first iPhone Steve Jobs decided he just didn’t love what they’d spent the past year sweating over. So to his teams dismay he started over with the design.

2. Berating the MobileMe team for damaging the Apple name
When MobileMe was criticized for being unreliable in the Wall Street Journal heads rolled and the team was berated for damaging the Apple name. Those that were left keenly aware that they’d let the team down.

3. Acknowledging the issues with the iPhone 4
With Antennagate Jobs (eventually) laid out the truth and stated that “we’re not perfect but we want to make our users happy”. In doing so he maintained, perhaps even increased, his integrity.

Actions like this, day in day out, clearly reinforced to the rest of the company that Apple was (and is still) about quality. It’s about the products not the profit.

In the software testing domain we all talk (and frankly little gets done) about identifying defects early in the product life cycle. The age old argument that if we identify defects in the requirements capture phase then we’ll save billions (slight exaggeration) of dollars in the long term. We’ll with Steve Jobs’ drive for quality this concept was taken to another level.

When your CEO advocates quality in this way it has to rub off on the rest of the organization. It’s a million times better than someone just standing up and saying we must identify defects at the requirements capture phase. This is about instilling a passion for quality in the whole company. It’s a drive for quality from the ultimate source; The CEO.

How many CEO’s of companies today can honestly say they want quality that much? You’re presented with the option to ship a nearly completed product (like the first iPhone) or stop and say .. “this isn’t good enough”. Would you ship and start taking revenue? Or back off and start again? Far too many companies are consumed by the need to ship products and start generating revenue. Revenue first, quality second.

I expect that many CEO’s would argue that quality is top of their list. That quality is what makes them a great company. That the CEO has a list of examples where he’s castigated teams for poor quality releases may be used to prove a point. Where he may have sacked someone as a consequence of a product release that didn’t go well. It’s usually a front though. Very few CEO’s have the real credibility needed to backup their commitment to quality. Far too many are consumed by the desire to chase profit and revenue.

For Apple though it was more about quality first and then the profit will follow. Jobs was quoted as saying “the product, not the profits, were the motivation”. And therein lies the a tricky dichotomy. You need profit in order to invest in quality products. Yet with quality products comes higher profits (or at least it did for Apple). It’s the same dichotomy we face in software testing.

Do we invest heavily in testing in order to drive up product quality? Can we really say that this investment makes a significant difference to the bottom line? Who knows? One thing I do know though. Quantifying our contribution to the bottom line is near on impossible. Yet with Apple we have a clear example that by putting product quality first you can build the world’s largest publicly traded company. Surely that holds some weight when it comes to arguing for more, not less, resources being allocated to software testing?

I came across an interesting report from Gartner on integrated software quality suites last week. Whilst the report is 12 months old it does provide some light on the increasingly competitive market of software testing tools. Gartner apply their Magic Quadrant research approach to assess the positions of the competing vendors within this market and determine how well the vendors deliver on their stated vision. The full report can be purchased and downloaded here..

http://www.gartner.com/id=1533716

For those of you that aren’t familiar with the Magic Quadrant methodology this is an approach for categorising positions of product vendors. The aim being to allow you to understand how technology providers deliver products that might best fit your needs. Gartner categorise the providers into Challengers, Leaders, Niche Players and Visionaries. Where challengers dominate the sector but may not be well placed for the market of tomorrow. Leaders are delivering against their vision for the market and look well placed for the market of tomorrow. Visionaries can see where the market is going or may even be dictating where it is going but are yet to capitalize on this vision. And niche players execute well in their segment but may not be placed well for competing in tomorrow’s market.

The report argues that companies are looking for solutions that enable automation, deliver traceability, improve workflow and deliver more comprehensive reporting. Another key point that they put across is that companies are looking for suppliers that have well established partner programs that can deliver as systems integrators and put consultants on the ground. It’s not unexpected to read that many of the market leaders (HP, IBM, etc) don’t have the most innovative products in the software testing arena but they typically catch up pretty quickly when they see the market move (usually by partnering or acquisitions). The visionaries in the sector are usually focused on a particular niche where their tools are purchased in conjunction with the market leaders.

The most interesting point they put across is that Gartner expect two or three companies to move into the leaders quadrant. This is clearly not an easy space to start competing in. The reasoning behind this that a few companies with composite and cloud based offerings will be well placed to take advantage of the changing technology landscape.

As expected HP is singled out for being the dominant player in the market. Whilst pressure is building on them to provide more competitively priced products they still deliver in most large enterprises. Whilst HP’s support, in the past, has been singled out for criticism this report notes that HP have made solid improvements in this area. Again HP deliver a tool set that leaves all the competitors mirroring their approach and aiming to provide integration to their tool set. No wonder then that most consultancies and system integrators purport to support the HP tool set. Yet there is some argument that the product line has been perceived as a little stagnant.

IBM is also up there as a dominant player. Again delivering a toolset that supports the full lifecycle. And again providing a strong global presence. Yet their Jazz platform, which delivers tools across the full development life cycle, comes in for a small amount of criticism. It’s argued that some of the products deliver overlapping functionality (although it’s known that this is being addressed). Ultimately this means it’s not always clear which products need to be deployed. The main strength for IBM? Delivering a quality based approach across all the project roles covering the whole project life cycle (e.g. from unit testing to build processes).

The other vendor I wanted to pull out from this report is SmartBear. They’ve been picked out as a Niche Player based on their focus to deliver cost-effective tools that fit teams working with agile development processes. With a product range that covers application lifecycle management (including test management), test automation, code review, load testing and continuous integration. The main point of caution raised here is that the product suite is best suited for small to mid sized teams but this is clearly compensated by the relative advantages of the price points they take in comparison to the likes of HP and IBM.

Whilst this report covers many other vendors and suppliers of software testing tools it confirms the market dominant position of of HP and IBM. Yet it notes the increased competitiveness in the market and the threat some of these niche players pose to the dominant players. Many of these new players are are doing well because they provide better support and better targeted solutions. It’ll be interesting to see in the next release of this report. Which, if any, of these new software testing tool providers will have made it into the Leaders quadrant?

Interesting article from Seth Eliot on the future of software testing in “The Testing Planet” this week. You will find a copy of the article here (although it does require a subscription)….

http://www.thetestingplanet.com/2012/03/march-2012-issue-7/

In summary Seth argues that the traditional product QA approach is changing. The concept of execute, record result then mark as pass/fail is flawed in the new world. Quite a brave argument to put forward when this concept is such a huge cornerstone of software testing. He puts forward a good case though. With the advent of large scale services (Google, Amazon, Facebook) Seth suggests that for the web this traditional concept is no longer adequate.

In order to assess the quality of web services we need to be analysing live production data. Our test input is the live users accessing the system. Our recorded results are the telemetry info being generated in the data centers that run our applications. Our pass/fail analysis will then be based on patterns or detail found in this data. This analysis then gives us insight into how the users are using the systems and the behaviour of the system itself.

Seth argues that using this approach finds defects that you’d never find in a test lab. It’s hard to disagree with this. Only real life analysis is likely to show up real life user experiences (e.g. how long it takes for a page to load across different operating systems and browsers). Effectively we’re software testing in production.

Seth calls this approach TestOps. Implying that we’re testing in the operational environment. As part of this live environment approach we have three key inputs we can work with. We can analyse our telemetry info for KPI’s (e.g. availability or performance). We can also deploy new features or releases into production where a small set of end users get access in the live environment. Essentially we’re running in a beta phase where the end user may not necessarily know they are part of a beta test. The third input can be testers that make back-end changes in an attempt to force test scenarios and then assess results.

This is a significant change in the software testing landscape for you and I. Our focus is shifting from working at the front end of the development process. In this setup we’re working with a live environment after the product has been released. Let’s be realistic though. We’re not going to see the traditional skills and practices change. This is more like an additional facet to our roles as software testers.

My biggest concern? This is a beta phase where the beta testers don’t know they’re beta testers. All this telemetry info may help us identify some issues we wouldn’t have caught in the traditional software testing phases. Who’s to say that what we miss with the telemetry info will be picked up by the end user? And even if they do pick it up will they tell us? I doubt it.

I also expect, with the usual pressures to ship, that we’ll see more of the QA effort pushed into this OpsTest phase. The problem with this is that we won’t catch many of the issues the end user sees (and that our telemetry data misses). We’ll become more and more reliant on the end user and telemetry data. This is not the same as a tester evaluating the user experience. For that we need dedicated software testers with the resources and time to complete their work in the traditional phases. I agree that this is an interesting new area of software testing, but I doubt it’ll change the landscape drastically.

Many software testing teams are happy just to bumble along with their existing software testing processes and tools. They have no desire to continually strive to make those small but significant incremental changes. For these groups it’s only when something nasty happens that the impetuous to improve and develop surfaces. Contrasting this are the QA groups where there is just this inherent desire to improve and develop. It’s part of their DNA. Working with these teams is empowering and immensely enjoyable.

Testers often quote lack of time and poor resources as the reason for not changing. This is usually a cover. The biggest problem is the overall desire to change. I’ve worked in teams where line managers were resistant to change. It’s sole destroying sitting there knowing that things could work much better. On the other side of the coin when you’re part of a team that loves to improve it’s one of the most empowering feelings you’ll ever experience. Successful groups make time. Successful groups find resources. Successful software testing teams build an attitude and momentum that relishes development and improvements.

Think of it like the seesaw you played on as a child. When you and a group of friends all jump on the seesaw at the same time. You end up with 5 people on one end and 3 on the other. With enough weight on one end there’s no way the lighter end is going anywhere. Improvement is about getting enough people and weight on to the end of the seesaw that embraces change. You need the balance to swing in favour of change.

Typically there will be a few people in a software testing team that want to embrace change. They want to improve processes and implement new tools. They want to see the team succeed and they enjoy the challenge of learning new things. The problem comes when the majority of the weight is on the side of the seesaw that doesn’t want change.

If you’re one of those people that loves working on improving and developing then take stock. If you’re feeling frustrated at the lack of progress then get off the seesaw and look at it objectively. Is your manager on the end that is resisting change? Are the majority of the team on the end that want everything to stay as it is? If they are then leave. Walk out of the playground. Find another job. Find a job where people will embrace your enthusiasm for change and your skills for implementing tools. Find a software testing environment to work in where you can make a difference.

There will be resistance to change in any software testing team. The key is to objectively evaluate your chances of success. Can you achieve what you want to achieve where you are? If you’ve got enough people on your side of the seesaw then great. Keep persuading others to join you on your end. If the weight distribution looks unfairly stacked against you then find another software testing group to work with. Find a group to work with that makes you feel empowered. Work where you can make a difference. There are other teams out there that will really appreciate your enthusiasm and talent. You just need to find them.

Software testing of APIs is becoming an ever more important aspect of the QA teams repertoire. With many companies understanding the importance of data sharing, APIs are an important part of many businesses strategies. In fact, for many companies, opening up data channels with APIs is the most important decision they’ve made since accepting the web in the first place.

As companies extend software applications by opening up and sharing data via APIs this puts additional demands on software testing teams. This type of testing is technical and requires an understanding of protocols and data formats that go well beyond the usual GUI stuff you run. So typically API testing ends up straddling both development and QA teams. As a result many QA teams have to work closely with a development team to implement their software testing objectives in this realm.

On a technical front many use tools like SoapUI to cover the tests. Yet even with tools like this you still need to bring this under control of your test management process and look to automate. The following video looks at an approach for linking SoapUI to TestComplete (TC8) and QAComplete.

Here we have SoapUI executing the API tests. TC8 covers the automated execution of SoapUI. TC8’s integration with QAComplete is then leveraged to provide a central reporting location for all your automated tests. With this setup you can schedule the execution of your automated SoapUI tests directly from QAComplete (great for managing regression).

If you are looking to try this software testing approach then the following steps will help. Please note that the following expects a reasonable level of knowledge of SoapUI, TC8 and QAComplete. If you’re not familiiar with these products then you can sign up for trials using these links.

     Product Trials

     Software Test Management with QAComplete
     Test Automation with TestComplete
     API Software Testing with SoapUI

QAComplete to TestComplete Integration

You will need to install the software bridge application on the TC8 machine. Download this from here.

     http://www.softwareplanner.com/downloads/AutomatedTestingBridgeSetup.msi

The user guide to help you setup can be found here.

     http://www.softwareplanner.com/UsersGuide_TC.pdf

TestComplete to SoapUI

1. Create a testedApp in TC8 which executes a command line like this.

“C:\Program Files (x86)\SmartBear\soapUI-Pro-4.0.1\bin\testrunner.bat” -sLoging_and_Get_Config -c”Get Config” -fC:\ -R”TestCase Report” -FXML C:\Users\Bill\Documents\QAC-soapui-project.xml

This is the SoapUI command line that will execute your API tests. This command line should run successfully in a command prompt before you try to include it as a TC8 tested App.

2. Create a keyword test in TC8 that does two things on the software testing front.

      a. runs the tested app (e.g. the command line in 1 above).

      b. waits for the SoapUI run to complete.

3. Create a script in TC8 that parses the log file from SoapUI. An example script can be found at the bottom of this web page (e.g. its the 3rd example).

     http://blog.smartbear.com/post/06-10-05/working-with-xml-files-in-testcomplete/

You may need to install this on your windows box to make this work.

     http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=19662

You will need to modify this script. You’ll need to modify it so that it picks out the parts of the SoapUI log file that are important to your software testing reporting requirements. The script we’ve given you will work in principal but you’ll need to modify it to present the required entries in the TestComplete log as you see fit.

4. Create a project in TC8 that contains (2) and (3) from above.

5. When you execute the project in TC8, this will run SoapUI, collate the results (and copy the log files) back to QAComplete.

Taking your software testing a step further you can then use QAComplete to schedule automated runs of SoapUI tests. To do this setup a test schedule that calls the TestComplete project that you created above. In this way you can drive everything from QAComplete.

1. QAComplete kicks off the TC8 test.
2. TC8 runs the SoapUI tests.
3. TC8 parses the SoapUI log/results file.
4. TC8 creates test results and logs based on SoapUI log file.
5. TC8 writes test results and log files back to QAComplete.

Rather than link these tools in this way you could link SoapUI directly with QAComplete to cover scheduling and result tracking. To do this you would need to utilise the QAComplete API and write the corresponding bridge application to link SoapUI to QAComplete. Probably a more efficient solution but it would have one major down side. That is you can’t combine your API automated tests with GUI and database automated scripts.

With the setup we’ve described, once you are driving SoapUI from TestComplete you can write automated tests that encompass multiple areas of your application. By this we mean that you can test the API (via SoapUI) and confirm the GUI operation of your application with TestComplete’s GUI automation. Then add to this connectivity to your back end database with a TestComplete database connection. Then you’ve got the ability to automate the cross checking over all the major aspects of your application. Then link TestComplete back up to QAComplete where you store all of your manual and automated test results. With this you have a well integrated test management and automated software testing framework.