Dorothy Graham

Experiences of Test Automation

Dorothy Graham & Mark Fewster
Experiences of Test Automation
The technology of test automation has moved on significantly since our last book. We wanted to find out what approaches have been successful, what types of applications are now being tested and how test automation has changed.
The case studies in this book show some approaches that were successful and some that were not. Many application areas, environments and platforms are described. The chapters cover commercial, open source and in-house tools, in traditional and agile development projects.
This book is more than a collection of essays. As part of our review process we have inserted comments to highlight points that we think should stand out. We include lessons learned and useful tips.
The experiences described by the authors are all true,. We have been impressed with the dedication and persistence of those who develop test automation within their companies. Unfortunately, we have also been struck by the difficulties that many of them encountered which sometimes resulted in failure.

Chapter Summaries

Click the chapter heading to see the summary. Click again to hide

Chapter 1, An Agile Team’s Test Automation Journey: The First Year
Lisa Crispin describes, in her very engaging style, what happened when an agile team decided to automate their testing. Given Lisa’s expertise in agile, you will not be surprised to see that this team really was agile in practice. One of the interesting things about this project is that everyone on the team (which was fairly small) was involved in the automation. Not only did they excel in agile development, they also developed the automation in an agile way—and they succeeded. Agile development was not the only component of this team’s success; other factors were equally important, including building a solid relationship with management through excellent communication and building the automation to help support creative manual testing. Another key factor was the team’s decision to build in process improvement along the way, including scheduling automation refactoring sprints twice a year. You are sure to agree that what Lisa and her team accomplished in their first year is remarkable. The project was done for a U.S. company in the finance sector.
Chapter 2, The Ultimate Database Automation
Henri van de Scheur tells a story that spans half a dozen years, relating what happened when he and his colleagues developed a tool for testing databases in multiple environments. They set good objectives for their automation and a good architecture for the tool. They automated so many tests that they developed a lifecycle for automated tests that included periodic weeding. Tests were run nightly, weekly, or with special scheduling. Despite great success, a number of problems were encountered, and Henri describes them honestly. The development of this database testing tool (now open source) was done in Norway by a small team, over several years, and it achieved a very impressive return on investment.
Chapter 3, Moving to the Cloud: The Evolution of TiP, Continuous Regression Testing in Production
Ken Johnston and Felix Deschamps from Microsoft describe how they moved from product-based to service-based automated testing by implementing the automation in the cloud. Testing of Microsoft Exchange servers was already extensively automated, and much of the existing automation was reusable. Testing in production seems a foreign concept to most testers, but this chapter explains why it was necessary and beneficial to move to continuous monitoring and contains useful tips for anyone considering a similar move. This experience takes place in the United States, over 3 years, and unsurprisingly, Microsoft tools were used.
Chapter 4, The Automator Becomes the Automated
Bo Roop takes us on a guided tour of attempting to automate the testing of a test automation tool. One of the first questions to ask a tool vendor is “Do you test the tool using the tool?” But the answer isn’t as straightforward as you might think! With his lively writing style, Bo gives an honest description of the difficulties and challenges he encountered, particularly in the verification of test results. It is a good idea to find out what others have tried, and Bo shows the advantages of doing so. His sensible approach to automation is to start by automating the easier components before tackling the more complex. Unfortunately, this story does not have a happy ending. It illustrates how presumably well-intentioned management actions can sabotage an automation effort. For reasons that become obvious when you read this chapter, the tool vendor is not identified: a fictitious company and tool name are used instead. This experience takes place in the United States with one automator (the author) and covers just over 1 year.
Chapter 5, Autobiography of an Automator: From Mainframe to Framework Automation
John Kent tells us how and when test automation started and offers surprising information about the origins of capture/replay technology. Understanding how automation worked on mainframes shows how some of the prevailing problems with test automation have developed; approaches that worked well in that environment did not work well with graphical user interfaces (GUIs) and the need to synchronize the test scripts with the software under test. The principles John discovered and put into practice, such as good error handling and reporting and the importance of testing the automation itself, are still relevant and applicable today. John’s explanation of the economic benefits of wrappers and levels of abstraction are compelling. He ends with some recent problem/solution examples of how Web elements can trip up the automation. This UK-based project involved mainly commercial tools.
Chapter 6, Project 1: Failure, Project 2: Success!
Ane Clausen tells of two experiences with test automation, the first one unsuccessful and the second one a solid success, largely due to what she learned from her first experience. Lessons are not always so well learned—which is a lesson in itself for everyone! Ane’s first story is told honestly and highlights the serious impact of insufficient management support and the importance of choosing the right area to automate. In her second story, Ane designed a 3-month pilot study with clear objectives and a good plan for achieving them. Many useful lessons are described in this chapter, such as good communication (including using the walls), limited scope of the early automation efforts, good use of standards in the automation, a good structure (looking for common elements), and keeping things simple. The continuing automation was then built on the established foundation. Ane’s experience was with pension and insurance applications in Denmark, using commercial tools.
Chapter 7, Automating the Testing of Complex Government Systems
Elfriede Dustin, well known in the test automation world, shares her experience of developing an automation framework for real-time, mission-critical systems for the U.S. Department of Defense. Because of the particular type of software that was being tested, there were specific requirements for a tool solution, and Elfriede and her colleagues needed to spend some time searching for and experimenting with different tools. Their clear statement of requirements kept them on track for a successful outcome, and their eventual solution used a mixture of commercial, open source, and inhouse tools. They met with some unexpected resistance to what was technically a very good system. This story covers hundreds of testers and tens of automators, testing millions of lines of code, over a period of 4½ years.
Chapter 8, Device Simulation Framework
Alan Page from Microsoft tells a story of discovery: how to automate hardware device testing. We all take for granted that our USB devices will work with our computers, but the number of different devices that need to be tested is very large and growing, and it was difficult to automate such actions as unplugging a device. However, a simulation framework was developed that has enabled much of this testing to be automated in a way that has found widespread use inside and outside of Microsoft. The chapter includes numerous examples showing both the problems encountered and the solutions implemented. This story is from the United States and was an inhouse development now used by hundreds of testers.
Chapter 9, Model-Based Test-Case Generation in ESA Projects
Stefan Mohacsi and Armin Beer describe their experience in using model-based testing (MBT) for the European Space Agency (ESA). Their team developed a test automation framework that took significant effort to set up but eventually was able to generate automated tests very quickly when the application changed. This chapter includes an excellent return-on-investment calculation applicable to other automation efforts (not just MBT). The team estimated break-even at four iterations/releases. The need for levels of abstraction in the testware architecture is well described. The application being tested was ESA’s Multi-Mission User Information Services. The multinational team met the challenges of automation in a large, complex system with strict quality requirements (including maintainability and traceability) in a waterfall development—yes, it can work! If you are thinking of using MBT, you will find much useful advice in this chapter. A mixture of inhouse, commercial, and open source tools were used by the team.
Chapter 10, Ten Years On and Still Going
Simon Mills updates his case study from our previous book, Software Test Automation (Addison-Wesley, 1999). Still automating 10 years on is a significant achievement! The original story is included in full and contains excellent lessons and ideas. The success and continued growth of this automation is a testament to the sound foundation on which it was built more than a decade ago. The case study describes many lessons learned the hard way and some amusing observations on Simon and his team’s first automation attempts. Their automation architecture separated their tests from the specific tools they were using—a wise move as was proved later. They devised a reliable way to document their tests that has stood the test of time. This story takes place in the United Kingdom, uses commercial tools, and covers about 15 years.
Chapter 11, A Rising Phoenix from The Ashes
Jason Weden tells a story of initial failure leading to later success. The failure of the first attempt at automation was not due to technical issues—the approach was very sound. However, it was a grassroots effort and was too dependent on its originator. When he left, the automation fell into disuse. But the phoenix did rise from the ashes, thanks to Jason and others who had the wisdom to build on what had gone before, making many improvements to ensure that it was more widely used by business users as well as technical people. Their “test selector” for choosing which tests to execute gave the test engineers flexibility, and they ensured their legitimacy by keeping stakeholders informed about bugs found by automated tests. The small team that implemented automated testing for home networking equipment is based in the United States.
Chapter 12, Automating the Wheels of Bureaucracy
Damon Yerg (a pseudonym) tells of experiences in automating large systems for a government agency, over more than 10 years, with hundreds of developers and testers and more than a dozen automators. After some uncertain starts, external pressure brought the right support to move the automation in the right way. The tests to be automated covered diverse applications from Web-based to mainframes, all with environmental factors. This story brings home the need for automation standards when many people are using the automation. Damon and his colleagues organized the regression library into core and targeted tests to enable them to be selective about which tests to run, and they related the automated tests to risk factors. The basic design of the automation supported business expert testers and offered technical support as needed. One of the most powerful things they did to ensure continuing management support was to develop a spreadsheet describing the benefits in a way that communicated effectively to stakeholders. This is a very successful large-scale automation project from Australia.
Chapter 13, Automated Reliability Testing Using Hardware Interfaces
Bryan Bakker tells of automating testing for medical devices, an area with stringent quality requirements and difficulties in accessing the software embedded in devices such as X-ray machines. Bakker and his team’s starting point was simple tests that assessed reliability; functional testing came later. The automation was developed in increments of increasing functionality. The chapter contains many interesting observations about management’s changing views toward the automation (e.g., management was surprised when the testers found a lot of new bugs, even though “finding bugs” is what management expected of them). The team developed a good abstraction layer, interfacing through the hardware, and were even able to detect hardware issues such as the machines overheating. The results in the test logs were analyzed with home-built tools. The reliability testing paid for itself with the first bug it prevented from being released—and, in all, 10 bugs were discovered. Subsequent functional testing was smoother, resulting in cutting the number of test cycles from 15 to 5. This story is from the Netherlands, and the project had excellent success using commercial and home-grown tools with just two people as the test automators.
Chapter 14, Model-Based GUI Testing of Android Applications
Antti Jääskeläinen, Tommi Takala and Mika Katara tell how they used model-based testing (MBT) in a substantial pilot study, testing smartphone applications—specifically Messaging, Contacts, and Calendar—on the Android platform. They used domain-specific tools rather than generic testing tools and a mix of commercial, open source, and inhouse tools, including two MBT tools. Testing in a huge state space was quite a challenge, but they give clear examples of the types of tests they were automating and good explanations of the algorithms used. They include many helpful ideas for making testing easier, such as using correct syntax for keywords and converting testware to more usable formats. This story covers a pilot project in Finland.
Chapter 15, Test Automation of SAP Business Processes
Christoph Mecke, Melanie Reinwarth, and Armin Gienger tell how automation is used in testing major application areas of SAP, specifically banking and health care. Because SAP applications are deployed in many client sites and have a long life, the test automation is on a large scale, with over 3 million automation lines of code and 2,400 users of the automation. The testware architecture of the tool they developed is very modular. The standards and guidelines put in place ensure the automation can be used in many areas and in many countries, and the tool can be used by SAP customers as well. The automated tests must be ready to go as soon as new software is delivered to enable the testing to help speed delivery rather than slow it down. Some of the problems they encountered concerned testing parallel versions and multiple releases, consistency of test data environments, setting of customized parameters, and country-specific legal issues. One particularly good idea the authors relate is to have a tool do static analysis on the test automation scripts. They also warn about ending up with “zombie scripts”: dead automation code in a script. This story takes place in Germany and India over several years.
Chapter 16, Test Automation of a SAP Implementation
Björn Boisschot tells how he developed a generic framework based on his observations while setting up automation for various clients. He explains how he used the framework and extended it in the automation of the testing for two SAP applications in the energy sector. The groundwork for the project is well laid, with a proof-of-concept forming the basis of the go/no-go decision. Björn gives good examples of communicating with managers, explains why capture/playback does not work, and emphasizes the importance of setting realistic expectations. The framework, now a commercial product, used good automation and software development principles to construct a modular structure that successfully scaled up. The layers of abstraction are well implemented, separating the automated tests from the tool and giving users access to tests without having to program. He ends by showing how multilingual tests were implemented through a translation table and some of the challenges of that project. This case study takes place in Belgium and is the origin of a commercial framework.
Chapter 17, Choosing the Wrong Tool
Michael Williamson tells the story of trying to use a tool (which he inherited) that he later realized was not suitable for what he was trying to test. He was trying to automate the testing of major functionality in Web development tools at Google, where he was new to testing and test automation. Some assumptions that seemed obvious at first turned out to be not as they seemed. His approach to taking over the use of an existing tool seemed sensible, yet he encountered a lot of problems (which are illustrated), particularly in automated comparison. Michael found it was difficult to abandon something you have put a lot of effort into already, yet in the end, this was the best approach (and he was surprised when he discovered what had really been hindering his efforts). This story is of one person in the United States attempting to use a commercial tool.
Chapter 18, Automated Tests for Marketplace Systems: Ten Years and Three Frameworks
Lars Wahlberg gives an insight into his 10 years of test automation for marketplace systems, including the development of three automation frameworks. One of the key messages in this chapter is the importance of having the right abstraction level between the technical aspects of the automated tests and the testers. Lars describes how they addressed this issue in each of the different frameworks and some of the problems that occurred if the abstraction layer was too thick or too thin. As Lars and his team moved into agile development, they found the process worked best when they had people who could be both tester and test automator, but the role of the test automation architect was critical for smooth implementation of automation. The chapter illustrates the progression of a test from manual to automated and the automated checking of preconditions. Lars also includes an illuminating assessment of return on investment for automation based on how often tests are run, the costs of automating, and the number of tests that are automated. This work was done in Sweden using open source tools.
Chapter 19, There’s More to Automation than Regression Testing: Thinking Outside the Box
Jonathan Kohl takes us through a set of short stories, each illustrating a variation on the theme of automating things other than what people usually think of, that is, thinking outside the box. The stories together demonstrate the value of applying ingenuity and creativity to solve problems by automating simple tasks, or processes other than test execution. Full-scale automation is not necessarily a practical option in some cases, but small and simple tool solutions can provide significant benefits and savings. Jonathan shows that even devising “disposable” automation scripts can be very useful. These experiences from Canada cover small to large applications, from Web-based to embedded systems, using commercial, open source, and inhouse tools.
Chapter 20, Software for Medical Devices and Our Need for Good Software Test Automation
Even if you are not working with medical devices, this chapter, written by Albert Farré Benet, Christian Ekiza Lujua, Helena Soldevila Grau, Manel Moreno Jáimez, Fernando Monferrer Pérez, and Celestina Bianco, has many interesting lessons for anyone in automation. For the safety-related applications they test, a strict, formal test process is used with a risk-based approach to the implementation of the automation. Their story shows how success can be mixed even across different projects in the same organization; attempts to resurrect previous automated tests that had fallen into disuse resulted in different outcomes in different projects. In some cases, unrealistic imposed objectives (“total automation”) virtually guaranteed failure. However, they did progress, devising a good list of what they wanted to use automation for and what they wanted to avoid automating. The problems encountered included some that are unique to regulated industries (where test tools need to be formally validated before their results can be used in the qualification testing of the software), problems with hardware interaction, and problems with custom controls (overcome by an abstraction layer). Their honest description of problems, solutions, and lessons learned in the four projects described is as useful as it is interesting. These stories from Spain involve commercial and inhouse-developed tools and cover projects lasting a few months to 5 years.
Chapter 21, Automation through the “Back Door” (by Supporting Manual Testing)
Seretta Gamba tells of the difficulties she and her team experienced in trying to progress automation, even though it had a good technical foundation. She shows how they addressed the real needs of the testers and provided support where testers needed it most, which was in manual testing. The really interesting twist is how they were able at the same time to harvest things that would progress the automation – win-win situations. Their approach is described in detail, showing how they implemented levels of abstraction, separating tests from application detail such as the graphical user interface, and they include an example of a user interface to the tests. They developed what they call command-driven testing, which is based on a form of keyword-driven testing and worked well for them. This case study takes place in the insurance industry in Germany.
Chapter 22, Test Automation as an Approach to Adding Value to Portability Testing
Wim Demey describes how he developed an inhouse tool to work with commercial and open source tools to enable parallel testing of different configurations. Testing the installation of packages that are developed with a common core but customized for different customers is important because failures in installation may have serious impacts not just on the system but on market perception as well. But testing the ports of the different packages to a large variety of configurations (operating systems, browsers, databases) can be a time-consuming and resource-hungry process, so it is a good candidate for automation. Wim used virtualization to run portability tests on many configurations at once through a custom-built tool that launches the portability tests in the virtual environments. The chapter offers many good lessons, including the importance of automating the activities surrounding test execution, for those considering a similar task. This story comes from Belgium and involves financial client software.
Chapter 23, Automated Testing in an Insurance Company: Feeling Our Way
Ursula Friede describes the experience of “feeling their way” to better automation. She and her team did not plan the steps they ended up taking, but it would have been a good plan if they had. They began by just experimenting with a tool but soon realized its limitations, so they decided to focus on the most pressing problems by changing their automation. In each of four phases, they successfully addressed an existing problem but then encountered new problems. They calculated an impressive saving per release after they had implemented their improvements. Ursula was based in Germany at the time and was automating tests in the insurance sector.
Chapter 24, Adventures with Test Monkeys
John Fodeh tells of his experiences with automated random test execution, also known as “monkey testing,” for medical devices used for diagnosis and in therapeutic procedures. John describes the limitations of automated regression testing and why few bugs are found that way. Using test monkeys enabled him to find many bugs in the devices that otherwise would not have been found before release. Inputs are selected from a list at random, and all crashes are investigated. Test monkeys can be implemented with commercial execution tools or inhouse tools. This approach borders on model-based testing and also is close to exploratory test automation. John’s team was able to calculate the reliability of the devices over time, which helped in the release decision. Many other interesting metrics are described in this chapter as well. An honest assessment of the limitations of the technique makes this a well-balanced description of an interesting use of tool support in testing. This story takes place in Denmark.
Chapter 25, System-of-Systems Test Automation at NATS
Mike Baxter, Nick Flynn, Christopher Wills, and Michael Smith describe the automation of testing for NATS (formerly called National Air Traffic Services Ltd.). Among its other responsibilities, NATS provides air traffic control services for the airspace over the eastern North Atlantic Ocean. Testing a safety-critical system where lives are at stake requires a careful approach, and the requirements included special technical factors, human factors, and commercial considerations. The tool used was able to test the software without affecting it, and it was also useful in some unexpected ways, such as in training. The authors provide a helpful checklist for deciding which tests to automate and describe lessons learned, both general and technical. This case study is from the United Kingdom and involves commercial, open source, and inhouse tools.
Chapter 26, Automating Automotive Electronics Testing
Ross Timmerman and Joseph Stewart tell about the inhouse testing tools they developed over the years to test automotive electronics systems at Johnson Controls. In 2000, no commercial tools were available for this niche area, so the Johnson Controls team chose to develop their own tools, which was done initially by the test team and later by an offshore team. Both benefits and problems arise from writing your own tools, but the team dramatically increased the number of tests they were able to run. They also discovered ways to automate things they initially thought had to be done manually. In their second initiative, they built on the success of the earlier tools but provided more functionality, using off-the-shelf hardware modules with their own software to get the best results. This case study covers 5 years and is based in the United States.
Chapter 27, BHAGs, Change, and Test Transformation
Ed Allen and Brian Newman tell of their experience in automating the testing for a customer relationship management system. The problems they encountered ranged from technical issues such as environments, testware architecture/framework, and “script creep” to people issues, including management support, working with developers, and when to ignore a consultant. After several false starts, they were challenged by some “Big Hairy Audacious Goals” (BHAGs) and were given support to meet them. They achieved good benefits in the end and provide some intriguing metrics about the number of defects found by different ways of testing (automated, manual scripted, exploratory testing, and bug fixes). This story is based in the United States with a team of 28 including 25 from QA and 3 from development.
Chapter 28, Exploratory Test Automation: An Example Ahead of Its Time
Harry Robinson and Ann Gustafson Robinson describe Harry’s experiences in doing what seems on the surface like an oxymoron. How can testing be both exploratory and automated? There are a number of requirements to make it possible, but when it can be done, it provides a way to explore test input spaces that are far too large to be tested in any other way. This chapter takes us step by step through Harry’s journey of implementing what has now become known as model-based testing. The effort was far more successful than anyone thought possible at the time, which had an unexpected side effect on Harry’s career! Although this story takes place rather a long time ago, the techniques described are now “coming into their own” because better tool support makes them applicable to more systems in a wider context. (Note: although this chapter has two authors, it is told from Harry’s viewpoint for easier reading.) This story comes from the United States.