Software Automation Test Tools and Trends

“If you want to teach people a new way of thinking, don’t bother trying to teach them. Instead, give them a tool, the use of which will lead to new ways of thinking.” — Richard Buckminster Fuller, 20th Century Inventor and Visionary

The 2016 World Quality Report from global technology consulting firms Capgemini and Sogeti, in partnership with HP Enterprise, is a large industry study of QA and Testing trends. The report should be of interest to all hiring managers, budget stakeholders, and testers involved with software development projects (“testers” is used here to incorporate the various professional titles in software development and in which testing software is the primary function). Current trends from the report indicate that organizations value test automation to help transform how they rapidly innovate and disrupt their markets. It also mentions that testing is essential for its primary function — preventing defects — but is valued for the impact on the customer experience these products deliver as well.

Challenges are also trending in the data. Costs related to automation testing environments continue to grow, and there is an emphasis on finding more efficiency in testing by adopting a “QA approach” and enabling the “shift left” movement of test automation functions earlier in the development lifecycle. Test automation is a fundamental part of the shift-left software development practices of DevOps, Agile, and the related methods of delivering software faster. The report mentions a key recommendation for investing in self-learning platforms and “as-a-service” solutions in all areas of application testing. Demand will continue for testers with the technical proficiency to capitalize on the recommendations that promote more test automation and the use of automation software tools. Experts from the test community point out that automation tools and technology do not replace all manual testing activity or the critical thinking of qualified test professionals. Learn more about automated software testing.

There are two schools of thought on how automation will impact job creation in the “High Tech” category of the report that includes software development. On the one hand, the data is clear: certain industries have and will continue to lose jobs to robots and advanced software that can perform tasks better than a person can manually.

However, another set of data that shows that the advent of automation may increase job growth for computer-driven occupations because of the growing demand for these products and services. In the academic paper How Computer Automation Affects Occupations: technology, jobs, and skills, author James Bessen supports this theory. Bessen is an economist and a lecturer at the Boston University School of Law. He provides occupational data and explores how computer automation led to job growth since 1980 in fields that use advancing computer technology.

At Google, the roles that test engineers and test management play changed when work shifted from Google employees producing labor to develop and test web applications, to the public community of Google users and “dogfooders.” The term dogfood is used by software companies to describe software that is not yet released, so dogfooders are users who test product (either from other product teams or early adopters).

In the book, How Google Tests Software, the authors describe a future of software testing in the hands of the user. Google’s “fail fast” mentality for rapid software releases depends on user feedback and would prefer not to rely on testers (“test engineers” at Google). According to the authors, “As doomed as we believe the role is, the work itself cannot go away.” Google reorganized their testers with new titles and emphasized the role developers and project managers should have on test development. In other words, test automation technology did not reduce the need for testers, but a change in philosophy and modern methods of rapid software releases shifted the work to others.

The authors recommend testers morph their expertise into more of a test design role. Developers and PMs will engineer quality software with minimal defects, and release quickly to fail fast. Google users, dogfooders, and early adopters will use the software and submit feedback. Testers will analyze and collect data by creating or using existing tools and “assess coverage, calculate risk impact, ensure it trends down, and adjust the testing activities accordingly.”

Under this recommendation, creating and executing tests would no longer be the tester’s job at Google. Instead, identifying instances where security, privacy performance, and exploratory testing needs to be directed to the capable specialist would be the more valued service. Testers would develop specialist skills for roles like those involving security, the top concern of the leaders in technology polled by the 2016 World Quality Report.

The basic concepts of test automation frameworks and automation tools for web application development are terms that are often interchanged, as many of the automation tools available offer a complete suite of features that serve as the testing framework. Also, hybrid frameworks combine various open-source automation tools to provide a framework composed of best-of-breed software. For a complete look at automation frameworks, including analysis and recommendations for utilizing the modern types of test automation frameworks, read A Guide to Automation Frameworks.

Test automation frameworks provide the foundation to support automation software tools used for a variety of software tests including:

• Unit testing
• Functional requirement testing
• Performance testing
• GUI (graphical user interface) testing
• Regression testing
• Integration testing
• System load testing
• Security testing

The development of test automation frameworks demands skills in critical thinking, design, project implementation, and training. It is evident that there are benefits to understanding how to design, develop, implement, and maintain a framework.

A test automation framework is a defined set of guidelines for organizing and developing the testing of a project. The tester uses test scripts they develop, or automation tools and their test interfaces, to facilitate the software automation testing of a web application. An interface is a connection between two components. Interface testing verifies the communication between systems (API's or web servers). The test automation interface provides a platform for the test automation tools to work within the automation framework. For example, automating tests to check communication between a server and the application, or the application and a database.

Test automation frameworks may include support programs, test objects, test libraries, a test scripting language, and other tools to aid in testing different components of the application under test (AUT). Objects are features (units of code combining data and methods), and an object repository is a collection of the features that gets recorded as the test automation tool interacts with an AUT. Using an object repository allows an automation framework to adapt in order to create complete applications. This system makes it easier to reuse the framework for a different AUT.

Automation frameworks and tools are different. A framework does not perform a particular task like an automation tool. Rather, the framework is the infrastructure serving the automation tool and empowering the tester's automation goals. A test framework has different features or sets of test classes and interfaces, and supports code reuse by controlling how objects interact. This reduces the amount of new test code required for developing new web applications.

The design of a test automation framework helps produce high-quality software for the AUT. Well-designed, flexible automation frameworks and the use of appropriate automation tools empowers testers to spend more time meeting the goals for test automation on the AUT. The goal for using test automation frameworks is to facilitate the test automation of a single AUT. However, the goal should be to develop a test automation framework that is reusable and easy to maintain under modern software development methods associated with Agile development. Methods such as Continuous Integration and Deployment and Behavior-Driven Development (BDD) rely on test automation to achieve the rapid software schedules of today.

In How Google Tests Software, the authors discuss the need for a test infrastructure overhaul because of the “expensive people and machine infrastructure.” This overhaul would include moving test repositories, test code editors, recorders, and execution into the cloud. At Google, test automation applies to web AUTs and non-web AUTs such as Android or iOS applications.

The need for fewer in-house, custom test frameworks and dedicated test execution machines is important to Google. Testers leverage and contribute to open-source projects, cobble them together, and execute them on shared cloud computing resources. Selenium WebDriver is the automation tool of choice for the authors. They highlight Selenium’s community-maintained and corporate-sponsored infrastructure development as the reason more open-source test frameworks with open-source tracking systems and controls will thrive.

The book explains that, at Google, “Sharing test data, test cases, and test infrastructure will be worth giving up imagined advantages of keeping everything secret or proprietary. Secret and proprietary testing infrastructure just means it is expensive, slow-moving, and often not reusable even within the company across projects.”

Google envisions a future where test code is shared along with as much “bug data” as possible. This form of “crowd-based testing and test creation” outweighs the “imagined benefits of keeping everything hidden.” The cloud-based approach to testing saves money and provides the appropriate test automation infrastructure to focus on testing that leads to “higher quality products and faster release cycles.”

Calculating the cost of purchasing an automation tool by adding the cost of commercial licensing to the labor cost of human capital required to implement and maintain the automated tests seems like a straightforward equation. But how do you consider the long-term return on investment for improvements that result from reusable test code or the potential increased costs that arise from test automation missteps?

Douglas Hoffman is a management consultant for software QA and author of Cost Benefits Analysis of Test Automation. In his analysis, Hoffman emphasizes setting realistic expectations on where you’ll see benefits from automating software tests. He considers the ROI for test automation investments in general terms of fixed and variable costs. You can calculate the financial impact of automation by comparing it with the alternatives of “manually testing or not testing at all and accepting the risk.”

Hoffman publishes equations that take into account the factors associated with automating tests. He concludes that while “test automation is not always necessary, appropriate, or cost effective,” estimating the financial risk of test automation is possible if you identify the relevant factors.

Careful consideration of which automation framework to use and the proper use of automation tools for your goals will determine cost effectiveness. Costs can vary depending on the tool and functionality used. The cost of test automation includes the development and maintenance of the automation framework, the tools used, and the people responsible for automating tests. Selecting the appropriate framework and automation techniques can help maintain lower costs associated with automation testing. The framework provides a structure for automation testing that keeps test resources accessible. Determining the ROI requires analysis of the desired outcomes of automated tests, the appropriate framework, and tools to use based on these goals, and who will develop and maintain the technology.

Test Automation Motivation

In 2016, LinkedIn added a new learning feature using Lynda.com that aims to help people keep up with automation and the ever-changing technology skills introduced into their work environments. Microsoft’s $26 billion purchase of LinkedIn was an investment driven by their enterprise productivity vision. An educated, connected, and creative workforce is a productive and profitable workforce. Productivity and the need to keep up with a market demanding rapid innovation and high caliber customer experiences motivated the rise of automation technology in software development.

Testers working in software development should familiarize themselves with the most common and widely used test automation tools available to protect against the threat of automation supplanting them from the workforce. To benefit from this demand, testers should focus on skill development that allows them to wield the tools for automation.

With regards to the impacts of automation on testers and software testing jobs, automation is changing the landscape for demand in technical skills. Test automation is a fundamental aspect of Agile development methods, along with the rise of Continuous Integration, Continuous Deployment, and Behavior-Driven Development (BDD). Test automation will continue to aid rapid software deployment goals and the demand for more, high-quality features that are secure and reliable.

John Kent specializes in building test automation frameworks using tools like Selenium, HP UFT, Fitnesse, Cucumber, and SpecFlow. His Test Liberation framework is a keyword-driven test automation platform designed to eliminate the programming of automated test code (also known as “scriptless automation”). Kent’s framework reduces the time required for implementation, the cost of test automation, and the risk of failure associated with test automation projects.

Kent has more than 20 years of experience building test automation frameworks, and he believes that the only way to achieve the amount of regression testing required for each sprint in Agile development is through test automation. He writes:  

“I feel that the only way we can get enough test coverage and solve these issues is to use test automation. It will never be a replacement for testers, but it can provide huge test coverage over masses of hardware platforms, browsers and combinations of these.”

Kent’s recommendations for successful test automation projects echoes the core elements of test automation frameworks required for Agile development. The appropriate framework provides guidelines for producing reusable, easily maintained automated tests. He also believes that testers should be able to use tools without complex programming requirements that require specialists.

Technical testers capable of designing, building, and maintaining their hybrid test automation frameworks and writing test automation scripts are in high demand. But achieving this level of technical proficiency and mastering a programming language isn’t the only way to stay ahead of automation trends. Testers who familiarize themselves with modern software automation tools are still valuable to organizations implementing test automation.

Keep in mind, some familiarity with programming skills is recommended to take full advantage of these tools. For the web application example below, you’ll need to have some knowledge of Java. LinkedIn Learning has online skill development courses in Java and other programming languages here.

This list features seven open source and commercial test automation tools. These licensed automation tools have free trial versions available.

 

Now that you’ve learned about the various tools used for automation testing, it’s essential that you pick the right tool to get started tracking and managing the results of your tests. One such tool is Smartsheet, a collaborative work management and automation platform that helps enterprises and teams work better.

Use Smartsheet to track the schedule and results of planned, current, and completed tests. Share the schedule with your team and collaborate on the details in real time, in one central location. Whether you’re running manual or automated tests, Smartsheet’s broad range of views – Calendar, Gantt, Card, and traditional Grid – allow you to manage progress the way you want. Organize test results with hierarchy and use comments to keep work in context.

Plus, with Smartsheet Sights you can create custom dashboards to monitor testing and provide a high-level view of progress and key metrics to management. Improve collaboration on the testing process, increase control of timing and resources, and gain unprecedented visibility into results with one location for the truth.

Discover how easy it can be to track and manage automation testing using a template in Smartsheet.

Empower your people to go above and beyond with a flexible platform designed to match the needs of your team — and adapt as those needs change. 

The Smartsheet platform makes it easy to plan, capture, manage, and report on work from anywhere, helping your team be more effective and get more done. Report on key metrics and get real-time visibility into work as it happens with roll-up reports, dashboards, and automated workflows built to keep your team connected and informed. 

When teams have clarity into the work getting done, there’s no telling how much more they can accomplish in the same amount of time. Try Smartsheet for free, today.