The Journey to True Test Automation is Shorter Than You Think

By JB Brockman | 3/13/18

Everything about software has changed—how it’s architected, developed and produced, what it does, what users want from it, and how often they expect new features. To keep up, organisations are turning to continuous delivery and DevOps. Yet product teams still do a lot of manual testing, which consumes a lot of time they don’t have, thanks to shrinking test windows. Incorporating automation into your testing approach is a great strategy, but figuring out where and how to start isn’t necessarily quick and easy.

Clearly, the testing space is ripe for modernization, but where is it headed? We asked two experts—Antony Edwards, CTO of Testplant, and Paul Bell, specialist test consultant at NCC Group—to weigh in.

What does the future of testing look like?

Antony Edwards (AE): The future of testing involves more than just technology and products. We need to embrace a new testing approach that involves five key elements:

  • Test through the eyes of the user. To improve the user experience (UX), we need to engage with devices the way users do, rather than through the code.
  • Continuously test all aspects of the digital UX. Functionality, performance, and usability are all critical product attributes associated with the UX. Testing those elements through the eyes of the user creates a simpler, more intuitive way to test.
  • Expand automation beyond test execution using AI, machine learning, and analytics. If you look at the entire testing process, we’ve really only automated one (yet time-consuming) aspect—test execution. If we want teams to be more productive and keep pace with DevOps, we need to automate other key aspects, like creating test scripts, setting up the test environment, and analyzing test results.
  • Report quality in terms of the user with predictive analytics. Modern testing has to be about technology metrics and business metrics. Rather than reporting quality status in terms of metrics pass rate and defect density, we need to reveal quality specifics in user terms. For example, releasing now will result in a 4 out of 5 app rating and consumer conversion will increase by 15 percent.
  • Take a coherent approach to monitoring and testing. Monitoring UX in production is especially important. By emulating real network conditions, you can get a true, end-to-end view of performance in production. One that helps you determine that while your traditional monitoring approach says that your app’s response time is two seconds, your users say it’s eight seconds.

Paul Bell (PB): What we’re seeing here at NCC Group, across both our existing customer base and new clients, is that the role of the tester is changing and I’d expect this to become increasingly evident over the next few years. Demand is growing for our testers to demonstrate a level of technical competence that we’ve never seen before. There is an expectation that our people have a knowledge of test automation as standard, that they understand the benefits that it can bring, and are capable of applying it in some way for our clients.

Agile has dominated our industry now for a number of years and with the rise of DevOps, we feel that the role of a tester will continue to evolve. The lines between development, test, operations, and other roles across the lifecycle are blurring even further, and as testers, it’s our job to ensure that quality is central to the products that we deliver. Delivery timescales and test windows in particular will continue to be squeezed, making automation an essential enabler for our teams. As a result, we would expect to see more testing being done within production (shift right, anyone?), and an increased dependency on monitoring tooling—from both a functional and non-functional perspective.

What technologies are driving testing’s evolution and why are they important?

PB: The continued rise of mobile and digital technology is increasing the pressure to very quickly deliver high-quality solutions to our customers. The demand for a rich UX means that the most fundamental issues are magnified and there is an expectation that fixes to these issues are delivered fast and seamlessly. In this scenario, automated testing is simply imperative if we’re to get the assurance that we need across the ever-expanding range of devices, operating systems, and technology stacks. The increased adoption of automation will help delivery teams keep pace with the market demands whilst maintaining the quality of the end product. It enables test coverage to be increased, meaning an improved digital experience for our end users.

The landscape surrounding test automation is hugely complex, with a vast array of tools and technology that can at times be a little overwhelming for our clients. What we are seeing of late, however, is the ability to integrate quite easily between some of the big players in the market. For instance, Testplant’s Eggplant Functional solution now integrates with Selenium WebDriver, whilst the majority of open-source and commercial tools provide the ability to integrate with key continuous integration and DevOps tools such as Jenkins and Git. Without a strategic approach to tool selection, it’s not uncommon to find that costly licenses are purchased but quickly become shelfware as the application landscape changes. Improved integration across automated solutions will help overcome this challenge by offering greater flexibility across the tools, the ability to use open source where relevant, and embed automated tests into the delivery pipeline.

And then, of course, we have AI and machine learning, which looks set to underpin the new wave of automated test solutions. AI will support our test processes in a whole new way, taking test automation to the next level and making it even more accessible to stakeholders from all areas of the delivery team. These game-changing technologies will provide significant time savings, not only with test execution but earlier activities such test design and preparation. It will allow us to make our automation more dynamic, taking what is currently an automated checking function more toward intelligent testing and defect hunting.

AE: Analytics, AI, machine learning, image recognition, and natural language are disrupting testing as we know it. With AI and machine learning, we can auto-generate test cases and optimize test execution to find bugs faster, achieve broad coverage of the UX, and be confident about coverage when you launch a release. Natural language processing, a component of AI, is huge for testing voice-recognition software, which is constantly advancing. With analytics, we can automate results analysis and root-cause analysis to improve productivity, accelerate time to market, and keep up with the pace of DevOps and continuous delivery. Image recognition is ideal to test what users are actually seeing when they interact with your app, and because you’re not testing at the code level, non-technical people can be effective testers. All of these technologies are key to boosting team efficiency and productivity, and helping organisations improve the UX as well as business-driven objectives.

What are the top three actions organisations can take to modernise their testing strategies?


  1. Use AI and analytics to automate more of the testing process.
  2. Test from the user’s perspective, not the code’s perspective.
  3. Use predictive analytics to understand the impact of a change or new release on UX and business objectives.

PB: What is so important when considering any change to your testing strategy is you do so in a controlled manner, without putting your current business processes at risk.

  1. Be aware of the digital platforms that your customers are using and, where sensible to do so, implement test automation alongside device labs or cloud infrastructure services to ensure that the big hitters are included in your testing coverage.
  2. Review your current test processes to understand where efficiencies could be gained through the use of automation. Similarly, if you have already started on your automation journey, take the time to stop and consider where you are and the direction that you’re heading. Don’t be afraid to adapt.
  3. Consider how AI solutions can be incorporated into your testing process. Look at the tools that are out there on the market at the moment, try them out, and see what they offer.

If a journey toward test automation is in your future, we invite you to join us for a free, interactive, in-person seminar with NCC Group and Testplant on 20 March, 2018. Hear from our experts and talk to them in more detail about how test automation can impact your business. Plus, learn how one test team used Eggplant AI to achieve automation a lot faster than you’d think.

Register now for “Smashing the Digital Automation Myths.”

Topics: DevOps, Digital Experience, Real User Experience, Software testing tools, test execution, User experience testing, artificial intelligence, user journey, analytics, continuous delivery, test automation strategy

JB Brockman

Written by JB Brockman

JB writes stuff as the content marketing lead at Testplant.

Stay up-to-date with the latest in test automation

Lists by Topic

see all