I recently presented at the Northern Lights conference in Manchester. This conference was hosted by the BCS (British Computing Society), and my talk was “Tools. Techniques. Trouble? Why test automation is getting more difficult, and what can be done about it.” You may have seen my blog post from ahead of the show, but if you haven’t yet, you can find it here.
I focussed on automating user interactions with the System under Test (SUT) and automating the creation of test scripts but not the automation of the testing process itself. I addressed both functional and load testing.
For those who weren’t there, here are the key points that I covered in my presentation:
1. There are conflicting pressures on testing. More frequent system releases increase the need for testing but technical changes (see 4) make test automation more difficult. The costs of system failure and poor performance are rising alongside downward pressure on testing budgets.
2. Defining the scope of testing and the boundaries of the System under Test is becoming more difficult to achieve and ever more important. Getting explicit agreement from stakeholders is essential to avoid recriminations and to align testing with real business objectives.
3. Multiple suppliers impact the testing effort. Suppliers should be contracted to support the testing of systems that they contribute to by making components “testable” and providing technical support.
4. Technology change is making test automation more difficult. These changes include diverse client software running on multiple platforms, “mashup” design and changes to Web protocols.
5. Moving the automation target up the technology stack can address these issues - in particular if image based GUI testing is employed.
6. Scalability challenges means that for load testing protocol level automation is still essential. Techniques to help scripting at the protocol level are also evolving. Mixing levels of automation is often the best approach for load testing.
The slides are available on SlideShare, I hope that you find them interesting!
The day itself was very enjoyable and was attended by an impressive and engaged group of professionals who mainly work in the North West of England. You will be able to access the slides for some of the presentations here.
Stuart Reid provide a comprehensive look at how the economic forces of Digital Disruption are impacting testing and provoked a lively discussion on several points including crowd testing and whether organisations should create testing centres of excellence. I’ve enjoyed hearing Chris Ambler speak before and wasn’t disappointed this time – he is always entertaining and thought provoking.
Also entertaining was Stuart Taylor from LateRooms who provided a “battle hardened” professional’s take on DevOps. He gave some good advice on what does work (comprehensive monitoring with alerts that link development and operations for rapid responses) and what doesn’t (Chaos Monkey).
Finally, Duncan Nisbet introduced the CyneFin framework and its application to decision making. See Duncan’s slides and this Wikipedia entry. This session created a lot of interest and discussion despite being the last session. I always start with a healthy degree of skepticism for presentations that deal with overarching methodological or philosophical themes at industry events. In this case though I’m grateful that Duncan introduced me to this topic and I mean to follow it up.
Hopefully this will prove to be the first of many similar events in the North of England that will complement those held in London.