Florian Gumhold
ERNI Switzerland

 

Introduction

IT software projects are mainly conducted using an agile and iterative methodology like Scrum, SAFe or Kanban. Such projects are usually set up in CI/CD environments, which support automated merging, building and deploying of software. Software is produced incrementally and frequently to receive fast feedback and to allow corrective actions to be taken if necessary. Frequent software increments also require reliable and fast quality assurance to inspect quality and stability. The implementation of Test Automation is intended to fulfil the task of frequent test execution of e.g., regression tests and smoke tests, without manual intervention. Test Automation can be applied at different levels, such as component, system and integration testing, and it can be integrated into the existing CI/CD environment to automate the test execution of a defined test set as soon as a new build is being deployed in the test environment (Linz, 2016, p. 11-20; p. 150).

 

What is Test Automation, and why is it important?

The ISTQB provides a concise and accurate definition of Test Automation: “The use of software to perform or support test activities, e.g., test management, test design, test execution and results checking.” Most likely, the segment of automated test execution is known to many people working on agile projects. The intention of Test Automation in the subarea of test execution is to delegate repeatable and monotonous tests to a machine instead of repeatedly executing tests manually. Three main goals are pursued by using a Test Automation approach:

  • Quality, speed and efficiency in test execution should be improved. Manual test execution is not scalable. If the tested software scope exceeds the test team capacity, quality decreases and test coverage cannot keep pace with the implemented requirements.
  • Test Automation reduces costs. Executing many regression tests repeatedly is time-consuming and costly. Investing resources in Test Automation will save money over the project runtime and beyond.
  • Test Automation enables faster feedback on the quality of a software increment. Merging, building and deploying software is already automated in most projects. Therefore, it makes sense to include automated tests in the CI/CD environment. This allows continuous monitoring of the quality of the software and taking necessary actions if required (Hristov, 2019).

Having explained the purpose and advantages of Test Automation, it is also necessary to mention its limitations and that it cannot be the solution for every problem. Test Automation is only an execution of pre-defined steps which have been implemented by a human. A Test Automation solution only executes these steps without doing any other actions or validations. Any act beyond the implemented test script will not be performed. In other words, this could end up in a situation where defects will not be detected by the automated tests but would have been detected by a manual tester. This leads to the fact that Test Automation will only partially replace manual testing. It can be an extension and supporting task to manual testing. However, it cannot replace a manual tester’s creative skills and experience, especially when it comes to exploratory or negative tests, which are based on the experience and inventiveness of a skilled tester (ISTQB, 2017).

 

Tools, Technologies and know-how

Commonly, project teams need fast, reliable and comprehensive feedback on the quality of a software increment. Therefore, many vendors of Test Automation tools have taken the chance to provide a solution for this issue. Tools can be divided into open-source and commercial solutions, many of which are designed as an all-in-one solution. This means that one tool can cover many different testing levels and types of applications. Such a tool covers testing of mobile, web or desktop applications and requires no additional testing framework to implement end-to-end tests. Hence, tests can be implemented by non-technical people by using a record-and-play functionality to click and validate actions without writing a single line of code. Commercial vendors primarily provide this type of Test Automation. There is also the possibility of implementing test cases by using programming languages. This requires technical know-how and is mainly used with open-source tools. Which of the two approaches and means to choose primarily depends on the project environment, financial aspects, programming know-how and the existing toolchain (Gillis, 2019).

 

Setup of Test Automation in agile projects

A Test Automation solution evaluation process can be divided into different steps. The first step is defining the criteria catalogue and researching possible market tools that fulfil the criteria. This catalogue should not only be limited to functional requirements but also include TCO (total cost of ownership) and integration into the existing tool chain to guarantee smooth communication between the tools used in the project. After this step, the next one is to perform a proof of concept with 3 to 5 test cases to determine if the tool candidates can cover all the specified requirements. It is recommended to use the most critical and extensive test cases for the proof of concept to ensure that these high-priority tests can be automated. Test Automation for a project always requires assessing the skills and know-how available within the project team. Having the know-how, defining a strategy and providing enough resources in the evaluation phase is crucial. It determines if Test Automation can be established in the project or if it is considered overhead without adding benefit to the quality of the product (Hristov, 2019).

Figure 1: Evaluation process of a Test Automation solution (own figure)

 

Conclusion

Test Automation can increase quality and reduce costs in agile project environments by delegating repeatable tasks to a Test Automation solution. Although Test Automation can improve software quality, it is only suitable for some projects. Before considering this approach for a specific project, it is necessary to check if there are test cases which can be automated, if the required know-how exists within the team and if the team members can commit themselves following the automation strategy to boost software testing.

 

Glossary (following ISTQB-Glossary)

CI/CD Environment: Continuous Integration/Continuous Deployment: Automated software development procedure that merges, integrates and tests all changes as soon as they are committed.

Defect: An imperfection or deficiency in a work product whereby it does not meet its requirements or specifications.

End-to-End Test: A test level which tests business processes from start to end in a production-like environment.

Exploratory Test: An approach to testing whereby the testers dynamically design and execute tests based on their knowledge, exploration of the test item and the results of previous tests.

ISTQB: International Software Testing Quality Board.

Kanban: A methodology in software engineering where the amount of parallel work (“work in progress”) is limited to achieve shortened cycle time.

Negative Test: Testing a component or system in a way for which it was not intended to be used.

Proof of Concept (PoC): This general approach involves testing a particular assumption to obtain confirmation of whether or not the idea is feasible.

Regression Test: A type of change-related testing to detect whether defects have been introduced or uncovered in unchanged areas of the software.

SAFe: A set of organisational and workflow patterns for implementing agile practices at an enterprise scale.

Scrum: A process model of the project and product management, in particular for agile software development.

Software Increment: A potentially deliverable product increment.

Test Level: A specific instantiation of a test process.

 

References

Gillis, Alexander (2019): Automated Testing. Available online: https://www.techtarget.com/searchsoftwarequality/definition/automated-software-testing

Hristov, Anton (2019): So unterstützt automatisiertes Testen DevOps. Available online: https://www.atlassian.com/de/devops/devops-tools/test-automation

ISTQB (2017): Certified Tester Foundation Level – Extension Syllabus Agile Tester. Available online: https://www.german-testing-board.info/wp-content/uploads/2022/01/GTB-CTFL-AT_Lehrplan_v2017_DE.pdf

Linz, Tilo (2016): Testen in Scrum-Projekten. 2nd Ed. Heidelberg: dpunkt.

News from ERNI

In our newsroom, you find all our articles, blogs and series entries in one place.

  • 06.12.2023.
    Newsroom

    Streamlining software development: The journey from multiple to unified requirements management tools

    Productivity in software development is slowed down by managing specifications across various requirements management (RM) tools. Although moving to a single, updated RM tool involves an upfront investment, the long-term benefits are considerable. These include increased process efficiency, enhanced collaboration, superior traceability, improved software specification quality, cost reductions, scalability and better integration with other RM tools, among others.

  • 22.11.2023.
    Newsroom

    Recognising trends: An insight into regression analysis

    Data plays a very important role in every area of a company. When it comes to data, a distinction is made primarily between operational data and dispositive data. Operational data play an important role, especially in day-to-day business. However, they are not nearly as relevant as dispositive data. This is because these data are collected over a longer period of time and provide an initial insight into the history or the past.

  • 08.11.2023.
    Newsroom

    Why do we need digital transformation for medical devices?

    For hospitals, it is not up for discussion as to whether they want to digitalise. The increasing age of the population in western countries and the progressive shortage of medical professionals mean that without digitalisation, the healthcare system will not be able to provide the quality that patients want in the future.

  • 25.10.2023.
    Newsroom

    Mastering the challenges of mobile app testing: Strategies for efficient quality assurance

    Discover the unique challenges faced in testing mobile applications and learn how to overcome them effectively. From selecting suitable devices and operating systems to leveraging cloud-based test platforms, test automation and emulators, this article provides seven essential strategies for optimising your mobile app testing process.

  • 11.10.2023.
    Newsroom

    Incorporating classical requirements engineering methods in agile software development for a laboratory automation system

    Traditional agile methodologies can sometimes struggle to accommodate the complexity and regulatory requirements of laboratory automation systems, leading to misalignment with stakeholder needs, scope creep, and potential delays. The lack of comprehensive requirements documentation can result in ambiguous expectations and hinder effective communication among cross-functional teams.

  • 27.09.2023.
    Newsroom

    Unveiling the power of data: Part III – Navigating challenges and harnessing insights in data-driven projects

    Transforming an idea into a successful machine learning (ML)-based product involves navigating various challenges. In this final part of our series, we delve into two crucial aspects: ensuring 24/7 operation of the product and prioritising user experience (UX).

  • 13.09.2023.
    Newsroom

    Exploring Language Models: An overview of LLMs and their practical implementation

    Generative AI models have recently amazed with unprecedented outputs, such as hyper-realistic images, diverse music, coherent texts, and synthetic videos, sparking excitement. Despite this progress, addressing ethical and societal concerns is crucial for responsible and beneficial utilization, guarding against issues like misinformation and manipulation in this AI-powered creative era.

  • 01.09.2023.
    Newsroom

    Peter Zuber becomes the new Managing Director of ERNI Switzerland

    ERNI is setting an agenda for growth and innovation with the appointment of Peter Zuber as Managing Director of the Swiss business unit. With his previous experience and expertise, he will further expand the positioning of ERNI Switzerland, as a leading consulting firm for software development and digital innovation.