Introduction
IT software projects are mainly conducted using an agile and iterative methodology like Scrum, SAFe or Kanban. Such projects are usually set up in CI/CD environments, which support automated merging, building and deploying of software. Software is produced incrementally and frequently to receive fast feedback and to allow corrective actions to be taken if necessary. Frequent software increments also require reliable and fast quality assurance to inspect quality and stability. The implementation of Test Automation is intended to fulfil the task of frequent test execution of e.g., regression tests and smoke tests, without manual intervention. Test Automation can be applied at different levels, such as component, system and integration testing, and it can be integrated into the existing CI/CD environment to automate the test execution of a defined test set as soon as a new build is being deployed in the test environment (Linz, 2016, p. 11-20; p. 150).
What is Test Automation, and why is it important?
The ISTQB provides a concise and accurate definition of Test Automation: “The use of software to perform or support test activities, e.g., test management, test design, test execution and results checking.” Most likely, the segment of automated test execution is known to many people working on agile projects. The intention of Test Automation in the subarea of test execution is to delegate repeatable and monotonous tests to a machine instead of repeatedly executing tests manually. Three main goals are pursued by using a Test Automation approach:
- Quality, speed and efficiency in test execution should be improved. Manual test execution is not scalable. If the tested software scope exceeds the test team capacity, quality decreases and test coverage cannot keep pace with the implemented requirements.
- Test Automation reduces costs. Executing many regression tests repeatedly is time-consuming and costly. Investing resources in Test Automation will save money over the project runtime and beyond.
- Test Automation enables faster feedback on the quality of a software increment. Merging, building and deploying software is already automated in most projects. Therefore, it makes sense to include automated tests in the CI/CD environment. This allows continuous monitoring of the quality of the software and taking necessary actions if required (Hristov, 2019).
Having explained the purpose and advantages of Test Automation, it is also necessary to mention its limitations and that it cannot be the solution for every problem. Test Automation is only an execution of pre-defined steps which have been implemented by a human. A Test Automation solution only executes these steps without doing any other actions or validations. Any act beyond the implemented test script will not be performed. In other words, this could end up in a situation where defects will not be detected by the automated tests but would have been detected by a manual tester. This leads to the fact that Test Automation will only partially replace manual testing. It can be an extension and supporting task to manual testing. However, it cannot replace a manual tester’s creative skills and experience, especially when it comes to exploratory or negative tests, which are based on the experience and inventiveness of a skilled tester (ISTQB, 2017).
Tools, Technologies and know-how
Commonly, project teams need fast, reliable and comprehensive feedback on the quality of a software increment. Therefore, many vendors of Test Automation tools have taken the chance to provide a solution for this issue. Tools can be divided into open-source and commercial solutions, many of which are designed as an all-in-one solution. This means that one tool can cover many different testing levels and types of applications. Such a tool covers testing of mobile, web or desktop applications and requires no additional testing framework to implement end-to-end tests. Hence, tests can be implemented by non-technical people by using a record-and-play functionality to click and validate actions without writing a single line of code. Commercial vendors primarily provide this type of Test Automation. There is also the possibility of implementing test cases by using programming languages. This requires technical know-how and is mainly used with open-source tools. Which of the two approaches and means to choose primarily depends on the project environment, financial aspects, programming know-how and the existing toolchain (Gillis, 2019).
Setup of Test Automation in agile projects
A Test Automation solution evaluation process can be divided into different steps. The first step is defining the criteria catalogue and researching possible market tools that fulfil the criteria. This catalogue should not only be limited to functional requirements but also include TCO (total cost of ownership) and integration into the existing tool chain to guarantee smooth communication between the tools used in the project. After this step, the next one is to perform a proof of concept with 3 to 5 test cases to determine if the tool candidates can cover all the specified requirements. It is recommended to use the most critical and extensive test cases for the proof of concept to ensure that these high-priority tests can be automated. Test Automation for a project always requires assessing the skills and know-how available within the project team. Having the know-how, defining a strategy and providing enough resources in the evaluation phase is crucial. It determines if Test Automation can be established in the project or if it is considered overhead without adding benefit to the quality of the product (Hristov, 2019).
Conclusion
Test Automation can increase quality and reduce costs in agile project environments by delegating repeatable tasks to a Test Automation solution. Although Test Automation can improve software quality, it is only suitable for some projects. Before considering this approach for a specific project, it is necessary to check if there are test cases which can be automated, if the required know-how exists within the team and if the team members can commit themselves following the automation strategy to boost software testing.
Glossary (following ISTQB-Glossary)
CI/CD Environment: Continuous Integration/Continuous Deployment: Automated software development procedure that merges, integrates and tests all changes as soon as they are committed.
Defect: An imperfection or deficiency in a work product whereby it does not meet its requirements or specifications.
End-to-End Test: A test level which tests business processes from start to end in a production-like environment.
Exploratory Test: An approach to testing whereby the testers dynamically design and execute tests based on their knowledge, exploration of the test item and the results of previous tests.
ISTQB: International Software Testing Quality Board.
Kanban: A methodology in software engineering where the amount of parallel work (“work in progress”) is limited to achieve shortened cycle time.
Negative Test: Testing a component or system in a way for which it was not intended to be used.
Proof of Concept (PoC): This general approach involves testing a particular assumption to obtain confirmation of whether or not the idea is feasible.
Regression Test: A type of change-related testing to detect whether defects have been introduced or uncovered in unchanged areas of the software.
SAFe: A set of organisational and workflow patterns for implementing agile practices at an enterprise scale.
Scrum: A process model of the project and product management, in particular for agile software development.
Software Increment: A potentially deliverable product increment.
Test Level: A specific instantiation of a test process.
References
Gillis, Alexander (2019): Automated Testing. Available online: https://www.techtarget.com/searchsoftwarequality/definition/automated-software-testing
Hristov, Anton (2019): So unterstützt automatisiertes Testen DevOps. Available online: https://www.atlassian.com/de/devops/devops-tools/test-automation
ISTQB (2017): Certified Tester Foundation Level – Extension Syllabus Agile Tester. Available online: https://www.german-testing-board.info/wp-content/uploads/2022/01/GTB-CTFL-AT_Lehrplan_v2017_DE.pdf
Linz, Tilo (2016): Testen in Scrum-Projekten. 2nd Ed. Heidelberg: dpunkt.