How a lack of quality assurance can lead to a loss of $400 million in 37 seconds

By Florian Gumhold (ERNI Switzerland)

Even though the importance of testing in software projects has been proven many times before, even large companies sometimes neglect it. In this article, with the help of two examples, I will emphasise the mistakes to avoid, followed by an introduction of the testing process set-up.

Why software testing is essential

Could you imagine losing $400 million in 37 seconds? That happened to the ESA in 1996 when their carrier rocket Ariane 5 exploded shortly after take-off. The disaster was caused by a foolish software bug and is one of the biggest software failures in history. The starting point of the disaster was the cast of a double variable of the ground acceleration to another data type. The converted value was too big to be displayed within the given range, which caused an exception in the code. This exception enabled a series of reactions which ended in the self-destruction of Ariane 5 (Auer, 2021; Wunderlich-Pfeiffer, 2015).

The disaster of the Ariane 5 took place more than 25 years ago, and it could be assumed that processes, standards and quality assurance in companies have been improved in the meantime. However, that is not valid for every single company on the globe. In 2015 Starbucks had to close almost all its stores in the USA and Canada for half a day due to a problem with the cash registers. The stores couldn’t proceed with any orders and take payments after an internal failure during a daily system update (Starbucks, 2015).

These two real-world examples show how a lack of quality assurance can lead to financial loss, reputational damage and even ending up fatal. But how can companies prevent such disasters? The key is an appropriate and extensive quality assurance process and mindset within the organisation.

Responsibilities for software quality in projects

Software projects are complex, fast moving and often time critical. This can lead to some trade-offs and shifts of priorities within the project team. The project needs to be managed by a project manager, the requirements need to be defined by requirements engineers, and the corresponding software needs to be developed by the development team. But who is responsible for testing and ensuring that the software meets all its defined criteria? In general, the whole project team is responsible for the overall outcome of the final product, including quality aspects. Depending on the methodology and team setup, the primary responsibility for defining and fulfilling quality aspects can be assigned to roles and people (ISTQB, 2021).

So, how can appropriate quality assurance be set up in projects? This will be answered after presenting some misbeliefs about testing and why this mindset is dangerous for conducting a successful project.

Common misbeliefs about testing

As mentioned above, the importance and responsibility of quality aspects are often not anchored in projects. Although testing and fixing bugs in the early stages is cheaper than in the later stages, as shown in Figure 1, some common misbeliefs and flawed approaches still exist such as the following:

  • Dedicated test engineers will cost money and cause delays. They do not add functional value to the software project.
  • The developers will test their functionality to ensure that the final product is working as expected.
  • Extensive testing will happen at the end when the final product has been developed.
  • Testing can be done casually by anyone without any strategy.
  • Writing unit tests is enough to release the product; the end users will do end-to-end testing.
Figure 1 – Relative cost to fix bugs (https://sandline.ro/wp-content/uploads/2021/05/chart-1.jpg)

The listed examples are from actual projects in different areas and sizes. Every single project is unique, but many share the same problem – insufficient quality level due to a missing professional quality assurance process (Ronald, 2020).

Testing process setup

To set up an appropriate testing process, some prerequisites are necessary. First, there is a need for expertise within the team in the section of software quality and testing. Quality assurance is a separate discipline in software projects requiring an understanding of quality concepts and technical know-how, especially when implementing test automation to support frequent test execution in agile environments. Depending on the project size, there is usually a test manager who is responsible for defining a test strategy and ensuring that quality criteria are met. Furthermore, test engineers conduct tests on software in different ways. Usually, there are black-box, white-box, manual and automated tests undertaken to assess the current state of a software release (Spillner, 2019).

Once there is know-how and a mindset established within the team, the next step is to define a test process. As a starting point, the structure of the ISTQB can be taken to set up a testing process. Each stage of the process involves several tasks and activities. Although the stages are listed sequentially, they will be executed iteratively, and there is also some overlap between the stages. Especially in agile projects, where software is built incrementally, the stages are aligned with the SDLC and conducted iteratively with many iterations. The testing process offers a basic framework which needs to be adapted to the project and software being built (ISTQB, 2021):

Figure 2 – Testing process (own figure)

Conclusion

Ignoring the importance of software testing can lead to tremendous damages in the aspects of financial loss, reputational damage and even loss of life. Many real-world examples show that the lack of software testing is a severe problem for companies around the globe. Setting up an appropriate and professional software testing process is crucial for project success and a good reputation. Although software testing cannot detect every single bug in a software product, it is a commitment showing that software quality is taken seriously, and that processes and standards are set up to deliver high quality to the customers.

All our quality assurance experts are certified by the ISTQB to apply a standardised and worldwide acknowledged testing approach in your project!

Glossary (following ISTQB-Glossary)

Black-Box Testing: A test technique which is aimed at testing the system without knowing the internal structure of the software

End-to-End Test: Test level which tests business processes from start to end in a production-like environment

ESA: European Space Agency

ISTQB: International Software Testing Quality Board

SDLC: Software Development Life Cycle – a list of activities performed at different stages in software development and the connection between them in a logical and chronological manner

Test Engineer: A person who inspects and reports the quality of a system or product through the entire production cycle

Test Manager: A person who is responsible for managing testing activities, resources and evaluation of a test object

Testing Process: A set of interrelated activities to specify and fulfil given quality criteria

Testware: Work products produced during the test process for use in planning, designing, executing, evaluating and reporting on testing

Unit Test: Test level which focuses on single methods or functions in isolation

White-Box Testing: A test technique which is aimed at testing the system with access to the internal structure of the software

References

Auer, Andrea (2021): Ariane 5. https://www.mathe-museum.uni-passau.de/digitale-exponate-zum-ausprobieren/ariane-5/,

ISTQB (2016): Certified Tester Advanced Level Syllabus Test Automation Engineer. https://istqb-main-web-prod.s3.amazonaws.com/media/documents/ISTQB-CT-TAE_Syllabus_v1.0_2016.pdf,

ISTQB (2021): Certified Tester Foundation Level Syllabus. https://istqb-main-web-prod.s3.amazonaws.com/media/documents/ISTQB-CTFL_Syllabus_2018_v3.1.1.pdf, zuletzt

Ronald, Jasmine (2020): 12 Common Software Testing Misconceptions That We Need to Clear. https://dzone.com/articles/12-common-software-testing-misconceptions-that-we,

Spillner, Andreas (2019): Basiswissen Softwaretest: Aus- und Weiterbildung zum Certified Tester – Foundation Level nach ISTQB®-Standard (iSQI-Reihe). 6. Ed. Heidelberg: dpunkt.

Starbucks (2015): Point of Sale Register Outage Resolved.  https://stories.starbucks.com/stories/2015/starbucks-point-of-sale-register-outage-resolved/

Wunderlich-Pfeiffer, Franz (2015): In den Neunzigern stürzte alles ab. O https://www.golem.de/news/softwarefehler-in-der-raumfahrt-in-den-neunzigern-stuerzte-alles-ab-1511-117537.html

Are you ready
for the digital tomorrow?
better ask ERNI

We empower people and businesses through innovation in software-based products and services.