Background
CI (continuous integration)/CD (continuous delivery) pipelines have become an indispensable tool in software engineering since they enable developers to automate many tedious processes to verify the quality of the source code and the testing of the applications. With a little effort, the software artefacts can be built and tested in various environments and with different toolchains. If all tests in the pipeline pass, the artefacts can be further deployed based on the project’s needs.
Since embedded software often runs on custom hardware, it brings its own additional challenges to the testing step, for example, the image deployment, side effects due to competition over limited and shared resources, and a tight interaction with the hardware which complicates the testing. These points make it essential to regularly test the already developed features on the hardware to detect regression as early as possible. Due to its complexity, the tests are often performed manually. This is very time consuming; therefore, it is easier to test only the corner cases of the just implemented feature instead of running the whole test suite. As a result of having a limited number of tests, side effects may remain undetected.
In addition to the mentioned challenges, the development of embedded devices carries the problem that the development of the software and the target hardware happens in parallel. Therefore, the final hardware is not yet available at the beginning, and it is challenging to detect if a bug is occurring in the hardware or the software. Furthermore, only a limited number of boards are available, which must be shared within the development team. To overcome these difficulties and to reduce the burden of executing the entire test suite regularly on the hardware, it would be beneficial to integrate the on-target tests as an additional stage in the CI pipeline. This stage could execute unit tests which run on the target, integration tests or even hardware acceptance tests from the software point of view. As a side effect, one or more complete hardware setups are made available to the whole development team. This article provides a conceptual overview of the chain between your CI/CD pipeline and the target hardware.
General setup
The setup consists of 3 main components:
- The CI/CD server executing your pipeline on specified triggers
- Local executor machine
- The hardware in the loop (HIL) connection
All three components have different requirements and need a different implementation based on the project’s needs.
CI/CD server
This hosted service (GitLab, GitHub, Azure DevOps, Jenkins…) executes a CI/CD pipeline for the configured triggers. Primarily, this is controlled by a configuration file which is part of the Git repository. Typical triggers to run the pipeline are merge requests, pushes, manual triggers and many more. Usually, this service does not run in the local facility and is shared within the company between multiple teams and projects.
Local executor machine
This machine must have direct access to the target hardware and must equip the pipeline executor with the permissions to access the hardware interaction channels. The pipeline executor is a piece of software provided by the CI/CD service that checks the CI/CD server for pending jobs and executes them locally, giving the pipeline execution access to the target hardware through the provided interfaces.
The connection between the pipeline executor and the CI/CD server depends on the service provider. It requires some local configurations (server URL/port, authentication) and server-side configurations (permissions, authentication).
HIL connection
The hardware in the loop connection is the core of the setup and is responsible for connecting the target hardware with the test execution hardware and providing this interface to the pipeline executor. This part of the architecture can be very complex and varies from product to product. It can consist of multiple hardware components or be a software module on the Local Executor machine.
At a minimum, it should provide a way to program or deploy the newly built software image either through a debugger connection or via a communication interface, the possibility to restart the target, and at least one communication channel to interact with the target. The complexity can grow from this point as much as the project requires it to. Failure injection, other communication channels, or hardware emulation of the connected devices could be added to create a test environment as close to reality as possible.
Final remarks
With the setup described, it will be possible to run your test cases targeting your hardware in the CI/CD pipeline and regularly get direct feedback on the results. To run tests on the target, the corresponding pipeline stage must execute a test script (part of the repository) containing the test cases which directly interact with the locally provided HIL interfaces, for example, to program or restart the board, to communicate through a serial port or to open an SSH connection to the target. An advantage is that the same test scripts can be executed locally, outside the pipeline, to achieve the same results on your desk.