Reviewer Group: Senior DevOps Architects and Embedded Systems Development Leads
Abstract:
This presentation advocates for the widespread adoption of Continuous Integration/Continuous Deployment (CI/CD) methodologies within embedded systems development, addressing common industry struggles with inefficient Software Development Life Cycles (SDLCs). The speaker, Mark Hermling from Ada Core, highlights critical issues such as prolonged feedback loops (weeks for test results) and extensive manual release testing, which lead to a "haystack effect" of accumulating defects. The core argument is that CI/CD, interpreted for embedded systems as achieving a "continuously deployable" state rather than immediate deployment, empowers developers with rapid feedback, fosters collaboration, and significantly improves product quality and regulatory compliance. Key recommendations include strategic automation, investment in tooling and infrastructure, a strong emphasis on layered software design to enable host-based testing and stubbing, and comprehensive metric collection for continuous improvement, rather than isolated management.
Escaping the Haystack: CI/CD for Embedded Systems
0:01 Introduction and Problem Statement: Mark Hermling from Ada Core discusses widespread struggles in embedded systems projects with SDLCs, often leading to despair. He aims to present thoughts on using CI/CD to improve these processes.
Automotive SDV Builder: Developers faced a one-week feedback loop for test results (unit, static analysis, hardware, software tests).
Industrial Manufacturing: Feedback loops extended to over two weeks, leading to significant context-switching costs for developers.
Embedded Firmware Company: Required three months of manual release testing after development, an unsustainable practice given modern regulatory demands (e.g., Cyber Resilience Act, CISA).
3:51 The "Haystack Effect": Delayed feedback on code changes (bugs, features, defects) leads to a large, unmanageable backlog, as developers move to other tasks before receiving results. Empowering developers with immediate feedback is crucial to preventing this accumulation.
5:18 Your Team is Your Biggest Asset: Developers desire pride in their work and dread uncertainty about code acceptance or potential breakage. Fast, private feedback in their workspace promotes ownership, accountability, and continuous learning, especially for junior engineers, and facilitates integration with AI tools.
6:38 Challenging Complexity: The assertion that embedded environments are "too complex" for automation is challenged. Significant automation is possible with smart environmental analysis.
7:18 Investment Required: Implementing a robust SDLC with CI/CD is not free and requires investment in tooling, infrastructure, automation, compute power, and critically, software design.
8:15 CI/CD for Embedded Systems Defined: This refers to "continuous integration, continuously deployable," meaning a consistent state where a main or integration branch is always ready for deployment, not necessarily immediate deployment to an end system. This is vital for rapid responses to vulnerabilities (e.g., Cyber Resilience Act).
10:18 Developer Empowerment & Testing: Developers should be encouraged to push code often into shareable, private environments where comprehensive tests run rapidly.
11:37 Comprehensive Testing: Includes coding standards, unit tests, regression tests (system tests), code coverage, Static Application Security Testing (SAST) for buffer overruns and undefined behavior, and system testing, potentially with stubbing.
12:44 Software Design Matters (Layered Architecture): A layered software design (hardware, drivers, OS, HAL, middleware, business logic, UI) is crucial. By abstracting hardware interaction (stubbing the HAL), testing can be shifted to host machines (e.g., Linux), enabling faster, cheaper pipeline execution.
15:20 Static Analysis Scoping: Static analysis can be compartmentalized to run on smaller, changed sections of code for faster feedback, rather than analyzing the entire codebase.
Pipeline Automation: GitHub Actions, GitLab CI/CD, etc. (focus on completion with feedback, ~15 min to 1 hour feedback loop).
18:28 Gather Metrics: Track project trends (Lines of Code, test pass/fail, coverage, findings/KLOC, code complexity). These metrics are relative to the project, not for isolated management, but for transparency and continuous improvement for developers.
Host-based: Cheap, scalable, parallelizable, fits well into pipelines.
Target-based: More complicated and potentially expensive; solutions like GitLab's "Device Cloud" allow pipelines to allocate and manage shared hardware targets on demand.
22:48 Compute Infrastructure: Make it scalable and on-demand (on-prem Kubernetes/VMs or cloud solutions). Smart investment is crucial; don't compute what's not necessary (e.g., static analysis on unchanged third-party code).
24:12 Specialized Automation Teams: Dedicated teams should manage pipelines and automation, as core software developers may lack infrastructure expertise.
25:07 Streamlined Developer Environment: Developers should ideally only install an IDE and Docker. Leverage VMs, dev containers, and web environments to enable immediate project engagement without extensive setup. Automate deployments and debugging.
26:48 The Litmus Test: A new hire or experienced engineer should be able to complete a trivial task (e.g., changing a GUI button text) and get it merged within half a day, ideally less than an hour. If not, the SDLC needs improvement.