IMPORTANCE OF SYSTEM ARCHITECTURE AND PLATFORM CHOICE ON SAFETY CERTIFICATION

Not many companies have the expertise to build software to meet the DO-178C (Aviation), IEC61508 (Industrial), or ISO26262 (Automotive) safety standards. Yet the demand for safety-certified software is growing as software control is used in more critical systems, including self-driving cars, drones, and pilotless air taxis.

Across our experience in partnering with customers on military and commercial aerospace system designs, Lynx has seen projects that successfully fly prototypes that are impractical to safety certify due to design decisions made at the start of the project. Choice of CPU, GPU, programming language, or peripheral can easily expand a project budget by a factor of 10. As most mission-critical applications run on an RTOS, its architecture and line count has a major impact on cost, program risk and development schedule. The goal of this blog is to demystify how safety-certified RTOS' are built and offer a few rules of thumb to successfully and cost-effectively use them. Through that, Lynx’s objective is for you to avoid nasty surprises, redesigns and unexpected costs in your safety cert project.

At its heart, safety certification is a design assurance process that aims to ensure your software does exactly what it is supposed to. All behavior is intentional, all edge cases have been considered, the code is error free and works as intended. Functionality is no less and no more than the design.

The software design assurance process for commercial avionics is defined by DO-178C. It prescribes a development process to ensure that the software matches its requirements. The process follows the V-Model, beginning by defining system requirements, then high level requirements, followed by functional requirements and, finally, to writing source code which is compiled into executable binary code. At every stage, testing is performed to verify that the implementation matches the requirements. This results in a vast library of documentation that dwarfs the source code in both size and cost. The cost and extra rigor is tremendous—typically between $100 - $250 per line of source code (Xen and the Art of Certification, Xen Developer Summit 2014 - Nathan Studer and Robert VanVossen).

Here are approximate line counts from representative components. Comments and blank lines are ignored, only active lines of code are counted.

Given the cost to certify a single line of code shown above, driving down the line count for both the application and the RTOS becomes paramount. Certification is performed at the system level. The provider of the RTOS is a supplier of artifacts into the process but it is at the system level where those documents, referred to as artifacts, are collated and supplied to the authorities to prove that the system does what it says on the tin. Our extensive experience with complex avionics projects is that there is invariably some new functionality at the system level such that existing artifacts from a supplier need to be augmented. The size of the certification costs means that safety artifacts are unlikely to be built speculatively; instead, the catalyst is a particular funded project. The first certification of a new RTOS or new hypervisor for a new CPU architecture (example; RISC-V), carries a significant funding burden. Once completed, the opportunity exists for the customer to ease certification costs for the current and subsequent projects. Therefore your choice of RTOS becomes a strategic decision for the company.

Over time, certified RTOS vendors aim to gradually build a core library of certified components. The art of running a certified RTOS company is partly how you manage that library. The focus is on reuse to enable the creation of certified components as cost- and as time-effectively as possible. Building this library from scratch is incredibly difficult, which makes breaking into the certified RTOS business incredibly hard. There are a handful of certified RTOS vendors, such as Lynx, Wind River, DDCI, and SYSGO and no new entrant has joined the market in 30 years. Ultimately of course, the focus is less about how the RTOS company runs its business, more about how the RTOS' architecture brings value to customers. That “reuse” comment above has to translate to value to customers’ designs. This led to standards such as AC 20-148 for Reusable Software Components, something that Lynx is immensely proud to have been granted for its RTOS.

There is a lot more focus across avionics platforms about software reuse across programs in order to reduce system costs, development time, and vendor lock-in. Meeting standards is really just the start of this work since an API standard such as FACE doesn’t define system behavior. The scope of this area is somewhat outside the theme for this article, but if you are interested, check out this other blog and more information on the MOSA page of our website. One of the reasons why Lynx is excited about unikernels, is that they provide the opportunity to package up kernel and application software into a block of technology whose characteristics can be more accurately defined and, therefore, be reused more effectively across programs. Unikernels are not new. They have been discussed for a 15+ years and have been deployed in IT infrastructure including on Amazon EC2 as well as being explored as a base for next generation network infrastructure. What IS new is the concept of bringing them into safety critical environments. 

Let’s now discuss a common misunderstanding about a certified RTOS. Companies advertise their products as being certified, and that is accurate. LynxOS-178 for example has indeed been certified many times. However, RTOSs are highly configurable, and the reader needs to ensure they understand what elements of that RTOS has been certified. This is because due to the magnitude of certification costs, the majority of companies will only take through a specific subset of the RTOS’s features and architecture support through certification. The common misconception is that any industry RTOS, whatever RTOS configuration you use, one simply purchases the certification artifacts and you are done. In reality, all major commercial RTOS providers have a development version of the RTOS and a certifiable one. the latter contains a subset of an RTOS’s features and target architectures. That subset is detailed and complex. Imagine the source code of an RTOS as a giant tree of thousands of source code files; every branch is a large component each with many sub-components. Imagine scores of trees in a row, representing the versions of the RTOS as it is developed over time. Each version has thousands of changes, minor fixes, medium-sized new features, and occasionally entire branches lopped off as subsystems are replaced. As one tangible example, Lynx is one of the few suppliers that have an IPv6 networking stack certified to DO-178C DAL A. Let’s say about every 10 versions, a subset of 30% of one of the trees is safety certified. You end up with a diverse collection of certified components on different versions and different target architectures.

Customers must therefore perform a careful examination of the components needed for a specific project to find the set of certified components that most closely matches. Considered compromises can dramatically reduce certification costs by improving the overlap with the RTOS vendor’s library of existing artifacts.

In the last 20 years every major RTOS provider has achieved several certifications on PowerPC platforms. This means that most safety artifacts available today are for PowerPC. Artifacts for x86 and Arm are less mainstream. We believe that Lynx is today the only RTOS company with DO-178C DAL A hypervisor and RTOS artifacts available for x86. At the time the legacy artifacts were developed chips were mostly 32-bit, so the RTOS certifications were 32-bit also. We continue to work with customers that wish to reuse proven hardware and software on new programs, even though there is no hardware product roadmap for this technology. It is prudent to look at how unified the RTOS is across processor architectures. Some vendors have different APIs, tool chains, workflows and capabilities for each processor architecture. This creates potential challenges and hidden costs in terms of software reuse, (re)training of staff and development inefficiencies. 

New CPUs are more powerful and feature-rich while new languages are more secure and efficient. While considering your right approach, our theme is that you need to think about safety certification right up front as part of that project planning process. Our best advice is to work with your RTOS vendor closely and early. Be sure to speak with them before choosing your target CPU and programming language. You need a system architecture optimized for reducing program risk, safety certification costs and development time. This means:

• Strong compatibility to POSIX, beyond PSE52: Why? Many companies develop software on Linux which means that the stronger the alignment to POSIX, the easier the migration will be for the software that needs to be taken through certification (Certifying a Linux environment is impractical given its size)
• Focus on line count for the software that will be taken through certification. Not the minimum numbers quoted by companies but the actual count given all of the software you wish to include in your system
• Focus on system architecture; our experience and customer feedback points to a need for strong separation of applications, to optimize the use of hardware resources while reducing security risk and simplifying the path to achieving system certification
• Don’t believe you are done if you hear FACE, CAST-32A, or other standards. It is a helpful starting point, but it is not the end of your certification due diligence. Some vendors claim FACE conformance but, on deeper inspection, this only relates to a subset of their portfolio. Some don't have FACE conformance for a bare metal RTOS as an example
• On the business side, you need to consider software provenance; precisely where is it developed, tested and artifacts created 

Safety certification is an area of expertise and active innovation for Lynx. We develop safety artifacts using US and NATO-friendly resources. Safety projects should be cautiously approached and carefully planned to minimize risks. Our most recent work has involved taking LYNX MOSA.ic (i.e. hypervisor and RTOS) through DO-178C DAL A certification. Early planning and focus on a platform-optimized-for-certification can avoid large pitfalls and balance compromises. We are engaged in many certified avionics designs and would be delighted to discuss certification strategies for your safety-critical project. As ever, Lynx remains at your service and we would be delighted to discuss your next project.