LYNX FAQ

LynxOS-178 is built to the high software design assurance standard of DO-178C, but the certification artifacts are optional, which means that all users of LynxOS-178 benefit from this high quality. If your project needs to be certified to some standard or if real-time determinism is needed, then LynxOS-178 is an excellent RTOS.

If you need certification to some standard (even if it’s not DO-178) you will need LynxOS-178. We develop LynxOS-178 using DO-178C processes. This is a rigorous standard, and customers that do not need DO-178 will still benefit from LynxOS-178’s certification package for their certification efforts.

If you need real-time determinism, then LynxOS-178 is currently the RTOS of choice on the LYNX MOSA.ic™ development framework. However, we will soon be supporting FreeRTOS as an out-of-box RTOS solution on LYNX MOSA.ic as well. You will then be able to choose between FreeRTOS and LynxOS-178 as off-the-shelf RTOS solution, according to which out-of-the-box RTOS best fits your system’s needs. We also have customers that have ported their own RTOS solutions in preference to LynxOS-178. We do not force customers to use LynxOS-178 even when they want an RTOS and need real-time determinism.

If neither certification nor real-time determinism is required for your project, then you do not need LynxOS-178. Buildroot, which is provided as part of LYNX MOSA.ic, or a 3rd party COTS OS (e.g. RedHat Linux or Windows) may be a better OS fit for your product.

Read: Lynx announces support for FreeRTOS

Read: Do you need an RTOS?

Read: How to choose an RTOS

• LynxOS-178 supports the C programming language for both production mode and development mode (an enhanced set of APIs and capabilities to aid in development)
• LynxOS-178 supports C++11/14 for development mode
• LynxOS-178 supports Ada from Lynx’s partners
• Lynx can also provide a Rust compiler to enable customers to compile and run Rust applications on LynxOS-178

Yes. LynxOS-178 supports up to 16 ARINC 653 time, space, and resource partitions per OS instance. Each partition has its own allocation of CPU time, memory resources, timer resources, inode (file system) resources, etc.
To learn more, visit the the "more information" page for LynxOS-178.

Yes. LynxOS-178 is a native POSIX RTOS. It implements POSIX.1-2008 (2016 Edition), which is POSIX's core functionality, POSIX.1b (the POSIX real-time extensions), and POSIX.1c (also known as POSIX pthreads extensions).

Click here to learn more.

LYNX MfA use the GNU Compiler Collection (gcc) version 7.1.0. The C11 version of the C programming language is supported, formally known as ISO/IEC 9899:2011. Lynx plans to upgrade to GCC version 11.3 during 2023. LynxOS-178 certification artifacts assume user applications are written in C. 

FACE compliance refers to software has been modified to support the FACE application programming interfaces (APIs). FACE conformance requires that the software has been tested by third parties and verified to truly meet the standard.

LynxOS-178 is conformant to the FACE 3.1 specification.

Yes. It is offered as a software module for LynxOS-178. This stack is certified to DO178C DAL A.

No. Lynx is willing to license this for use with other real-time operating systems.

LynxOS-178 implements a time-slice scheduling algorithm that gives each partition fixed execution time so that the system can be deterministically safe. Additionally, the system allows multiple applications of differing criticality levels within partitions to execute, completely isolated, on the same hardware resource. With LynxOS-178, each task runs protected in its own space for uncompromising reliability within an ARINC 653 partition, enabling easier application certification.

LynxOS-178 partitions under MFA 2021.11.2 conform to the ARINC 653-1 Application Executive Software (APEX) Interface defined by the ARINC 653-1 standard. LynxOS-178 provides the system service groups: ARINC 653 Partition Management, ARINC 653 Process Management, ARINC 653 Time Management, ARINC 653 Interpartition Communication, ARINC 653 Intrapartition Communication, and ARINC 653 Health Monitoring.

LynxOS-178 conforms to the ARINC 653-1 APEX Interface defined by the ARINC 653-1 standard, ensuring application portability, software reuse, and interoperability between embedded systems. LynxOS-178 provides the following system service groups in accordance with the ARINC 653-1 standard: Partition Management, Process Management, Time Management, Inter-partition Communication (Sampling and Queuing Ports), Intra-partition Communication (Buffers, Blackboards, Semaphores, and Events), and Health Monitoring.

Yes, LynxOS-178 must be certified to the highest DAL level of any of the applications running on it. Drivers are part of the OS, and so fall under the same rule. LynxOS-178 drivers are separate binaries that are loaded at boot time. This reduces certification costs because a driver can be added and certified without touching (redoing) the kernel artifacts.

The LynxOS-178 RTOS conforms to the ARINC 653-1 Application Executive Software (APEX) Interface defined by the ARINC 653-1 standard. LynxOS-178 and LynxOS-SE provide the following system service groups in accordance with the ARINC 653-1 standard:

• ARINC 653 Partition Management: services related to partition management. GET_PARTITION_STATUS and SET_PARTITION_MODE are Partition Management service requests
  
  
• ARINC 653 Process Management: services related to process management. GET_PROCESS_ID and GET_PROCESS_STATUS are Process Management service requests
  
  
• ARINC 653 Time Management: services related to time management. TIMED_WAIT and PERIODIC_WAIT are Time Management service requests
  
  
• ARINC 653 Interpartition Communication: services responsible for communication between processes residing in different partitions. There are two types of Interpartition Communication services:
  • Sampling Port Services: A sampling port is a communication object allowing a partition to access a channel of communication configured to operate in sampling mode. CREATE_SAMPLING_PORT and WRITE_SAMPLING_MESSAGE are Sampling Port Services service requests
    
    
  • Queuing Port Services: A queuing port is a communication object allowing a partition to access a channel of communication configured to operate in queuing mode. CREATE_QUEUING_PORT and SEND_QUEUING_MESSAGE are Queuing Port Services service requests
    
    

• ARINC 653 Intrapartition Communication: services responsible for communication between processes residing in the same partition. There are four types of Intrapartition Communication service requests:
  • Buffer Services: A buffer is a communication object used by processes of a same partition to send or receive messages. CREATE_BUFFER and SEND_BUFFER are Buffer Services service requests
    
    
  • Blackboard Services: A blackboard is a communication object used by processes of the same partition to send or receive messages. CREATE_BLACKBOARD and DISPLAY_BLACKBOARD are Blackboard Services service requests
    
    
  • Semaphore Services: A counting semaphore is a synchronization object commonly used to provide access to partition resources. CREATE_SEMAPHORE and WAIT_SEMAPHORE are Semaphore Service service requests
    
    
  • Event Services: An event is a synchronization object used to notify the occurrence of a condition to processes that may wait for it. CREATE_EVENT and SET_EVENT are Event Services service requests
    
    
    
• ARINC 653 Health Monitoring: The Health Monitor (HM) is invoked by an application calling the RAISE_APPLICATION_ERROR service or by the OS or hardware detecting a fault.

ARINC 653 partitions are always enabled in LynxOS-178, but by default everything is in a single partition.

Software can be in an ARINC 653 partition and not use ARINC ports. In this sense, there is nothing “special” about a partition—it is just a normal LynxOS-178 application (that is, a set of processes, threads, a filesystem and device nodes). It could be multiple exe files where one calls another, which reads config files from disk and writes output to a network. Each exec file has access to the POSIX API as well as the ARINC API. This whole menagerie (the partition) has a restricted schedule and memory footprint.

The ARINC 653 partitions in LynxOS-178 are configured in a text file called the VCT (Virtual machine Configuration Table). It defines the CPU schedule for the partitions, how much RAM each gets, what FS each has and what device nodes are visible. VM0 is a privileged partition, think of it as the “root” partition, it can see into the other partitions.

We also provide an example VCT file with the maximum (16) partitions defined, the VCT file is easy to use, it is made up of about 50 parameters with relatively intuitive names. For additional and related information, please consult the LynxOS-178 User’s Guide.

By default the FS includes lots of LynxOS-178 ports of standard Unix utilities [see below]. RSH (remote shell) is supported as well as serial, so you can login to bash from serial or network. The intention is this Unix-like command line setup is used for development. Then, when you go to certification, you remove 95% of the file system contents, leaving only the applications you wrote and any support files (config files, log files) they need. You change the start-up script to call your application instead of bash, leaving you with a system with just the minimal necessary components, and hence less expensive to certify.

Here is the complete list of every command Lynx includes in LynxOS-178 that customers can optionally populate their file system with:

No, the LynxOS-178 file system is not power-fail safe. Instead, the filesystem is mounted as read-only. Power-fail safety is incredibly difficult to do reliably and expensive (lots of code to certify). So we rely on you setting up just the small bits of the FS you need as writable, specifically for use by your application(s). For eg /tmp is writable by default. It is expected that the system is setup to load the kernel and, let’s say, 5 custom applications from a RAM disk. Then those applications might mount a writable FS on an SSD for bulk data access (large map database for eg) or logging. In this situation, the OS and your apps will always run, and any FS power-fail corruption will be limited to writable areas on the SSD. And, since you have the OS guaranteed to be uncorrupted, you should build in ways to recover if the disk does get corrupted (scripts, the chkdsk utility, etc, called programmatically from your application). Bottom line, if you need to be writing heavily to large files on disk, consider buying a fail-safe filesystem from a 3rd party, or, even better using an embedded database instead of a FS. Disks are just not 100% reliable, they are very good, but think about how many external USB HDDs you have had over the years, and how they occasionally fail.

First, LynxSecure runs Initial built in test (IBIT) and continuous built in test (CBIT) to validate the crucial hardware registers that control the partitioning of the system remain valid. By design it should be impossible to programmatically change these, so CBIT is intended for things like single-event upset (SEU), ie cosmic rays, but it also protects against a hw faults or weakness (like rowhammer).

Secondly, LynxSecure has an audit log. Guest VMs can be granted privilege to access the audit log and to receive an interrupt when events are stored. There are about 60 audit events, things like VM lacks permission to set the time, change the schedule, make hypercall X. The events can be filtered and actions defined should you wish to halt or reset a VM that generates event X. This detects badly behaved VMs or VMs attempting to “break out”.

LynxOS-178 contains two environments: The Production Environment and the Development Environment. The Production version of LynxOS-178 has a feature set which supports certification to DO-178C Design Assurance Level A. The Development Environment (a superset of the Production Environment) has additional features that assist in application development and debugging on LynxOS-178 including;

• Kernel Features – TTY and ptrace for example
•  Shells and Utilities - bash, tftp, ssh, ps, gzip, and additional file system utilities for example
• Debuggers - Standard gdbserver for working with GDB
• Additional Drivers - USB, AHCI and network interface drivers 

These development mode features make LynxOS-178 development as productive and efficient as UNIX or Linux. Lynx does not have supporting DO-178C artifacts for Development Environment features. When application development is complete the project should switch to the Production Environment to eliminate these productivity elements and ready the system for safety certification. 

When setting up a RAM disk between two subjects, the subjects only have read-only access to the shared memory region. The Lynx Simple Application (LSA) has read-write access. We deliver a new image up to the LSA using the FIFO Image Transfer Protocol (FITP) down into the shared memory region, mount the file systems and create the shared read-only RAM disk. 

FITP uses lock-step messages to issue commands to the receiver and transfer data between subjects. It has commands to overwrite or zeros the boot image regions of the LynxOS-178 (or LynxElement) subjects. It also supports commands to stop, restart, pause and resume subjects. Finally, there are commands to change the scheduling policy of LynxSecure. 

Read our blog about RAM disks here.

The shared memory region must be declared in the autoconfig script so it is identifiable by the LynxOS-178 subjects, meaning that:

• The LynxOS-178 subjects must have access to the memory region
• The shared memory region must be mapped to a fixed guest physical address (GPA)
  
  

Each LynxOS-178 instance’s RAM disk driver needs to have an entry for the RAM disk’s GPA and size. By adding an entry to the LynxOS-178’s VCT file, the RAM disk can autoload on boot.

Read more here and learn specific commands we use to initiate the creation of this shared RAM disk.

LynxSecure is a separation kernel, which is a minimal hypervisor. It is a static hypervisor that is configured on your host PC so that when it boots, it allocates target hw resources (cores, memory, peripherals) into fixed immutable virtual machines. The privileged setup code is discarded (for security) so that all that is left of LynxSecure is a set of event handlers to respond to and redirect interrupts and handle management calls like “shutdown”. The separation is achieved by hardware virtualization, which once configured, does not require active management (emulation, scheduling, etc). The goal of a separation kernel is to be minimal, elegant and efficient (LynxSecure is 15K on Arm). Initially (Rushby, 1981) this was so that formal methods could be used to prove separation for security, but more recently these properties have gained traction in the safety arena, to reduce safety certification costs and to provide multicore partitioning.

Additional reading:

• LynxSecure Product Page
• What is a separation kernel?

No. In contrast to other popular hypervisors built by software vendors targeting the embedded market, LynxSecure is not an OS, nor is it "built on" an RTOS. LynxSecure nether runs on top of an existing RTOS or explicitly leverages a helper RTOS. It is not a microkernel and only requires only a very minimal board support package (BSP). Additionally, LynxSecure never loads any high-level drivers (these interfaces are left to the Guest OS of your choosing), and does not run any system services (leaving it with no known attack vectors).

LynxSecure was created in response to global events in the early 2000's which emphasized the need for improved safety and security. The technology was designed to satisfy real-time high assurance computing requirements in support of the NIST, NSA Common Criteria, and NERC CIP evaluation processes which are used to regulate military and industrial computing environments. It harnessed decades of university, government and corporate research which identified three fundamental principles for a separation kernel hypervisor, namely:

• Separate disparate applications into different domains
• Ensure each domain runs unaffected by another system
• Security-policy enforced information flow between subjects

There are about 20k lines of certifiable source code in this product.

LynxSecure is a separation kernel. LYNX MOSA.ic is a software framework which includes LynxSecure as a foundational element. LYNX MOSA.ic also includes Buildroot Linux, bare-metal tools, and other operating systems (from Lynx, from 3rd parties and soon from open source) which customers can harness to get a head start in their development work.

Short answer: Lynx offers a NIST 800-53 package for LynxSecure, the current NIAP-supported certification and accredited process for critical systems. No new evaluations have been accepted for years (nor will they be accepted, as the SKPP is effectively dead).

Detailed answer: The Common Criteria for Information Technology Security Evaluation (ISO/IEC 15408) is an International standard for IT products. Common Criteria defines security evaluation assurance levels (EALs) ranging from EAL1 (least secure) to EAL7 (most secure) to describe the security level achieved by Security Targets (components) being evaluated (tested) against a Protection Profile.

For or a few years beginning in 2007, the Common Criteria included a profile for separation kernels called the Separation Kernel Protection Profile (SKPP) that real-time operating system (RTOS) providers such as Lynx, Wind River Systems, and Green Hills Software built products toward. The SKPP was sunset in 2011 by NIAP after problems with the first evaluation. The Common Criteria remains a useful standard, but no further SKPP evaluations will be accepted, and the SKPP is effectively dead. Instead, NIAP is directly supporting the certification and accreditation process for critical systems. NIST Special Publication 800-53, “Security and Privacy Controls for Federal Information Systems and Organizations” is a more general security approach that partly fills the SKPP gap. We offer a NIST 800-53 package for LynxSecure.

LynxSecure was designed to satisfy real-time, high assurance computing requirements used to regulate military and industrial computing environments, such as NIST, NSA Common Criteria, and NERC CIP. Developed and maintained in San Jose, California in accordance with FAA DO-178 Safety Quality Standards and DoD Risk Management Framework guidelines, LynxSecure is certified, fielded, and maintained on classified DoD networks. The product has undergone many security assessments including penetration testing and design review by independent government security authorities.

For the last several years, LynxSecure has gone through numerous delta certifications showcasing significant cost savings from the reuse of previously certified components that have remained unmodified throughout the lifecycle of a program’s tech refresh period. The technology has also enabled programs to effortless spawn derivative platforms into adjacent programs further maximizing component reuse. Lynx offers a DoD Risk Management Framework guide to aid the US Army’s security evaluation of the security enforcing properties of the platform. The package includes NIST security control traceability and Common Criteria Security Target traceability into the underlying kernel design requirements.

Our NIST security artefacts are for LynxSecure on x86. A block of services comes with the artefacts to train users how to adjust them for additional hardware. If desired, Lynx can engage in a discussion about professional services work to enact custom work for a specific customer.

LynxSecure doesn’t partition software, it partitions hardware into virtual machines in which software executes. Each virtual machine can be granted very fine-grained privileges; the system can use one guest to perform one critical system function, and another guest to perform another critical system function. There is no need to bestow all of the system trust in a single guest operating system.

In LynxSecure there is no root OS, helper OS, master OS, trusted OS, etc. Developers can however choose to emulate this approach by assigning all key system functionality to a guest OS of their choice. LynxSecure configures all partitioning in the system prior to the start of any guest OS. All hardware registers defining partitioning attributes (MMU, interrupt vectors, SMMU, PCIe, etc.) are set to the values required for the hardware to enforce the defined partitioning scheme. This includes CPU schedules, interrupt controllers, memory, devices, PCIe end points, DMA masters, etc. Once these registers are written/set, the code which performed the write is removed from the LynxSecure binary. From this point forward, it is literally not possible for LynxSecure to modify the defined partitioning, privileges, and security policies. Additional details are available in the LynxSecure Architecture guide (product documentation)

For every architecture, LynxSecure defines one or more shared memory regions between guest operating systems.

Each region is uni-directional/single-write.

Each region is zero copy from the perspective of the Separation Kernel. LynxSecure has no role in copying or moving data from user to supervisor to user privilege modes. This clean approach provides a high performance, secure memory region as the underlying foundation for more advanced protocols. For Linux, Lynx Simple Applications and LynxOS-178, Lynx also provides drivers for FIFOs and vEthernet (Intel now, Arm mid 2020) communications between guests, relying on the underlying zero copy shared memory as transport.

Solutions related to security are very SoC specific and dependent on key storage, cryptographic accelerators, and more. As such, it would be false to state that we have an off-the-shelf solution, when any solution will require a level of hardware-specific customization to leverage available resources.

Solutions that are based on an underlying RTOS may implement platform-specific solutions which can then be exposed by that RTOS via higher level APIs. However, this will likely require that a comprehensive RTOS BSP is available. LynxSecure is not an RTOS and it relies on fine-grain privileged operating systems to use the software stack selected by our customers for each purpose.

Customers using LynxSecure have created upgrade mechanisms which are platform or SoC specific solutions customized to their needs.

Yes, our hypervisor, LynxSecure (the foundation of LYNX MOSA.ic) does support hardware cache partitioning on Intel architecture. We use the Intel RDT (resource director technology) CAT (cache allocation technology) feature to split the last level cache. Each CPU core or individual VCPU (CPU timeslice, ie VM) can be assigned a cache capacity bitmask to split the cache with the granularity supported by the hardware. Software cache-coloring is not supported today.

Read blog post: Multicore Cache Allocation Technology (CAT) Demo

The LYNX MOSA.ic framework and architecture allows customers to deploy their favored network protection solution within the guest OS that they prefer and have consequently assigned with the necessary hardware privileges and security policies. Lynx has not invested to incorporate these capabilities within LynxOS-178 as we feel there are more fully featured solutions available at lower cost. On Intel platforms, the Virtual Device Server (VDS), a privilege-restricted Linux guest OS, provides an ideal location where additional network or packet inspection services could be easily added on behalf of the full system

LYNX MOSA.ic is described as "a software framework for building and integrating complex multi-core safety- or security-critical systems using independent application modules." What, then, are "independent application modules?"

Independent application modules are isolated, static virtual machine environments (and their guests) created by the separation kernel which enable system architects to simplify their system designs by better managing software complexity inheritance. Certain applications may only require simpler bare-metal environments. Others may require more complex Linux environments. Application environments can be (1) bare-metal; (2) RTOS, Windows, or other 3rd-Party OS; (3) Linux. They are governed by an information flow policy that ensures system security and they cannot be dynamically altered during runtime.

Yes. LynxSecure is a Separation Kernel that sets up and locks-down device partitioning at boot, prior to the start of guest OS. All security policies, hardware partitioning, and inter-guest memory access privileges are defined according to the engineer’s needs, with few design impositions due to the hypervisor. This allows builders to fine-tune the levels of complexity and safety- or security-criticality for each application module and to certify each module—and only the module(s) that require certification—independently.

The Architecture Configuration Policy is setup on the host PC using a modeling language, and compiled on the host into a bootable binary. That binary is self-contained and configures the target resources as described in your model immediately on power-on. The privileged code that does that is eliminated after the job is done, leaving the system secure.

Firstly, LynxSecure runs Initial built in test (IBIT) and continuous built in test (CBIT) to validate the crucial hardware registers that control the partitioning of the system remain valid. By design it should be impossible to programmatically change these, so CBIT is intended for things like single-event upset (SEU), ie cosmic rays, but it also protects against a hardware faults or weakness (like rowhammer). Secondly, LynxSecure has an audit log. Guest VMs can be granted privilege to access the audit log and to receive an interrupt when events are stored. There are about 60 audit events. The events can be filtered and actions defined should you wish to halt or reset a VM that generates event X.

On x86 there is a module—a bare-metal virtual machine (VM)—called LSAstore that intercepts a block device (disk or partition) and provides transparent encrypted disk storage. But otherwise this is not a standalone feature because this can be trivially done with a bare-metal VM.

There is a LynxSecure demo where a VM is given access to an encrypted storage area and a Yubikey USB digital key to unlock it and provide secure access to control a robot via IPC (FIFO). Shmem or virtNet IPC is also supported.

On x86 there is a module—a bare-metal virtual machine (VM)—called LSAstore that intercepts a block device (disk or partition) and provides transparent encrypted disk storage. LSAstore uses the OpenSSL FIPS object module.

But, in general, LynxSecure does NOT contain any certified cryptographic libraries. It is deliberately minimal (no console, no create-VM APIs, no login).

No! There are no users. There are no drivers. The hypervisor only understands serial ports, VGA text-mode display, EHCI USB debugging (no USB stack), the virtualization technologies and control of the various system busses, including PCI. The only interface into the hypervisor - hypercalls - is very tightly controlled.

Safety certification of a separation kernel hypervisor is different because the HW register settings to implement the customer-specific multi-VM configuration are compiled into the hypervisor binary. Lynx uses a hybrid safety certification approach to certify LynxSecure. Lynx certifies the regular code of the hypervisor, such as startup code, interrupt handlers, context management, time management and scheduler following the classical DO-178C V-model (waterfall model). However, the HW register settings and page table data responsible for programming the HW to implement the customer specific configuration of VMs are certified following the DO-178C Parameter Data Item (PDI) approach. PDI allows a data file to be verified independently from the software if the data is in a form that is directly usable by the processing unit of the target computer. The PDI approach helps minimize LynxSecure's source lines of code (SLOC) count while enabling cost-effective certification of customer specific VM configurations. It allows the use of standard LynxSecure artifacts prepared by Lynx ahead of time for whatever choice of VM configuration the project chooses.

MfA provides 2 containerization options that may be used independently or in concert. Both enable remote 3rd party creation of binary payloads to be loaded and executed on the target. 

Hypervisor Dynamic VM Update makes use of existing MfA functionality to allow regular LynxSecure VMs to be dynamically stopped, replaced and restarted in a running system. Dynamic VM Update is controlled programmatically from a purpose-built orchestration application running as a LynxSecure VM. No remote orchestration infrastructure is provided. The orchestration application is set up with a library of pre-prepared VM images, either in its file system or network accessible (e.g., NFS, TFTP, SCP). The orchestration application runs in a privileged VM, with both write access to the boot memory regions of other VMs and control over restarting them. 

Standard Linux Containers; Buildroot Linux is a full featured embedded Linux distribution that is part of MfA and capable of running the Docker Engine and Linux infrastructure necessary to host and run 3rd party created containers. As an embedded Linux it is smaller than desktop Linux distributions, since its default configuration avoids a graphical desktop, office applications or a package manager. However, Buildroot Linux can run the necessary Docker components: 

• runC, the minimal CLI for running Linux containers
• docker-containerd, the daemon and API for runC
• docker-engine, the cli and api for the Docker application engine

A system configuration can use partitions to restrict access to resources and interpretation, and global system capabilities. On Intel based hardware platforms, LYNX MOSA.ic™ can support over a thousand partitions to create fine-grained access control policy enforcement.

Partition level authentication is achieved in the hypervisor through the monitoring of system management calls. Each system management call is handled by the hypervisor which contains the access control policy encoded as a permission lookup table created during the system build process. Partition access is granted by matching the hardware context ID of the calling partition against the authorized capabilities listed in the permission table. All system management calls are logged in a protected security log buffer. Policy violations can be programmed to trigger mode switching events to precisely recover from the event.   

The diagram below shows an example of a general-purpose LYNX MOSA.ic™ system configuration to provide clarity on how the BSP is composed to support the target system requirements. This general-purpose configuration specifies four virtual machines that are each allocated a dedicated CPU core, and the LynxOS-178 RTOS guest is loaded into each virtual machine. LYNX MOSA.ic™ permits many alternative options of configuration, allowing different CPU core allocation schemes and can host a variety of guest OS types 
Notice the indication of virtual machine (VM) types:
•    Application VM – Guest environment primarily used to host applications.
•    Control VM – Guest environment primarily used to host asynchronous drivers and hard real-time applications.
•    Data Service VM – Guest environment primarily used to host high speed DMA device drivers and device sharing services.
 


Figure 8 - General Platform Configuration

The virtual machine types are only a characterization profile of the VM. All VMs are permitted to host any device driver or application that the guest OS permits. This example shows that devices can be assigned to arbitrary VMs. It is important to note that for every device assignment, the hypervisor guarantees that the impact of hazardous events created by devices such as erroneous DMA and interrupt pre-emption, is constrained to the VM assigned to the device, protecting the integrity and timing of the other VMs. 

Lauterbach and Lynx have enjoyed a longstanding relationship over 10+ years and have performed engineering work to support prior versions of Lynx’s products. Lauterbach is currently performing engineering work to proof out their debugger series with LYNX MOSA.ic. More specific details on status and availability can be provided on request.

• Permissions for each virtual machine are established at boot up and are immutable
• The hypervisor, LynxSecure, is out of the dataplane during operation. Its role is to simply establish permissions for each virtual machine and ensure strong isolation between the applications that are executing in those virtual machines

The LynxSecure component of LYNX MOSA.ic is not an OS, is not based on an OS, and intentionally does not perform dynamic resource management. All subjects and resources allocated to subjects are fixed at build time according to a model. LynxSecure is thus immune to resource exhaustion and denial of service attacks.

The Buildroot Linux component has whatever protections against exhaustion or denial of service attacks are available in Linux, or in the applications written to run on Linux. The LynxOS-178 safety-critical RTOS supports multiple ARINC 653 partitions (up to 16), and resources (e.g. memory, file system nodes, processes, threads, etc.) and time allocated to one partition cannot be used or stolen by another partition.   

LynxSecure supports the use of a logging facility and can be used to capture events (e.g. a subject requesting inappropriate hypercalls or restarting). A subject with appropriate permissions can read from these logs and appropriate actions can be taken. When the system is booted, LynxSecure’s initialization code runs a startup BIT (SBIT) to verify the hardware is operating correctly. LynxSecure can also run the BIT tests as an initiated BIT (IBIT) and/or as a continuous BIT (CBIT) testing if desired.

Our NIST security artifacts are for LynxSecure on x86. Our business model is that a block of services comes with the artifacts to train users how to tweak them for additional hardware. This task is expected to be easy enough to do within the block-of-time for all but really unusual devices. It would be more work for Lynx to update the NIST package for Arm, possibly a few months of work. But, we are looking for an excuse to do that work, and would be delighted to undertake it should you be interested in NIST security artifacts for LynxSecure on Arm.

Four examples help to illustrate security aspects of the executable:

• There is no means to load any application or modify the LynxSecure executable
• Guests cannot access LynxSecure memory
• LynxSecure cannot access guest memory
• No shared kernel memory exists between guests and LynxSecure

PCI Base Address Register sharing is a feature where the LynxSecure hypervisor can create a special shared memory region to share portions of a PCI device’s address range. It can be used to share the HWTimeStamp registers of a PTP NIC, or the framebuffer of a graphics card for example. This allows other VMs to share the accurate hardware timer in a simple and efficient way. The VGA usecase allows a physical graphics card to be shared so that multiple VMs can have their own (reduced size) framebuffer and share the screen. Of simultaneous shared access to the screen is useless, the way this works is that LynxSecure provides a mechanism to switch which the screen between each VMs framebuffer.

For e.g. on an Intel x557 NIC, these 3 registers are shared in read only mode:

Tx Time Sync Control Register - TSYNCTXCTL (0x00008C00)

Tx Timestamp Value Low - TXSTMPL (0x00008C04)

Tx Timestamp Value High - TXSTMPH (0x00008C08)

PCI BAR sharing is done with the memregion autoconfig command. This example shares address 8000 of the PCI device NET0.IOMEM0 as a 0x1000 long memory region called NET0.TIMESYSNCREGS.

autoconfig mksrp $TARGET \

  --subject-vds0=NET0,...\

  --subject-fv0=NET0#1,memregion=NET0.TIMESYNCREGS:0x1000:RESERVED:RW:auto:auto:UNCACHEABLE:NET0.IOMEM0:00008000

The hwtimestamp registers are only available to the NIC PF (NET0), which is owned by the vds0 VM. The other VM, fv0, must use PCI BAR sharing to access the hwtimestamp because it is only assigned a virtual function (it has virtual function 1 of NET0, shown as NET0#1 in LynxSecure syntax).

• Resources and security policies are defined at boot
• User-space and zero copy memory ensures security-policy enforced guest-to-guest communication
• Data does not pass through LynxSecure
• Communications are Peer-to-Peer

We are familiar with and have supported customers that have used design tools like Simulink and SCADE to formally model systems. Architectural details that transcend functionality such as spatial, relational, and timing requirements of system subjects and objects will natively plug into the configuration interface of LynxSecure.

LynxSecure is developed and maintained in San Jose, California in accordance with FAA DO-178 Safety Quality Standards and United States DoD Risk Management Framework guidelines. LynxSecure is certified, fielded and maintained on classified DoD networks. The product has undergone many security assessments including penetration testing and design review by independent government security authorities.

Commonly, critical security function—such as a crypto algorithm, or data filter—and a unique information flow configuration must be established and protected to achieve a secure system. Critical functions can be placed in guests that leverage services of the separation kernel or independent trusted guests to monitor the work, liveliness, or integrity of the critical function. LynxSecure provides the following reference monitor features:

• Audit Event Manager;: the kernel contains a runtime state transition manager that can change the execution behavior of the system when a policy exception event occurs. Policy exceptions can come from guest generated hypercall events and hardware generated events – E.g. MMU/IOMMU exceptions. All events are recorded in an audit buffer.
• Watchdog: guests can be configured to strobe a virtual watchdog timer that the separation kernel will monitor. If a guest fails to strobe the watchdog in the permitted window of time the separation kernel will trigger a policy exception event.
• Signal: guests can us: ge the kernel to trigger hardware events to other guests to wake up interrupt handlers.
• Message: guests can use the kernel to send a 64 byte message to another guest in a single direction. Guests must be configured in the mandatory access control policy to have access to the messaging capability and must specifically declare which guest it is permitted to message.

Lynx developed Xilinx FPGA assisted boot and credential protection prototypes to serve as exemplar of fundamental boot and system initialization security design elements. Using the Xilinx SDK, Lynx integrated two security capabilities:

• Using the Xilinx bootgen utilities (pat of SDK), the LynxSecure image was processed to create a signature of Separation Kernel and guest images. This signature is used by the Xilinx MPSoC boot process to ensure the system image has not been subverted. It can only execute on a specific hardware target
• In addition to basic boot image and guest image integrity and authenticity protection, an additional capability allows guests to use the Xilinx encryption engine to protect keys used by guest applications

In addition to cert documents and workflows that we are happy to provide upon request (and in addition to offering NIST 800-53 security artifacts), LynxSecure offers other specific security features:

• An autoconfig option for the kernel and / or VMs, for Indirect Branch Restricted Speculation (IBRS) mitigation, that is, against Spectre variant 2
  
  
• An autoconfig option for the kernel and / or VMs, to prevent Foreshadow, also known as L1 Terminal Fault (L1TF)
  
  
• A high performance IRQ (interrupt) based audit system where the instant a VM attempts to access unauthorized memory or objects, it is frozen by LynxSecure and an IRQ generated into a privileged VM that can view a log and decide what action to take
  
  
• A hypervisor Module feature that allows binary modules to be added to the hypervisor at build time to add new hypercalls, callable by privileged VMs, to extend it for things like additional health monitoring, and custom firmware calls (like re-program FPGA)
  
  
• Memory sanitization. GOS memory is cleared before and after every guest restart. Extendable to sanitize user-resources as well
  
  
• Random number generator disable. VM access to the CPU’s built in Random number generator can be disabled to prevent it being used as a covert information channel

LynxSecure relies on an external bootloader, and that is where secure boot begins. Lynx does provide components from which a secure boot system can be built. Setting up secure boot is target specific. For example, we have a how-to for x86 that describes how to digitally sign your LynxSecure SRP binary and add keys to the BIOS to enable secure boot.

Following the signed SRP, secure boot can be continued using the LynxSecure segmented boot feature. Additional details regarding this feature can be shared upon request.

SR-IOV VF (virtual functions) do not support PTP. That is, the hwtimestamp capability, which is how PTP and 1588 work, is only available in the NIC’s PF (physical function).

IEEE 1588 (PTP) lets you synchronize clocks over a LAN. It is important to have a single master clock per network node (PC or target SoC). The standard operation of PTP is that it continuously disciplines (adjusts) slave clocks to keep them accurate with the master. Once a node (SoC) is disciplined, then you want to share that accurate clock with all the LynxSecure VMs running on it. This is where PCI BAR Sharing comes in. It is a LynxSecure feature that allows windows in the PCI Register Address range to be shared by multiple VMs. It does exactly what is required, it directly shares the PTP HW Clock registers with other VMs within a single target.

Lynx’s intention is to eventually have PTP support in LynxOS-178 guests running on top of LynxSecure. This is not available yet, but it means:

Extend the LynxOS-178 network stack to support PTP. This will allow it to be a PTP master (provide time) or PTP slave (receive time). At the moment a Linux guest can be used to import PTP time onto the SoC. Once the SoC has an accurate PTP clock, PCI BAR sharing is used to give other VMs access the timer registers. On LynxOS-178 the memmap driver is used to map them into user space, then the adjtime() API is called to synchronize the local VMs time.

Shared memory is a standard feature of LynxSecure. Memory windows of configurable size, writability, cache ability etc, can be defined and shared between multiple VMs. Accessing shared memory regions the VMs is done:

• In Linux we provide a driver that maps memregions as a device node into /dev/lsk/memregion.
• In bare metal applications we provide macros that assist querying the RO (read only) page so your code can discover shared memory regions.
• In LynxOS-178 the memmap driver is used to map the memory into userspace.

LynxOS and indeed LynxOS-178 are perfectly valid solutions. We have customers using it today for industrial applications including medical imaging equipment. FreeRTOS is the most widely used real-time operating system in the embedded industry. With an increasing amount of those systems becoming connected to the cloud, Lynx felt It was prudent to attach itself to the software ecosystems that are delivering this functionality. This is FreeRTOS (Amazon) and AzureRTOS (Microsoft). Since Lynx is a relatively new player in the industrial market, the pragmatic approach is to

• Show how companies can reuse their existing code bases around FreeRTOS
• Articulate a relatively quick an straightforward path to IEC61508 certification via Wittenstein that does this around FreeRTOS
• Ride on coattails of cloud companies offering up container frameworks, cloud connectivity etc rather than us doing it ourselves on our RTOS

Three primary boot “stages” to consider:

1. Power-on First Stage or Primary Bootloader
2. LynxSecure Separation Kernel Initialization and Boot
3. Guest OS Booting

 
Power-on First Stage or Primary Bootloader is in reference to the board’s native, power-on firmware and bootloader for the system, i.e. BIOS/EFI/SlimBoot or custom firmware and bootloader. It is the responsibility of this first stage bootloader to initialize the processor and board as needed and copy the LynxSecure System Runtime Package (SRP) binary boot image into main memory from whatever persistent storage device it is kept on, i.e. Flash, SSD, etc. Once loaded into memory, the first stage bootloader will jump to and hand-off control of the system to LynxSecure. Lynx has no control over this part of the boot sequence.


LynxSecure Separation Kernel Initialization and Boot is the time need by LynxSecure to initialize and configure the partitioning of the processor and system resources, plus the loading of the Guest OS images into their respective execution domains or “rooms”.


Guest OS Booting is the time needed by a particular Guest OS to boot to some meaningful startup point, which could be anything from when the OS init function is ready to start the first user specified application process to when a user’s application environment is fully operational and ready to perform its stated work. From a LynxOS-178 perspective, we can reasonably specify the startup time to the point at which the OS can launch the first user-specified process that is responsible for the remaining startup of the entire application.

 
Some factors that can greatly affect that boot time:

• Processor operating frequency
• DRAM operating frequency
• Amount of physical memory in the system
• Overall size and number of Guest OS subjects configured to run on the system
• Number of physical devices used and assigned to Guest OS subjects in the system configuration

The best resource to address this question is this technical white paper here.

Yes! LynxSecure enables a virtual machine to update software running in another VM at runtime by granting a privileged VM permission to write into the boot memory region of other VMs. The privileged VM is prepared with a library of OS images in its filesystem, or over the network (NFS, TFTP, SCP). It over-writes the OS image of the VM to be replaced and then makes a hypercall to reboot it which picks up and launches the new OS image. Dynamic VM (also referred to as “dynamic segmented boot” may be combined with multiple hypervisor schedules to allow a set of VMs to be staged with new OS images and then launched via a schedule switch. 

VLAN support was added to LYNX MOSA.ic for Avionics (MfA) 2022.12. A VLAN - virtual LAN - allows multiple network interfaces (eth0, eth1, etc) to be connected to a single physical NIC, allowing that NIC to have multiple IP addresses. A maximum of 4095 VLANs per trunk interface are supported. VLANs are represented as separate LCS network interfaces and identified as interface-name.VLAN-ID, e.g “eth0.100”, consistent with Linux and UNIX systems. VLANs are implemented by adding a VLAN Identification (ID) TAG into the Ethernet packet header. This field, as well as the VLAN Priority field, allows VLAN-aware switches and routers to direct and filter network traffic to create networks with the desired priority and architecture.

LynxOS-178 networking performance matches Linux on the same HW. Hypervisor slowdown is statistically insignificant. 30 NIC chipsets (greater than 30 NICs) are supported. PTP (IEEE 1588) is supported. VLAN and Time Sensitive Networking (TSN), IEEE 802.1AS, time-aware shaper (IEEE 802.1Qbv) and credit-based shaper (IEEE 802.1Qav) are supported in MFA 2022.12.

IC means “Integration Center”. We strong believe our focus should be assisting our customers to combine legacy software, Lynx technology, open source assets and 3^rd party applications in a safe and secure way.

No. Lynx offers a range of SKUs. Some of these include pre-integrated products from 3^rd parties. Others do not. Contact your point of contact in Lynx’s commercial team for more information.

LynxOS-178 for PowerPC was previously awarded a Reusable Software Certification (RSC) by the FAA, the first and only RTOS to do so. The LynxOS-178 safety-critical RTOS includes ARINC 653 Health Management services, restarting ARINC 653 partitions, or the RTOS should a fault occur.

The LynxSecure separation kernel has a logging and state transition machine that can cause a transition change from secure mode to insecure mode should an event occur. This can optionally change the system’s active runtime schedule (thus changing the core allocations to the subjects and allowing the system to continue operation, potentially with reduced functionality), restart the offending subject or the system, and/or shutdown the offending subject or the entire system should it be deemed necessary. 

We are working to migrate our XML based architecture configuration tool to MBSE/AADL to improve traceability to system/component level modeling. Lynx is collaborating w/multiple MBSE tool vendors around two things (a) migration of MBSE appropriate airworthiness/security lifecycle data to MBSE format (b) conversion of our LYNX MOSA.ic configuration tool to AADL.

Yes, it can. LynxSecure protects data in use by separating the memory regions of the different subjects at build time according to the customer’s system model. Lynx also partners with RunSafe Security to implement their technology into the runtime environment, for additional in-use data integrity. Data integrity at rest can be accomplished through the use of encryption subjects as previously done with Lynx’s LSA.store and LSA.connect add-on technologies for data at rest and data in transit respectively. These types of encryption subjects can leverage hardware encryption technologies and/or be layered on top of hardware encryption technologies, as well

Yes.

External monitoring: Critical functions can be placed in guests that leverage services of the separation kernel or independent trusted guests to monitor the work, liveliness, or integrity of the critical function.

Watchdog: guests can be configured to strobe a virtual watchdog timer that the separation kernel will monitor. If a guest fails to strobe the watchdog in the permitted window of time the separation kernel will trigger a policy exception event.  

Yes. The LynxOS-178 Health Monitor can restart an ARINC 653 partition or the OS instance itself, and the LynxSecure separation kernel’s system state machine and logging capability can be used to identify and correct a failing subject so appropriate action can be taken.