Introducing Dust DDS – A native Rust implementation of the Data Distribution Service (DDS) middleware

Summary

This article introduces Dust DDS, a native Rust implementation of the Data Distribution Service (DDS) middleware and explores the trade-offs for the design and implementation of the API and the internals of the library. This introduction covers the following topics:

1. DDS API mapping and implementation: The DDS standard defines the API method inside interfaces in IDL. We explored the possibility of mapping this in Rust to traits with associated types, traits with methods returning opaque types and implementing the methods for objects. For the ease of use and implementation we have opted for the solution in which the DDS entities are objects, and the API are methods implemented on those objects.
2. DDS Listener mapping: The DDS standard defines the mechanism of Listener which can be installed on the different entities to react on events. In Dust DDS the Listeners interfaces are provided as traits and the user-created objects can be installed on the entities as optional Boxed trait objects.
3. DDS Reader and Writer type safety and interoperability: The DDS standards specifies that the Reader and Writer objects must be type safe and allow only to publish/subscribe the type they are originally created for. In the Dust DDS implementation this is achieved with generics on the reader and writer. For a custom type to be transmitted on the wire, it must implement the type support traits. Dust DDS provides a derive macro for easy implementation of these traits suitable for most use cases. The wire protocol is RTPS over UDP for interoperability between different vendors.
4. Internal library architecture: For the library internal architecture we have explored implementations with fine-grained locking, participant-level locking and the actor model. After observing the high risk of deadlocks in the fine-grained locking implementation, the high lock contention and low performance of the participant-level locking we have settled on the actor model using Tokio as the runtime.
5. Async and Sync API: Given the API method definitions in the DDS standard and the event-driven nature of DDS with listeners and wait sets, Dust DDS regards the “Sync” API as the primary interface. However, given that the actor model we have used for the internal implementation is inherently async and there are many relevant Rust use cases in which DDS might be integrated into async applications an identical version of the async API is provided. Ultimately the sync API achieves its synchronous characteristics by using blocking calls to the analogous async methods, with object mappings as needed.

Introduction

In this article we are introducing Dust DDS, a native Rust implementation of the OMG Data Distribution Service (DDS) middleware. We will give a brief introduction to DDS and the relevant standards and walk through the high-level architecture and main trade-offs we have made when implementing the specification.

DDS is a middleware protocol and API standard designed for data-centric connectivity. At its core, DDS aims to facilitate the seamless sharing of pertinent data precisely where and when it’s needed, even across publishers and subscribers operating asynchronously in time. With DDS, applications can exchange information through the reading and writing of data-objects identified by user-defined names (Topics) and keys. One of its defining features is the robust control it offers over Quality-of-Service (QoS) parameters, encompassing reliability, bandwidth, delivery deadlines, and resource allocations.

The DDS standard “defines both the Application Interfaces (APIs) and the Communication Semantics (behaviour and quality of service) that enable the efficient delivery of information from information producers to matching consumer”. Complementing this standard is the DDSI-RTPS specification, which defines an interoperability wire protocol for DDS. Its primary aim is to ensure that applications based on different vendors’ implementations of DDS can interoperate. The implementation of Dust DDS primarily centres around the DDS and DDSI-RTPS standards.

At its core DDS is organized around entities which handle the information flow in the system with Publishers and Data Writers on the sending side and Subscribers and Data Readers on the receiving side. A Data Writer acts as a typed accessor to a publisher. The DataWriter is the object the application must use to communicate to a publisher the existence and value of data-objects of a given type. To access the received data, the application must use a typed Data Reader attached to the Subscriber. Topic objects conceptually fit between publications and subscriptions. A Topic associates a name with (unique in the domain) with a data-type. Each Data Reader and Data Writer must have a Topic associated with it. All these entities are attached to a Domain Participant which represents the local membership of the application in a domain. A domain is a distributed concept that links all the applications able to communicate with each other. It represents a communication plane: only the publishers and the subscribers attached to the same domain may interact. The Figure below gives a summary of this architecture.

DDS API mapping and implementation

The DDS standards defines the Application Programming Interface (API) for the entities that allow an application to interact with the DDS databus. The methods of the API are defined using the OMG Interface Description Language (IDL). For the entities these methods are defined in IDL as interfaces. Here is an example of a function of the API in IDL.

interface DomainParticipant : Entity {
    Publisher create_publisher(
        in PublisherQos qos,
        in PublisherListener a_listener,
        in StatusMask mask);
}

Regarding the methods, the description defines that a nil pointer should be returned in case of failure when an object is created. Since Rust provides error handling by returning, we have opted for returning the Result type throughout the API except where a ReturnCode is not required. The methods that do not have a ReturnCode are implemented as infallible. Here is the mapping for the example method.

pub fn create_publisher(&self, qos: PublisherQoS, a_listener: PublisherListener, mask: StatusMask) -> Result<Publisher, DdsError>;

For the implementation of these methods, the most natural mapping of an IDL interface to Rust would be a trait. However, to use those methods would require the traits to be imported by the end user. Since Rust 1.75 it is also possible to return opaque objects from methods in traits. We considered this possibility for the API design but since the traits would also have to be imported for using the methods and it is not easy to store opaque objects in other struct we have opted for what we believe is the simplest solution from a Rust usage perspective. The following code gives simplified examples of all the different options.

/// This is the solution used in Dust DDS. Here the entities are represented as structs and the API is implemented as methods on those structs. This is the simplest solution to use and probably the least verbose to implement. It exposes the implementation objects and all theirs methods to the users of the library.
pub struct DomainParticipant;
pub struct Publisher;

impl DomainParticipant {
    pub fn create_publisher(&self, qos: PublisherQoS, a_listener: PublisherListener, mask: StatusMask) -> Result<Publisher, DdsError> {
        todo!()
    }
}

/// In this solution the entities are represented as structs and their methods are traits which are implemented on those structs. It decouples the API definition from the implementation but is more verbose, requires more names and the user has to bring the traits into scope for the methods they want to use. We decided agains it since many traits would have to be imported for the user to call the methods in the object.
pub struct DomainParticipantImpl;
pub struct PublisherImpl;

pub trait DomainParticipant {
    type Publisher;

    fn create_publisher(&self, qos: PublisherQoS, a_listener: PublisherListener, mask: StatusMask) -> Result<Self::Publisher, DdsError>;
}

impl DomainParticipant for DomainParticipantImpl {
    type Publihser = PublisherImpl;

    fn create_publisher(&self, qos: PublisherQoS, a_listener: PublisherListener, mask: StatusMask) -> Result<Self::Publisher, DdsError> {
        todo!()
    }
}

/// In this solution the entities are opaque object around the traits representing the API. This reduces the verbosity compared to the previous implmentation but still requires the user to bring the traits into scope for using the methods. On top of this it makes it also complicated to store the DDS objects in other structs.
pub struct DomainParticipantImpl;

pub trait DomainParticipant {
    fn create_publisher(&self, qos: PublisherQoS, a_listener: PublisherListener, mask: StatusMask) -> Result<impl Publisher, DdsError>;
}

pub trait Publisher{}

impl DomainParticipant for DomainParticipantImpl {
    fn create_publisher(&self, qos: PublisherQoS, a_listener: PublisherListener, mask: StatusMask) -> Result<impl Publisher, DdsError> {
        todo!()
    }
}

DDS Listener mapping

The DDS Listener mechanism allows the user application to execute functions based on defined DDS events such as on_data_available or on_requested_deadline_missed. These listeners are defined in the IDL API as interfaces. Here is a simplified example of the Data Reader Listener IDL definition.

interface DataReaderListener {
    void on_requested_deadline_missed(
        in DataReader the_reader,
        in RequestedDeadlineMissedStatus status);

    void on_data_available(in DataReader the_reader);
}

These Listeners are mapped in Dust DDS as traits. In this case the decision is more clear-cut than for the rest of the API since it is up to the user of the library to implement this functionality. Since it is often the case that only a subset of the listener methods are of interest we provided an empty default implementation. Following the description of the DDS standard, the Listeners provide the Data Reader object to enable the user to directly call DDS functions such as read() within the listener function.

pub trait DataReaderListener: 'static {
    /// Type of the DataReader with which this Listener will be associated.
    type Foo;

    fn on_requested_deadline_missed(
        &mut self,
        _the_reader: DataReader<Self::Foo>,
        _status: RequestedDeadlineMissedStatus,
    ) {
    }

    fn on_data_available(&mut self, _the_reader: DataReader<Self::Foo>) {}
}

Once the user has created the listener objects they can be installed in the respective entities by passing an Option<Box<dyn DataReaderListener>> object either on construction or by calling the set_listener method on the entity.

DDS Reader and Writer type safety and interoperability

The DDS standard guarantees type safety on the reader and writer to ensure at compile time that only the type that is specified for the actual reader/writer can be used to publish/subscribe data. On Dust DDS this is achieved by using generics with trait bounds on the DataReader<Foo> and DataWriter<Foo> objects.

For a user type to be published or subscribed by Dust DDS it must implement the type support traits which inform the middleware how the type can be serialized and deserialized and whether it has a key or not. Using traits offers maximum flexibility because they can be automatically implemented by using the #[derive(DdsType)] macro provided by Dust DDS. If a special serialization is needed the trait can be manually implemented by the user.

On the wire, Dust DDS implements the RTPS wire protocol over the UDP transport which ensures interoperability with different vendors.

Internal library architecture

The implementation of the internal architecture of the DDS library is challeging because information is being added at runtime simultaneously from the user API and from the network interface and both of these data flows affect the same reader and writer objects. Additionally to this it must be possible to trigger events on listeners and wait sets. To implement this functionality we have experimented with three types of architecture:

1. Fine-grained locking: In a fine-grained locking architecture, locks are applied at a very granular level, often at the level of individual fields or smaller units of data within a data structure such as a struct or an object. Since each lock guards access to a very section of data, it allows for more concurrency and potentially reduces contention. The drawback is that it is easier to get into deadlock situations.
2. Participant-level locking: In a participant-level locking model a single lock is applied to the Domain Participant since this is the main entry point for DDS entities. This simplifies the structure of fine-grained locking at the cost of increasing the amount of contention and reducing performance.
3. Actor model: In the actor model, concurrent computation is structured around autonomous units called actors. Each actor encapsulates state and behaviour, communicating exclusively through asynchronous message passing. The actors can execute concurrently without the need for explicit locks since they process messages sequentially, which ensures thread safety and minimizes the risk of race conditions. The drawback is the additional overhead of message passing and the additional structs needed to implement the model.

After evaluating the three possibilities, we have discarded both lock based architectures. Fine-grained locking caused very hard to predict deadlock problems due to the high-number of locks and undefined order of operations. Problems were particularly prevalent when listeners were used. Participant-level locking was simpler to implement but introduced too much lock contention which made the performance too slow even for very typical DDS use cases. Therefore we have picked the actor model for the internal implementation of Dust DDS.

To implement the actor model we have decided that each method of the actor object should represent a message that can be trasmitted to the actor. To ensure consistency in the implementation a proc macro was created to generated the messages and convinience methods that send the message to the actor with the arguments needed to invoke the method and a one-shot channel for receiving the output reply. Here is a really good article giving an overview of actor with Tokio which explains much of the implementation.

Async and Sync API

In Rust development, a significant consideration is whether to offer “sync” or “async” methods within library APIs. Given the API method definitions in the DDS standard IDL and the event-driven nature of DDS with listeners and wait sets, the Sync API serves as the primary interface to Dust DDS.

However, recognizing scenarios where users may wish to integrate DDS functionality into existing asynchronous environments, such as web servers, we’ve opted to provide both variants of the API. These two APIs are largely a duplication, differing only in the coloring of functions as async. Notably, the internal architecture of the Dust DDS middleware inherently employs bounded message channels, facilitating asynchronous behaviour. The Sync API achieves its synchronous characteristics by using blocking calls to the analogous async methods, with object mappings as needed.