Implementing Python Bindings for Dust DDS with PyO3

Summary

Dust DDS is a native Rust implementation of the Data Distribution Service (DDS) middleware. While Rust is a great choice when doing systems programming, for its safety and performance, there are many use cases such as quick prototyping, testing and validation and interaction with other libraries which are better accomplished by Python. To enable those use-cases, Dust DDS provides a feature complete Python binding to its original Rust API. In this article we explore how we implemented the binding using PyO3, discuss the key decisions and trade-offs, and detail the process we used for generating documentation using the original API.

A brief introduction to Dust DDS

Data Distribution Service (DDS) is a middleware protocol and API standard designed for data-centric connectivity. At its core, DDS aims to facilitate the seamless sharing of pertinent data precisely where and when it’s needed, even across publishers and subscribers operating asynchronously in time. With DDS, applications can exchange information through the reading and writing of data-objects identified by user-defined names (Topics) and keys. One of its defining features is the robust control it offers over Quality-of-Service (QoS) parameters, encompassing reliability, bandwidth, delivery deadlines, and resource allocations. The DDS standard defines the Application Interfaces (APIs) that enable the efficient delivery of information from information producers to matching consumer.

Rust, with its strong safety guarantees, zero-cost abstractions, and high performance made it an ideal choice for implementing Dust DDS, our native Rust implementation of the DDS middleware.

However, despite Rust’s strengths, some use cases benefit from the simplicity and flexibility of Python. Python’s ease of use makes it the preferred language for rapid prototyping, testing, validation, and integration with third-party libraries. To make Dust DDS accessible for these scenarios, we created a fully-featured Python binding that mirrors the original Rust API. In this article, we’ll walk through how we implemented these bindings using the PyO3 crate, the choices and trade-offs made along the way, and how we generated comprehensive documentation.

Implementing the Dust DDS bindings

In this section we explain the process of creating the Dust DDS python bindings, starting from the choice of crate to generate the bindings, moving through the API design decisions and trade-offs and concluding with the generation of documentation.

Creating Python bindings from Rust

PyO3 is a mature Rust crate that facilitates the creation of Python bindings. It simplifies the process of exposing Rust functionality to Python by automatically handling many of the tedious details involved in foreign function interfacing. It boasts a nice list of well known and widely used projects like Polars so the choice here was relatively straightforward.

Binding implementation considerations

Our goal with the Dust DDS Python bindings was to maintain as much as possible API consistency between the Rust and Python versions of Dust DDS while still giving users a native Python experience. Since Dust DDS is a building block for high-performance, critical systems, we aim to keep the dependencies of the original crate to a minimum. This led us early on to creating a separate crate to realize the Python bindings. This crate is mainly composed of wrapper types on the original Dust DDS types and a copy of the funcions of the original API. Here is a snippet of code of the Domain Participant Python binding wrapper which gives an idea of the general structure:

#[pyclass]
pub struct DomainParticipant(dust_dds::domain::domain_participant::DomainParticipant);

impl From<dust_dds::domain::domain_participant::DomainParticipant> for DomainParticipant {
    fn from(value: dust_dds::domain::domain_participant::DomainParticipant) -> Self {
        Self(value)
    }
}

impl AsRef<dust_dds::domain::domain_participant::DomainParticipant> for DomainParticipant {
    fn as_ref(&self) -> &dust_dds::domain::domain_participant::DomainParticipant {
        &self.0
    }
}

#[pymethods]
impl DomainParticipant {
    #[pyo3(signature = (qos=None, a_listener=None, mask=Vec::new()))]
    pub fn create_publisher(
        &self,
        qos: Option<PublisherQos>,
        a_listener: Option<Py<PyAny>>,
        mask: Vec<StatusKind>,
    ) -> PyResult<Publisher> {
        let qos = match qos {
            Some(q) => dust_dds::infrastructure::qos::QosKind::Specific(q.into()),
            None => dust_dds::infrastructure::qos::QosKind::Default,
        };

        let listener: Option<
            Box<dyn dust_dds::publication::publisher_listener::PublisherListener + Send>,
        > = match a_listener {
            Some(l) => Some(Box::new(PublisherListener::from(l))),
            None => None,
        };
        let mask: Vec<dust_dds::infrastructure::status::StatusKind> = mask
            .into_iter()
            .map(dust_dds::infrastructure::status::StatusKind::from)
            .collect();

        match self.0.create_publisher(qos, listener, &mask) {
            Ok(p) => Ok(p.into()),
            Err(e) => Err(into_pyerr(e)),
        }
    }

    pub fn delete_publisher(&self, a_publisher: &Publisher) -> PyResult<()> {
        match self.0.delete_publisher(a_publisher.as_ref()) {
            Ok(_) => Ok(()),
            Err(e) => Err(into_pyerr(e)),
        }
    }
}

While this approach is more verbose and may require additional maintenance, it allows us to deviate from the original Rust implementation where necessary to provide a more intuitive Python interface. The example above demonstrates the use of default parameters, Python exceptions, and Python-specific types.

Error handling and other API considerations

Rust’s Result and Option types offer powerful error-handling mechanisms, but they don’t have direct equivalents in Python. To handle this, we converted Rust errors into Python exceptions, preserving the meaningful error messages of the original crate. This approach ensures a smooth Python experience while maintaining Rust’s error reporting integrity.

Default parameters are common in Python, so we implemented them wherever they made sense to provide a more idiomatic Python API.

Serialization and deserialization of the custom data types

A key feature of DDS is its ability to transmit user-defined data types, ensuring data safety and consistency across publishers and subscribers. In Rust, this is achieved by implementing specific type support traits. However, Python’s dynamic typing lacks an equivalent mechanism.

To address this, we use Python introspection capabilities to iterate through fields, types, and values to establish correct serialization/deserialization for each type. This is done by annotating struct fields with the desired types and decorating them with @dataclass. Here is an example:

@dataclass
class MyDataType:
    id : dust_dds.TypeKind.uint32
    state : dust_dds.TypeKind.boolean
    data: bytes
    msg: str
    def _key():
        return ["id"]

The _key function identifies the fields that form the key, allowing users to manage separate instances of the type.

Documentation Generation

Once the Python bindings were complete, we focused on generating comprehensive documentation and IDE helpers. We wanted Python users to have access to the same detailed, high-quality documentation as our Rust users.

Python documentation and type support are achieved by providing a .pyi file alongside the library. We wrote a custom build.rs script using the syn crate to parse the original Dust DDS Rust code and generate the .pyi file. Since our Python bindings interface mirrors the Rust API, this approach allowed us to synchronize documentation updates across both languages, reducing the risk of mismatches or outdated references.

The code used to generate the documentation mostly concerns parsing details and is too broad to discuss in this article but you can find it on our Dust DDS repo.

Installation and usage

Installing Dust DDS is straightforward, since it is available on PyPI. You can easily add Dust DDS to your Python environment by running the following command:

pip install dust-dds

This doesn’t require any further compilation or dependencies and is available on all major platforms. To help you get started quickly, you can refer to the example on PyPi or the test cases provided in our Dust DDS repository.

Conclusion

By creating Python bindings for Dust DDS using PyO3, we combined the performance and safety of Rust with the flexibility and ease of Python. By bridging these two ecosystems, Dust DDS can now support a wider range of applications, from high-performance, safety-critical systems in Rust to rapid prototyping and integration with Python’s vast ecosystem of libraries and tools. We already made use of these bindings to create an exciting Edge AI face recognition application with the Jetson Nano. Now we look forward to seeing what our users build with it!