The rclc package

Table of contents

• Introduction

• Analysis of rclcpp standard Executor
  • Architecture
  • Scheduling Semantics
• Analysis of processing patterns
  • Sense-plan-act pipeline in robotics
  • Synchronization of multiple rates
  • High-priority processing path
  • Real-time embedded applications
• rclc Executor
  • Features
    • Trigger condition
    • Sequential execution
    • LET-Semantics
    • Multi-threading and scheduling configuration
  • Executor API
    • Configuration phase
    • Running phase
  • Examples
    • Sense-plan-act pipeline in robotics example
    • Synchronization of multiple rates example
    • High-priority processing path example
    • Real-time embedded applications example
    • ROS 2 Executor Workshop Reference System
  • Future work
  • Download
• Callback-group-level Executor
  • API Changes
  • Test Bench
• Related Work
  • Fawkes Framework
• References

Introduction

Predictable execution under given real-time constraints is a crucial requirement for many robotic applications. While the service-based paradigm of ROS allows a fast integration of many different functionalities, it does not provide sufficient control over the execution management. For example, there are no mechanisms to enforce a certain execution order of callbacks within a node. Also the execution order of multiple nodes is essential for control applications in mobile robotics. Cause-effect-chains comprising of sensor acquisition, evaluation of data and actuation control should be mapped to ROS nodes executed in this order, however there are no explicit mechanisms to enforce it. Furthermore, when data recordings collected in field tests as ROS-bags are re-played, then the results are often surprisingly different due to non-determinism of process scheduling.

Manually setting up a particular execution order of subscriptions and publishing topics as well as defining use-case specific priorities of the corresponding Linux processes is always possible. However, this approach is error-prone, difficult to extend and requires an in-depth knowledge of the deployed ROS 2 packages in the system.

Therefore the goal of the Executor in micro-ROS is to support roboticists with practical and easy-to-use real-time mechanisms which provide solutions for:

• Deterministic execution
• Real-time guarantees
• Integration of real-time and non real-time functionalities on one platform
• Specific support for RTOS and microcontrollers

In ROS 1 a network thread is responsible for receiving all messages and putting them into a FIFO queue (in roscpp). That is, all callbacks were called in a FIFO manner, without any execution management. With the introduction of DDS (data distribution service) in ROS 2, the messages are buffered in DDS. In ROS 2, an Executor concept was introduced to support execution management. At the rcl-layer, a wait-set is configured with handles to be received and in a second step, the handles are taken from the DDS-queue. A handle is a generic term defined in rcl-layer for timers, subscriptions, services, clients and guard conditions.

The standard implementation of the ROS 2 Executor for the C++ API (rclcpp) has, however, certain unusual features, like precedence of timers over all other DDS handles, non-preemptive round-robin scheduling for non-timer handles and considering only one input data for each handle (even if multiple could be available). These features have the consequence, that in certain situations the standard rclcpp Executor is not deterministic and it makes guaranteeing real-time requirements very hard [CB2019]. We have not looked at the ROS 2 Executor implementation for Python Frontend (rclpy) because we consider a micro-controllers platform, on which typically C or C++ appliations will run.

Given the goals for a Real-Time Executor and the limitations of the ROS 2 standard rclcpp Executor, the challenges are:

• to develop an adequate and well-defined scheduling mechanisms for the ROS 2 framework and the real-time operating system (RTOS)
• to define an easy-to-use interface for ROS developers
• to model requirements (like latencies, determinism in subsystems)
• mapping of ROS 2 framework and operating system schedulers (semi-automated and optimized mapping is desired as well as generic, well-understood framework mechanisms)

Our approach is to provide a real-time-capable Executor for the rcl+rclc layer (as described in section Introduction to Client Library.) in the C programming language .

As the first step, we propose the rclc Executor for the rcl-layer in C programming language with several new features to support real-time and deterministic execution: It supports 1.) user-defined static sequential execution, 2) conditional execution semantics, 3) multi-threaded execution with scheduling configuration, and 4) logical execution semantics (LET). Sequential execution refers to the runtime behavior, that all callbacks are executed in a pre-defined order independent of the arrival time of messages. Conditional execution is available with a trigger condition which enables typical processing patterns in robotics (which are analyzed in detail in section Analysis of processing patterns. Configuration of scheduling parameters for multi-threaded application accomplishes prioritized execution. The logical execution time concept (LET) provides data synchronization for fixed periodic task scheduling of embedded applications.

Beyond the advanced execution management mechanisms for micro-ROS, we also contributed to improving and extending the Executor concept in rclcpp for standard ROS 2: the callback group-level Executor. It is not a new Executor but rather a refinement of the ROS 2 Executor API allowing to prioritize a group of callbacks which is not possible with the ROS 2 default Executor in its current Galactic release.

Analysis of rclcpp standard Executor

ROS 2 allows to bundle multiple nodes in one operating system process. To coordinate the execution of the callbacks of the nodes of a process, the Executor concept was introduced in rclcpp (and also in rclpy).

The ROS 2 design defines one Executor (instance of rclcpp::executor::Executor) per process, which is typically created either in a custom main function or by the launch system. The Executor coordinates the execution of all callbacks issued by these nodes by checking for available work (timers, services, messages, subscriptions, etc.) from the DDS queue and dispatching it to one or more threads, implemented in SingleThreadedExecutor and MultiThreadedExecutor, respectively.

The dispatching mechanism resembles the ROS 1 spin thread behavior: the Executor looks up the wait sets, which notifies it of any pending callback in the DDS queue. If there are multiple pending callbacks, the ROS 2 Executor executes them in an in the order as they were registered at the Executor.

Architecture

The following diagram depicts the relevant classes of the standard ROS 2 Executor implementation:

Note that an Executor instance maintains weak pointers to the NodeBaseInterfaces of the nodes only. Therefore, nodes can be destroyed safely, without notifying the Executor.

Also, the Executor does not maintain an explicit callback queue, but relies on the queue mechanism of the underlying DDS implementation as illustrated in the following sequence diagram:

The Executor concept, however, does not provide means for prioritization or categorization of the incoming callback calls. Moreover, it does not leverage the real-time characteristics of the underlying operating-system scheduler to have finer control on the order of executions. The overall implication of this behavior is that time-critical callbacks could suffer possible deadline misses and a degraded performance since they are serviced later than non-critical callbacks. Additionally, due to the FIFO mechanism, it is difficult to determine usable bounds on the worst-case latency that each callback execution may incur.

Scheduling Semantics

In a recent paper [CB2019], the rclcpp Executor has been analyzed in detail and a response time analysis of cause-effect chains has been proposed under reservation-based scheduling. The Executor distinguishes four categories of callbacks: timers, which are triggered by system-level timers, subscribers, which are triggered by new messages on a subscribed topic, services, which are triggered by service requests, and clients, which are triggered by responses to service requests. The Executor is responsible for taking messages from the input queues of the DDS layer and executing the corresponding callback. Since it executes callbacks to completion, it is a non-preemptive scheduler, However it does not consider all ready tasks for execution, but only a snapshot, called readySet. This readySet is updated when the Executor is idle and in this step it interacts with the DDS layer updating the set of ready tasks. Then for every type of task, there are dedicated queues (timers, subscriptions, services, clients) which are processed sequentially. The following undesired properties were pointed out:

File truncated at 100 lines see the full file

Changelog for package rclc

2.0.6 (2022-01-25)

• [backport galactic, foxy] data_available optimization (#212)
• Fix data_available reset for timer (#215) (#216)
• Executor ignore canceled timers (#220) (#221)
• Resolved error in unit test see issue #230 (#231) (#232)
• Updated documentation README.md (#229)

2.0.5 (2021-11-08)

• Fix printf in executor spin
• Fix init options handling

2.0.4 (2021-08-19)

• Refactoring: remove callback_type
• Improvement: Checking for valid ROS context in spin_some
• Bug fix: Ignoring unsuccessful SERVICE_TAKE
• Bug fix: Updated ci workflow dependency on galactic
• Improvement: Updated codecov configuration to ignore unit tests

2.0.3 (2021-07-26)

• Updated codecov to ignore test folders
• Updated bloom release status table

2.0.2 (2021-07-17)

• Added rclc_parameter package
• Added quality of service entity creation API
• Addded executor_prepare API
• Added support for removing subscription from executor
• Added support for subscription with context
• Updated compatability function for sleep
• Removed duplicate NOTICE files

2.0.1 (2021-05-28)

• added quality declaration

2.0.0 (2021-04-23)

• added codecov support
• new API of rcl_lifecycle in Rolling required major version bump

1.0.1 (2021-03-29)

• Windows port
• Compatibility sleep function (Windows, POSIX-OS)
• Fixed RCL lifecycle API change for Rolling

1.0.0 (2021-03-04)

• service callbacks with context
• fixed minor issues unit tests
• upgraded setup_ros action (ci jobs)
• removed Eloquent from ci jobs

0.1.7 (2021-01-20)

• Corrected corrupted changelog

0.1.6 (2021-01-20)

• Fixed issues due to Github Action timing delays on cloud build
• Fixed missing package dependency in Github Action for Eloquent

0.1.5 (2020-12-11)

• Added support for services,clients and guard_conditions to rclc executor
• Added table for bloom release status in README file

0.1.4 (2020-11-25)

• Fixed error in bloom release

0.1.3 (2020-11-23)

• Added rclc_lifecycle package
• Change maintainer information
• Minor fixes, updated unit tests

0.1.2 (2020-05-19)

• Fixed compiler errors for bloom release

0.1.1 (2020-05-14)

• Initial release