ros2_medkit_fault_manager

Central fault manager node for the ros2_medkit fault management system.

Overview

The FaultManager node provides a central point for fault aggregation and lifecycle management. It receives fault reports from multiple sources, aggregates them by fault_code, and provides query and clearing interfaces.

Quick Start

By default, faults are confirmed immediately when reported - no additional configuration needed.

# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py

# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
  "{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"

# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
  "{statuses: ['CONFIRMED']}"

# Clear a fault
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
  "{fault_code: 'MOTOR_OVERHEAT'}"

Services

Features

• Multi-source aggregation: Same fault_code from different sources creates a single fault
• Occurrence tracking: Counts total reports and tracks all reporting sources
• Severity escalation: Fault severity is updated if a higher severity is reported
• Persistent storage: SQLite backend ensures faults survive node restarts
• Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity threshold overrides
• Snapshot capture: Captures topic data when faults are confirmed for debugging (snapshots are deleted when fault is cleared)
• Fault correlation (optional): Root cause analysis with symptom muting and auto-clear

Parameters

Snapshot Parameters

Snapshots capture topic data when faults are confirmed for post-mortem debugging.

Topic Resolution Priority:

1. fault_specific - Exact match for fault code (configured via YAML config file)
2. patterns - Regex pattern match (configured via YAML config file)
3. default_topics - Fallback for all faults

Example YAML config file (snapshots.yaml):

fault_specific:
  MOTOR_OVERHEAT:
    - /joint_states
    - /motor/temperature
patterns:
  "MOTOR_.*":
    - /joint_states
    - /cmd_vel

Storage Backends

SQLite (default): Faults are persisted to disk and survive node restarts. Uses WAL mode for optimal performance.

Memory: Faults are stored in memory only. Useful for testing or when persistence is not required.

Usage

File truncated at 100 lines see the full file

Changelog for package ros2_medkit_fault_manager

0.4.0 (2026-03-20)

• Per-entity confirmation and healing thresholds via manifest configuration (#269)
• Default rosbag storage format changed from sqlite3 to mcap
• Support for namespaced fault manager nodes - gateway resolves service/topic names when the fault manager runs in a custom namespace
• Build: use shared cmake modules from ros2_medkit_cmake package
• Build: centralized clang-tidy configuration
• Contributors: \@bburda

0.3.0 (2026-02-27)

• Accurate HIGHEST_SEVERITY reassignment and stale fault_to_cluster_ cleanup (#221)
• Clean up pending_clusters_ when fault cleared before min_count (#211)
• Multi-distro CI support for ROS 2 Humble, Jazzy, and Rolling (#219, #242)
• Contributors: \@bburda, \@eclipse0922

0.2.0 (2026-02-07)

• Initial rosdistro release
• Central fault management node with ROS 2 services:
  • ReportFault - report FAILED/PASSED events with debounce filtering
  • GetFaults - query faults with filtering by severity, status, correlation
  • ClearFault - clear/acknowledge faults
• Debounce filtering with configurable thresholds:
  • FAILED events decrement counter, PASSED events increment
  • Configurable confirmation_threshold (default: -1, immediate)
  • Optional healing support (healing_enabled, healing_threshold)
  • Time-based auto-confirmation (auto_confirm_after_sec)
  • CRITICAL severity bypasses debounce
• Dual storage backends:
  • SQLite persistent storage with WAL mode (default)
  • In-memory storage for testing/lightweight deployments
• Snapshot capture on fault confirmation:
  • Topic data captured as JSON with configurable topic resolution
  • Priority: fault_specific > patterns > default_topics
  • Stored in SQLite with indexed fault_code lookup
  • Auto-cleanup on fault clear
• Rosbag capture with ring buffer:
  • Configurable duration, post-fault recording, topic selection
  • Lazy start mode (start on PREFAILED) or immediate
  • Auto-cleanup of bag files, storage limits (max_bag_size_mb)
  • GetRosbag service for bag file metadata
• Fault correlation engine:
  • Hierarchical mode: root cause to symptom relationships
  • Auto-cluster mode: group similar faults within time window
  • YAML-based configuration with pattern wildcards
  • Muted faults tracking, auto-clear on root cause resolution
• FaultEvent publishing on ~/events topic for SSE streaming
• Wall clock timestamps (compatible with use_sim_time)
• Contributors: Bartosz Burda, Michal Faferek