Package Summary
| Version | 0.7.2 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Description | |
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-11-22 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_fault_manager
Central fault manager node for the ros2_medkit fault management system.
Overview
The FaultManager node provides a central point for fault aggregation and lifecycle management.
It receives fault reports from multiple sources, aggregates them by fault_code, and provides
query and clearing interfaces.
Quick Start
By default, faults are confirmed immediately when reported - no additional configuration needed.
# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py
# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
"{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"
# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
"{statuses: ['CONFIRMED']}"
# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"
# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"
Note: The
skip_correlation_auto_clearrequest field was added post-0.4.0. Adding a request field changes the service type hash, so callers built againstros2_medkit_msgs0.4.0 or earlier must rebuild to keep talking tofault_manager.
Services
| Service | Type | Description |
|---|---|---|
~/report_fault |
ros2_medkit_msgs/srv/ReportFault |
Report a fault occurrence |
~/list_faults |
ros2_medkit_msgs/srv/ListFaults |
Query faults with filtering |
~/clear_fault |
ros2_medkit_msgs/srv/ClearFault |
Clear/acknowledge a fault |
~/get_snapshots |
ros2_medkit_msgs/srv/GetSnapshots |
Get topic snapshots for a fault |
Features
-
Multi-source aggregation: Same
fault_codefrom different sources creates a single fault - Occurrence tracking: Counts outages, not reports - the count starts at one and rises only when a cleared fault is raised again - and tracks all reporting sources
- Severity escalation: Fault severity is updated if a higher severity is reported
- Persistent storage: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
- Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
-
Snapshot capture: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless
snapshots.retain_on_clearis set) - Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
-
Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across
clear_fault(see below) - Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
- Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
storage_type |
string | "sqlite" |
Storage backend: "sqlite", "memory" or "postgres"
|
database_path |
string | "/var/lib/ros2_medkit/faults.db" |
Path to SQLite database file |
database_url |
string | "" |
PostgreSQL connection string; empty = libpq environment variables |
confirmation_threshold |
int | -1 |
Counter value at which faults are confirmed |
healing_enabled |
bool | false |
Enable automatic healing via PASSED events |
healing_threshold |
int | 3 |
Counter value at which faults are healed |
auto_confirm_after_sec |
double | 0.0 |
Auto-confirm PREFAILED faults after timeout (0 = disabled) |
report_fault_queue_depth |
int | 100 |
Requests ~/report_fault holds until read (1-65536, read-only); past it they are dropped without an answer |
entity_thresholds.config_file |
string | "" |
Path to YAML file with per-entity debounce threshold overrides |
fault_thresholds.config_file |
string | "" |
Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
near_miss.max_per_fault |
int | 200 |
Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
Snapshot Parameters
Snapshots capture topic data when faults are confirmed for post-mortem debugging.
Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.
Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.
That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.
| Parameter | Type | Default | Description |
|---|---|---|---|
snapshots.enabled |
bool | true |
Enable/disable snapshot capture |
snapshots.background_capture |
bool | false |
Cache each configured topic’s latest message vs on-demand capture; a topic absent at startup is cached once it appears |
snapshots.timeout_sec |
double | 1.0 |
Timeout waiting for topic message (on-demand mode) |
snapshots.max_message_size |
int | 65536 |
Maximum message size in bytes (larger messages skipped) |
snapshots.default_topics |
string[] | [] |
Topics to capture for all faults |
snapshots.config_file |
string | "" |
Path to YAML config for fault_specific and patterns
|
snapshots.recapture_cooldown_sec |
double | 60.0 |
Min seconds between captures for the same fault code. |
snapshots.max_per_fault |
int | 10 |
Max snapshots retained per fault. |
snapshots.capture_pool_size |
int | 2 |
Max concurrent capture threads under a fault storm (>= 1). Parallelizes snapshot capture only; rosbag stays single-writer. |
snapshots.capture_queue_depth |
int | 16 |
Max pending captures before the full-queue policy applies (>= 1). |
snapshots.capture_queue_full_policy |
string | reject_newest |
Policy when the queue is full: reject_newest or drop_oldest. |
File truncated at 100 lines see the full file
Changelog for package ros2_medkit_fault_manager
0.7.2 (2026-11-19)
- No changes. Version aligned with the rest of the stack
0.7.1 (2026-11-18)
- Recordings: every bag gets a unique path and the recording just stored is kept, a recording is hashed and stored off the executor thread, and a clear that arrives while a row is being stored takes that row (#785)
- A recording's reported size is the sum of the storage files its metadata names, and a bag's metadata is followed only to a storage file inside the bag. The snapshots payload no longer advertises a route that was removed (#679)
-
report_faultholds a burst of requests from one reporter instead of dropping part of it (#789) - A fault code can be debounced by the code, not only by who reported it (#690)
- A background snapshot topic that does not exist at startup is subscribed once it appears (#775)
- Fault clusters: the cluster severity stays current on re-reports, an active cluster's representative is reassigned without a pending cluster (#772), and the positive debounce counter is reset when a critical fault confirms immediately (#773)
- A time-based confirmation is as visible as a reported one, and
muting is honoured on the timer path.
healing_thresholdis 0 (it was 3) inconfig/fault_manager.yamland in the gateway'sconfig/bringup_params.yaml.bringup_params.yamlturns healing on, so with it one PASSED report now heals a fault: a fault from the action status bridge heals when its action recovers, where 0.7.0 kept it CONFIRMED.fault_manager.yamlkeeps healing off, so there the new value counts once healing is turned on (#643) - Optional PostgreSQL storage backend, off by default
(
-DPOSTGRES_SUPPORT=ON). It reconnects after a database outage (#651) -
Breaking (C++ API): the
FaultStoragevirtuals infault_storage.hppchanged:report_fault_eventtakesReportKindandint64_tnanoseconds, andcheck_time_based_confirmationtakesint64_t. A storage written for 0.7.0 no longer overrides them. Storage, thresholds, correlation, the audit log and the report, clear, list and get handling live in a ROS-free core (#740, #741, #742, #743, #744) - Parameters have upper bounds, for example 64 for
snapshots.capture_pool_size(#742, #744) - Contributors: \@bburda, \@mfaferek93, \@ahmedsleem109, \@gstavrinos, \@onur-ulusoy, \@Akinzou
0.7.0 (2026-08-27)
- Rosbag black-box recordings are no longer limited to one per fault
code. A fault that re-confirms keeps a bounded history of recordings
instead of overwriting the previous one, controlled by the new
snapshots.rosbag.max_bags_per_fault(default1, which reproduces the previous behaviour exactly;0= unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally therosbag_filesgrain changed from "one row per fault" to "one row per (fault, recording) link":recording_idis now a stored, indexed column, and the legacy column-levelUNIQUE(fault_code)is replaced by aUNIQUE INDEXon(fault_code, file_path)through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording,get_rosbag_filehad noORDER BYand would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and bothdelete_rosbag_file/delete_rosbag_filesread only the firstfile_pathof a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620) - Optional append-only, hash-chained audit log of fault state
transitions: each transition appends one immutable row
(
record_hash = sha256(prev_hash + canonical(event))via OpenSSL EVP SHA-256) with a persisted chain head, averifyroutine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited.verifyreads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering.BEFORE UPDATE/BEFORE DELETEtriggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, soverifydetects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_fault_manager at Robotics Stack Exchange
Package Summary
| Version | 0.7.2 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Description | |
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-11-22 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_fault_manager
Central fault manager node for the ros2_medkit fault management system.
Overview
The FaultManager node provides a central point for fault aggregation and lifecycle management.
It receives fault reports from multiple sources, aggregates them by fault_code, and provides
query and clearing interfaces.
Quick Start
By default, faults are confirmed immediately when reported - no additional configuration needed.
# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py
# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
"{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"
# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
"{statuses: ['CONFIRMED']}"
# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"
# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"
Note: The
skip_correlation_auto_clearrequest field was added post-0.4.0. Adding a request field changes the service type hash, so callers built againstros2_medkit_msgs0.4.0 or earlier must rebuild to keep talking tofault_manager.
Services
| Service | Type | Description |
|---|---|---|
~/report_fault |
ros2_medkit_msgs/srv/ReportFault |
Report a fault occurrence |
~/list_faults |
ros2_medkit_msgs/srv/ListFaults |
Query faults with filtering |
~/clear_fault |
ros2_medkit_msgs/srv/ClearFault |
Clear/acknowledge a fault |
~/get_snapshots |
ros2_medkit_msgs/srv/GetSnapshots |
Get topic snapshots for a fault |
Features
-
Multi-source aggregation: Same
fault_codefrom different sources creates a single fault - Occurrence tracking: Counts outages, not reports - the count starts at one and rises only when a cleared fault is raised again - and tracks all reporting sources
- Severity escalation: Fault severity is updated if a higher severity is reported
- Persistent storage: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
- Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
-
Snapshot capture: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless
snapshots.retain_on_clearis set) - Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
-
Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across
clear_fault(see below) - Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
- Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
storage_type |
string | "sqlite" |
Storage backend: "sqlite", "memory" or "postgres"
|
database_path |
string | "/var/lib/ros2_medkit/faults.db" |
Path to SQLite database file |
database_url |
string | "" |
PostgreSQL connection string; empty = libpq environment variables |
confirmation_threshold |
int | -1 |
Counter value at which faults are confirmed |
healing_enabled |
bool | false |
Enable automatic healing via PASSED events |
healing_threshold |
int | 3 |
Counter value at which faults are healed |
auto_confirm_after_sec |
double | 0.0 |
Auto-confirm PREFAILED faults after timeout (0 = disabled) |
report_fault_queue_depth |
int | 100 |
Requests ~/report_fault holds until read (1-65536, read-only); past it they are dropped without an answer |
entity_thresholds.config_file |
string | "" |
Path to YAML file with per-entity debounce threshold overrides |
fault_thresholds.config_file |
string | "" |
Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
near_miss.max_per_fault |
int | 200 |
Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
Snapshot Parameters
Snapshots capture topic data when faults are confirmed for post-mortem debugging.
Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.
Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.
That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.
| Parameter | Type | Default | Description |
|---|---|---|---|
snapshots.enabled |
bool | true |
Enable/disable snapshot capture |
snapshots.background_capture |
bool | false |
Cache each configured topic’s latest message vs on-demand capture; a topic absent at startup is cached once it appears |
snapshots.timeout_sec |
double | 1.0 |
Timeout waiting for topic message (on-demand mode) |
snapshots.max_message_size |
int | 65536 |
Maximum message size in bytes (larger messages skipped) |
snapshots.default_topics |
string[] | [] |
Topics to capture for all faults |
snapshots.config_file |
string | "" |
Path to YAML config for fault_specific and patterns
|
snapshots.recapture_cooldown_sec |
double | 60.0 |
Min seconds between captures for the same fault code. |
snapshots.max_per_fault |
int | 10 |
Max snapshots retained per fault. |
snapshots.capture_pool_size |
int | 2 |
Max concurrent capture threads under a fault storm (>= 1). Parallelizes snapshot capture only; rosbag stays single-writer. |
snapshots.capture_queue_depth |
int | 16 |
Max pending captures before the full-queue policy applies (>= 1). |
snapshots.capture_queue_full_policy |
string | reject_newest |
Policy when the queue is full: reject_newest or drop_oldest. |
File truncated at 100 lines see the full file
Changelog for package ros2_medkit_fault_manager
0.7.2 (2026-11-19)
- No changes. Version aligned with the rest of the stack
0.7.1 (2026-11-18)
- Recordings: every bag gets a unique path and the recording just stored is kept, a recording is hashed and stored off the executor thread, and a clear that arrives while a row is being stored takes that row (#785)
- A recording's reported size is the sum of the storage files its metadata names, and a bag's metadata is followed only to a storage file inside the bag. The snapshots payload no longer advertises a route that was removed (#679)
-
report_faultholds a burst of requests from one reporter instead of dropping part of it (#789) - A fault code can be debounced by the code, not only by who reported it (#690)
- A background snapshot topic that does not exist at startup is subscribed once it appears (#775)
- Fault clusters: the cluster severity stays current on re-reports, an active cluster's representative is reassigned without a pending cluster (#772), and the positive debounce counter is reset when a critical fault confirms immediately (#773)
- A time-based confirmation is as visible as a reported one, and
muting is honoured on the timer path.
healing_thresholdis 0 (it was 3) inconfig/fault_manager.yamland in the gateway'sconfig/bringup_params.yaml.bringup_params.yamlturns healing on, so with it one PASSED report now heals a fault: a fault from the action status bridge heals when its action recovers, where 0.7.0 kept it CONFIRMED.fault_manager.yamlkeeps healing off, so there the new value counts once healing is turned on (#643) - Optional PostgreSQL storage backend, off by default
(
-DPOSTGRES_SUPPORT=ON). It reconnects after a database outage (#651) -
Breaking (C++ API): the
FaultStoragevirtuals infault_storage.hppchanged:report_fault_eventtakesReportKindandint64_tnanoseconds, andcheck_time_based_confirmationtakesint64_t. A storage written for 0.7.0 no longer overrides them. Storage, thresholds, correlation, the audit log and the report, clear, list and get handling live in a ROS-free core (#740, #741, #742, #743, #744) - Parameters have upper bounds, for example 64 for
snapshots.capture_pool_size(#742, #744) - Contributors: \@bburda, \@mfaferek93, \@ahmedsleem109, \@gstavrinos, \@onur-ulusoy, \@Akinzou
0.7.0 (2026-08-27)
- Rosbag black-box recordings are no longer limited to one per fault
code. A fault that re-confirms keeps a bounded history of recordings
instead of overwriting the previous one, controlled by the new
snapshots.rosbag.max_bags_per_fault(default1, which reproduces the previous behaviour exactly;0= unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally therosbag_filesgrain changed from "one row per fault" to "one row per (fault, recording) link":recording_idis now a stored, indexed column, and the legacy column-levelUNIQUE(fault_code)is replaced by aUNIQUE INDEXon(fault_code, file_path)through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording,get_rosbag_filehad noORDER BYand would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and bothdelete_rosbag_file/delete_rosbag_filesread only the firstfile_pathof a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620) - Optional append-only, hash-chained audit log of fault state
transitions: each transition appends one immutable row
(
record_hash = sha256(prev_hash + canonical(event))via OpenSSL EVP SHA-256) with a persisted chain head, averifyroutine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited.verifyreads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering.BEFORE UPDATE/BEFORE DELETEtriggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, soverifydetects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_fault_manager at Robotics Stack Exchange
Package Summary
| Version | 0.7.2 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Description | |
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-11-22 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_fault_manager
Central fault manager node for the ros2_medkit fault management system.
Overview
The FaultManager node provides a central point for fault aggregation and lifecycle management.
It receives fault reports from multiple sources, aggregates them by fault_code, and provides
query and clearing interfaces.
Quick Start
By default, faults are confirmed immediately when reported - no additional configuration needed.
# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py
# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
"{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"
# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
"{statuses: ['CONFIRMED']}"
# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"
# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"
Note: The
skip_correlation_auto_clearrequest field was added post-0.4.0. Adding a request field changes the service type hash, so callers built againstros2_medkit_msgs0.4.0 or earlier must rebuild to keep talking tofault_manager.
Services
| Service | Type | Description |
|---|---|---|
~/report_fault |
ros2_medkit_msgs/srv/ReportFault |
Report a fault occurrence |
~/list_faults |
ros2_medkit_msgs/srv/ListFaults |
Query faults with filtering |
~/clear_fault |
ros2_medkit_msgs/srv/ClearFault |
Clear/acknowledge a fault |
~/get_snapshots |
ros2_medkit_msgs/srv/GetSnapshots |
Get topic snapshots for a fault |
Features
-
Multi-source aggregation: Same
fault_codefrom different sources creates a single fault - Occurrence tracking: Counts outages, not reports - the count starts at one and rises only when a cleared fault is raised again - and tracks all reporting sources
- Severity escalation: Fault severity is updated if a higher severity is reported
- Persistent storage: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
- Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
-
Snapshot capture: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless
snapshots.retain_on_clearis set) - Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
-
Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across
clear_fault(see below) - Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
- Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
storage_type |
string | "sqlite" |
Storage backend: "sqlite", "memory" or "postgres"
|
database_path |
string | "/var/lib/ros2_medkit/faults.db" |
Path to SQLite database file |
database_url |
string | "" |
PostgreSQL connection string; empty = libpq environment variables |
confirmation_threshold |
int | -1 |
Counter value at which faults are confirmed |
healing_enabled |
bool | false |
Enable automatic healing via PASSED events |
healing_threshold |
int | 3 |
Counter value at which faults are healed |
auto_confirm_after_sec |
double | 0.0 |
Auto-confirm PREFAILED faults after timeout (0 = disabled) |
report_fault_queue_depth |
int | 100 |
Requests ~/report_fault holds until read (1-65536, read-only); past it they are dropped without an answer |
entity_thresholds.config_file |
string | "" |
Path to YAML file with per-entity debounce threshold overrides |
fault_thresholds.config_file |
string | "" |
Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
near_miss.max_per_fault |
int | 200 |
Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
Snapshot Parameters
Snapshots capture topic data when faults are confirmed for post-mortem debugging.
Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.
Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.
That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.
| Parameter | Type | Default | Description |
|---|---|---|---|
snapshots.enabled |
bool | true |
Enable/disable snapshot capture |
snapshots.background_capture |
bool | false |
Cache each configured topic’s latest message vs on-demand capture; a topic absent at startup is cached once it appears |
snapshots.timeout_sec |
double | 1.0 |
Timeout waiting for topic message (on-demand mode) |
snapshots.max_message_size |
int | 65536 |
Maximum message size in bytes (larger messages skipped) |
snapshots.default_topics |
string[] | [] |
Topics to capture for all faults |
snapshots.config_file |
string | "" |
Path to YAML config for fault_specific and patterns
|
snapshots.recapture_cooldown_sec |
double | 60.0 |
Min seconds between captures for the same fault code. |
snapshots.max_per_fault |
int | 10 |
Max snapshots retained per fault. |
snapshots.capture_pool_size |
int | 2 |
Max concurrent capture threads under a fault storm (>= 1). Parallelizes snapshot capture only; rosbag stays single-writer. |
snapshots.capture_queue_depth |
int | 16 |
Max pending captures before the full-queue policy applies (>= 1). |
snapshots.capture_queue_full_policy |
string | reject_newest |
Policy when the queue is full: reject_newest or drop_oldest. |
File truncated at 100 lines see the full file
Changelog for package ros2_medkit_fault_manager
0.7.2 (2026-11-19)
- No changes. Version aligned with the rest of the stack
0.7.1 (2026-11-18)
- Recordings: every bag gets a unique path and the recording just stored is kept, a recording is hashed and stored off the executor thread, and a clear that arrives while a row is being stored takes that row (#785)
- A recording's reported size is the sum of the storage files its metadata names, and a bag's metadata is followed only to a storage file inside the bag. The snapshots payload no longer advertises a route that was removed (#679)
-
report_faultholds a burst of requests from one reporter instead of dropping part of it (#789) - A fault code can be debounced by the code, not only by who reported it (#690)
- A background snapshot topic that does not exist at startup is subscribed once it appears (#775)
- Fault clusters: the cluster severity stays current on re-reports, an active cluster's representative is reassigned without a pending cluster (#772), and the positive debounce counter is reset when a critical fault confirms immediately (#773)
- A time-based confirmation is as visible as a reported one, and
muting is honoured on the timer path.
healing_thresholdis 0 (it was 3) inconfig/fault_manager.yamland in the gateway'sconfig/bringup_params.yaml.bringup_params.yamlturns healing on, so with it one PASSED report now heals a fault: a fault from the action status bridge heals when its action recovers, where 0.7.0 kept it CONFIRMED.fault_manager.yamlkeeps healing off, so there the new value counts once healing is turned on (#643) - Optional PostgreSQL storage backend, off by default
(
-DPOSTGRES_SUPPORT=ON). It reconnects after a database outage (#651) -
Breaking (C++ API): the
FaultStoragevirtuals infault_storage.hppchanged:report_fault_eventtakesReportKindandint64_tnanoseconds, andcheck_time_based_confirmationtakesint64_t. A storage written for 0.7.0 no longer overrides them. Storage, thresholds, correlation, the audit log and the report, clear, list and get handling live in a ROS-free core (#740, #741, #742, #743, #744) - Parameters have upper bounds, for example 64 for
snapshots.capture_pool_size(#742, #744) - Contributors: \@bburda, \@mfaferek93, \@ahmedsleem109, \@gstavrinos, \@onur-ulusoy, \@Akinzou
0.7.0 (2026-08-27)
- Rosbag black-box recordings are no longer limited to one per fault
code. A fault that re-confirms keeps a bounded history of recordings
instead of overwriting the previous one, controlled by the new
snapshots.rosbag.max_bags_per_fault(default1, which reproduces the previous behaviour exactly;0= unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally therosbag_filesgrain changed from "one row per fault" to "one row per (fault, recording) link":recording_idis now a stored, indexed column, and the legacy column-levelUNIQUE(fault_code)is replaced by aUNIQUE INDEXon(fault_code, file_path)through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording,get_rosbag_filehad noORDER BYand would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and bothdelete_rosbag_file/delete_rosbag_filesread only the firstfile_pathof a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620) - Optional append-only, hash-chained audit log of fault state
transitions: each transition appends one immutable row
(
record_hash = sha256(prev_hash + canonical(event))via OpenSSL EVP SHA-256) with a persisted chain head, averifyroutine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited.verifyreads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering.BEFORE UPDATE/BEFORE DELETEtriggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, soverifydetects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_fault_manager at Robotics Stack Exchange
Package Summary
| Version | 0.7.2 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Description | |
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-11-22 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_fault_manager
Central fault manager node for the ros2_medkit fault management system.
Overview
The FaultManager node provides a central point for fault aggregation and lifecycle management.
It receives fault reports from multiple sources, aggregates them by fault_code, and provides
query and clearing interfaces.
Quick Start
By default, faults are confirmed immediately when reported - no additional configuration needed.
# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py
# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
"{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"
# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
"{statuses: ['CONFIRMED']}"
# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"
# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"
Note: The
skip_correlation_auto_clearrequest field was added post-0.4.0. Adding a request field changes the service type hash, so callers built againstros2_medkit_msgs0.4.0 or earlier must rebuild to keep talking tofault_manager.
Services
| Service | Type | Description |
|---|---|---|
~/report_fault |
ros2_medkit_msgs/srv/ReportFault |
Report a fault occurrence |
~/list_faults |
ros2_medkit_msgs/srv/ListFaults |
Query faults with filtering |
~/clear_fault |
ros2_medkit_msgs/srv/ClearFault |
Clear/acknowledge a fault |
~/get_snapshots |
ros2_medkit_msgs/srv/GetSnapshots |
Get topic snapshots for a fault |
Features
-
Multi-source aggregation: Same
fault_codefrom different sources creates a single fault - Occurrence tracking: Counts outages, not reports - the count starts at one and rises only when a cleared fault is raised again - and tracks all reporting sources
- Severity escalation: Fault severity is updated if a higher severity is reported
- Persistent storage: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
- Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
-
Snapshot capture: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless
snapshots.retain_on_clearis set) - Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
-
Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across
clear_fault(see below) - Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
- Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
storage_type |
string | "sqlite" |
Storage backend: "sqlite", "memory" or "postgres"
|
database_path |
string | "/var/lib/ros2_medkit/faults.db" |
Path to SQLite database file |
database_url |
string | "" |
PostgreSQL connection string; empty = libpq environment variables |
confirmation_threshold |
int | -1 |
Counter value at which faults are confirmed |
healing_enabled |
bool | false |
Enable automatic healing via PASSED events |
healing_threshold |
int | 3 |
Counter value at which faults are healed |
auto_confirm_after_sec |
double | 0.0 |
Auto-confirm PREFAILED faults after timeout (0 = disabled) |
report_fault_queue_depth |
int | 100 |
Requests ~/report_fault holds until read (1-65536, read-only); past it they are dropped without an answer |
entity_thresholds.config_file |
string | "" |
Path to YAML file with per-entity debounce threshold overrides |
fault_thresholds.config_file |
string | "" |
Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
near_miss.max_per_fault |
int | 200 |
Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
Snapshot Parameters
Snapshots capture topic data when faults are confirmed for post-mortem debugging.
Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.
Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.
That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.
| Parameter | Type | Default | Description |
|---|---|---|---|
snapshots.enabled |
bool | true |
Enable/disable snapshot capture |
snapshots.background_capture |
bool | false |
Cache each configured topic’s latest message vs on-demand capture; a topic absent at startup is cached once it appears |
snapshots.timeout_sec |
double | 1.0 |
Timeout waiting for topic message (on-demand mode) |
snapshots.max_message_size |
int | 65536 |
Maximum message size in bytes (larger messages skipped) |
snapshots.default_topics |
string[] | [] |
Topics to capture for all faults |
snapshots.config_file |
string | "" |
Path to YAML config for fault_specific and patterns
|
snapshots.recapture_cooldown_sec |
double | 60.0 |
Min seconds between captures for the same fault code. |
snapshots.max_per_fault |
int | 10 |
Max snapshots retained per fault. |
snapshots.capture_pool_size |
int | 2 |
Max concurrent capture threads under a fault storm (>= 1). Parallelizes snapshot capture only; rosbag stays single-writer. |
snapshots.capture_queue_depth |
int | 16 |
Max pending captures before the full-queue policy applies (>= 1). |
snapshots.capture_queue_full_policy |
string | reject_newest |
Policy when the queue is full: reject_newest or drop_oldest. |
File truncated at 100 lines see the full file
Changelog for package ros2_medkit_fault_manager
0.7.2 (2026-11-19)
- No changes. Version aligned with the rest of the stack
0.7.1 (2026-11-18)
- Recordings: every bag gets a unique path and the recording just stored is kept, a recording is hashed and stored off the executor thread, and a clear that arrives while a row is being stored takes that row (#785)
- A recording's reported size is the sum of the storage files its metadata names, and a bag's metadata is followed only to a storage file inside the bag. The snapshots payload no longer advertises a route that was removed (#679)
-
report_faultholds a burst of requests from one reporter instead of dropping part of it (#789) - A fault code can be debounced by the code, not only by who reported it (#690)
- A background snapshot topic that does not exist at startup is subscribed once it appears (#775)
- Fault clusters: the cluster severity stays current on re-reports, an active cluster's representative is reassigned without a pending cluster (#772), and the positive debounce counter is reset when a critical fault confirms immediately (#773)
- A time-based confirmation is as visible as a reported one, and
muting is honoured on the timer path.
healing_thresholdis 0 (it was 3) inconfig/fault_manager.yamland in the gateway'sconfig/bringup_params.yaml.bringup_params.yamlturns healing on, so with it one PASSED report now heals a fault: a fault from the action status bridge heals when its action recovers, where 0.7.0 kept it CONFIRMED.fault_manager.yamlkeeps healing off, so there the new value counts once healing is turned on (#643) - Optional PostgreSQL storage backend, off by default
(
-DPOSTGRES_SUPPORT=ON). It reconnects after a database outage (#651) -
Breaking (C++ API): the
FaultStoragevirtuals infault_storage.hppchanged:report_fault_eventtakesReportKindandint64_tnanoseconds, andcheck_time_based_confirmationtakesint64_t. A storage written for 0.7.0 no longer overrides them. Storage, thresholds, correlation, the audit log and the report, clear, list and get handling live in a ROS-free core (#740, #741, #742, #743, #744) - Parameters have upper bounds, for example 64 for
snapshots.capture_pool_size(#742, #744) - Contributors: \@bburda, \@mfaferek93, \@ahmedsleem109, \@gstavrinos, \@onur-ulusoy, \@Akinzou
0.7.0 (2026-08-27)
- Rosbag black-box recordings are no longer limited to one per fault
code. A fault that re-confirms keeps a bounded history of recordings
instead of overwriting the previous one, controlled by the new
snapshots.rosbag.max_bags_per_fault(default1, which reproduces the previous behaviour exactly;0= unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally therosbag_filesgrain changed from "one row per fault" to "one row per (fault, recording) link":recording_idis now a stored, indexed column, and the legacy column-levelUNIQUE(fault_code)is replaced by aUNIQUE INDEXon(fault_code, file_path)through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording,get_rosbag_filehad noORDER BYand would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and bothdelete_rosbag_file/delete_rosbag_filesread only the firstfile_pathof a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620) - Optional append-only, hash-chained audit log of fault state
transitions: each transition appends one immutable row
(
record_hash = sha256(prev_hash + canonical(event))via OpenSSL EVP SHA-256) with a persisted chain head, averifyroutine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited.verifyreads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering.BEFORE UPDATE/BEFORE DELETEtriggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, soverifydetects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_fault_manager at Robotics Stack Exchange
Package Summary
| Version | 0.7.2 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Description | |
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-11-22 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_fault_manager
Central fault manager node for the ros2_medkit fault management system.
Overview
The FaultManager node provides a central point for fault aggregation and lifecycle management.
It receives fault reports from multiple sources, aggregates them by fault_code, and provides
query and clearing interfaces.
Quick Start
By default, faults are confirmed immediately when reported - no additional configuration needed.
# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py
# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
"{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"
# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
"{statuses: ['CONFIRMED']}"
# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"
# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"
Note: The
skip_correlation_auto_clearrequest field was added post-0.4.0. Adding a request field changes the service type hash, so callers built againstros2_medkit_msgs0.4.0 or earlier must rebuild to keep talking tofault_manager.
Services
| Service | Type | Description |
|---|---|---|
~/report_fault |
ros2_medkit_msgs/srv/ReportFault |
Report a fault occurrence |
~/list_faults |
ros2_medkit_msgs/srv/ListFaults |
Query faults with filtering |
~/clear_fault |
ros2_medkit_msgs/srv/ClearFault |
Clear/acknowledge a fault |
~/get_snapshots |
ros2_medkit_msgs/srv/GetSnapshots |
Get topic snapshots for a fault |
Features
-
Multi-source aggregation: Same
fault_codefrom different sources creates a single fault - Occurrence tracking: Counts outages, not reports - the count starts at one and rises only when a cleared fault is raised again - and tracks all reporting sources
- Severity escalation: Fault severity is updated if a higher severity is reported
- Persistent storage: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
- Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
-
Snapshot capture: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless
snapshots.retain_on_clearis set) - Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
-
Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across
clear_fault(see below) - Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
- Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
storage_type |
string | "sqlite" |
Storage backend: "sqlite", "memory" or "postgres"
|
database_path |
string | "/var/lib/ros2_medkit/faults.db" |
Path to SQLite database file |
database_url |
string | "" |
PostgreSQL connection string; empty = libpq environment variables |
confirmation_threshold |
int | -1 |
Counter value at which faults are confirmed |
healing_enabled |
bool | false |
Enable automatic healing via PASSED events |
healing_threshold |
int | 3 |
Counter value at which faults are healed |
auto_confirm_after_sec |
double | 0.0 |
Auto-confirm PREFAILED faults after timeout (0 = disabled) |
report_fault_queue_depth |
int | 100 |
Requests ~/report_fault holds until read (1-65536, read-only); past it they are dropped without an answer |
entity_thresholds.config_file |
string | "" |
Path to YAML file with per-entity debounce threshold overrides |
fault_thresholds.config_file |
string | "" |
Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
near_miss.max_per_fault |
int | 200 |
Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
Snapshot Parameters
Snapshots capture topic data when faults are confirmed for post-mortem debugging.
Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.
Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.
That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.
| Parameter | Type | Default | Description |
|---|---|---|---|
snapshots.enabled |
bool | true |
Enable/disable snapshot capture |
snapshots.background_capture |
bool | false |
Cache each configured topic’s latest message vs on-demand capture; a topic absent at startup is cached once it appears |
snapshots.timeout_sec |
double | 1.0 |
Timeout waiting for topic message (on-demand mode) |
snapshots.max_message_size |
int | 65536 |
Maximum message size in bytes (larger messages skipped) |
snapshots.default_topics |
string[] | [] |
Topics to capture for all faults |
snapshots.config_file |
string | "" |
Path to YAML config for fault_specific and patterns
|
snapshots.recapture_cooldown_sec |
double | 60.0 |
Min seconds between captures for the same fault code. |
snapshots.max_per_fault |
int | 10 |
Max snapshots retained per fault. |
snapshots.capture_pool_size |
int | 2 |
Max concurrent capture threads under a fault storm (>= 1). Parallelizes snapshot capture only; rosbag stays single-writer. |
snapshots.capture_queue_depth |
int | 16 |
Max pending captures before the full-queue policy applies (>= 1). |
snapshots.capture_queue_full_policy |
string | reject_newest |
Policy when the queue is full: reject_newest or drop_oldest. |
File truncated at 100 lines see the full file
Changelog for package ros2_medkit_fault_manager
0.7.2 (2026-11-19)
- No changes. Version aligned with the rest of the stack
0.7.1 (2026-11-18)
- Recordings: every bag gets a unique path and the recording just stored is kept, a recording is hashed and stored off the executor thread, and a clear that arrives while a row is being stored takes that row (#785)
- A recording's reported size is the sum of the storage files its metadata names, and a bag's metadata is followed only to a storage file inside the bag. The snapshots payload no longer advertises a route that was removed (#679)
-
report_faultholds a burst of requests from one reporter instead of dropping part of it (#789) - A fault code can be debounced by the code, not only by who reported it (#690)
- A background snapshot topic that does not exist at startup is subscribed once it appears (#775)
- Fault clusters: the cluster severity stays current on re-reports, an active cluster's representative is reassigned without a pending cluster (#772), and the positive debounce counter is reset when a critical fault confirms immediately (#773)
- A time-based confirmation is as visible as a reported one, and
muting is honoured on the timer path.
healing_thresholdis 0 (it was 3) inconfig/fault_manager.yamland in the gateway'sconfig/bringup_params.yaml.bringup_params.yamlturns healing on, so with it one PASSED report now heals a fault: a fault from the action status bridge heals when its action recovers, where 0.7.0 kept it CONFIRMED.fault_manager.yamlkeeps healing off, so there the new value counts once healing is turned on (#643) - Optional PostgreSQL storage backend, off by default
(
-DPOSTGRES_SUPPORT=ON). It reconnects after a database outage (#651) -
Breaking (C++ API): the
FaultStoragevirtuals infault_storage.hppchanged:report_fault_eventtakesReportKindandint64_tnanoseconds, andcheck_time_based_confirmationtakesint64_t. A storage written for 0.7.0 no longer overrides them. Storage, thresholds, correlation, the audit log and the report, clear, list and get handling live in a ROS-free core (#740, #741, #742, #743, #744) - Parameters have upper bounds, for example 64 for
snapshots.capture_pool_size(#742, #744) - Contributors: \@bburda, \@mfaferek93, \@ahmedsleem109, \@gstavrinos, \@onur-ulusoy, \@Akinzou
0.7.0 (2026-08-27)
- Rosbag black-box recordings are no longer limited to one per fault
code. A fault that re-confirms keeps a bounded history of recordings
instead of overwriting the previous one, controlled by the new
snapshots.rosbag.max_bags_per_fault(default1, which reproduces the previous behaviour exactly;0= unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally therosbag_filesgrain changed from "one row per fault" to "one row per (fault, recording) link":recording_idis now a stored, indexed column, and the legacy column-levelUNIQUE(fault_code)is replaced by aUNIQUE INDEXon(fault_code, file_path)through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording,get_rosbag_filehad noORDER BYand would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and bothdelete_rosbag_file/delete_rosbag_filesread only the firstfile_pathof a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620) - Optional append-only, hash-chained audit log of fault state
transitions: each transition appends one immutable row
(
record_hash = sha256(prev_hash + canonical(event))via OpenSSL EVP SHA-256) with a persisted chain head, averifyroutine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited.verifyreads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering.BEFORE UPDATE/BEFORE DELETEtriggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, soverifydetects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_fault_manager at Robotics Stack Exchange
Package Summary
| Version | 0.7.2 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Description | |
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-11-22 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_fault_manager
Central fault manager node for the ros2_medkit fault management system.
Overview
The FaultManager node provides a central point for fault aggregation and lifecycle management.
It receives fault reports from multiple sources, aggregates them by fault_code, and provides
query and clearing interfaces.
Quick Start
By default, faults are confirmed immediately when reported - no additional configuration needed.
# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py
# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
"{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"
# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
"{statuses: ['CONFIRMED']}"
# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"
# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"
Note: The
skip_correlation_auto_clearrequest field was added post-0.4.0. Adding a request field changes the service type hash, so callers built againstros2_medkit_msgs0.4.0 or earlier must rebuild to keep talking tofault_manager.
Services
| Service | Type | Description |
|---|---|---|
~/report_fault |
ros2_medkit_msgs/srv/ReportFault |
Report a fault occurrence |
~/list_faults |
ros2_medkit_msgs/srv/ListFaults |
Query faults with filtering |
~/clear_fault |
ros2_medkit_msgs/srv/ClearFault |
Clear/acknowledge a fault |
~/get_snapshots |
ros2_medkit_msgs/srv/GetSnapshots |
Get topic snapshots for a fault |
Features
-
Multi-source aggregation: Same
fault_codefrom different sources creates a single fault - Occurrence tracking: Counts outages, not reports - the count starts at one and rises only when a cleared fault is raised again - and tracks all reporting sources
- Severity escalation: Fault severity is updated if a higher severity is reported
- Persistent storage: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
- Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
-
Snapshot capture: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless
snapshots.retain_on_clearis set) - Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
-
Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across
clear_fault(see below) - Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
- Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
storage_type |
string | "sqlite" |
Storage backend: "sqlite", "memory" or "postgres"
|
database_path |
string | "/var/lib/ros2_medkit/faults.db" |
Path to SQLite database file |
database_url |
string | "" |
PostgreSQL connection string; empty = libpq environment variables |
confirmation_threshold |
int | -1 |
Counter value at which faults are confirmed |
healing_enabled |
bool | false |
Enable automatic healing via PASSED events |
healing_threshold |
int | 3 |
Counter value at which faults are healed |
auto_confirm_after_sec |
double | 0.0 |
Auto-confirm PREFAILED faults after timeout (0 = disabled) |
report_fault_queue_depth |
int | 100 |
Requests ~/report_fault holds until read (1-65536, read-only); past it they are dropped without an answer |
entity_thresholds.config_file |
string | "" |
Path to YAML file with per-entity debounce threshold overrides |
fault_thresholds.config_file |
string | "" |
Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
near_miss.max_per_fault |
int | 200 |
Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
Snapshot Parameters
Snapshots capture topic data when faults are confirmed for post-mortem debugging.
Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.
Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.
That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.
| Parameter | Type | Default | Description |
|---|---|---|---|
snapshots.enabled |
bool | true |
Enable/disable snapshot capture |
snapshots.background_capture |
bool | false |
Cache each configured topic’s latest message vs on-demand capture; a topic absent at startup is cached once it appears |
snapshots.timeout_sec |
double | 1.0 |
Timeout waiting for topic message (on-demand mode) |
snapshots.max_message_size |
int | 65536 |
Maximum message size in bytes (larger messages skipped) |
snapshots.default_topics |
string[] | [] |
Topics to capture for all faults |
snapshots.config_file |
string | "" |
Path to YAML config for fault_specific and patterns
|
snapshots.recapture_cooldown_sec |
double | 60.0 |
Min seconds between captures for the same fault code. |
snapshots.max_per_fault |
int | 10 |
Max snapshots retained per fault. |
snapshots.capture_pool_size |
int | 2 |
Max concurrent capture threads under a fault storm (>= 1). Parallelizes snapshot capture only; rosbag stays single-writer. |
snapshots.capture_queue_depth |
int | 16 |
Max pending captures before the full-queue policy applies (>= 1). |
snapshots.capture_queue_full_policy |
string | reject_newest |
Policy when the queue is full: reject_newest or drop_oldest. |
File truncated at 100 lines see the full file
Changelog for package ros2_medkit_fault_manager
0.7.2 (2026-11-19)
- No changes. Version aligned with the rest of the stack
0.7.1 (2026-11-18)
- Recordings: every bag gets a unique path and the recording just stored is kept, a recording is hashed and stored off the executor thread, and a clear that arrives while a row is being stored takes that row (#785)
- A recording's reported size is the sum of the storage files its metadata names, and a bag's metadata is followed only to a storage file inside the bag. The snapshots payload no longer advertises a route that was removed (#679)
-
report_faultholds a burst of requests from one reporter instead of dropping part of it (#789) - A fault code can be debounced by the code, not only by who reported it (#690)
- A background snapshot topic that does not exist at startup is subscribed once it appears (#775)
- Fault clusters: the cluster severity stays current on re-reports, an active cluster's representative is reassigned without a pending cluster (#772), and the positive debounce counter is reset when a critical fault confirms immediately (#773)
- A time-based confirmation is as visible as a reported one, and
muting is honoured on the timer path.
healing_thresholdis 0 (it was 3) inconfig/fault_manager.yamland in the gateway'sconfig/bringup_params.yaml.bringup_params.yamlturns healing on, so with it one PASSED report now heals a fault: a fault from the action status bridge heals when its action recovers, where 0.7.0 kept it CONFIRMED.fault_manager.yamlkeeps healing off, so there the new value counts once healing is turned on (#643) - Optional PostgreSQL storage backend, off by default
(
-DPOSTGRES_SUPPORT=ON). It reconnects after a database outage (#651) -
Breaking (C++ API): the
FaultStoragevirtuals infault_storage.hppchanged:report_fault_eventtakesReportKindandint64_tnanoseconds, andcheck_time_based_confirmationtakesint64_t. A storage written for 0.7.0 no longer overrides them. Storage, thresholds, correlation, the audit log and the report, clear, list and get handling live in a ROS-free core (#740, #741, #742, #743, #744) - Parameters have upper bounds, for example 64 for
snapshots.capture_pool_size(#742, #744) - Contributors: \@bburda, \@mfaferek93, \@ahmedsleem109, \@gstavrinos, \@onur-ulusoy, \@Akinzou
0.7.0 (2026-08-27)
- Rosbag black-box recordings are no longer limited to one per fault
code. A fault that re-confirms keeps a bounded history of recordings
instead of overwriting the previous one, controlled by the new
snapshots.rosbag.max_bags_per_fault(default1, which reproduces the previous behaviour exactly;0= unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally therosbag_filesgrain changed from "one row per fault" to "one row per (fault, recording) link":recording_idis now a stored, indexed column, and the legacy column-levelUNIQUE(fault_code)is replaced by aUNIQUE INDEXon(fault_code, file_path)through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording,get_rosbag_filehad noORDER BYand would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and bothdelete_rosbag_file/delete_rosbag_filesread only the firstfile_pathof a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620) - Optional append-only, hash-chained audit log of fault state
transitions: each transition appends one immutable row
(
record_hash = sha256(prev_hash + canonical(event))via OpenSSL EVP SHA-256) with a persisted chain head, averifyroutine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited.verifyreads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering.BEFORE UPDATE/BEFORE DELETEtriggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, soverifydetects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_fault_manager at Robotics Stack Exchange
Package Summary
| Version | 0.7.2 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Description | |
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-11-22 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_fault_manager
Central fault manager node for the ros2_medkit fault management system.
Overview
The FaultManager node provides a central point for fault aggregation and lifecycle management.
It receives fault reports from multiple sources, aggregates them by fault_code, and provides
query and clearing interfaces.
Quick Start
By default, faults are confirmed immediately when reported - no additional configuration needed.
# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py
# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
"{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"
# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
"{statuses: ['CONFIRMED']}"
# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"
# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"
Note: The
skip_correlation_auto_clearrequest field was added post-0.4.0. Adding a request field changes the service type hash, so callers built againstros2_medkit_msgs0.4.0 or earlier must rebuild to keep talking tofault_manager.
Services
| Service | Type | Description |
|---|---|---|
~/report_fault |
ros2_medkit_msgs/srv/ReportFault |
Report a fault occurrence |
~/list_faults |
ros2_medkit_msgs/srv/ListFaults |
Query faults with filtering |
~/clear_fault |
ros2_medkit_msgs/srv/ClearFault |
Clear/acknowledge a fault |
~/get_snapshots |
ros2_medkit_msgs/srv/GetSnapshots |
Get topic snapshots for a fault |
Features
-
Multi-source aggregation: Same
fault_codefrom different sources creates a single fault - Occurrence tracking: Counts outages, not reports - the count starts at one and rises only when a cleared fault is raised again - and tracks all reporting sources
- Severity escalation: Fault severity is updated if a higher severity is reported
- Persistent storage: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
- Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
-
Snapshot capture: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless
snapshots.retain_on_clearis set) - Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
-
Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across
clear_fault(see below) - Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
- Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
storage_type |
string | "sqlite" |
Storage backend: "sqlite", "memory" or "postgres"
|
database_path |
string | "/var/lib/ros2_medkit/faults.db" |
Path to SQLite database file |
database_url |
string | "" |
PostgreSQL connection string; empty = libpq environment variables |
confirmation_threshold |
int | -1 |
Counter value at which faults are confirmed |
healing_enabled |
bool | false |
Enable automatic healing via PASSED events |
healing_threshold |
int | 3 |
Counter value at which faults are healed |
auto_confirm_after_sec |
double | 0.0 |
Auto-confirm PREFAILED faults after timeout (0 = disabled) |
report_fault_queue_depth |
int | 100 |
Requests ~/report_fault holds until read (1-65536, read-only); past it they are dropped without an answer |
entity_thresholds.config_file |
string | "" |
Path to YAML file with per-entity debounce threshold overrides |
fault_thresholds.config_file |
string | "" |
Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
near_miss.max_per_fault |
int | 200 |
Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
Snapshot Parameters
Snapshots capture topic data when faults are confirmed for post-mortem debugging.
Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.
Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.
That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.
| Parameter | Type | Default | Description |
|---|---|---|---|
snapshots.enabled |
bool | true |
Enable/disable snapshot capture |
snapshots.background_capture |
bool | false |
Cache each configured topic’s latest message vs on-demand capture; a topic absent at startup is cached once it appears |
snapshots.timeout_sec |
double | 1.0 |
Timeout waiting for topic message (on-demand mode) |
snapshots.max_message_size |
int | 65536 |
Maximum message size in bytes (larger messages skipped) |
snapshots.default_topics |
string[] | [] |
Topics to capture for all faults |
snapshots.config_file |
string | "" |
Path to YAML config for fault_specific and patterns
|
snapshots.recapture_cooldown_sec |
double | 60.0 |
Min seconds between captures for the same fault code. |
snapshots.max_per_fault |
int | 10 |
Max snapshots retained per fault. |
snapshots.capture_pool_size |
int | 2 |
Max concurrent capture threads under a fault storm (>= 1). Parallelizes snapshot capture only; rosbag stays single-writer. |
snapshots.capture_queue_depth |
int | 16 |
Max pending captures before the full-queue policy applies (>= 1). |
snapshots.capture_queue_full_policy |
string | reject_newest |
Policy when the queue is full: reject_newest or drop_oldest. |
File truncated at 100 lines see the full file
Changelog for package ros2_medkit_fault_manager
0.7.2 (2026-11-19)
- No changes. Version aligned with the rest of the stack
0.7.1 (2026-11-18)
- Recordings: every bag gets a unique path and the recording just stored is kept, a recording is hashed and stored off the executor thread, and a clear that arrives while a row is being stored takes that row (#785)
- A recording's reported size is the sum of the storage files its metadata names, and a bag's metadata is followed only to a storage file inside the bag. The snapshots payload no longer advertises a route that was removed (#679)
-
report_faultholds a burst of requests from one reporter instead of dropping part of it (#789) - A fault code can be debounced by the code, not only by who reported it (#690)
- A background snapshot topic that does not exist at startup is subscribed once it appears (#775)
- Fault clusters: the cluster severity stays current on re-reports, an active cluster's representative is reassigned without a pending cluster (#772), and the positive debounce counter is reset when a critical fault confirms immediately (#773)
- A time-based confirmation is as visible as a reported one, and
muting is honoured on the timer path.
healing_thresholdis 0 (it was 3) inconfig/fault_manager.yamland in the gateway'sconfig/bringup_params.yaml.bringup_params.yamlturns healing on, so with it one PASSED report now heals a fault: a fault from the action status bridge heals when its action recovers, where 0.7.0 kept it CONFIRMED.fault_manager.yamlkeeps healing off, so there the new value counts once healing is turned on (#643) - Optional PostgreSQL storage backend, off by default
(
-DPOSTGRES_SUPPORT=ON). It reconnects after a database outage (#651) -
Breaking (C++ API): the
FaultStoragevirtuals infault_storage.hppchanged:report_fault_eventtakesReportKindandint64_tnanoseconds, andcheck_time_based_confirmationtakesint64_t. A storage written for 0.7.0 no longer overrides them. Storage, thresholds, correlation, the audit log and the report, clear, list and get handling live in a ROS-free core (#740, #741, #742, #743, #744) - Parameters have upper bounds, for example 64 for
snapshots.capture_pool_size(#742, #744) - Contributors: \@bburda, \@mfaferek93, \@ahmedsleem109, \@gstavrinos, \@onur-ulusoy, \@Akinzou
0.7.0 (2026-08-27)
- Rosbag black-box recordings are no longer limited to one per fault
code. A fault that re-confirms keeps a bounded history of recordings
instead of overwriting the previous one, controlled by the new
snapshots.rosbag.max_bags_per_fault(default1, which reproduces the previous behaviour exactly;0= unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally therosbag_filesgrain changed from "one row per fault" to "one row per (fault, recording) link":recording_idis now a stored, indexed column, and the legacy column-levelUNIQUE(fault_code)is replaced by aUNIQUE INDEXon(fault_code, file_path)through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording,get_rosbag_filehad noORDER BYand would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and bothdelete_rosbag_file/delete_rosbag_filesread only the firstfile_pathof a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620) - Optional append-only, hash-chained audit log of fault state
transitions: each transition appends one immutable row
(
record_hash = sha256(prev_hash + canonical(event))via OpenSSL EVP SHA-256) with a persisted chain head, averifyroutine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited.verifyreads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering.BEFORE UPDATE/BEFORE DELETEtriggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, soverifydetects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_fault_manager at Robotics Stack Exchange
Package Summary
| Version | 0.7.2 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Description | |
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-11-22 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_fault_manager
Central fault manager node for the ros2_medkit fault management system.
Overview
The FaultManager node provides a central point for fault aggregation and lifecycle management.
It receives fault reports from multiple sources, aggregates them by fault_code, and provides
query and clearing interfaces.
Quick Start
By default, faults are confirmed immediately when reported - no additional configuration needed.
# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py
# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
"{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"
# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
"{statuses: ['CONFIRMED']}"
# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"
# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"
Note: The
skip_correlation_auto_clearrequest field was added post-0.4.0. Adding a request field changes the service type hash, so callers built againstros2_medkit_msgs0.4.0 or earlier must rebuild to keep talking tofault_manager.
Services
| Service | Type | Description |
|---|---|---|
~/report_fault |
ros2_medkit_msgs/srv/ReportFault |
Report a fault occurrence |
~/list_faults |
ros2_medkit_msgs/srv/ListFaults |
Query faults with filtering |
~/clear_fault |
ros2_medkit_msgs/srv/ClearFault |
Clear/acknowledge a fault |
~/get_snapshots |
ros2_medkit_msgs/srv/GetSnapshots |
Get topic snapshots for a fault |
Features
-
Multi-source aggregation: Same
fault_codefrom different sources creates a single fault - Occurrence tracking: Counts outages, not reports - the count starts at one and rises only when a cleared fault is raised again - and tracks all reporting sources
- Severity escalation: Fault severity is updated if a higher severity is reported
- Persistent storage: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
- Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
-
Snapshot capture: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless
snapshots.retain_on_clearis set) - Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
-
Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across
clear_fault(see below) - Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
- Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
storage_type |
string | "sqlite" |
Storage backend: "sqlite", "memory" or "postgres"
|
database_path |
string | "/var/lib/ros2_medkit/faults.db" |
Path to SQLite database file |
database_url |
string | "" |
PostgreSQL connection string; empty = libpq environment variables |
confirmation_threshold |
int | -1 |
Counter value at which faults are confirmed |
healing_enabled |
bool | false |
Enable automatic healing via PASSED events |
healing_threshold |
int | 3 |
Counter value at which faults are healed |
auto_confirm_after_sec |
double | 0.0 |
Auto-confirm PREFAILED faults after timeout (0 = disabled) |
report_fault_queue_depth |
int | 100 |
Requests ~/report_fault holds until read (1-65536, read-only); past it they are dropped without an answer |
entity_thresholds.config_file |
string | "" |
Path to YAML file with per-entity debounce threshold overrides |
fault_thresholds.config_file |
string | "" |
Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
near_miss.max_per_fault |
int | 200 |
Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
Snapshot Parameters
Snapshots capture topic data when faults are confirmed for post-mortem debugging.
Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.
Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.
That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.
| Parameter | Type | Default | Description |
|---|---|---|---|
snapshots.enabled |
bool | true |
Enable/disable snapshot capture |
snapshots.background_capture |
bool | false |
Cache each configured topic’s latest message vs on-demand capture; a topic absent at startup is cached once it appears |
snapshots.timeout_sec |
double | 1.0 |
Timeout waiting for topic message (on-demand mode) |
snapshots.max_message_size |
int | 65536 |
Maximum message size in bytes (larger messages skipped) |
snapshots.default_topics |
string[] | [] |
Topics to capture for all faults |
snapshots.config_file |
string | "" |
Path to YAML config for fault_specific and patterns
|
snapshots.recapture_cooldown_sec |
double | 60.0 |
Min seconds between captures for the same fault code. |
snapshots.max_per_fault |
int | 10 |
Max snapshots retained per fault. |
snapshots.capture_pool_size |
int | 2 |
Max concurrent capture threads under a fault storm (>= 1). Parallelizes snapshot capture only; rosbag stays single-writer. |
snapshots.capture_queue_depth |
int | 16 |
Max pending captures before the full-queue policy applies (>= 1). |
snapshots.capture_queue_full_policy |
string | reject_newest |
Policy when the queue is full: reject_newest or drop_oldest. |
File truncated at 100 lines see the full file
Changelog for package ros2_medkit_fault_manager
0.7.2 (2026-11-19)
- No changes. Version aligned with the rest of the stack
0.7.1 (2026-11-18)
- Recordings: every bag gets a unique path and the recording just stored is kept, a recording is hashed and stored off the executor thread, and a clear that arrives while a row is being stored takes that row (#785)
- A recording's reported size is the sum of the storage files its metadata names, and a bag's metadata is followed only to a storage file inside the bag. The snapshots payload no longer advertises a route that was removed (#679)
-
report_faultholds a burst of requests from one reporter instead of dropping part of it (#789) - A fault code can be debounced by the code, not only by who reported it (#690)
- A background snapshot topic that does not exist at startup is subscribed once it appears (#775)
- Fault clusters: the cluster severity stays current on re-reports, an active cluster's representative is reassigned without a pending cluster (#772), and the positive debounce counter is reset when a critical fault confirms immediately (#773)
- A time-based confirmation is as visible as a reported one, and
muting is honoured on the timer path.
healing_thresholdis 0 (it was 3) inconfig/fault_manager.yamland in the gateway'sconfig/bringup_params.yaml.bringup_params.yamlturns healing on, so with it one PASSED report now heals a fault: a fault from the action status bridge heals when its action recovers, where 0.7.0 kept it CONFIRMED.fault_manager.yamlkeeps healing off, so there the new value counts once healing is turned on (#643) - Optional PostgreSQL storage backend, off by default
(
-DPOSTGRES_SUPPORT=ON). It reconnects after a database outage (#651) -
Breaking (C++ API): the
FaultStoragevirtuals infault_storage.hppchanged:report_fault_eventtakesReportKindandint64_tnanoseconds, andcheck_time_based_confirmationtakesint64_t. A storage written for 0.7.0 no longer overrides them. Storage, thresholds, correlation, the audit log and the report, clear, list and get handling live in a ROS-free core (#740, #741, #742, #743, #744) - Parameters have upper bounds, for example 64 for
snapshots.capture_pool_size(#742, #744) - Contributors: \@bburda, \@mfaferek93, \@ahmedsleem109, \@gstavrinos, \@onur-ulusoy, \@Akinzou
0.7.0 (2026-08-27)
- Rosbag black-box recordings are no longer limited to one per fault
code. A fault that re-confirms keeps a bounded history of recordings
instead of overwriting the previous one, controlled by the new
snapshots.rosbag.max_bags_per_fault(default1, which reproduces the previous behaviour exactly;0= unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally therosbag_filesgrain changed from "one row per fault" to "one row per (fault, recording) link":recording_idis now a stored, indexed column, and the legacy column-levelUNIQUE(fault_code)is replaced by aUNIQUE INDEXon(fault_code, file_path)through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording,get_rosbag_filehad noORDER BYand would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and bothdelete_rosbag_file/delete_rosbag_filesread only the firstfile_pathof a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620) - Optional append-only, hash-chained audit log of fault state
transitions: each transition appends one immutable row
(
record_hash = sha256(prev_hash + canonical(event))via OpenSSL EVP SHA-256) with a persisted chain head, averifyroutine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited.verifyreads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering.BEFORE UPDATE/BEFORE DELETEtriggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, soverifydetects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_fault_manager at Robotics Stack Exchange
Package Summary
| Version | 0.7.2 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Description | |
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-11-22 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_fault_manager
Central fault manager node for the ros2_medkit fault management system.
Overview
The FaultManager node provides a central point for fault aggregation and lifecycle management.
It receives fault reports from multiple sources, aggregates them by fault_code, and provides
query and clearing interfaces.
Quick Start
By default, faults are confirmed immediately when reported - no additional configuration needed.
# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py
# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
"{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"
# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
"{statuses: ['CONFIRMED']}"
# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"
# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"
Note: The
skip_correlation_auto_clearrequest field was added post-0.4.0. Adding a request field changes the service type hash, so callers built againstros2_medkit_msgs0.4.0 or earlier must rebuild to keep talking tofault_manager.
Services
| Service | Type | Description |
|---|---|---|
~/report_fault |
ros2_medkit_msgs/srv/ReportFault |
Report a fault occurrence |
~/list_faults |
ros2_medkit_msgs/srv/ListFaults |
Query faults with filtering |
~/clear_fault |
ros2_medkit_msgs/srv/ClearFault |
Clear/acknowledge a fault |
~/get_snapshots |
ros2_medkit_msgs/srv/GetSnapshots |
Get topic snapshots for a fault |
Features
-
Multi-source aggregation: Same
fault_codefrom different sources creates a single fault - Occurrence tracking: Counts outages, not reports - the count starts at one and rises only when a cleared fault is raised again - and tracks all reporting sources
- Severity escalation: Fault severity is updated if a higher severity is reported
- Persistent storage: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
- Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
-
Snapshot capture: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless
snapshots.retain_on_clearis set) - Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
-
Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across
clear_fault(see below) - Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
- Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
storage_type |
string | "sqlite" |
Storage backend: "sqlite", "memory" or "postgres"
|
database_path |
string | "/var/lib/ros2_medkit/faults.db" |
Path to SQLite database file |
database_url |
string | "" |
PostgreSQL connection string; empty = libpq environment variables |
confirmation_threshold |
int | -1 |
Counter value at which faults are confirmed |
healing_enabled |
bool | false |
Enable automatic healing via PASSED events |
healing_threshold |
int | 3 |
Counter value at which faults are healed |
auto_confirm_after_sec |
double | 0.0 |
Auto-confirm PREFAILED faults after timeout (0 = disabled) |
report_fault_queue_depth |
int | 100 |
Requests ~/report_fault holds until read (1-65536, read-only); past it they are dropped without an answer |
entity_thresholds.config_file |
string | "" |
Path to YAML file with per-entity debounce threshold overrides |
fault_thresholds.config_file |
string | "" |
Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
near_miss.max_per_fault |
int | 200 |
Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
Snapshot Parameters
Snapshots capture topic data when faults are confirmed for post-mortem debugging.
Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.
Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.
That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.
| Parameter | Type | Default | Description |
|---|---|---|---|
snapshots.enabled |
bool | true |
Enable/disable snapshot capture |
snapshots.background_capture |
bool | false |
Cache each configured topic’s latest message vs on-demand capture; a topic absent at startup is cached once it appears |
snapshots.timeout_sec |
double | 1.0 |
Timeout waiting for topic message (on-demand mode) |
snapshots.max_message_size |
int | 65536 |
Maximum message size in bytes (larger messages skipped) |
snapshots.default_topics |
string[] | [] |
Topics to capture for all faults |
snapshots.config_file |
string | "" |
Path to YAML config for fault_specific and patterns
|
snapshots.recapture_cooldown_sec |
double | 60.0 |
Min seconds between captures for the same fault code. |
snapshots.max_per_fault |
int | 10 |
Max snapshots retained per fault. |
snapshots.capture_pool_size |
int | 2 |
Max concurrent capture threads under a fault storm (>= 1). Parallelizes snapshot capture only; rosbag stays single-writer. |
snapshots.capture_queue_depth |
int | 16 |
Max pending captures before the full-queue policy applies (>= 1). |
snapshots.capture_queue_full_policy |
string | reject_newest |
Policy when the queue is full: reject_newest or drop_oldest. |
File truncated at 100 lines see the full file
Changelog for package ros2_medkit_fault_manager
0.7.2 (2026-11-19)
- No changes. Version aligned with the rest of the stack
0.7.1 (2026-11-18)
- Recordings: every bag gets a unique path and the recording just stored is kept, a recording is hashed and stored off the executor thread, and a clear that arrives while a row is being stored takes that row (#785)
- A recording's reported size is the sum of the storage files its metadata names, and a bag's metadata is followed only to a storage file inside the bag. The snapshots payload no longer advertises a route that was removed (#679)
-
report_faultholds a burst of requests from one reporter instead of dropping part of it (#789) - A fault code can be debounced by the code, not only by who reported it (#690)
- A background snapshot topic that does not exist at startup is subscribed once it appears (#775)
- Fault clusters: the cluster severity stays current on re-reports, an active cluster's representative is reassigned without a pending cluster (#772), and the positive debounce counter is reset when a critical fault confirms immediately (#773)
- A time-based confirmation is as visible as a reported one, and
muting is honoured on the timer path.
healing_thresholdis 0 (it was 3) inconfig/fault_manager.yamland in the gateway'sconfig/bringup_params.yaml.bringup_params.yamlturns healing on, so with it one PASSED report now heals a fault: a fault from the action status bridge heals when its action recovers, where 0.7.0 kept it CONFIRMED.fault_manager.yamlkeeps healing off, so there the new value counts once healing is turned on (#643) - Optional PostgreSQL storage backend, off by default
(
-DPOSTGRES_SUPPORT=ON). It reconnects after a database outage (#651) -
Breaking (C++ API): the
FaultStoragevirtuals infault_storage.hppchanged:report_fault_eventtakesReportKindandint64_tnanoseconds, andcheck_time_based_confirmationtakesint64_t. A storage written for 0.7.0 no longer overrides them. Storage, thresholds, correlation, the audit log and the report, clear, list and get handling live in a ROS-free core (#740, #741, #742, #743, #744) - Parameters have upper bounds, for example 64 for
snapshots.capture_pool_size(#742, #744) - Contributors: \@bburda, \@mfaferek93, \@ahmedsleem109, \@gstavrinos, \@onur-ulusoy, \@Akinzou
0.7.0 (2026-08-27)
- Rosbag black-box recordings are no longer limited to one per fault
code. A fault that re-confirms keeps a bounded history of recordings
instead of overwriting the previous one, controlled by the new
snapshots.rosbag.max_bags_per_fault(default1, which reproduces the previous behaviour exactly;0= unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally therosbag_filesgrain changed from "one row per fault" to "one row per (fault, recording) link":recording_idis now a stored, indexed column, and the legacy column-levelUNIQUE(fault_code)is replaced by aUNIQUE INDEXon(fault_code, file_path)through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording,get_rosbag_filehad noORDER BYand would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and bothdelete_rosbag_file/delete_rosbag_filesread only the firstfile_pathof a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620) - Optional append-only, hash-chained audit log of fault state
transitions: each transition appends one immutable row
(
record_hash = sha256(prev_hash + canonical(event))via OpenSSL EVP SHA-256) with a persisted chain head, averifyroutine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited.verifyreads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering.BEFORE UPDATE/BEFORE DELETEtriggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, soverifydetects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_fault_manager at Robotics Stack Exchange
Package Summary
| Version | 0.7.2 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Description | |
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-11-22 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_fault_manager
Central fault manager node for the ros2_medkit fault management system.
Overview
The FaultManager node provides a central point for fault aggregation and lifecycle management.
It receives fault reports from multiple sources, aggregates them by fault_code, and provides
query and clearing interfaces.
Quick Start
By default, faults are confirmed immediately when reported - no additional configuration needed.
# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py
# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
"{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"
# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
"{statuses: ['CONFIRMED']}"
# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"
# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
"{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"
Note: The
skip_correlation_auto_clearrequest field was added post-0.4.0. Adding a request field changes the service type hash, so callers built againstros2_medkit_msgs0.4.0 or earlier must rebuild to keep talking tofault_manager.
Services
| Service | Type | Description |
|---|---|---|
~/report_fault |
ros2_medkit_msgs/srv/ReportFault |
Report a fault occurrence |
~/list_faults |
ros2_medkit_msgs/srv/ListFaults |
Query faults with filtering |
~/clear_fault |
ros2_medkit_msgs/srv/ClearFault |
Clear/acknowledge a fault |
~/get_snapshots |
ros2_medkit_msgs/srv/GetSnapshots |
Get topic snapshots for a fault |
Features
-
Multi-source aggregation: Same
fault_codefrom different sources creates a single fault - Occurrence tracking: Counts outages, not reports - the count starts at one and rises only when a cleared fault is raised again - and tracks all reporting sources
- Severity escalation: Fault severity is updated if a higher severity is reported
- Persistent storage: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
- Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
-
Snapshot capture: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless
snapshots.retain_on_clearis set) - Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
-
Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across
clear_fault(see below) - Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
- Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
storage_type |
string | "sqlite" |
Storage backend: "sqlite", "memory" or "postgres"
|
database_path |
string | "/var/lib/ros2_medkit/faults.db" |
Path to SQLite database file |
database_url |
string | "" |
PostgreSQL connection string; empty = libpq environment variables |
confirmation_threshold |
int | -1 |
Counter value at which faults are confirmed |
healing_enabled |
bool | false |
Enable automatic healing via PASSED events |
healing_threshold |
int | 3 |
Counter value at which faults are healed |
auto_confirm_after_sec |
double | 0.0 |
Auto-confirm PREFAILED faults after timeout (0 = disabled) |
report_fault_queue_depth |
int | 100 |
Requests ~/report_fault holds until read (1-65536, read-only); past it they are dropped without an answer |
entity_thresholds.config_file |
string | "" |
Path to YAML file with per-entity debounce threshold overrides |
fault_thresholds.config_file |
string | "" |
Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
near_miss.max_per_fault |
int | 200 |
Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
Snapshot Parameters
Snapshots capture topic data when faults are confirmed for post-mortem debugging.
Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.
Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.
That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.
| Parameter | Type | Default | Description |
|---|---|---|---|
snapshots.enabled |
bool | true |
Enable/disable snapshot capture |
snapshots.background_capture |
bool | false |
Cache each configured topic’s latest message vs on-demand capture; a topic absent at startup is cached once it appears |
snapshots.timeout_sec |
double | 1.0 |
Timeout waiting for topic message (on-demand mode) |
snapshots.max_message_size |
int | 65536 |
Maximum message size in bytes (larger messages skipped) |
snapshots.default_topics |
string[] | [] |
Topics to capture for all faults |
snapshots.config_file |
string | "" |
Path to YAML config for fault_specific and patterns
|
snapshots.recapture_cooldown_sec |
double | 60.0 |
Min seconds between captures for the same fault code. |
snapshots.max_per_fault |
int | 10 |
Max snapshots retained per fault. |
snapshots.capture_pool_size |
int | 2 |
Max concurrent capture threads under a fault storm (>= 1). Parallelizes snapshot capture only; rosbag stays single-writer. |
snapshots.capture_queue_depth |
int | 16 |
Max pending captures before the full-queue policy applies (>= 1). |
snapshots.capture_queue_full_policy |
string | reject_newest |
Policy when the queue is full: reject_newest or drop_oldest. |
File truncated at 100 lines see the full file
Changelog for package ros2_medkit_fault_manager
0.7.2 (2026-11-19)
- No changes. Version aligned with the rest of the stack
0.7.1 (2026-11-18)
- Recordings: every bag gets a unique path and the recording just stored is kept, a recording is hashed and stored off the executor thread, and a clear that arrives while a row is being stored takes that row (#785)
- A recording's reported size is the sum of the storage files its metadata names, and a bag's metadata is followed only to a storage file inside the bag. The snapshots payload no longer advertises a route that was removed (#679)
-
report_faultholds a burst of requests from one reporter instead of dropping part of it (#789) - A fault code can be debounced by the code, not only by who reported it (#690)
- A background snapshot topic that does not exist at startup is subscribed once it appears (#775)
- Fault clusters: the cluster severity stays current on re-reports, an active cluster's representative is reassigned without a pending cluster (#772), and the positive debounce counter is reset when a critical fault confirms immediately (#773)
- A time-based confirmation is as visible as a reported one, and
muting is honoured on the timer path.
healing_thresholdis 0 (it was 3) inconfig/fault_manager.yamland in the gateway'sconfig/bringup_params.yaml.bringup_params.yamlturns healing on, so with it one PASSED report now heals a fault: a fault from the action status bridge heals when its action recovers, where 0.7.0 kept it CONFIRMED.fault_manager.yamlkeeps healing off, so there the new value counts once healing is turned on (#643) - Optional PostgreSQL storage backend, off by default
(
-DPOSTGRES_SUPPORT=ON). It reconnects after a database outage (#651) -
Breaking (C++ API): the
FaultStoragevirtuals infault_storage.hppchanged:report_fault_eventtakesReportKindandint64_tnanoseconds, andcheck_time_based_confirmationtakesint64_t. A storage written for 0.7.0 no longer overrides them. Storage, thresholds, correlation, the audit log and the report, clear, list and get handling live in a ROS-free core (#740, #741, #742, #743, #744) - Parameters have upper bounds, for example 64 for
snapshots.capture_pool_size(#742, #744) - Contributors: \@bburda, \@mfaferek93, \@ahmedsleem109, \@gstavrinos, \@onur-ulusoy, \@Akinzou
0.7.0 (2026-08-27)
- Rosbag black-box recordings are no longer limited to one per fault
code. A fault that re-confirms keeps a bounded history of recordings
instead of overwriting the previous one, controlled by the new
snapshots.rosbag.max_bags_per_fault(default1, which reproduces the previous behaviour exactly;0= unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally therosbag_filesgrain changed from "one row per fault" to "one row per (fault, recording) link":recording_idis now a stored, indexed column, and the legacy column-levelUNIQUE(fault_code)is replaced by aUNIQUE INDEXon(fault_code, file_path)through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording,get_rosbag_filehad noORDER BYand would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and bothdelete_rosbag_file/delete_rosbag_filesread only the firstfile_pathof a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620) - Optional append-only, hash-chained audit log of fault state
transitions: each transition appends one immutable row
(
record_hash = sha256(prev_hash + canonical(event))via OpenSSL EVP SHA-256) with a persisted chain head, averifyroutine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited.verifyreads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering.BEFORE UPDATE/BEFORE DELETEtriggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, soverifydetects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by
File truncated at 100 lines see the full file