rosbag2_storage_sqlite3

Storage implementation plugin for rosbag2 providing SQLite3 .db3 files as the bag storage backend.

Storage Configuration File

The --storage-config-file option for this plugin takes files in the following format:

read:
  pragmas: <list of SQLite pragma settings for read-only>
write:
  pragmas: <list of SQLite pragma settings for write modes>

By default, SQLite settings are significantly optimized for performance. This might have consequences of bag data being corrupted after an application or system-level crash. This consideration only applies to current bagfile in case bag splitting is on (through --max-bag-* parameters). If increased crash-caused corruption resistance is necessary, use resilient option for --storage-preset-profile setting.

Settings are fully exposed to the user and should be applied with understanding. Please refer to documentation of pragmas.

An example configuration file could look like this:

write:
  pragmas: ["journal_mode = MEMORY", "synchronous = OFF", "schema.cache_size = 1000", "schema.page_size = 4096"]

Changelog for package rosbag2_storage_default_plugins

0.26.5 (2024-09-06)

• Fix incorrect zero size for sqlite storage (#1759) (#1761) Co-authored-by: Michael Orlov <<michael.orlov@apex.ai>> (cherry picked from commit 86f681d9f525a4b7c3b4133d6b657e342283aad8) Co-authored-by: Roman <<rsokolkov@gmail.com>>
• Fix for failing throws_on_invalid_pragma_in_config_file on Windows (#1742) (#1746) (cherry picked from commit 055935d33dd2fed2772657c9dc1f2173eaa7f752) Co-authored-by: Michael Orlov <<michael.orlov@apex.ai>>
• Contributors: mergify[bot]

0.26.4 (2024-06-27)

• Add topics with zero message counts to the SQLiteStorage::get_metadata(). (#1725) (#1731) Co-authored-by: Michael Orlov <<michael.orlov@apex.ai>> Co-authored-by: Tomoya Fujita <<Tomoya.Fujita@sony.com>>
• Contributors: mergify[bot]

0.26.3 (2024-05-15)

0.26.2 (2024-04-24)

0.26.1 (2024-04-17)

0.26.0 (2024-04-16)

• Support service 2/2 --- rosbag2 service play (#1481)
• Use middleware send and receive timestamps from message_info during recording (#1531)
• Update to use yaml-cpp version 0.8.0. (#1605)
• Contributors: Barry Xu, Chris Lalancette, jmachowinski, Michael Orlov

0.25.0 (2024-03-27)

• Make some changes for newer versions of uncrustify. (#1578)
• Add topic_id returned by storage to the TopicMetadata (#1538)
• Remove rcpputils::fs dependencies from rosbag2_storages (#1558)
• Change an incorrect TSA annotation. (#1552)
• Improve performance in SqliteStorage::get_bagfile_size() (#1516)
• Update rosbag2_storage_sqlite3 to C++17. (#1501)
• Use enum values for offered_qos_profiles in code and string names in serialized metadata (#1476)
• Stop inheriting from std::iterator. (#1424)
• Contributors: Chris Lalancette, Michael Orlov, Patrick Roncagliolo, Roman Sokolkov

0.24.0 (2023-07-11)

• Implement storing and loading ROS_DISTRO from metadata.yaml and mcap files (#1241)
• Store metadata in db3 file (#1294)
• Contributors: Emerson Knapp, Michael Orlov

0.23.0 (2023-04-28)

0.22.0 (2023-04-18)

• Add type_hash in MessageDefinition struct (#1296)
• Store message definitions in SQLite3 storage plugin (#1293)
• Add message definition read API (#1292)
• rosbag2_storage: add type description hash to topic metadata (#1272)
• Contributors: Michael Orlov, james-rms

0.21.0 (2023-04-12)

• rosbag2_cpp: move local message definition source out of MCAP plugin (#1265)
• Update rosbag2 to C++17. (#1238)
• Use target_link_libraries instead of ament_target_dependencies (#1202)
• Contributors: Chris Lalancette, Daisuke Nishimatsu, Michael Orlov, james-rms

0.20.0 (2023-02-14)

• CLI: Get storage-specific values from plugin (#1209)
• Contributors: Emerson Knapp

0.19.0 (2023-01-13)

• Add Michael Orlov as maintainer in rosbag2 packages (#1215)
• Remove sqlite3-specific info from main README, make it more storage agnostic and point to plugin-specific README (#1193)
• set_read_order: return success (#1177)
• Add [update_metadata(BagMetadata)]{.title-ref} API for storage plugin interface (#1149)
• Store db schema version and ROS_DISTRO name in db3 files (#1156)
• Contributors: Emerson Knapp, Michael Orlov, james-rms

0.18.0 (2022-11-15)

• ros2bag: move storage preset validation to sqlite3 plugin (#1135)
• Move sqlite3 storage implementation to rosbag2_storage_sqlite3 package (#1113)
• Contributors: Emerson Knapp, james-rms

0.17.0 (2022-07-30)

• Use a single variable for evaluating the filter regex (#1053)
• Renamed --topics-regex to --regex and -e in Player class to be consistent with Recorder (#1045)
• Added support for filtering topics via regular expressions on Playback (#1034)
• Contributors: Esteve Fernandez

0.16.0 (2022-05-11)

0.15.1 (2022-04-06)

0.15.0 (2022-04-05)

0.14.1 (2022-03-29)

• Bump version number to avoid conflict
• Contributors: Chris Lalancette

0.14.0 (2022-03-29)

• Install headers to include/${PROJECT_NAME} (#958)
• Contributors: Shane Loretz

0.13.0 (2022-01-13)

• Emit a warning rather than crash when a message is too big for sqlite (#919)
• Contributors: William Woodall

0.12.0 (2021-12-17)

• Enable YAML encoding/decoding for RecordOptions and StorageOptions (#916)
• Contributors: Emerson Knapp

0.11.0 (2021-11-08)

• Update package maintainers (#899)
• Contributors: Michel Hidalgo

0.10.1 (2021-10-22)

0.10.0 (2021-10-19)

• added seek interface (#836)
• Contributors: sonia

0.9.0 (2021-05-17)

0.8.0 (2021-04-19)

• Remove -Werror from builds, enable it in Action CI (#722)
• Explicitly add emersonknapp as maintainer (#692)
• Reindexer core (#641) Add a new C++ Reindexer class for reconstructing metadata from bags that are missing it.
• Contributors: Emerson Knapp, jhdcs

0.7.0 (2021-03-18)

0.6.0 (2021-02-01)

• Fix build issues when rosbag2_storage is binary installed (#585)
• Mutex protection for db writing and stl collections in writer & storage (#603)
• SQLite storage optimized by default (#568)
• Contributors: Adam Dąbrowski, P. J. Reed

0.5.0 (2020-12-02)

0.4.0 (2020-11-19)

• read yaml config file (#497)
• add storage_config_uri (#493)
• Update the package.xml files with the latest Open Robotics maintainers (#535)
• Contributors: Karsten Knese, Michael Jeronimo

0.3.2 (2020-06-03)

0.3.1 (2020-06-01)

0.3.0 (2020-05-26)

0.2.8 (2020-05-18)

0.2.7 (2020-05-12)

0.2.6 (2020-05-07)

0.2.5 (2020-04-30)

• Add filter for reading selective topics (#302)
• Transaction based sqlite3 inserts (#225)
• Add QoS profiles field to metadata struct and provide serialization utilities (#330)
• Replace rcutils_get_file_size with rcpputils::fs::file_size (#291)
• code style only: wrap after open parenthesis if not in one line (#280)
• Improve SQLite error messages (#269)
• remove rosbag2 filesystem helper (#249)
• Sanitize bagfile splitting CLI input (#226)
• make ros tooling working group maintainer (#211)
• Contributors: Dirk Thomas, Emerson Knapp, Karsten Knese, Mabel Zhang, Prajakta Gokhale, Sriram Raghunathan, Zachary Michaels

0.2.4 (2019-11-18)

0.2.3 (2019-11-18)

• Enhance rosbag writer capabilities to split bag files. (#185)
• Contributors: Zachary Michaels

0.2.2 (2019-11-13)

• (API) Generate bagfile metadata in Writer (#184)
• Contributors: Zachary Michaels

0.2.1 (2019-10-23)

• Add get_identifier to storage io-interface for support in bagfile splitting. (#183)
• Change storage interfaces for bagfile splitting feature (#170)
• Add error checking on SqliteWrapper deconstructor. (#169)
• Contributors: Zachary Michaels

0.2.0 (2019-09-26)

0.1.2 (2019-05-20)

• Indexing of messages.timestamp to avoid runtime-copy. (#121) Extended SqliteStorage::initialize() to add an index for the message table's timestamp column. Without this, the ORDER BY query in prepare_for_reading() causes expensive table duplication, which also has potential for out-of-disk or out-of-memory errors.
• Fixes an init race condition (#93) * This could probably be a race condition, for ex: When we've create a subscriber in the API, and the subscriber has the data already available in the callback (Cause of existing publishers) the db entry for the particular topic would not be availalble, which in turn returns an SqliteException. This is cause write_->create_topic() call is where we add the db entry for a particular topic. And, this leads to crashing before any recording. Locally I solved it by adding the db entry first, and if create_subscription fails, remove the topic entry from the db and also erase the subscription. Signed-off-by: Sriram Raghunathan <<rsriram7@visteon.com>> * Fix comments for pull request https://github.com/ros2/rosbag2/pull/93 Signed-off-by: Sriram Raghunathan <<rsriram7@visteon.com>> * Added unit test case for remove_topics from db Signed-off-by: Sriram Raghunathan <<rsriram7@visteon.com>> * Fix unit tests failing by adding dependent test macros Signed-off-by: Sriram Raghunathan <<rsriram7@visteon.com>>
  • Fixes the linter errors
• Contributors: Felix-El, Sriram Raghunathan

0.1.1 (2019-05-09)

0.1.0 (2019-05-08)

• fix line length of logging macros (#110)
• fix logging signature (#107)
• Contributors: Dirk Thomas, Karsten Knese

0.0.5 (2018-12-27)

0.0.4 (2018-12-19)

• 0.0.3
• Play old bagfiles (#69)
• Contributors: Karsten Knese, Martin Idel

0.0.2 (2018-12-12)

• update maintainer email
• fix unused variable warning when in release
• Contributors: Karsten Knese

0.0.1 (2018-12-11)

• rename topic_with_types to topic_metadata
• GH-142 replace map with unordered map where possible (#65)
• Use converters when recording a bag file (#57)
• use uint8 for serialized message (#61)
• Renaming struct members for consistency (#64)
• Display bag summary using [ros2 bag info]{.title-ref} (#45)
• Use directory as bagfile and add additonal record options (#43)
• Introduce rosbag2_transport layer and CLI (#38)
• Add correct timing behaviour for rosbag play (#32)
• Improve sqlite iterator interface (#33)
• Improve sqlite usage and test stability (#31)
• Record all topics (#30)
• Record and play multiple topics (#27)
• Allow an arbitrary topic to be recorded (#26)
• Use serialized message directly (#24)
• add visibility macros (#28)
• initial version of plugin based storage api (#7)
• Contributors: Alessandro Bottero, Andreas Greimel, Andreas Holzner, Karsten Knese, Martin Idel