Expand description
Custom data persistence: shared helpers and orchestration.
Centralizes the logic for appending the data_type column and metadata to Arrow batches
(Parquet/Feather), and custom-data write preparation, path construction, and decode logic
so the catalog delegates here instead of inlining custom-specific branching.
Functionsยง
- augment_
batch_ with_ data_ type_ column - Appends a
data_typecolumn (JSON string per row) and type_name + optional metadata to the batch schema. Used by both the Parquet catalog and Feather writer for catalog-compatible output. - custom_
data_ path_ components - Returns path components for custom data:
["data", "custom", type_name, ...identifier segments]. Used by the catalog to build full object-store paths viamake_object_store_path_owned. - decode_
batch_ to_ data - Decodes a RecordBatch to Data objects based on metadata.
- decode_
custom_ batches_ to_ data - Decodes multiple RecordBatches (e.g. from custom data files) into a single
Vec<Data>. Optionally replacests_initcolumn withts_eventbefore decoding each batch. - prepare_
custom_ data_ batch - Prepares a batch of custom data for writing: encodes to Arrow, augments with data_type column, and returns type identity and timestamp range so the catalog can build path and perform I/O.
- schema_
with_ data_ type_ column - Builds a schema that adds the
data_typecolumn andtype_namemetadata to a base schema. Used when creating a Feather buffer for custom data (single type per writer).