Module catalog 

Parquet data catalog for efficient storage and retrieval of financial market data.

This module provides a data catalog implementation that uses Apache Parquet format for storing financial market data with object store backends. The catalog supports various data types including quotes, trades, bars, order book data, and other market events.

Key Features

• Object Store Integration: Works with local filesystems, S3, and other object stores.
• Data Type Support: Handles all major financial data types (quotes, trades, bars, etc.).
• Time-based Organization: Organizes data by timestamp ranges for efficient querying.
• Consolidation: Merges multiple files to optimize storage and query performance.
• Validation: Ensures data integrity with timestamp ordering and interval validation.

Architecture

The catalog organizes data in a hierarchical structure:

data/
├── quotes/
│   └── INSTRUMENT_ID/
│       └── start_ts-end_ts.parquet
├── trades/
│   └── INSTRUMENT_ID/
│       └── start_ts-end_ts.parquet
└── bars/
    └── INSTRUMENT_ID/
        └── start_ts-end_ts.parquet

Usage

use std::path::Path;

use nautilus_persistence::backend::parquet::catalog::ParquetDataCatalog;

// Create a new catalog
let catalog = ParquetDataCatalog::new(
    Path::new("/path/to/data"),
    None,       // storage_options
    Some(5000), // batch_size
    None,       // compression (defaults to SNAPPY)
    None,       // max_row_group_size (defaults to 131,072)
);

// Write data to the catalog
// catalog.write_to_parquet(&data, None, None, None)?;

Re-exports

pub use crate::backend::parquet::intervals::are_intervals_contiguous;

pub use crate::backend::parquet::intervals::are_intervals_disjoint;

pub use crate::backend::parquet::paths::CatalogPathPrefix;

pub use crate::backend::parquet::paths::extract_identifier_from_path;

pub use crate::backend::parquet::paths::extract_path_components;

pub use crate::backend::parquet::paths::extract_sql_safe_filename;

pub use crate::backend::parquet::paths::local_to_object_store_path;

pub use crate::backend::parquet::paths::make_local_path;

pub use crate::backend::parquet::paths::make_object_store_path;

pub use crate::backend::parquet::paths::make_sql_safe_identifier;

pub use crate::backend::parquet::paths::parse_filename_timestamps;

pub use crate::backend::parquet::paths::safe_directory_identifier;

pub use crate::backend::parquet::paths::timestamps_to_filename;

pub use crate::backend::parquet::paths::urisafe_instrument_id;

Structs

ParquetDataCatalog

A high-performance data catalog for storing and retrieving financial market data using Apache Parquet format.

Constants

QUERY_OPTIMIZE_FILE_LOADING

Optional Parquet query file-loading hint parameter key.

WRITE_SKIP_DISJOINT_CHECK

Optional Parquet write overlap bypass parameter key.