mod parquet_source

module parquet_source

ParquetMarkerSource: marker stream backed by a Parquet temp file.

Used when the predicted working set would not safely fit in RAM. The Parquet payload has the same schema as the in-memory variant (id, sequence, ind1, …, indN) so the per-row decoding is shared via super::arrow_source::ArrowMarkerSource::from_batches.

Enums

enum ParquetSourceError

EmptyInput

TempFile(String)

Write(String)

Traits implemented

impl std::fmt::Display for ParquetSourceError

impl std::error::Error for ParquetSourceError

Structs and Unions

struct ParquetMarkerSource

Spilled-to-Parquet marker source. Owns the underlying temp file so it is removed automatically when the source is dropped.

Implementations

impl ParquetMarkerSource

Functions

fn path(&self) -> &std::path::Path

Path of the on-disk Parquet payload (e.g. for diagnostics).

fn spill_from_arrow(in_mem: &ArrowMarkerSource) -> Result<Self, ParquetSourceError>

Materialise an in-memory ArrowMarkerSource as a Parquet temp file and wrap it as a ParquetMarkerSource. The original batches are dropped once the file is written.