Stream output to an IPC file — sink

This function allows to stream a LazyFrame that is larger than RAM directly to an IPC file without collecting it in the R session, thus preventing crashes because of too small memory.

Usage

sink_ipc(
  .data,
  path,
  ...,
  compression = "zstd",
  maintain_order = TRUE,
  type_coercion = TRUE,
  predicate_pushdown = TRUE,
  projection_pushdown = TRUE,
  simplify_expression = TRUE,
  slice_pushdown = TRUE,
  no_optimization = FALSE
)

Arguments

.data: A Polars LazyFrame.
path: Output file.
...: Ignored.
compression: NULL or a character of the compression method, "uncompressed" or "lz4" or "zstd". NULL is equivalent to "uncompressed". Choose "zstd" for good compression performance. Choose "lz4" for fast compression/decompression.
maintain_order: Whether maintain the order the data was processed (default is TRUE). Setting this to FALSE will be slightly faster.
type_coercion: Coerce types such that operations succeed and run on minimal required memory (default is TRUE).
predicate_pushdown: Applies filters as early as possible at scan level (default is TRUE).
projection_pushdown: Select only the columns that are needed at the scan level (default is TRUE).
simplify_expression: Various optimizations, such as constant folding and replacing expensive operations with faster alternatives (default is TRUE).
slice_pushdown: Only load the required slice from the scan. Don't materialize sliced outputs level. Don't materialize sliced outputs (default is TRUE).
no_optimization: Sets the following optimizations to FALSE: predicate_pushdown, projection_pushdown, slice_pushdown, simplify_expression. Default is FALSE.

Value

The input LazyFrame.