Arrow batches in, Explorer out — and, for Latu.create_dataframe/3, the other direction.
This module is Latu's Explorer boundary in both directions; nothing else touches Arrow.
It also owns the shape a schema comes back as. Latu.schema/1 and Latu.parse_ddl/2 return
a list of field/0, each carrying Spark's own name for its type, as a string: there is
no client-side type model in either direction — Latu.create_dataframe/3 takes a DDL string
going out, and this is what comes back (M8.1 and M10.1; docs/decisions.md). Nested types
render into the type string — array<int>, struct<a:int,b:string> — so a schema stays a
flat, pattern-matchable list however deep the data is.
A schema is also checked before any bytes are decoded: Spark's interval types, and two columns
of one name, panic inside Polars' NIF rather than failing cleanly, so a result carrying either
is refused, by column name, using the DataType the server sends ahead of the first batch.
Summary
Functions
The frame's columns in this order — the order a schema names them, for
Latu.create_dataframe/3. Every name must be a column.
Decode Arrow batches into one Explorer DataFrame.
An Explorer DataFrame from column data — {name, values} pairs, in the given order.
Column names, in the frame's order.
The one cell of a 1x1 result, positionally — PySpark's table[0][0].
The one cell of a 1x1 result, as ShowString and HtmlString produce.
The rows of a decoded frame, as maps.
Row count, so callers outside the Explorer boundary need no Explorer call.
One Arrow IPC stream for the whole frame — exactly the bytes LocalRelation.data carries.
The frame as row slices, each dumped as its own complete IPC stream.
Types
Functions
@spec arrange(Explorer.DataFrame.t(), [String.t()]) :: Explorer.DataFrame.t()
The frame's columns in this order — the order a schema names them, for
Latu.create_dataframe/3. Every name must be a column.
@spec decode( [Latu.Client.batch()], keyword() ) :: {:ok, Explorer.DataFrame.t()} | {:error, Latu.Error.t()}
Decode Arrow batches into one Explorer DataFrame.
One load_ipc_stream per batch. Each batch is a complete IPC stream with its own end-of-
stream marker, so concatenating the binaries first would silently keep the first batch's rows
and drop the rest — the decoded row count is checked against the server's for that reason.
Options: :columns, passed to Explorer at load time so pruned columns are never deserialized.
A payload Polars cannot parse raises rather than returning an error; Spark's interval types are the live hazard, and the schema guard described above is what heads them off — callers run it on the execution's schema before handing bytes here.
@spec from_columns([{atom() | String.t(), list()}] | Explorer.DataFrame.t()) :: Explorer.DataFrame.t()
An Explorer DataFrame from column data — {name, values} pairs, in the given order.
The encode side of Latu.create_dataframe/3; ragged or untypeable columns raise, as
Explorer.DataFrame.new/1 does.
@spec names(Explorer.DataFrame.t()) :: [String.t()]
Column names, in the frame's order.
@spec only(Explorer.DataFrame.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
The one cell of a 1x1 result, positionally — PySpark's table[0][0].
@spec only(Explorer.DataFrame.t(), String.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
The one cell of a 1x1 result, as ShowString and HtmlString produce.
@spec rows(Explorer.DataFrame.t(), :atoms | :strings) :: [map()]
The rows of a decoded frame, as maps.
Streamed out in chunks rather than through Explorer.DataFrame.to_rows/2, which converts
every column to a full list before emitting one row. :atoms interns the
column names — bounded by the schemas ever selected, not by data.
@spec size(Explorer.DataFrame.t()) :: non_neg_integer()
Row count, so callers outside the Explorer boundary need no Explorer call.
@spec to_ipc(Explorer.DataFrame.t()) :: binary()
One Arrow IPC stream for the whole frame — exactly the bytes LocalRelation.data carries.
Raises on a frame Explorer cannot serialize; that is an argument problem, not a transport one.
@spec to_ipc_chunks(Explorer.DataFrame.t(), pos_integer()) :: [binary()]
The frame as row slices, each dumped as its own complete IPC stream.
Every chunk must independently decode — the server caches each as a separate artifact and
reads them back by hash (ChunkedCachedLocalRelation) — which is why this slices and
re-dumps rather than splitting to_ipc/1's bytes.