Latu.Result.Arrow (latu v0.4.0)

Copy Markdown View Source

An Arrow IPC streaming-format reader: the bytes the server sends, without Explorer.

Latu.to_explorer/2 hands its batches to Polars, which is the right answer for a DataFrame and the wrong one for a tensor — it decodes every column into a Series first. This reads the same bytes far enough to hand back, per column, sub-binaries of the batch: no copy, and no interpretation beyond what Arrow itself says. Latu.Result.Nx is what turns those into tensors, and is the only caller that needs to.

It reads only what Spark sends. A dictionary batch, a compressed body, or a stream whose schema and batches disagree is refused rather than guessed at; refusals are plain strings, because the caller has the column name and the verb and can say more than this can.

What a column comes back as

%{
  name: "features",
  type: {:float, 64} | {:int, 32, :signed} | :bool | :utf8 | :list | :struct | {:other, _},
  length: 4,
  null_count: 0,
  buffers: [validity_or_nil, ...],
  children: [column]
}

Buffers are in Arrow's own order for the type — [validity, data] for a primitive, [validity, offsets] plus one child for a list, [validity] plus N children for a struct — and a buffer of length zero comes back as nil, which is how Arrow spells "no nulls".

Summary

Types

batch()

One record batch: how many rows, and the top-level columns.

column()

One column of one batch: its type, its Arrow buffers, and its children.

type()

A column's type, as Arrow declares it. Anything not named here is {:other, name}.

Functions

read(stream)

Every record batch in one IPC stream.

schema(stream)

The schema's top-level column names and types, without reading a batch.

Types

batch()

@type batch() :: %{rows: non_neg_integer(), columns: [column()]}

One record batch: how many rows, and the top-level columns.

column()

@type column() :: %{
  name: String.t(),
  type: type(),
  length: non_neg_integer(),
  null_count: non_neg_integer(),
  buffers: [binary() | nil],
  children: [column()]
}

One column of one batch: its type, its Arrow buffers, and its children.

type()

@type type() ::
  {:int, pos_integer(), :signed | :unsigned}
  | {:float, 16 | 32 | 64}
  | :bool
  | :utf8
  | :list
  | :struct
  | :null
  | {:other, atom()}

A column's type, as Arrow declares it. Anything not named here is {:other, name}.

Functions

read(stream)

@spec read(binary()) :: {:ok, [batch()]} | {:error, String.t()}

Every record batch in one IPC stream.

The stream is [schema][batch]…[end-of-stream], which is what one element of Latu.to_arrow/2 holds. A stream with no batches — an empty result still carries its schema — answers {:ok, []}.

schema(stream)

@spec schema(binary()) :: {:ok, [{String.t(), type()}]} | {:error, String.t()}

The schema's top-level column names and types, without reading a batch.