Latu.Result.Nx (latu v0.4.0)

Copy Markdown View Source

Latu.Result.Arrow's buffers as Nx tensors.

Two shapes, and nothing else:

  • a numeric column with no nulls becomes a 1-D tensor of its own type. The Arrow buffer is already the tensor's binary, so a single batch costs no copy at all.
  • a column of equal-length numeric lists, and a column of dense Vectors, become one {rows, width} tensor. Both are one contiguous buffer in Arrow with an offsets buffer beside it, so this is a check that the offsets are regular and then a reshape.

Everything else is refused by name, because a tensor has one type and one shape and there is no honest default for a column that has neither: nulls, strings, booleans (Arrow packs them as a bitmap, not a byte per value), ragged lists, sparse vectors, and anything nested beyond the two shapes above.

A Vector column is the reason this exists. Spark describes it as a UDT with no SQL type, so Latu.collect/2 and Latu.to_explorer/2 both refuse it (docs/deviations.md) — but the Arrow stream carries its own schema, and in there it is an ordinary struct whose values child is a list of doubles.

Summary

Types

tensors()

Every requested column, keyed by name.

Functions

decode(streams, opts \\ [])

Decode Arrow IPC streams — Latu.to_arrow/2's batches — into one tensor per column.

Types

tensors()

@type tensors() :: %{required(String.t()) => Nx.Tensor.t()}

Every requested column, keyed by name.

Functions

decode(streams, opts \\ [])

@spec decode(
  [binary()],
  keyword()
) :: {:ok, tensors()} | {:error, String.t()}

Decode Arrow IPC streams — Latu.to_arrow/2's batches — into one tensor per column.

Options: :columns, a list of names to keep. Pruning copies: an Arrow buffer is a sub-binary of the whole batch and holds it alive, so keeping one column of a wide result without copying would retain every other column's bytes too. Taking the whole batch does not copy, because there is nothing left to release.