Contributing

Copy Markdown View Source

Requires Elixir ~> 1.20, Docker, protoc, and Python 3 for the plan oracle.

mix deps.get
docker compose up -d --wait            # both spark servers; the reattach profile is not optional
python3 -m venv dev/.venv && \
    dev/.venv/bin/pip install \
    -r dev/requirements.txt            # the plan oracle
mix fixtures                           # the integration suite's data files, once per checkout

mix check                              # format + warnings-as-errors + offline tests
mix check.all                          # the same, plus the integration suite

Both halt on the first failure and exit non-zero. docker compose up -d spark-connect alone starts the ordinary server on :15002; the second profile on :15003 runs with a five-second senderMaxStreamDuration, which is what makes the reattach tests mean anything.

mix fixtures needs the oracle's venv. The data files it writes are generated rather than committed, so the suite stays hermetic and carries no third-party data licence. mix check needs none of them; mix check.all refuses to start without them, naming this command.

Conventions — naming, the shapes a builder and an action return, how errors are worded, how a verb is documented — live in CLAUDE.md. Per-decision rationale is in docs/decisions.md, a dated log; every place the public API departs from PySpark is in docs/deviations.md, with why.

Tests

Unit tests mirror lib/ under test/. Anything touching the network lives in test/integration/ behind an :integration tag, excluded by default.

Four of them are checks on the repo rather than on Latu, and are worth knowing about before you trip one:

  • test/latu/docs_test.exs — every public function in a documented module has a @doc; every documented module has a groups_for_modules entry; every guide in docs/guides/ is in mix.exs's :extras and its Guides group; and every relative link in every extra resolves, to the file it names.
  • test/latu/function_calls_test.exs — every F.<name>/<arity> call site anywhere in lib, test or dev matches a real export, read from the AST so the arity is exact.
  • test/latu/examples_test.exs — every code block in every docstring and prose doc parses as Elixir and names only functions that exist at the arity shown, including inline Latu. calls in a sentence or a table cell. It also insists that every markdown file in the repo is checked, run as a guide, or excused there by name.
  • test/latu/options_test.exs — every option any verb accepts is explained under ## Options in the facade's own docstring, with every value of a closed set spelled out. It reads the Keyword.validate! and lookup(@const, ...) calls in lib/ rather than a list anyone maintains.

Documentation is executed

An example that is not executed is a lie with a shelf life, so every example is run one of three ways:

  • a doctest, for anything pure — no server, runs in mix check;
  • an elixir fence in docs/guides/*.md, for anything that needs a server, run in order by test/integration/guides_test.exs with one shared binding. A guide asserts by matching, so [%{id: 5} | _] = rows is documentation and a test at once;
  • otherwise examples_test.exs above, which cannot run the example but proves every function it names is real.

A fence that cannot run against the test server — a merge needs an Iceberg or Delta target — is preceded by a visible line beginning

> **Not executed.**

with the reason on the same line. The runner skips it and guides_test.exs asserts the whole skipped set by guide and reason, so a new one fails until somebody writes down why. The marker is a blockquote rather than an HTML comment because the page tells readers its snippets are executed: the exceptions belong on the page, not in its source.

dev/example.exs is Latu's own smoke test — mix run dev/example.exs with a server up — not the source of the README's snippets.

The plan oracle and the golden fixtures

Latu's correctness rests on its plan being byte-identical to the one PySpark builds for the same pipeline. dev/pyspark_oracle.py generates those expectations and dev/README.md explains the workflow, the two servers, and how to read a plan.

Protobuf modules under lib/latu/protocol/generated/ are checked in and generated from the vendored Spark 4.2.0 protos in priv/proto/. Do not edit them by hand:

mix proto.generate

Release checklist

Maintainer only. The nested notes are the traps each step exists for; the reasoning behind them is in docs/decisions.md.

Before the release commit

  • [ ] every branch belonging in this version is merged to main
    • a branch touching lib/ is a release change however it is named: git diff --stat main..<branch> -- lib/
  • [ ] no ## Unreleased section left in CHANGELOG.md — fold it into this version's entry while the version is still unpublished
  • [ ] CHANGELOG.md's top entry is the new version and date, migration in one line
  • [ ] @version in mix.exs matches it
  • [ ] the README's install snippet names the new minor
  • [ ] mix check.all green, both compose profiles up
  • [ ] git rev-parse main origin/main match

Release

  • [ ] mix hex.build, and read the file list it prints
  • [ ] working tree clean — mix hex.publish builds the docs from the tree, not from the tarball
  • [ ] git tag v<version> and git push origin v<version>, before publishing
  • [ ] mix hex.publish

Within the hour

mix hex.publish --revert <version> works for one hour on a version that is not the package's first; after that, retire only. Check the versioned URLs — the unversioned pages sit behind a CDN cache and go on showing the previous release for a while.

  • [ ] hex.pm/packages/latu/<version> renders
  • [ ] readme.hex.pm/latu/<version>
  • [ ] latu.hexdocs.pm/<version>/readme.html
  • [ ] a "view source" link on hexdocs lands at v<version>
  • [ ] every README link works on hex.pm, not only on GitHub