Spark versions

Latu targets Spark 4.2.0. The vendored protos, the golden plans and the servers mix check.all runs against are 4.2.0, and that is the only version the README claims.

This page is what happens against another Spark, measured rather than promised. The integration suite ran against other apache/spark images with .github/workflows/spark-versions.yml; every table names the image tag and the date. It is a record of runs, not a support matrix, and a version not listed here was not run.

How a 4.2 client fails against an older server

Spark Connect is protobuf over gRPC, and protobuf is forgiving in a way that matters here. Three things can happen when Latu sends something the server predates. Two are refused, one is not.

An RPC the server lacks is refused. gRPC answers UNIMPLEMENTED, and %Latu.Error{} has status: 12 with a message naming the method, such as Method not found: spark.connect.SparkConnectService/GetStatus. Latu adds that the server does not implement the RPC.

A field the server lacks is dropped. An unknown protobuf field is skipped on decode, not refused. When that field was the arm of a oneof, the server sees the oneof unset and reports that as its own error: This oneOf field in spark.connect.Relation is not set: RELTYPE_NOT_SET on 4.1, Expected Relation to be set, but is empty. on 4.0, with INTERNAL_ERROR as the class where there is one. Latu recognises these phrasings and adds that it sent a node the server does not know. Latu.nearest_by_join/4 and Latu.Catalog.drop_table/3 fail this way on both versions below.

An enum value the server lacks is refused. It decodes as UNRECOGNIZED and the server says so: Unknown SubqueryType UNRECOGNIZED is what 4.0 answers an IN subquery with.

The one that is not refused is the residual risk. A dropped field that was optional on a message the server does know leaves nothing unset, so the server runs the request without it, with its own default in place, and reports nothing. Latu offers no knob that would fail this way: write_stream/2 has no real_time trigger for exactly this reason, since real_time_batch_duration is 4.2-only and a 4.1 server would run a plain processing-time trigger while the caller believed otherwise. The list of 4.2-only optional fields is Spark's, and this page cannot prove it empty; it can say the suite found no case where a result differed silently.

Beyond the wire, an error class can move between versions, and matching on one is the right thing to do, so the moves are listed. And a conf's string can change shape: 4.1 reports spark.sql.session.localRelationSizeLimit as 3221225472b where 4.2 reports a bare count, which is why Latu parses Spark's byte strings rather than integers.

Spark 4.1.3

Measured 2026-09-11, apache/spark:4.1.3, Latu at fb941a3 plus the fixes this run produced. 13 failures in a suite of 1229, by the surface each uses:

Everything else passes: create_dataframe/3 over the cache threshold, since ChunkedCachedLocalRelation is 4.1 rather than 4.2 once its byte-string conf is parsed; IN subqueries; clone_session/2; structured streaming apart from the two classes above; and the whole relational, functions, analysis, config and result surface.

Spark 4.0.4

Measured 2026-09-11, apache/spark:4.0.4, same Latu, locally. 27 failures in a suite of 1230: 4.1.3's 13 and these:

The two streaming refusals in 4.1.3's table carry no class at all here: error_class is nil. The two dropped-field surfaces read differently, Expected Relation to be set, but is empty. and CATTYPE_NOT_SET not supported., and Latu recognises both phrasings.

Rendering is unchanged on both. The nine result goldens in test/sql, which pin the schema and the rendered table of deterministic queries over numeric widening, decimals, temporals, nulls, complex types, casts, intervals, string functions and aggregates, are byte-identical on 4.0.4 and on 4.1.3. Whatever 4.2 changed in how a value prints, it is not in that set.

Spark 3.5

Not measured. The apache/spark:3.5.x images ship Scala 2.12 without the Connect server in the assembly, so the compose file cannot start one; a 3.5 measurement needs the -scala2.13 distribution with --packages org.apache.spark:spark-connect_2.13:3.5.x, or a managed cluster that runs 3.5. Every failure above applies, and 3.5 predates more: Latu.zip_with_index/2, Latu.transpose/2, Latu.lateral_join/3, Latu.table_changes/3, Latu.parse/2 and the time type all arrive in 4.x. Treat 3.5 as unsupported until a run says otherwise.

Newer than 4.2

Not measured either, and the direction is different: what breaks against a newer server is invisible to a golden that pins what Latu sends, because the bytes are unchanged and the answer moves. The Spark versions workflow runs the integration suite against any tag given as its input, and the result goldens in test/sql report a changed rendering into the run summary rather than failing. 4.3.0-rc1 existed upstream on 2026-09-07 with no image published; the first newer run happens when there is a tag to give it.

Support calendar

From spark.apache.org's release policy: 4.0 is supported until 2026-11, 4.1 until 2027-06, 4.2 until 2028-01, and 3.5 as LTS until 2027-11. Latu re-vendors its protos at the designated 4.x LTS, 4.5.0, around January 2027.

Managed platforms

A managed endpoint is an sc:// URL with headers, and Latu.Session's URL parser turns any ;key=value it does not recognise into a gRPC metadata header, so none of these needs code. None of them has been run by Latu's suite; the URL shapes are the platforms' own.

Databricks, a classic cluster: sc://<workspace-host>:443/;use_ssl=true;token=<pat>;x-databricks-cluster-id=<cluster-id>. The token becomes authorization: Bearer <pat>, and Latu refuses to send one over cleartext, which use_ssl=true satisfies. Serverless compute provisions the session client-side in Databricks' own connector and exposes no URL of this shape.

EMR Serverless: GetSessionEndpoint hands out sc://<endpoint-host>:443/;use_ssl=true;x-aws-proxy-auth=<token>, which is the same mechanism. emr-7.13.0 runs Spark 3.5.6, so the 3.5 section applies.

Dataproc provisions the session client-side, like Databricks serverless, and is not reachable by URL.

Latu.spark_version!/1 is the first thing to run against any of them.

Running the suite against another version

docker-compose.yml reads SPARK_VERSION for both servers, and so does the one test that asserts the version. Locally:

export SPARK_VERSION=4.1.3
docker compose up -d --force-recreate --wait
mix test --include integration 2>&1 | tee tmp/versions-$SPARK_VERSION.log
LATU_GOLDEN=report mix test --include golden
for f in test/sql/*.actual; do diff -u "${f%.actual}.answer" "$f"; done
unset SPARK_VERSION && docker compose up -d --force-recreate --wait

The .actual files are gitignored, and a golden that differs is a rendering the other version changed, not a Latu defect. On GitHub, gh workflow run spark-versions.yml -f versions='["4.1.3"]' does the same and writes the failing tests and the golden diffs into the run summary.