Hey @CEH,

What you’re running into looks like a Spark Connect gRPC message-size limit, not a computational failure with the union itself. Even with smallish row counts, the serialized payload (either the inlined query plan or Arrow batch results) can blow past the default 128 MB gRPC cap and trigger a RESOURCE_EXHAUSTED error — the classic “Sent message larger than max (… vs. 134217728)”

Why this happens

• Local-relation inlining explodes plan size.

  If df1/df2 are created from local Python data, Spark inlines that data as a local_relation. When you union (or self-join) them, that data is duplicated in the plan, and the serialized plan can easily exceed 128 MB even if the row count looks tiny.

• Arrow batches can exceed the limit.

  display() or collect() sends Arrow batches back to the client. A single batch with wide string columns or many rows can tip the scale.

• Limit is hard-coded at 128 MB.

  Some configs exist to raise it, but Databricks environments may not honor them yet.

Practical fixes (pick one – or layer a few)

1️⃣ Materialize first, then union via SQL.

This swaps local_relation for a cached relation.

df1.createOrReplaceTempView("df1")
df2.createOrReplaceTempView("df2")

results = spark.sql("""
  SELECT c1, c2, c3, c4, c5, c6, c7 FROM df1
  UNION ALL
  SELECT c1, c2, c3, c4, c5, c6, c7 FROM df2
""")

display(results.limit(1000))

2️⃣ Persist and read back (Delta or managed tables).

Catalog-backed relations avoid the message-size constraint.

df1.write.mode("overwrite").saveAsTable("tmp.df1")
df2.write.mode("overwrite").saveAsTable("tmp.df2")

results = spark.sql("""
  SELECT * FROM tmp.df1
  UNION ALL
  SELECT * FROM tmp.df2
""")

display(results.limit(1000))

3️⃣ Reduce Arrow batch size.

Smaller batches → smaller gRPC messages.

spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "2000")
display(results)

4️⃣ Truncate wide string columns before display.

from pyspark.sql.functions import col, substring

small = results.select(
    col("c1"),
    *[substring(col(c), 1, 2000).alias(c) for c in ["c2","c3","c4","c5","c6"]],
    col("c7")
).limit(1000)

display(small)

5️⃣ Avoid big local Python objects.

Parallelize before creating DataFrames:

rdd = spark.sparkContext.parallelize(local_rows)
df1 = spark.createDataFrame(rdd, schema=my_schema)

6️⃣ (Advanced) Try raising the limit — if allowed:

spark.conf.set("spark.connect.grpc.maxInboundMessageSize", 268435456)  # 256 MB

May or may not be honored depending on your workspace setup.

💡 Why repartition() didn’t help

Repartitioning changes distribution, not the size of the serialized Arrow batch or query plan. The failure happens during serialization when the message crosses the 128 MB threshold — not during computation.

✅ Quick checklist to unblock you

• Convert df1/df2 to temp views and UNION ALL via SQL.

• Lower spark.sql.execution.arrow.maxRecordsPerBatch.

• Truncate long string columns before display.

• If data comes from local Python, use parallelize() or persist & read back.

Hope this helps, Lou.