As part of a function I create df1 and df2 and aim to stack them and output the results.  But the results do not display within the function, nor if I output the results and display after.

results = df1.unionByName(df2, allowMissingColumns=False)

display(results)

This is the error:

SparkConnectGrpcException: <_InactiveRpcError of RPC that terminated with:

status = StatusCode.RESOURCE_EXHAUSTED

details = "CLIENT: Sent message larger than max (200529144 vs. 134217728)"

df1 is 350 rows and df2 is 1500 rows.  They share the same seven ordered columns, and I have checked they share the same schema.  Though df1 does have pure nulls for c3 and c4.

 |-- c1: long (nullable = true)

 |-- c2: string (nullable = true)

 |-- c3: string (nullable = true)

 |-- c4: string (nullable = true)

 |-- c5: string (nullable = true)

 |-- c6: string (nullable = true)

 |-- c7: double (nullable = true)

The input df into the function is 3800 rows and 8 columns, the function is not complicated, and yet resource is exhausted trying to union two tiny outputs.  I can display df1 and df2 before the union, it is the union that crashes Databricks.

I tried manually inputting the data and creating two dataframes, and they union in less than 1s.  In the same project I have used unionByName to union larger dataframes, as part of more complicated functions.  Plus, this function used to work when I used test data smaller than 350 and 1500 rows.

What solutions could I try to fix this, repartitioning doesn't help.  Thank you.