Hi @Malthe,

Since you have confirmed this is vanilla PySpark with no external libraries on serverless runtime environment version 5, this narrows things down considerably. Here are some additional observations and recommendations beyond what Louis shared.

WHAT THE STACK TRACE TELLS US

The crash path is inside pyspark/sql/connect/streaming/query.py at line 479, which is the StreamingQueryListenerBus handler thread. Serverless compute runs exclusively through Spark Connect (the gRPC-based protocol), and the "Channel closed" / StatusCode.CANCELLED messages indicate that the server-side Spark session terminated (or the gRPC channel was recycled) while the client-side listener bus thread was still active. The segfault then occurs when the Python process tries to access memory through a now-invalid gRPC channel reference.

In other words, this looks like a timing issue in the Spark Connect streaming listener cleanup sequence, not a problem with your MERGE logic itself. The MERGE completes successfully on the server side, but the client-side Python process crashes during teardown.

THE TASK TIMING GAP

The discrepancy you see (1.39m accounted for vs 4.14m total) is consistent with this theory. The "unaccounted" time likely includes:

1. The Spark Connect session setup and teardown overhead
2. The listener bus polling interval before the crash
3. Any retry/backoff in the gRPC layer before the fatal signal

This gap alone does not indicate a problem with your MERGE performance.

RECOMMENDED NEXT STEPS

1. Check whether the job actually succeeds despite the segfault. In many Spark Connect streaming teardown crashes, the data is written correctly and the checkpoint is committed before the Python process crashes. Verify your target Delta table has the expected data after the run. If the data is correct, the segfault is happening during cleanup, not during the write.

2. Try wrapping your streaming query with explicit lifecycle management to give the listener bus time to shut down cleanly:

query = (
  df.writeStream
  .foreachBatch(merge_function)
  .trigger(availableNow=True)
  .option("checkpointLocation", checkpoint_path)
  .start()
)
query.awaitTermination()
# Add a brief pause before the Python process exits
import time
time.sleep(5)

The sleep gives the background gRPC threads time to close gracefully before Python tears down the process.

3. If you are running this as a Databricks Job task, check whether the task is marked as failed or succeeded in the job run history. If the task succeeds despite the segfault warning in stdout, the crash is cosmetic (happening after the streaming query has already finished).

4. File a support ticket. Since this is a crash in Databricks-managed native code on the serverless runtime with no user-installed libraries, the Databricks engineering team is best positioned to investigate and address it. Include:

 - The workspace URL and job run URL
 - The full driver logs from the failed run
 - The serverless environment version (v5 as you noted)
 - Approximate data volumes (rows and size of the source micro-batch, and the size of the target Delta table)

DOCUMENTATION REFERENCES

- Serverless compute limitations (note: only Spark Connect APIs are supported, and only Trigger.AvailableNow is supported for streaming):

https://docs.databricks.com/en/compute/serverless/limitations.html

- foreachBatch with Delta Lake merge in structured streaming:

https://docs.databricks.com/en/structured-streaming/foreach.html

- Structured streaming production best practices:

https://docs.databricks.com/en/structured-streaming/production.html

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.