Python segmentation fault in serverless job

Malthe
Valued Contributor II

We're getting a Python segmentation fault in a serverless job that uses Delta Table merge inside a foreachBatch step in structured streaming (trigger once).

/databricks/python/lib/python3.12/site-packages/pyspark/sql/connect/streaming/query.py:479: UserWarning: StreamingQueryListenerBus Handler thread received exception, all client side listeners are removed and handler thread is terminated. The error is: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.CANCELLED
	details = "Channel closed!"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Channel closed!", grpc_status:1, created_time:"2026-03-05T06:33:35.843237805+00:00"}"
>
  warnings.warn(
/databricks/python/lib/python3.12/site-packages/pyspark/sql/connect/streaming/query.py:479: UserWarning: StreamingQueryListenerBus Handler thread received exception, all client side listeners are removed and handler thread is terminated. The error is: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.CANCELLED
	details = "Channel closed!"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2026-03-05T06:34:03.441994764+00:00", grpc_status:1, grpc_message:"Channel closed!"}"
>
  warnings.warn(
Fatal Python error: Segmentation fault

 Looking at the serverless query UI, more time is spent in the task than can be accounted for (1.39m vs 4.14m).

Screenshot 2026-03-05 at 11.01.39.png

Could this be related to the segmentation fault? We have lots of warnings and errors in the logs, but that's par for the course with Databricks in general.