Hi @Malthe,

There are two separate things going on here, and they are likely independent of each other. Let me break them down.

TASK TIME vs EXECUTION TIME DISCREPANCY

The gap between aggregated task time (1.39m) and total execution time (4.14m) is expected behavior and is not caused by the segfault. Aggregated task time only measures the wall-clock time that Spark tasks spend actively computing on executors. It does not include:

- Scheduler delay (time between task submission and actual execution)
- Serialization and deserialization of task results
- Shuffle read/write I/O wait time
- Driver-side orchestration overhead
- Spark Connect gRPC round-trip communication (this is especially relevant on serverless)

On serverless compute, all Spark interactions go through the Spark Connect API over gRPC. This adds a communication layer between your driver code and the Spark execution engine. For a MERGE operation inside foreachBatch, there are multiple planning, optimization, and coordination steps that happen on the driver side and over gRPC that do not count as "task time" but do count toward total execution duration.

Additionally, the MERGE operation itself has phases (reading the target table, computing the join, writing deletes/updates/inserts) where the time between phases is driver coordination, not active task execution.

You can verify this by looking at the Spark UI for that run. Under the SQL/DataFrame tab, you will see the query plan and can see time spent in different stages. The "Duration" column in the Stages tab will also show scheduling delay and other overhead separately from task compute time.

SEGMENTATION FAULT AND gRPC CHANNEL CLOSED ERRORS

The "Fatal Python error: Segmentation fault" combined with the StreamingQueryListenerBus gRPC "Channel closed" warnings points to an issue in the serverless runtime itself, not in your PySpark code. Since you confirmed you are using vanilla PySpark with no external libraries on environment version 5, this is a runtime-level crash.

The gRPC "Channel closed" messages are a symptom of the crash, not the cause. When the underlying process crashes (SIGSEGV), the Spark Connect gRPC channel is forcibly closed, which triggers those StreamingQueryListenerBus warnings.

A few things to check and try:

1. Trigger type: if you are using Trigger.Once, note that it has been deprecated since Databricks Runtime 11.3 LTS. Switch to Trigger.AvailableNow, which is the supported trigger for incremental batch processing on serverless.
https://docs.databricks.com/en/structured-streaming/triggers.html

2. Caching in foreachBatch: the Databricks documentation recommends caching the batch DataFrame before a MERGE to avoid reading the input multiple times. However, on serverless compute, DataFrame caching (df.cache(), spark.catalog.cacheTable()) is not supported and will throw exceptions. If your foreachBatch code includes any caching calls, remove them.
https://docs.databricks.com/en/structured-streaming/foreach.html

3. MERGE idempotency: make sure your MERGE statement inside foreachBatch is idempotent, as restarts can re-apply the same batch.
https://docs.databricks.com/en/structured-streaming/delta-lake.html

4. Batch size: if the MERGE is processing a very large batch, try limiting the input with maxBytesPerTrigger or maxFilesPerTrigger to reduce per-batch memory pressure. Heavy memory use during native Delta I/O operations can contribute to OS-level process termination that surfaces as SIGSEGV.

5. File a support ticket: since this is a native crash in the managed serverless runtime with vanilla PySpark, the root cause is most likely in the runtime itself. Open a support ticket and include the workspace URL, the job run URL, and the exact environment version. The engineering team can examine the crash dump and determine whether this is a known issue with a fix in a newer environment version.

The task time discrepancy is normal and unrelated to the crash. The segfault is the issue that needs attention, and a support ticket is the best path to resolution for a native crash on serverless.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.