Hi everyone,

We're facing a persistent issue with our production streaming pipelines where executors are being forcefully killed with the following error:

Executor got terminated abnormally due to FORCE_KILL

📌 Screenshot for reference:

Context:

• Our pipelines create streaming tables using Delta Live Tables.

• This issue only started happening after Databricks migrated from GKE to GCE.

• We initially ran the job on 2 workers with 16 cores each, but due to failures, we tried scaling up gradually:

  • 3×16-core

  • 2×32-core (equivalent to 4×16-core)

  • even tried 5×32-core workers.

• Despite the aggressive scaling, executors still get force-killed.

• When we monitor resource usage, we notice executors are only using ~70% CPU, and the job is killed before even completing the first batch.

Questions:

1. Has anyone experienced a similar behavior after the move to GCE?

2. What could be causing FORCE_KILL on relatively idle executors (only ~70% utilization)?

3. Are there known configurations or cluster policies in GCE that could trigger such early termination?

4. Could this be related to DLT’s retry policy or hidden limits at the infrastructure level?

Any insights or recommendations are greatly appreciated!

Thanks in advance.