Databricks Community

minhhung0507 · 09-02-2025

I am running a pipeline job in Databricks and it failed with the following message:Run failed with error message Could not reach driver of cluster 5824-145411-p65jt7uo. This message is not very descriptive, and I am not able to identify the root ca...

minhhung0507 · 08-11-2025

We have a Databricks pipeline where the layer reads from several Silver tables to detect PK/FK changes and trigger updates to Gold tables. Normally, this near real-time job has ~3 minutes latency per micro-batch.Recently, we noticed that each batch i...

minhhung0507 · 08-04-2025

We are in the process of migrating a pipeline from legacy publishing mode (LIVE virtual schema) to default publishing mode, following the steps outlined in the documentation:Enable the default publishing mode in a pipeline (Public Preview)However, wh...

minhhung0507 · 06-25-2025

Hi everyone,We're facing a persistent issue with our production streaming pipelines where executors are being forcefully killed with the following error:Executor got terminated abnormally due to FORCE_KILL Screenshot for reference:Context:Our pipeli...

minhhung0507 · 06-04-2025

I'm running into an issue where my Spark driver keeps pausing and eventually restarting due to excessive garbage collection (GC), even though I’ve already scaled up the cluster memory. Below is an example from the driver logs:Driver/192.168.231.23 pa...

minhhung0507 · 08-05-2025

Yes, we did refer to the documentation and followed the recommended troubleshooting steps exactly as described:We ran a successful update after setting only pipelines.setMigrationHints = true, and then continued from step 3 by re-adding pipelines.ena...

minhhung0507 · 06-04-2025

Thank you very much for your detailed analysis and helpful recommendations.We have reviewed your suggestions, and I’d like to share a quick update:We have already tried most of the mitigation strategies you mentioned — including increasing driver mem...

minhhung0507 · 05-22-2025

minhhung0507 · 04-21-2025

Hi @RiyazAliM ,I have two quick questions I’d appreciate your help with:Why do we need to use an interactive (all‑purpose) cluster when submitting a job, rather than a job cluster?I can’t find any reference to a parameter called job_cluster_details i...

minhhung0507 · 04-17-2025

@RiyazAliM FYI@RiyazAliM FYI:400 --- {"error_code":"INVALID_PARAMETER_VALUE","message":"The cluster 0417-102632-cuhsyfm0 is not an all-purpose cluster. existing_cluster_id only supports all-purpose cluster IDs.","details":[{"@type":"type.googleapis....

Databricks Community

User Stats

User Activity

Could not reach driver of cluster

Slow batch processing in Databricks job due to high deletion vector and unified unified cache overhe

Error MISSING_EXPECTED_PROPERTY during migration to default publishing mode

Executors Getting FORCE_KILL After Migration to GCE – Resource Scaling Not Helping

Spark Driver keeps restarting due to high GC pressure despite scaling up memory

Re: Error MISSING_EXPECTED_PROPERTY during migration to default publishing mode

Re: Spark Driver keeps restarting due to high GC pressure despite scaling up memory

Re: DLT Flow Failed Due to Missing Flow Checkpoints Directory When Using Unity Catalog

Re: API for Restarting Individual Failed Tasks within a Job?

Re: API for Restarting Individual Failed Tasks within a Job?