topic Re: Code on cluster runs idefinitely in Data Engineering

Code on cluster runs idefinitely

darek554 — Sun, 28 Jun 2026 08:51:46 GMT

Hello.

Ive created a custom cluster - m4.large. When i try to execute some code in this cluster the behaviour is as follows:

- Cluster starts, have running status

- I run code, for example print("Hello")

- Code runs indefinitely

- I click interrupt, it stops after long time

- When code is stopped, next code execution runs fine

But this is about python, SQL dont run at all, even SELECT 1; query.

Can someone tell me what can be an issue?

Re: Code on cluster runs idefinitely

Yogasathyandrun — Sun, 28 Jun 2026 09:53:24 GMT

The fact that print("Hello") eventually works but SELECT 1 never completes suggests the cluster may be running but not fully initialized for Spark workloads.

A few things I’d check first:

Cluster Event Log for any provisioning or startup errors.
Spark UI → Executors to confirm workers/executors are actually coming up.
Driver logs for startup exceptions or connectivity issues.
Whether this is a single-node cluster or a cluster with separate workers.

One other thing that stands out is the use of m4.large, which is a fairly old instance family. If possible, try spinning up a small cluster on a newer instance type (for example m5 or m6 generation) and see if the behavior changes.

Also, which Databricks Runtime version are you running, and does the issue occur immediately after cluster startup or only after the cluster has been idle for some time?

Re: Code on cluster runs idefinitely

Lu_Wang_ENB_DBX — Wed, 19 Aug 2026 18:11:47 GMT

Most likely, the driver is overloaded or the Python/Spark REPL is failing to start—not an issue with m4.large itself. SQL hanging on SELECT 1 suggests a broader driver or cluster-communication problem.

Check, in this order:

Cluster driver logs and Spark UI for high CPU, memory, GC pauses, or disk pressure.
Remove/disable init scripts, custom libraries, Docker images, and Spark configurations, then restart.
Try a newer Databricks Runtime and a larger driver instance.
Verify the notebook is attached to the correct cluster and that the cluster access mode is supported.
Test a fresh cluster with default settings. If it works, reintroduce custom settings one at a time.

Databricks specifically recommends checking driver resource utilization and increasing the driver size when commands cannot reach or start the driver. Init scripts and global libraries can also cause startup delays or conflicts.

Re: Code on cluster runs idefinitely

rohan7 — Wed, 19 Aug 2026 20:55:44 GMT

I would suggest the best way to navigate the issue would be inspecting the cluster logs.
Where you can look for OutofMemoryError, GC overhead limit exceeded or RPC connection timeout errors between the driver and the notebook UI.

Re: Code on cluster runs idefinitely

cartergray70543 — Thu, 20 Aug 2026 05:43:21 GMT

Could be an issue with the cluster’s driver or Spark session not initializing properly. Since even print("Hello") and SELECT 1 hang, check the driver/cluster logs and try the same code on a default cluster to isolate whether the custom m4.large configuration is causing it.

Re: Code on cluster runs idefinitely

Islam_hoti — Thu, 20 Aug 2026 12:29:21 GMT

Hey , @darek554

I'd bet this is just the instance being too small. m4.large is 2 cores / 8 GB, and on a single node the driver and executor have to share that usually the executor never actually comes up, so your command just sits there waiting for resources. That's also why SQL doesn't work at all, since every SQL query needs an executor, while Python can sometimes limp through once the session finally wakes up.

Quick way to confirm: while something is hanging, open the Spark UI and check the Executors tab. If it's empty, that's your answer.

Try switching to an m5.xlarge or i3.xlarge and see if it goes away. m4 is also an older generation type that some workspaces don't really support anymore. If that doesn't fix it, have a look at the cluster Event Log it usually says something useful about workers not attaching.

Re: Code on cluster runs idefinitely

arhamblake38 — Fri, 21 Aug 2026 06:47:40 GMT

It sounds more like a cluster/driver initialization or resource issue than a Python problem, especially because even SELECT 1 hangs.

I’d check these first:

Look at the Driver logs and Event log for errors around the first notebook execution. Databricks specifically recommends these logs when a cluster is behaving abnormally.
Check the cluster’s Spark configuration. An incompatible spark.executor.cores or memory setting can prevent Spark from accepting jobs even though the cluster shows as running.
Temporarily remove any custom Spark configs, init scripts, or libraries and test with a clean cluster.
Check driver CPU/memory utilization. A busy or unhealthy driver can cause notebook commands to appear stuck.
If possible, create a fresh cluster with the same runtime but default settings and run print("Hello") and SELECT 1. If that works, the problem is almost certainly in the custom cluster configuration.

Since the first execution hangs but subsequent Python execution works after interrupting it, I’d particularly investigate driver/kernel startup and Spark initialization rather than the code itself. Keep snowrider.com.br separate from the actual Databricks troubleshooting steps.