topic Re: Can not open socket to local (127.0.0.1) in Data Engineering

Can not open socket to local (127.0.0.1)

HansAdriaans — Wed, 18 Sep 2024 09:37:51 GMT

Hi, I'm running a databricks pipeline hourly using python notebooks checked out from git with on-demand compute (using r6gd.xlarge 32GB + 4 CPU's Gravaton). Most of the times the pipeline runs without problems. However, sometimes the first notebook fails with the error after starting the cluster

```Can not open socket: ["tried to connect to ('127.0.0.1', 36349), but an error occurred: [Errno 111] Connection refused"].```

This happens with the first interaction between the notebook and spark, running the command

```

km_per_nm = (

spark.read.table(assumptions_table)

.where(f.col("item") == "km_per_nm")

.collect()[0]

.asDict()

.get("value")

)

```

I know that this could be a sign of an OOM issue, but the result of that query is a single value from 1 row. The entire table is just 1.5 kb anyway and the driver has 32GiB of mem.

Can you guys help me with ideas where to look at ?

Re: Can not open socket to local (127.0.0.1)

HansAdriaans — Thu, 19 Sep 2024 09:19:49 GMT

Short update, I changed the script a bit by simply adding a display function just before the running the collect and this seems to work for now

Re: Can not open socket to local (127.0.0.1)

prasad_dhongade — Tue, 03 Feb 2026 17:56:27 GMT

HI I am facing similar error, the cluster runs 24/7 and this issue is observed for a few runs in the day. The data volume being processed is not huge but the logic that this needs to go though is complex. I do not want to include the display in production code. Please let me know your thoughts