topic Re: ConnectException: Connection refused (Connection refused) This is often caused by an OOM error in Data Engineering

ConnectException: Connection refused (Connection refused) This is often caused by an OOM error

Rnmj — Mon, 25 Oct 2021 12:25:36 GMT

I am trying to run a python code where a json file is flattened to pipe separated file . The code works with smaller files but for huge files of 2.4 GB I get below error:

ConnectException: Connection refused (Connection refused)

Error while obtaining a new communication channel

ConnectException error: This is often caused by an OOM error that causes the connection to the Python REPL to be closed. Check your query's memory usage.

Databricks version 9.1 LTS

The cluster is 5 node Standard_DS4_V2

Re: ConnectException: Connection refused (Connection refused) This is often caused by an OOM error

-werners- — Mon, 25 Oct 2021 13:40:55 GMT

Can you check this topic?

It might be what you are looking for:

https://community.databricks.com/s/question/0D53f00001Q0Rq9CAF/bufferholder-exceeded-on-json-flattening

Re: ConnectException: Connection refused (Connection refused) This is often caused by an OOM error

jose_gonzalez — Tue, 26 Oct 2021 20:33:36 GMT

hi @RN mj ,

Could you provide more details? how do you read your JSON file? are you using an autoscaling cluster? what is the full error stack-trace?

Re: ConnectException: Connection refused (Connection refused) This is often caused by an OOM error

Rnmj — Fri, 29 Oct 2021 03:58:14 GMT

Hi @Jose Gonzalez , @Werner Stinckens @Kaniz Fatma ,

Thanks for your response .Appreciate a lot.

The issue was in the code, it was a python /panda code running on Spark. Due to this only driver node was being used. i did validate this by increasing the driver configuration. The next steps is to revisit the code and use RDD/dataframes so code has some parallel processing