OSError: [Errno 107] Transport endpoint is not connected

nihar_ghude — Mon, 11 Mar 2024 17:26:43 GMT

Hi,

I am facing this error when performing write operation in foreach() on a dataframe. The piece of code was working fine for over 3 months but started failing since last week.

To give some context, I have a dataframe extract_df which contains 2 columns xml_full_name and content.

I use the below code to write these xmls to the target folder in ADLS.

extract_df.foreach(write_file)

'write_file' is defined as:

def write_file(row):

with open(row.extract_path, "wb") as f:

f.write(row.content)

The notebook also uses spark.write command to write some parquets which is working fine.

On investigating, I found that this issue could be related to parallelism and as a work around I tried the below:

for row in extract_df.collect():

with open(row.extract_path, "wb") as f:

f.write(row.content)

This works which means the connection is fine but parallelism isn't working. However, this can't be used as a fix though as it will bring down the performance.

Anyone here has faced this issue? If this is related to some configuration, any suggestions on what and where I can check. All inputs are welcome.

Thanks.

Re: OSError: [Errno 107] Transport endpoint is not connected

nihar_ghude — Fri, 15 Mar 2024 11:52:49 GMT

Thanks for the suggestion. I am not using structured streaming in the code. Not sure, if i can use `foreachBatch()` without it.

I have added a `try` block around the `foreach()` logic and added the `foreachPartition()` logic in an `except` block.

Also, the error resolved itself without any changes from our side. We are still awaiting response from Databricks support as to why the error intermittently appeared.

topic OSError: [Errno 107] Transport endpoint is not connected in Data Engineering

OSError: [Errno 107] Transport endpoint is not connected

Re: OSError: [Errno 107] Transport endpoint is not connected