topic dataframe.rdd.isEmpty() is throwing error in 9.1 LTS in Data Engineering

dataframe.rdd.isEmpty() is throwing error in 9.1 LTS

thushar — Wed, 19 Jan 2022 06:00:40 GMT

Loaded a csv file with five columns into a dataframe, and then added around 15+ columns using dataframe.withColumn method.

After adding these many columns, when I run the query df.rdd.isEmpty() - which throws the below error.

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 32.0 failed 4 times, most recent failure: Lost task 0.3 in stage 32.0 (TID 28) (10.139.64.4 executor 9): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

Any idea what is the issue?

Re: dataframe.rdd.isEmpty() is throwing error in 9.1 LTS

Anonymous — Wed, 19 Jan 2022 16:40:28 GMT

Hello again, @Thushar R - I'm sorry to hear that you're having this difficulty also. Let's give the community a chance to respond first. Thanks in advance for your patience.

Re: dataframe.rdd.isEmpty() is throwing error in 9.1 LTS

Hubert-Dudek — Thu, 20 Jan 2022 11:45:32 GMT

Please check your logs as it can be some other issue.

Please check also using bool(df.head(1)) instead.

Re: dataframe.rdd.isEmpty() is throwing error in 9.1 LTS

thushar — Fri, 21 Jan 2022 07:57:41 GMT

Thanks for the workaround. But why this particular piece of code fails in 9.0 LTS runtime and run in 8.3 without issues. Any idea. Please see the code below.

from pyspark.sql.functions import lit,col,row_number,floor,trim

df = spark.read.option("header", "true").csv(filePath)

df2 = df.select(col("cc"),col("ac"),col("an"),\

col("ag"),col("at")).distinct()

lstOfMissingColumns = ['col1', 'col2', 'col3', 'col4', 'col5', 'col6', 'col7', 'col8', 'col8', 'col9','col9', 'col10', 'col11', 'col12', 'col13',

'col14', 'col15', 'col16', 'col17']

for c in lstOfMissingColumns:

df2 = df2.withColumn(c,lit(''))

df2.rdd.isEmpty()

Re: dataframe.rdd.isEmpty() is throwing error in 9.1 LTS

Anonymous — Wed, 16 Feb 2022 17:04:24 GMT

@Thushar R - Thank you for your patience. We are looking for the best person to help you.

Re: dataframe.rdd.isEmpty() is throwing error in 9.1 LTS

jose_gonzalez — Thu, 24 Feb 2022 01:11:43 GMT

Hi @Thushar R ,

Are you using the same CSV file?

the error message is

"Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages" which could be a OOM error. How big is your CSV file? have you check the executor's 9 logs?