topic Spark read CSV does not throw Exception if the file path is not available in Databricks 14.3 in Get Started Discussions

Spark read CSV does not throw Exception if the file path is not available in Databricks 14.3

thilanka02 — Fri, 19 Apr 2024 08:06:24 GMT

We were using this method and this was working as expected in Databricks 13.3.

def read_file(): try: df_temp_dlr_kpi = spark.read.load(raw_path,format="csv", schema=kpi_schema) return df_temp_dlr_kpi except Exception as e: error_message = str(e) print(error_message) dbutils.notebook.exit('Fail')

But now we have upgraded databricks to 14.3 and now method get returned without catching the exception when the file is not available.

When we call the method, unhandled exception generates as shown below.

Can anyone explain why we are experiencing this behaviour, and the solution to catch the exception?

Thanks in advance

Re: Spark read CSV does not throw Exception if the file path is not available in Databricks 14.3

daniel_sahal — Fri, 19 Apr 2024 09:13:30 GMT

@thilanka02
It looks like there is a bug in Spark 3.5, nothing much you can do about it rn.

https://issues.apache.org/jira/browse/SPARK-47708

Re: Spark read CSV does not throw Exception if the file path is not available in Databricks 14.3

thilanka02 — Fri, 19 Apr 2024 10:24:41 GMT

Thank you @daniel_sahal for the reply

Re: Spark read CSV does not throw Exception if the file path is not available in Databricks 14.3

databricks100 — Wed, 17 Jul 2024 19:05:57 GMT

Hi, has this been resolved? I am still seeing this issue with Runtime 14.3 LTS

Thanks in advance.