Spark read CSV does not throw Exception if the file path is not available in Databricks 14.3

thilanka02
New Contributor II

We were using this method and this was working as expected in Databricks 13.3.

 

 

def read_file():
  try:
    df_temp_dlr_kpi = spark.read.load(raw_path,format="csv", schema=kpi_schema)
    return df_temp_dlr_kpi                          
  except Exception as e:
    error_message = str(e)
    print(error_message)
    dbutils.notebook.exit('Fail')

 

 

But now we have upgraded databricks to 14.3 and now method get returned without catching the exception when the file is not available.

When we call the method, unhandled exception generates as shown below.

Screenshot 2024-04-19 at 13.29.19.png

Can anyone explain why we are experiencing this behaviour, and the solution to catch the exception?

Thanks in advance