cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Spark read CSV does not throw Exception if the file path is not available in Databricks 14.3

thilanka02
New Contributor II

We were using this method and this was working as expected in Databricks 13.3.

 

 

def read_file():
  try:
    df_temp_dlr_kpi = spark.read.load(raw_path,format="csv", schema=kpi_schema)
    return df_temp_dlr_kpi                          
  except Exception as e:
    error_message = str(e)
    print(error_message)
    dbutils.notebook.exit('Fail')

 

 

But now we have upgraded databricks to 14.3 and now method get returned without catching the exception when the file is not available.

When we call the method, unhandled exception generates as shown below.

Screenshot 2024-04-19 at 13.29.19.png

Can anyone explain why we are experiencing this behaviour, and the solution to catch the exception?

Thanks in advance

1 ACCEPTED SOLUTION

Accepted Solutions

daniel_sahal
Esteemed Contributor

@thilanka02 
It looks like there is a bug in Spark 3.5, nothing much you can do about it rn.

 https://issues.apache.org/jira/browse/SPARK-47708

View solution in original post

3 REPLIES 3

daniel_sahal
Esteemed Contributor

@thilanka02 
It looks like there is a bug in Spark 3.5, nothing much you can do about it rn.

 https://issues.apache.org/jira/browse/SPARK-47708

thilanka02
New Contributor II

Thank you @daniel_sahal for the reply

databricks100
New Contributor II

Hi, has this been resolved? I am still seeing this issue with Runtime 14.3 LTS

Thanks in advance.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group