Data bricks Internal error

fzlrfk
New Contributor II

‎12-05-2024 04:19 PM

I am new to data bricks. I am trying to debug an existing notebook which fails

intermittently (few times a month). Once it is re-run it runs fine. 
Any help will be appreciated. I have attached sample code. environment:
Data bricks Runtime Version : 14.3  (includes Apache Spark 3.5.0, Scala 2.12)
Preview file
40 KB
Preview file
45 KB
0 Kudos

Alberto_Umana
Databricks Employee
Databricks Employee

‎12-06-2024 06:05 AM

Hello @fzlrfk,

I see that this is a bug already reported internally  and being worked on. I will keep you posted on latest updates.

0 Kudos

fzlrfk
New Contributor II

‎12-06-2024 06:20 AM

Hi

Thanks for your response. As I said the assistant give recommendation on changing the code. Would changing the code help ? When will a fix be released ?

 

error: NoSuchElementException: None.get at scala.None$.get(Option.scala:527)assistant fix:
The error is likely due to the use of a subquery in the SELECT clause. Instead, you can join in the main query).

runError": "[INTERNAL_ERROR] The \"count\" action failed. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000"

assistant:
To fix the error, you need to separate the CACHE TABLE statement from the rest of the query. The CACHE TABLE statement should be executed independently. Here is the corrected code:

Thanks

0 Kudos