Data bricks Internal error

[fzlrfk]

I am new to data bricks. I am trying to debug an existing notebook which fails

intermittently (few times a month). Once it is re-run it runs fine. 

Any help will be appreciated. I have attached sample code. environment:

Data bricks Runtime Version : 14.3  (includes Apache Spark 3.5.0, Scala 2.12)

[Alberto_Umana]

Hello @fzlrfk,

I see that this is a bug already reported internally  and being worked on. I will keep you posted on latest updates.

[fzlrfk]

Hi

Thanks for your response. As I said the assistant give recommendation on changing the code. Would changing the code help ? When will a fix be released ?

error: NoSuchElementException: None.get at scala.None$.get(Option.scala:527)assistant fix:
The error is likely due to the use of a subquery in the SELECT clause. Instead, you can join in the main query).

runError": "[INTERNAL_ERROR] The \"count\" action failed. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000"

assistant:
To fix the error, you need to separate the CACHE TABLE statement from the rest of the query. The CACHE TABLE statement should be executed independently. Here is the corrected code:

Thanks