topic Re: Data bricks Internal error in Data Engineering

Data bricks Internal error

fzlrfk — Fri, 06 Dec 2024 00:19:11 GMT

I am new to data bricks. I am trying to debug an existing notebook which fails

intermittently (few times a month). Once it is re-run it runs fine.

Any help will be appreciated. I have attached sample code. environment:

Data bricks Runtime Version : 14.3 (includes Apache Spark 3.5.0, Scala 2.12)

Re: Data bricks Internal error

Alberto_Umana — Fri, 06 Dec 2024 14:05:23 GMT

Hello @fzlrfk,

I see that this is a bug already reported internally and being worked on. I will keep you posted on latest updates.

Re: Data bricks Internal error

fzlrfk — Fri, 06 Dec 2024 14:20:28 GMT

Thanks for your response. As I said the assistant give recommendation on changing the code. Would changing the code help ? When will a fix be released ?

error: NoSuchElementException: None.get at scala.None$.get(Option.scala:527)assistant fix:
The error is likely due to the use of a subquery in the SELECT clause. Instead, you can join in the main query).

runError": "[INTERNAL_ERROR] The \"count\" action failed. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000"

assistant:
To fix the error, you need to separate the CACHE TABLE statement from the rest of the query. The CACHE TABLE statement should be executed independently. Here is the corrected code:

Thanks