Issue with Databricks Jobs: SQLSTATE: XXKST

[BMex]

Hi,

we have our Databricks Jobs deployed via DABs, and they have been running fine for a while now (approximately 1 month since we migrated from ADF). However, since yesterday, we are getting a weird issue while writing. See error below:

[STREAM_FAILED] Query [id = query_id, runId = run_id] terminated with exception: Do not know how to convert JArray(List(JString(some_key_string))) into class java.lang.String SQLSTATE: XXKST

For context, we use AutoLoader File-Notification mode to fetch the data from ADLS, and store it into our bronze layer. Nothing changed - infrastructure wise - and I cannot find the reason why this is happening. It started in a couple of tasks - some working some not - and after a while, my whole bronze layer is not working anymore.

I would appreciate if you can nudge me in the right direction.

[WiliamRosa]

Hi @BMex 
There are more users reporting similar issues recently:
https://community.databricks.com/t5/data-engineering/intermittent-failures-with-auto-loader-on-azure...
I’ll wait for others in the community to help shed more light on this.

[WiliamRosa]

Hi @BMex,
The link I shared with a similar issue contains some solutions — did any of them work for you?

[BMex · Accepted solution]

Hi @WiliamRosa,

not really. However, the issue resolved itself after 2 days. I guess it was because of an internal change (Microsoft’s or Databricks’ - not sure which). It appeared again one time last week, but I’m hoping it went away for good now.