pyspark streaming failed for now reason

[patojo94]

Hi everyone, I have a pyspark streaming reading from an aws kinesis that suddenly failed for no reason (I mean, we did not make any changes in the last time).

It is giving the following error:

ERROR MicroBatchExecution: Query kinesis_events_prod_bronze [id = 06233cfc-e27d-410d-858b-7c2546c5004f, runId = ace41ec4-c18b-421f-9e5b-bf5f75c96b12] terminated with error
java.lang.IllegalStateException: The transaction log has failed integrity checks. We recommend you contact Databricks support for assistance. To disable this check, set spark.databricks.delta.state.corruptionIsFatal to false

Do you have any idea of what could have happened or how to fix it?

Thank you!

[Hubert-Dudek · Accepted solution]

@patricio tojo​ , It just seems that some record coming from AWS kinesis is corrupted. I think you can debug it on Kinesis side.

[jose_gonzalez]

Hi @patricio tojo​,

Did you increase/reduce your Kinesis shards? or did you remove your checkpoint?

[jcasanella]

@patricio tojo​ I've the same problem, however in my case is after migrating into unity catalog. Need to investigate a little more but adding this to my spark job, it works:

spark.conf.set("spark.databricks.delta.state.corruptionIsFatal", False)