topic Re: pyspark streaming failed for now reason in Data Engineering

pyspark streaming failed for now reason

patojo94 — Sat, 07 May 2022 20:05:23 GMT

Hi everyone, I have a pyspark streaming reading from an aws kinesis that suddenly failed for no reason (I mean, we did not make any changes in the last time).

It is giving the following error:

ERROR MicroBatchExecution: Query kinesis_events_prod_bronze [id = 06233cfc-e27d-410d-858b-7c2546c5004f, runId = ace41ec4-c18b-421f-9e5b-bf5f75c96b12] terminated with error
java.lang.IllegalStateException: The transaction log has failed integrity checks. We recommend you contact Databricks support for assistance. To disable this check, set spark.databricks.delta.state.corruptionIsFatal to false

Do you have any idea of what could have happened or how to fix it?

Thank you!

Re: pyspark streaming failed for now reason

Hubert-Dudek — Sun, 08 May 2022 09:28:03 GMT

@patricio tojo , It just seems that some record coming from AWS kinesis is corrupted. I think you can debug it on Kinesis side.

Re: pyspark streaming failed for now reason

jose_gonzalez — Thu, 02 Jun 2022 00:31:31 GMT

Hi @patricio tojo,

Did you increase/reduce your Kinesis shards? or did you remove your checkpoint?

Re: pyspark streaming failed for now reason

jcasanella — Mon, 07 Nov 2022 23:45:40 GMT

@patricio tojo I've the same problem, however in my case is after migrating into unity catalog. Need to investigate a little more but adding this to my spark job, it works:

spark.conf.set("spark.databricks.delta.state.corruptionIsFatal", False)