I'm using PyDeequ data quality checks in one of our jobs.
After adding this check, I noticed that the job does not complete and keeps running indefinitely after PyDeequ checks are completed and results are returned.
As stated in Pydeequ documentation here, I've added the calls below at the end after all processing is done.
spark.sparkContext._gateway.shutdown_callback_server()
spark.stop()
However, the job continues to run and has to be eventually cancelled.
Has anyone else faced this while integrating with pydeequ on databricks.
Would appreciate any pointers.
Thanks.