Job runs indefinitely after integrating with PyDeequ

[JD410993]

I'm using PyDeequ data quality checks in one of our jobs.

After adding this check, I noticed that the job does not complete and keeps running indefinitely after PyDeequ checks are completed and results are returned.

As stated in Pydeequ documentation here, I've added the calls below at the end after all processing is done.

spark.sparkContext._gateway.shutdown_callback_server()
spark.stop()

However, the job continues to run and has to be eventually cancelled.

Has anyone else faced this while integrating with pydeequ on databricks.

Would appreciate any pointers.

Thanks.

[Hubert-Dudek]

I don't think databricks support it.

[-werners-]

Hm, deequ certainly works as I have read about multiple people using it.

And when reading the issues (open/closed) on the github pages of pydeequ, databricks is mentioned in some issues so it might be possible after all.

But I think you need to check your spark version etc as there is an open issue regarding recent spark versions (https://github.com/awslabs/python-deequ/issues/93).

[-werners-]

to add to this:

do not create your own sparksession or stop it. Databricks handles the sparksession.

Here are the announcements of the pydeequ page:

• With PyDeequ v0.1.8+, we now officially support Spark3 ! Just make sure you have an environment variable SPARK_VERSION to specify your Spark version!
• We've release a blogpost on integrating PyDeequ onto AWS leveraging services such as AWS Glue, Athena, and SageMaker! Check it out: Monitor data quality in your data lake using PyDeequ and AWS Glue.
• Check out the PyDeequ Release Announcement Blogpost with a tutorial walkthrough the Amazon Reviews dataset!
• Join the PyDeequ community on PyDeequ Slack to chat with the devs!