Job runs indefinitely after integrating with PyDeequ

JD410993
New Contributor II

‎01-16-2023 11:26 PM

I'm using PyDeequ data quality checks in one of our jobs.

After adding this check, I noticed that the job does not complete and keeps running indefinitely after PyDeequ checks are completed and results are returned.

As stated in Pydeequ documentation here, I've added the calls below at the end after all processing is done.

spark.sparkContext._gateway.shutdown_callback_server()
spark.stop()

However, the job continues to run and has to be eventually cancelled.

Has anyone else faced this while integrating with pydeequ on databricks.

Would appreciate any pointers.

Thanks.

Labels:

Hubert-Dudek
Databricks MVP

‎01-17-2023 12:56 AM

I don't think databricks support it.


My blog: https://databrickster.medium.com/
0 Kudos

-werners-
Esteemed Contributor III

‎01-17-2023 02:26 AM

Hm, deequ certainly works as I have read about multiple people using it.

And when reading the issues (open/closed) on the github pages of pydeequ, databricks is mentioned in some issues so it might be possible after all.

But I think you need to check your spark version etc as there is an open issue regarding recent spark versions (https://github.com/awslabs/python-deequ/issues/93).

-werners-
Esteemed Contributor III
In response to -werners-

‎01-17-2023 02:30 AM

to add to this:

do not create your own sparksession or stop it. Databricks handles the sparksession.

Here are the announcements of the pydeequ page:

0 Kudos