Pydeequ is not working with shared mode clusters on a unity catalog enabled databricks account

parasupadhyay
New Contributor

‎12-02-2024 04:41 AM

Hi folks,

Recently i was doing a poc on pydeequ. I found that it is not working with shared mode clusters in a databricks account where unity catalog is enabled. It throws the below error :

parasupadhyay_0-1733142604026.png

It works fine with the single mode clusters. Does anyone have a solution to make it work for shared mode clusters as well?
I am using pydeequ 1.4.0 and com.amazon.deequ:deequ:2.0.8-spark-3.5.

0 Kudos

szymon_dybczak
Esteemed Contributor III

‎12-02-2024 06:36 AM - edited ‎12-02-2024 06:36 AM

Hi @parasupadhyay ,

I don't think there is much you can do. For security reasons, you can't use shared access mode in UC to access sparkContext. Your only option to make it work is to use single mode cluster.

0 Kudos