Pydeequ is not working with shared mode clusters on a unity catalog enabled databricks account

[parasupadhyay]

Hi folks,

Recently i was doing a poc on pydeequ. I found that it is not working with shared mode clusters in a databricks account where unity catalog is enabled. It throws the below error :

It works fine with the single mode clusters. Does anyone have a solution to make it work for shared mode clusters as well?
I am using pydeequ 1.4.0 and com.amazon.deequ:deequ:2.0.8-spark-3.5.

[szymon_dybczak]

Hi @parasupadhyay ,

I don't think there is much you can do. For security reasons, you can't use shared access mode in UC to access sparkContext. Your only option to make it work is to use single mode cluster.