https://community.databricks.com/t5/data-engineering/pydeequ-is-not-working-with-shared-mode-clusters-on-a-unity/m-p/100612#M40351 <P>Hi folks,<BR /><BR />Recently i was doing a poc on pydeequ. I found that it is not working with shared mode clusters in a databricks account where unity catalog is enabled. It throws the below error :</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="parasupadhyay_0-1733142604026.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/13272iE5866E2E0DAC6A83/image-size/medium?v=v2&amp;px=400" role="button" title="parasupadhyay_0-1733142604026.png" alt="parasupadhyay_0-1733142604026.png" /></span></P><P>It works fine with the single mode clusters. Does anyone have a solution to make it work for shared mode clusters as well?<BR />I am using pydeequ 1.4.0 and&nbsp;<A target="_blank">com.amazon.deequ:deequ:2.0.8-spark-3.5.</A></P> Mon, 02 Dec 2024 12:41:14 GMT parasupadhyay 2024-12-02T12:41:14Z https://community.databricks.com/t5/data-engineering/pydeequ-is-not-working-with-shared-mode-clusters-on-a-unity/m-p/100612#M40351 <P>Hi folks,<BR /><BR />Recently i was doing a poc on pydeequ. I found that it is not working with shared mode clusters in a databricks account where unity catalog is enabled. It throws the below error :</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="parasupadhyay_0-1733142604026.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/13272iE5866E2E0DAC6A83/image-size/medium?v=v2&amp;px=400" role="button" title="parasupadhyay_0-1733142604026.png" alt="parasupadhyay_0-1733142604026.png" /></span></P><P>It works fine with the single mode clusters. Does anyone have a solution to make it work for shared mode clusters as well?<BR />I am using pydeequ 1.4.0 and&nbsp;<A target="_blank">com.amazon.deequ:deequ:2.0.8-spark-3.5.</A></P> Mon, 02 Dec 2024 12:41:14 GMT https://community.databricks.com/t5/data-engineering/pydeequ-is-not-working-with-shared-mode-clusters-on-a-unity/m-p/100612#M40351 parasupadhyay 2024-12-02T12:41:14Z https://community.databricks.com/t5/data-engineering/pydeequ-is-not-working-with-shared-mode-clusters-on-a-unity/m-p/100637#M40363 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/134567">@parasupadhyay</a>&nbsp;,</P><P>I don't think there is much you can do. For security reasons, you can't use shared access mode in UC to access sparkContext. Your only option to make it work is to use single mode cluster.</P> Mon, 02 Dec 2024 14:36:20 GMT https://community.databricks.com/t5/data-engineering/pydeequ-is-not-working-with-shared-mode-clusters-on-a-unity/m-p/100637#M40363 szymon_dybczak 2024-12-02T14:36:20Z