topic Re: Why Shared Access Mode for Unity Catalog enabled DLT pipeline? in Data Engineering

Why Shared Access Mode for Unity Catalog enabled DLT pipeline?

hayden_blair — Wed, 17 Jul 2024 16:34:14 GMT

Hello all,

I am trying to use an RDD API in a Unity Catalog enabled Delta Live Tables pipeline.

I am getting an error because Unity Catalog enabled DLT can only run on "shared access mode" compute, and RDD APIs are not supported on shared access compute for security reasons.

Is there a reason that Unity Catalog enabled DLT is shared access mode only? Can we expect to see Unity Catalog enabled DLT pipelines that can run on single user compute anytime soon?

Unity Catalog with DLT doc (says shared compute only)

Compute access mode restrictions for Unity Catalog (says no RDD API on shared access mode)

Thank you!

Hayden

Re: Why Shared Access Mode for Unity Catalog enabled DLT pipeline?

szymon_dybczak — Thu, 18 Jul 2024 06:14:59 GMT

Hi @hayden_blair ,

The error you are encountering is related to Py4J security settings in Apache Spark. In Shared access mode, Py4J security is enabled by default for security reasons, which restricts certain methods from being called on the Spark RDD object.

To put it simply, security around Unity Catalog is strict but necessary 🙂

Re: Why Shared Access Mode for Unity Catalog enabled DLT pipeline?

hayden_blair — Fri, 19 Jul 2024 15:02:17 GMT

Thank you for the response @szymon_dybczak. Do you know if single user clusters are inherently less secure? I am still curious about why single user access mode is not allowed for DLT + Unity Catalog.