topic Disable access to mount point for client code in Data Engineering

Disable access to mount point for client code

Yatoom — Tue, 27 Dec 2022 13:02:38 GMT

We are building a platform where we automatically execute Databricks jobs using Python packages delivered by our end-users.

We want to create a mount point so that we can deliver the cluster's driver logs to an external storage. However, we don't want the client code to have access to this mount point. Because then we can not:

guarantee isolation between jobs (the code of one end-user project can read the logs of another project)
ensure immutability to the logs (users can override )

Is it possible to set some access control, so that the cluster can only write the driver logs there?

Re: Disable access to mount point for client code

daniel_sahal — Tue, 27 Dec 2022 13:10:06 GMT

It depends which Cloud provider you're using. For AWS S3 you'll need to create IAM role and create a bucket policy that provides access to the role.

For Azure Databricks no longer recommends mounting external data locations to Databricks Filesystem (https://docs.databricks.com/external-data/azure-storage.html#deprecated-patterns-for-storing-and-accessing-data-from-databricks) And there's no possibility to manage permissions when using mounts.

Re: Disable access to mount point for client code

Aviral-Bhardwaj — Wed, 28 Dec 2022 04:34:51 GMT

Check with cloud providers