Disable access to mount point for client code

[Yatoom]

We are building a platform where we automatically execute Databricks jobs using Python packages delivered by our end-users.

We want to create a mount point so that we can deliver the cluster's driver logs to an external storage. However, we don't want the client code to have access to this mount point. Because then we can not:

• guarantee isolation between jobs (the code of one end-user project can read the logs of another project)
• ensure immutability to the logs (users can override )

Is it possible to set some access control, so that the cluster can only write the driver logs there?

[daniel_sahal]

It depends which Cloud provider you're using. For AWS S3 you'll need to create IAM role and create a bucket policy that provides access to the role.

For Azure Databricks no longer recommends mounting external data locations to Databricks Filesystem (https://docs.databricks.com/external-data/azure-storage.html#deprecated-patterns-for-storing-and-accessing-data-from-databricks) And there's no possibility to manage permissions when using mounts.

[Aviral-Bhardwaj]

Check with cloud providers