topic Re: Mount Azure Blob Storage with Cluster access control in Data Engineering

Mount Azure Blob Storage with Cluster access control

LukaszJ — Wed, 17 Aug 2022 17:26:10 GMT

Hello.

I want to mount and share for the one group the container from Azure Blob Storage (It could be simple blob storage or Azure Data Lake Storage gen 2). But I am not able to do it because I am using Cluster with Table Access Control.

This is my code and the error:

storage_name = "***"
container_name = "***"
conf_key = "***"
mount_point = "/mnt/***"
 
dbutils.fs.mount(
    source = f"wasbs://{container_name}@{storage_name}.blob.core.windows.net",
    mount_point = mount_point,
    extra_configs = {f"fs.azure.account.key.{storage_name}.blob.core.windows.net": conf_key}
)

py4j.security.Py4JSecurityException: Constructor public com.databricks.backend.deamon.dbutils.FSUtilsParallel(org.apache.spark.SparkContext) is not whitelisted

How can I mount a container? When I am using standard cluster everything is okay however, then people (on ACL cluster) cannot see the container.

Best regards,

Łukasz

Re: Mount Azure Blob Storage with Cluster access control

Vivian_Wilfred — Wed, 17 Aug 2022 20:32:18 GMT

Hi @Łukasz Jaremek , dbutils.fs.* commands wouldn’t work on Table ACL clusters since users do not have direct storage access. This is expected behavior.

Re: Mount Azure Blob Storage with Cluster access control

Vivian_Wilfred — Wed, 17 Aug 2022 20:35:08 GMT

Refer to this document-

https://docs.databricks.com/security/access-control/table-acls/table-acl.html#enable-table-access-control-for-a-cluster

Re: Mount Azure Blob Storage with Cluster access control

LukaszJ — Thu, 18 Aug 2022 05:51:24 GMT

Dear @Vivian Wilfred ,

Thanks you for the answer.

I see. So what can I do when people want to save DataFrames to the Azure Blob Storage container as csv?

Best regards,

Łukasz

Re: Mount Azure Blob Storage with Cluster access control

Vidula — Mon, 12 Sep 2022 06:21:18 GMT

Hi @Łukasz Jaremek

Does @Vivian Wilfred response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Re: Mount Azure Blob Storage with Cluster access control

LukaszJ — Thu, 17 Nov 2022 09:05:16 GMT

I have a good solution to the problem:

I am using Python library.

There are some documentation.

Topic to be closed.

Best regards,

Łukasz