topic Unity Catalog - Storage Account Data Access in Data Engineering

Unity Catalog - Storage Account Data Access

Mailendiran — Sat, 27 Apr 2024 15:35:45 GMT

I was exploring on unity catalog option on Databricks premium workspace.

I understood that i need to create storage account credentials and external connection in workspace.

Later, i can access the cloud data using 'abfss://storage_account_details' .

I was wondering if we can access the data using mount point instead of using full 'abfss:' path.

Please guide me if there is any option to do it in premium workspace.

Re: Unity Catalog - Storage Account Data Access

Palash01 — Sun, 28 Apr 2024 16:57:28 GMT

Hey @Mailendiran

In Databricks, mounting storage to DBFS (Databricks File System) using the `abfss` protocol is a common practice for accessing data stored externally in Azure Blob Storage. While you typically use the full `abfss` path to access data, you can indeed simplify this process by mounting the storage and accessing it via a mount point.

You can mount the Azure Blob Storage to DBFS using the `dbutils.fs.mount` command.
Specify the mount point (a directory in DBFS) and provide the Azure Blob Storage URI.

to do this create a notebook in databricks environment and use the below commands after updating your container name and other required things:

PYTHON

storage_account_name = "your_storage_account_name" container_name = "your_container_name" mount_point = "/mnt/storage" dbutils.fs.mount( source = f"abfss://$storage_account_name@{storage_account_name}.dfs.core.windows.net/{container_name}", mount_point = mount_point, extra_configs = {"fs.azure.account.key."+storage_account_name+".dfs.core.windows.net":dbutils.secrets.get(scope = "your_scope_name", key = "your_storage_key")})

to access data now you can use:

data_df = spark.read.csv("/mnt/storage/path_to_your_data.csv")

Hope this helps, thanks for posting.

Re: Unity Catalog - Storage Account Data Access

DouglasMoore — Tue, 30 Apr 2024 19:21:08 GMT

Databricks strategic direction is to deprecate mount points in favor of Unity Catalog Volumes.
Setup an STORAGE CREDENTIAL and EXTERNAL LOCATION to access and define how to get to your cloud storage account. To access data on the account, define a Table or a Volume (file based access).

Re: Unity Catalog - Storage Account Data Access

ashish31negi — Tue, 25 Mar 2025 06:09:07 GMT

is this is also applicable to aws databricks becoz i'm not seeing option to setup credentials, just having s3 configuration only.