Unity Catalog - Storage Account Data Access
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-27-2024 08:35 AM
I was exploring on unity catalog option on Databricks premium workspace.
I understood that i need to create storage account credentials and external connection in workspace.
Later, i can access the cloud data using 'abfss://storage_account_details' .
I was wondering if we can access the data using mount point instead of using full 'abfss:' path.
Please guide me if there is any option to do it in premium workspace.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-28-2024 09:55 AM - edited 04-28-2024 09:57 AM
Hey @Mailendiran
In Databricks, mounting storage to DBFS (Databricks File System) using the `abfss` protocol is a common practice for accessing data stored externally in Azure Blob Storage. While you typically use the full `abfss` path to access data, you can indeed simplify this process by mounting the storage and accessing it via a mount point.
- You can mount the Azure Blob Storage to DBFS using the `dbutils.fs.mount` command.
- Specify the mount point (a directory in DBFS) and provide the Azure Blob Storage URI.
to do this create a notebook in databricks environment and use the below commands after updating your container name and other required things:
PYTHON
storage_account_name = "your_storage_account_name"
container_name = "your_container_name"
mount_point = "/mnt/storage"
dbutils.fs.mount(
source = f"abfss://$storage_account_name@{storage_account_name}.dfs.core.windows.net/{container_name}",
mount_point = mount_point,
extra_configs = {"fs.azure.account.key."+storage_account_name+".dfs.core.windows.net":dbutils.secrets.get(scope = "your_scope_name", key = "your_storage_key")})
to access data now you can use:
data_df = spark.read.csv("/mnt/storage/path_to_your_data.csv")
Hope this helps, thanks for posting.
Palash
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-30-2024 12:21 PM
Databricks strategic direction is to deprecate mount points in favor of Unity Catalog Volumes.
Setup an STORAGE CREDENTIAL and EXTERNAL LOCATION to access and define how to get to your cloud storage account. To access data on the account, define a Table or a Volume (file based access).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-24-2025 11:09 PM
is this is also applicable to aws databricks becoz i'm not seeing option to setup credentials, just having s3 configuration only.