topic Re: Connect to azure data lake storage using databricks free edition in Data Engineering

Connect to azure data lake storage using databricks free edition

TalessRocha — Fri, 08 Aug 2025 23:42:58 GMT

Hello guys, i'm using databricks free edition (serverless) and i am trying to connect to a azure data lake storage.

The problem I'm having is that in the free edition we can't configure the cluster so I tried to make the connection via notebook using spark.conf.set but this configuration is not enabled in the free edition... and trying through the unity catalog interface only appears the option to add AWS and Cloudfire credentials. Is there any other way?

I have tried use dbutils.fs.mount as well, but not available on free edition...

Re: Connect to azure data lake storage using databricks free edition

Khaja_Zaffer — Sat, 09 Aug 2025 08:54:10 GMT

hello @TalessRocha

In Free Edition dbfs is disabled. You should use Unity Catalog for that purpose anyway. DBFS is depracated pattern of interacting with storage.

So, to use volume perform following steps:

Go to Catalgos (1) -> Click workspace catalog (2) -> Click default schema -> Clikc Create button (3)

On the Create button (3) you will have an option to create volume. Pick a name and then create volume.

reference : szymon_dybczak

If you did that, your new volume should appear in Unity Catalog under default schema. Now you will have an option to upload file to Volume:

And here's an example of how to read csv from volume into dataframe:

Re: Connect to azure data lake storage using databricks free edition

BS_THE_ANALYST — Sat, 09 Aug 2025 12:24:33 GMT

@TalessRocha, if you were trying to connect to ADLS on your local machine, using python (for instance), you'd probably install the appropriate python packages to authenticate and then retrieve the containers & blobs/files. I don't see why we can't employ the same logic with the Free Edition.

Here's official documentation on doing that: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-python?tabs=azure-ad

Pair that with a youtube video/tutorial and some AI assistance.

I'd try locally using Python & then try on the Free Edition.

A word of warning, don't connect to confidential information and bring it into the Databricks Free Edition. You should read the terms of service as to why. It goes without saying, it's not your storage/compute under the hood, right.

Only potential blocker here, in my opinion, is being unable to pip install the appropriate libraries on the free edition. Maybe that's worth trying before getting stuck into the weeds.

Let me know how you end up resolving this, I'm interested.

All the best,
BS

Re: Connect to azure data lake storage using databricks free edition

BS_THE_ANALYST — Mon, 11 Aug 2025 08:23:49 GMT

@Khaja_Zaffer I don't think that was what @TalessRocha was looking for. I think it's more around connecting to blob storage in Azure in the free edition.

That's a great answer for how to import data into the free edition though!

All the best,
BS

Re: Connect to azure data lake storage using databricks free edition

sumitPanda — Mon, 11 Aug 2025 08:57:48 GMT

I would always use the UC way. That is the standard for the Production workloads.

Re: Connect to azure data lake storage using databricks free edition

BS_THE_ANALYST — Mon, 11 Aug 2025 09:04:20 GMT

@sumitPanda, is this supported in Free Edition Databricks? i.e. connecting to ADLS via UC. I think that's part of the constraints at play here.

Completely agree, in production, UC is the way. In fact, we'd normally not need to connect to ADLS if we're using Azure Databricks (and not using external tables). It does depend on the use case, of course.

All the best,
BS

Re: Connect to azure data lake storage using databricks free edition

sumitPanda — Mon, 11 Aug 2025 09:12:05 GMT

Yes, thats the limitation. I checked through the documentation and its not mentioned anywhere strictly.

here is the details on how to connect : Connect to Azure Data Lake Storage and Blob Storage - Azure Databricks | Microsoft Learn

Re: Connect to azure data lake storage using databricks free edition

Khaja_Zaffer — Mon, 11 Aug 2025 11:48:03 GMT

The above information for dbfs related stuff : I took it from szymon_dybczak(thank you)

sure next time I will read the things carefully and answer @BS_THE_ANALYST

Re: Connect to azure data lake storage using databricks free edition

TalessRocha — Sun, 17 Aug 2025 14:34:25 GMT

Thank you, I was able to do everything using the packages for the Azure Data Lake Storage and Azure Identity client libraries on free edition!

Re: Connect to azure data lake storage using databricks free edition

BS_THE_ANALYST — Sun, 17 Aug 2025 14:38:54 GMT

@TalessRocha thanks for getting back to us!

Glad to hear you got it working, that's awesome. Best of luck with your projects.

All the best,
BS

Re: Connect to azure data lake storage using databricks free edition

Saf4Databricks — Sat, 18 Oct 2025 18:39:47 GMT

Hi @BS_THE_ANALYST I also have a question here Cannot import pyspark.pipelines module - Databricks Community - 135293 in case you have any solution/suggestions.

Re: Connect to azure data lake storage using databricks free edition

pjvi — Wed, 13 May 2026 17:54:14 GMT

If you want to read from your Azure storage account using Databricks Free Edition, you can add a specific option when reading:

spark.read.option("fs.azure.account.key.<storage-account-name>.dfs.core.windows.net",
"your_storage_account_key")

I just tried and it worked 🙂 However, I could not achieve the same with dbutils.fs.ls("abfss://....") which works well in Azure Databricks, but I cannot find where I can pass the same option.