Accessing data bricks data outside data bricks

maikel
Contributor III

‎10-21-2025 07:10 AM

Hi!

What is the best way to access data bricks data, outside data bricks e.g. from Python code? The main problem is authentication so that I can access data to which I have permissions but I would like to generate token outside data bricks (e.g. via REST endpoint).
The data which I would like to access are files but also tables. 

I will be very thankful for your help and if possible examples how to achieve this.

 

 

0 Kudos

Chiran-Gajula
New Contributor III

‎10-21-2025 07:35 AM

Hello Maikel,

You can use your token generated in Databricks and use databricks SQL connector for python.
More details here: Databricks SQL Connector for Python | Databricks on AWS

 

G.Chiranjeevi

maikel
Contributor III

‎10-21-2025 08:18 AM

Hi!
The problem is, I have to authenticate from python code as well with my corporation account. Not copy and paste token to the code.

Any ideas? 🙂

0 Kudos

dkushari
Databricks Employee
Databricks Employee

‎10-21-2025 08:39 AM

Hi @maikel - You can set up a Service Principal in Databricks and a client ID and Client Secret. Then set up a Databricks profile and use Python code with that profile. Look at the profile section in step 2, how the profile can be set up with client ID and secret for workspace-level operation.

Databricks uses OAuth 2.0 as the preferred protocol for service principal authorization and authentication outside of the UI. Unified client authentication automates token generation and refresh. When a service principal signs in and is granted consent, OAuth issues an access token for the CLI, SDK, or other tool to use on its behalf. Each access token is valid for one hour, after which a new token is automatically requested.

Here is my example -

Profile -

[fielddemo-sp]
host             = https://workspace.cloud.databricks.com/
client_id        = XXXXXXXXXXXXX
client_secret    = dosedXXXXXXXXXXXXXXXX

Code -

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(profile="fielddemo-sp")

filepath=f"{w.volumes.read("<<catalog>>.<<schema>>.<<volumename>>").as_dict()['storage_location']}"
print(filepath)
vol_path = "/Volumes/catalog/schema/volumename"
for f in w.files.list_directory_contents(vol_path):
    print(f)

 

View solution in original post