Accessing data bricks data outside data bricks

[maikel]

Hi!

What is the best way to access data bricks data, outside data bricks e.g. from Python code? The main problem is authentication so that I can access data to which I have permissions but I would like to generate token outside data bricks (e.g. via REST endpoint).
The data which I would like to access are files but also tables. 

I will be very thankful for your help and if possible examples how to achieve this.

[Chiran-Gajula]

Hello Maikel,

You can use your token generated in Databricks and use databricks SQL connector for python.
More details here: Databricks SQL Connector for Python | Databricks on AWS

[maikel]

Hi!
The problem is, I have to authenticate from python code as well with my corporation account. Not copy and paste token to the code.

Any ideas? 🙂

[dkushari]

Hi @maikel - You can set up a Service Principal in Databricks and a client ID and Client Secret. Then set up a Databricks profile and use Python code with that profile. Look at the profile section in step 2, how the profile can be set up with client ID and secret for workspace-level operation.

Databricks uses OAuth 2.0 as the preferred protocol for service principal authorization and authentication outside of the UI. Unified client authentication automates token generation and refresh. When a service principal signs in and is granted consent, OAuth issues an access token for the CLI, SDK, or other tool to use on its behalf. Each access token is valid for one hour, after which a new token is automatically requested.

Here is my example -

Profile -

[fielddemo-sp]
host             = https://workspace.cloud.databricks.com/
client_id        = XXXXXXXXXXXXX
client_secret    = dosedXXXXXXXXXXXXXXXX

Code -

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(profile="fielddemo-sp")

filepath=f"{w.volumes.read("<<catalog>>.<<schema>>.<<volumename>>").as_dict()['storage_location']}"
print(filepath)
vol_path = "/Volumes/catalog/schema/volumename"
for f in w.files.list_directory_contents(vol_path):
    print(f)