cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Calling Databricks API from Databricks notebook

ThomasKastl
Contributor

A similar question has already been added, but the reply is very confusing to me.

Basically, for automated jobs, I want to log the following information from inside a Python notebook that runs in the job:

- What is the cluster configuration (most imporant, what machine type and how many?)

- Which user triggered the run

- And maybe some other stuff.

I know that all this information can be accessed by using the Cluster API ( https://docs.databricks.com/dev-tools/api/latest/clusters.html#get ) - however, I am not sure how to do this while on the cluster itself. Is there an easier way, like going through dbutils? If not, how would authorization work, since the job might be started by anyone on our instance? I would highly appreciate some sample code that always works when on the cluster itself... Thanks!

4 REPLIES 4

ThomasKastl
Contributor

All the examples seem to use a .netrc file, but it looks like on the cluster, there is none by default - surely, there is some way to check what the cluster config of the current cluster is without going through lots of authentication hoops?

Atanu
Esteemed Contributor

@Thomas Kastl​ First and foremost, api auth is needed to call https://docs.databricks.com/dev-tools/api/latest/clusters.html?&_ga=2.69428824.930568569.1652408683-...

like this - https://docs.databricks.com/dev-tools/api/latest/authentication.html . though you can call this through notebook, terminal or IDE (https://docs.databricks.com/dev-tools/index.html#developer-tools-and-guidance)

jose_gonzalez
Moderator
Moderator

hi @Thomas Kastl​,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

Thanks for the follow-up. We resolved this a different way, but the input with the dbutils.secret was helpful.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group