Calling Databricks API from Databricks notebook

[ThomasKastl]

A similar question has already been added, but the reply is very confusing to me.

Basically, for automated jobs, I want to log the following information from inside a Python notebook that runs in the job:

- What is the cluster configuration (most imporant, what machine type and how many?)

- Which user triggered the run

- And maybe some other stuff.

I know that all this information can be accessed by using the Cluster API ( https://docs.databricks.com/dev-tools/api/latest/clusters.html#get ) - however, I am not sure how to do this while on the cluster itself. Is there an easier way, like going through dbutils? If not, how would authorization work, since the job might be started by anyone on our instance? I would highly appreciate some sample code that always works when on the cluster itself... Thanks!

[ThomasKastl]

All the examples seem to use a .netrc file, but it looks like on the cluster, there is none by default - surely, there is some way to check what the cluster config of the current cluster is without going through lots of authentication hoops?

[Kaniz]

Hi @ ThomasKastl ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the community have an answer to your question first. Or else I will follow up with my team and get back to you soon.Thanks.

[Atanu]

@Thomas Kastl​ First and foremost, api auth is needed to call https://docs.databricks.com/dev-tools/api/latest/clusters.html?&_ga=2.69428824.930568569.1652408683-...

like this - https://docs.databricks.com/dev-tools/api/latest/authentication.html . though you can call this through notebook, terminal or IDE (https://docs.databricks.com/dev-tools/index.html#developer-tools-and-guidance)

[jose_gonzalez]

hi @Thomas Kastl​,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

[ThomasKastl]

Thanks for the follow-up. We resolved this a different way, but the input with the dbutils.secret was helpful.