Calling Databricks API from Databricks notebook
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-11-2021 07:16 AM
A similar question has already been added, but the reply is very confusing to me.
Basically, for automated jobs, I want to log the following information from inside a Python notebook that runs in the job:
- What is the cluster configuration (most imporant, what machine type and how many?)
- Which user triggered the run
- And maybe some other stuff.
I know that all this information can be accessed by using the Cluster API ( https://docs.databricks.com/dev-tools/api/latest/clusters.html#get ) - however, I am not sure how to do this while on the cluster itself. Is there an easier way, like going through dbutils? If not, how would authorization work, since the job might be started by anyone on our instance? I would highly appreciate some sample code that always works when on the cluster itself... Thanks!
- Labels:
-
Notebook
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-11-2021 07:20 AM
All the examples seem to use a .netrc file, but it looks like on the cluster, there is none by default - surely, there is some way to check what the cluster config of the current cluster is without going through lots of authentication hoops?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-12-2022 08:22 PM
@Thomas Kastl First and foremost, api auth is needed to call https://docs.databricks.com/dev-tools/api/latest/clusters.html?&_ga=2.69428824.930568569.1652408683-...
like this - https://docs.databricks.com/dev-tools/api/latest/authentication.html . though you can call this through notebook, terminal or IDE (https://docs.databricks.com/dev-tools/index.html#developer-tools-and-guidance)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-25-2022 02:06 PM
hi @Thomas Kastl,
Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2022 12:30 AM
Thanks for the follow-up. We resolved this a different way, but the input with the dbutils.secret was helpful.

