topic Re: Calling Databricks API from Databricks notebook in Data Engineering

Calling Databricks API from Databricks notebook

ThomasKastl — Wed, 11 Aug 2021 14:16:16 GMT

A similar question has already been added, but the reply is very confusing to me.

Basically, for automated jobs, I want to log the following information from inside a Python notebook that runs in the job:

- What is the cluster configuration (most imporant, what machine type and how many?)

- Which user triggered the run

- And maybe some other stuff.

I know that all this information can be accessed by using the Cluster API ( https://docs.databricks.com/dev-tools/api/latest/clusters.html#get ) - however, I am not sure how to do this while on the cluster itself. Is there an easier way, like going through dbutils? If not, how would authorization work, since the job might be started by anyone on our instance? I would highly appreciate some sample code that always works when on the cluster itself... Thanks!

Re: Calling Databricks API from Databricks notebook

ThomasKastl — Wed, 11 Aug 2021 14:20:17 GMT

All the examples seem to use a .netrc file, but it looks like on the cluster, there is none by default - surely, there is some way to check what the cluster config of the current cluster is without going through lots of authentication hoops?

Re: Calling Databricks API from Databricks notebook

Atanu — Fri, 13 May 2022 03:22:42 GMT

@Thomas Kastl First and foremost, api auth is needed to call https://docs.databricks.com/dev-tools/api/latest/clusters.html?&_ga=2.69428824.930568569.1652408683-480220406.1638459894&_gac=1.113179766.1652199842.Cj0KCQjwmuiTBhDoARIsAPiv6L8UwsWrnD6YaYVaeDvIdOpt0XSNFvM1KNup4vlU7eylk4fFmx8TucgaAlrNEALw_wcB#get

like this - https://docs.databricks.com/dev-tools/api/latest/authentication.html . though you can call this through notebook, terminal or IDE (https://docs.databricks.com/dev-tools/index.html#developer-tools-and-guidance)

Re: Calling Databricks API from Databricks notebook

jose_gonzalez — Mon, 25 Jul 2022 21:06:39 GMT

hi @Thomas Kastl,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

Re: Calling Databricks API from Databricks notebook

ThomasKastl — Tue, 26 Jul 2022 07:30:20 GMT

Thanks for the follow-up. We resolved this a different way, but the input with the dbutils.secret was helpful.