topic Start Up Notebook in Databricks in Data Engineering

Start Up Notebook in Databricks

Soma — Sat, 07 May 2022 05:55:59 GMT

Hi Team,

Is it possible to have a startup notebook in databricks similar to init sql script (

This need to run on start of every cluster )

Re: Start Up Notebook in Databricks

Hubert-Dudek — Sat, 07 May 2022 10:43:48 GMT

There is a way to do that using terraform but it will be complicated... https://github.com/databrickslabs/terraform-provider-databricks

If you need something less complicated: for jobs cluster is easy just set task on which all others will depend.

Another way which I think of is to set the azure function which runs every minute and checks for new running clusters and then triggers their notebook.

In Azure using event grid and logic apps you can also monitor new VM deployment ad than trigger notebook.

Re: Start Up Notebook in Databricks

Soma — Sat, 07 May 2022 11:30:27 GMT

@Hubert Dudek thanks but it will be helpful if Databricks provides something out of the box

as I am planning to set some startup tasks for auditing purpose (for eg secret detection etc ) .

Another easy way is to have %run on top of every notebook but devs can escape by not executing it

Also I need it on interactive cluster

Re: Start Up Notebook in Databricks

Prabakar — Sun, 08 May 2022 21:40:14 GMT

@somanath Sankaran , Running a notebook while the cluster is starting won't be possible. If your requirement is for auditing, you can configure the audit logs.

https://docs.databricks.com/administration-guide/account-settings/audit-logs.html

Re: Start Up Notebook in Databricks

Soma — Wed, 18 May 2022 07:29:09 GMT

@Kaniz Fatma I think we dont have the option to do the same in Databricks

Re: Start Up Notebook in Databricks

Soma — Thu, 26 May 2022 12:18:15 GMT

@Kaniz Fatma sure will do