Start Up Notebook in Databricks

[Soma]

Hi Team,

Is it possible to have a startup notebook in databricks similar to init sql script (

This need to run on start of every cluster )

[Hubert-Dudek]

There is a way to do that using terraform but it will be complicated... https://github.com/databrickslabs/terraform-provider-databricks

If you need something less complicated: for jobs cluster is easy just set task on which all others will depend.

Another way which I think of is to set the azure function which runs every minute and checks for new running clusters and then triggers their notebook.

In Azure using event grid and logic apps you can also monitor new VM deployment ad than trigger notebook.

[Soma]

@Hubert Dudek​ thanks but it will be helpful if Databricks provides something out of the box

as I am planning to set some startup tasks for auditing purpose (for eg secret detection etc ) .

Another easy way is to have %run on top of every notebook but devs can escape by not executing it

Also I need it on interactive cluster

[Prabakar · Accepted solution]

@somanath Sankaran​ , Running a notebook while the cluster is starting won't be possible. If your requirement is for auditing, you can configure the audit logs.

https://docs.databricks.com/administration-guide/account-settings/audit-logs.html

[Kaniz_Fatma]

Hi @somanath Sankaran​ , Just a friendly follow-up. Do you still need help, or @Prabakar Ammeappin​ 's response help you to find the solution? Please let us know.

[Soma]

@Kaniz Fatma​ I think we dont have the option to do the same in Databricks

[Kaniz_Fatma]

Hi @somanath Sankaran​ , Well, In that case, would you like to raise a feature request at the Databricks portal here?

[Soma]

@Kaniz Fatma​ sure will do

[Kaniz_Fatma]

Great, Thanks!