cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Start Up Notebook in Databricks

Soma
Valued Contributor

Hi Team,

Is it possible to have a startup notebook in databricks similar to init sql script (

This need to run on start of every cluster )

1 ACCEPTED SOLUTION

Accepted Solutions

Prabakar
Esteemed Contributor III
Esteemed Contributor III

@somanath Sankaran​ , Running a notebook while the cluster is starting won't be possible. If your requirement is for auditing, you can configure the audit logs.

https://docs.databricks.com/administration-guide/account-settings/audit-logs.html

View solution in original post

8 REPLIES 8

Hubert-Dudek
Esteemed Contributor III

There is a way to do that using terraform but it will be complicated... https://github.com/databrickslabs/terraform-provider-databricks

If you need something less complicated: for jobs cluster is easy just set task on which all others will depend.

Another way which I think of is to set the azure function which runs every minute and checks for new running clusters and then triggers their notebook.

In Azure using event grid and logic apps you can also monitor new VM deployment ad than trigger notebook.

Soma
Valued Contributor

@Hubert Dudek​ thanks but it will be helpful if Databricks provides something out of the box

as I am planning to set some startup tasks for auditing purpose (for eg secret detection etc ) .

Another easy way is to have %run on top of every notebook but devs can escape by not executing it

Also I need it on interactive cluster

Prabakar
Esteemed Contributor III
Esteemed Contributor III

@somanath Sankaran​ , Running a notebook while the cluster is starting won't be possible. If your requirement is for auditing, you can configure the audit logs.

https://docs.databricks.com/administration-guide/account-settings/audit-logs.html

Kaniz
Community Manager
Community Manager

Hi @somanath Sankaran​ , Just a friendly follow-up. Do you still need help, or @Prabakar Ammeappin​ 's response help you to find the solution? Please let us know.

Soma
Valued Contributor

@Kaniz Fatma​ I think we dont have the option to do the same in Databricks

Kaniz
Community Manager
Community Manager

Hi @somanath Sankaran​ , Well, In that case, would you like to raise a feature request at the Databricks portal here?

Soma
Valued Contributor

@Kaniz Fatma​ sure will do

Kaniz
Community Manager
Community Manager

Great, Thanks!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.