topic Automate Cluster creation in missing-QuestionPost

Automate Cluster creation

Vidisha — Tue, 13 Jun 2023 10:41:58 GMT

I am new to databricks and my lead told me that we create clusters for running the notebook manually. So please write one python script to do this automation i.e to create the clusters automatically.

Can anyone help me to write the script using PySpark in Databricks. I have to use Azure Cloud Services for this.

Re: Automate Cluster creation

-werners- — Tue, 13 Jun 2023 12:59:07 GMT

I see 2 easy ways:

use databricks workflows. this creates job clusters for you
use data factory and the Databricks notebook activity: this does the same

Basically you tell databricks to create short-lived clusters which are alive during the execution of a spark program. When the program is finished, the cluster is terminated.

There is also something like cluster pools, which keeps instances warm.

Re: Automate Cluster creation

Vidisha — Wed, 14 Jun 2023 20:29:58 GMT

Actually we already have job clusters created. So now we need to automate existing manually created clusters which is used in running multiple notebooks by multiple users. Please suggest

Re: Automate Cluster creation

-werners- — Thu, 15 Jun 2023 07:14:47 GMT

So you created interactive clusters which you want to use to run notebooks in a scheduled way?

If so: you can schedule notebooks on an existing cluster (the schedule button) or in data factory use an existing cluster to run a notebook. So my answer stays pretty much the same.

If that is not what you are looking for, please explain your use case.