topic Have you ever wondered how to automate your #databricks jobs and workflows without using the UI? If you want to manage your Databricks resources as co... in Data Engineering

Have you ever wondered how to automate your #databricks jobs and workflows without using the UI? If you want to manage your Databricks resources as co...

Hubert-Dudek — Wed, 03 May 2023 12:12:00 GMT

Have you ever wondered how to automate your #databricks jobs and workflows without using the UI? If you want to manage your Databricks resources as code, you should check out Terraform.

Here is a simple example of creating a job that runs a notebook on a new cluster using Terraform.

resource "databricks_job" "example" {
 
  name = "example-job"
 
  new_cluster {
    num_workers   = 2
    spark_version = "13.0.x-scala2.12"
    node_type_id  = "Standard_DS3_v2"
  }
 
  notebook_task {
    notebook_path = "/Users/example@example.com/MyNotebook"
  }
}

Re: Have you ever wondered how to automate your #databricks jobs and workflows without using the UI? If you want to manage your Databricks resources as co...

-werners- — Wed, 03 May 2023 13:09:58 GMT

or use ADF 🙂 Still waiting for actual added value on using Databricks Workflow over ADF.

Re: Have you ever wondered how to automate your #databricks jobs and workflows without using the UI? If you want to manage your Databricks resources as co...

Hubert-Dudek — Wed, 03 May 2023 16:55:35 GMT

at least logical if/no is coming 🙂