Databricks Running Jobs and Terraform

[scottbisaillon]

What happens to a currently running job when a workspace is deployed again using Terraform? Are the jobs paused/resumed, or are they left unaffected without any down time? Searching for this specific scenario doesn't seem to come up with anything and some initial testing seems lead to no disruption, but knowing specifically what happens would be good for reference. Thanks!

[Kaniz_Fatma]

Hi @scottbisaillon, When deploying a workspace again using Terraform, the behaviour regarding currently running jobs depends on the specific Terraform version and the platform you are using.

Let’s explore the details:

1. Terraform Cloud (formerly known as Terraform Enterprise):

   • In Terraform Cloud, when you deploy a workspace again using Terraform, it no longer triggers the run...¹. This means that existing runs are not paused or resumed; they continue to run without any disruption.
   • You can downgrade a workspace with existing runs to use a prior Terraform version without causing an error. However, the run tasks won’t be triggered during the downgrade process.

2. Databricks:

   • In Databricks, there is a community discussion about running jobs and Terraform. Unfortunately, there isn’t a clear-cut answer to your specific scenario. Some initial testing suggests that there is no disruption to currently running jobs when deploying a...².

3. Best Practices:

   • When managing multiple environments (such as dev, staging, and production) using Terraform, consider adopting best practices like using workspaces, environment-specific configurations, and modules. This approach helps maintain a clean, organized, and efficient infrastructure-as-code (IaC) workflow³.
   • Integrating with CI/CD pipelines further enhances the security, reliability, and speed of your deployments.

Remember that the behaviour might vary based on the specific tools and configurations you’re using.  😊🚀