@Mars Su​ :

Yes, in a blue-green deployment scenario, both the blue and green versions of the Spark Structured Streaming job would be running at the same time, with traffic gradually shifted from the blue to the green version.

Regarding the checkpoint location, it is generally recommended to use separate checkpoint locations for each version of the job in order to avoid potential conflicts or data corruption. This is because the checkpoint location stores the state of the streaming query, including the current offset, which is used to resume the query in case of failures or restarts.

To achieve this in Terraform, you can define two separate checkpoint locations for the blue and green versions of the job, and specify them in the

checkpoint_location parameter of the spark_conf block for each job. For example:

# Blue job
resource "databricks_job" "blue_job" {
  # ...
 
  new_cluster {
    # ...
 
    spark_conf = {
      "spark.sql.streaming.checkpointLocation" = "/blue/checkpoints"
    }
  }
 
  # ...
}
 
# Green job
resource "databricks_job" "green_job" {
  # ...
 
  new_cluster {
    # ...
 
    spark_conf = {
      "spark.sql.streaming.checkpointLocation" = "/green/checkpoints"
    }
  }
 
  # ...
}

In this example, the blue job would use the checkpoint location /blue/checkpoints

, while the green job would use /green/checkpoints. Note that you would also need to ensure that any output or intermediate data is written to separate locations for the blue and green versions of the job, to avoid conflicts or data corruption.