Hi @dpc ,

• Managing DDL changes accross environments (dev, test, prod) will be a part of your CI/CD pipeline.
• Create a Deployments folder, with subfolders. Each deployment will be represented by a timestamped folder. Example folder structure:

/Deployments
    /Deployments_20240924
        /01_add_column4_table1.sql
        /02_alter_other_table.sql
    /Deployments_20240930
        /01_create_new_table.sql
        /02_update_table1.sql

• Inside the folder, each script (e.g., 01_add_column4_table1.sql) represents individual DDL changes (like adding a column or altering a table).
• You can then integrate your CI/CD pipeline with the deployment folder
  1. Add deployment folder name to your CI/CD pipeline
  2. After your notebooks are deployed, add to the pipeline the step to run the Notebooks that are inside the provided deployment pipeline
• Alternatively, for every deployment you can create a deployment Job. Each job will have multiple tasks, where each task represent a specific DDL change. In the example, you would create Deployment_20240924 job, that would have 2 steps: to run 01_add_column4_table1.sql and then to run 02_alter_other_table.sql.
  1. Running the job could also be a step in your CI/CD pipeline
  2. Alternatively, the job could already be scheduled to run at a given time (but you need to know when you deployment is going to happen
  3. Alternatively, the job could be run manually after the CI/CD pipeline finishes deploying the notebooks

Thanks.

I'll step through this solution and see if I can get it working