Production vs Development DLT Schema

Dave_Nithio — Mon, 21 Oct 2024 15:00:16 GMT

My organization is currently ingesting data utilizing a Delta Live Table pipeline. This pipeline points to a production Storage location and Target schema. This means that whenever we make changes to this pipeline, it directly impacts the production data. Is there a way to point to a different storage location / target schema when making changes to test or would this require a completely separate DLT with a different schema?

Re: Production vs Development DLT Schema

VZLA — Thu, 31 Oct 2024 17:20:30 GMT

To test changes to your Delta Live Table (DLT) pipeline without impacting production data, you can point to a different storage location and target schema. This does not require creating a completely separate DLT pipeline. Here are the steps:

Create a Test Environment: Set up a separate storage location and target schema for testing. This can be done by creating a new storage account or bucket and a new schema within your data catalog.
Modify Pipeline Configuration: Update the configuration of your DLT pipeline to point to the test storage location and target schema. This can be done by modifying the pipeline settings in the Databricks UI or by updating the pipeline configuration file.
Parameterize Your Pipeline: Use parameters to easily switch between production and test environments. You can use environment variables or configuration files to manage different settings for storage locations and schemas.
Run Tests: Execute your pipeline with the test configuration to ensure that changes do not affect production data. Validate the results in the test environment before applying the changes to the production pipeline.

topic Production vs Development DLT Schema in Data Engineering

Production vs Development DLT Schema

Re: Production vs Development DLT Schema