Cost Optimization for serverless Delta Live Table Implementation

[Vetrivel]

I am currently using serverless Delta Live Tables for our silver layer, specifically leveraging the apply changes API method for SCD Type 2. However, we have observed that the costs are higher than initially anticipated, and I would like to seek your guidance on on reducing costs, particularly with optimizing DBU consumption during data loads.

[Mounika_Tarigop]

To optimize DBU consumption and reduce costs while using serverless Delta Live Tables (DLT) for your silver layer, particularly with the apply changes API method for Slowly Changing Dimension (SCD) Type 2, consider the following options:

 - Instead of performing full data loads, use incremental data loading strategies. This approach reduces the amount of data processed in each run, thereby lowering DBU consumption. Delta Merge operations can be optimized for incremental loads, which is essential for SCD Type 2 tables.

• Delta Merge operations can be resource-intensive. Ensure that your merge conditions are well-optimized to minimize the amount of data scanned and processed. This can be achieved by using efficient join conditions and filtering out unnecessary data early in the pipeline. 
• Instead of materializing intermediate results as Delta tables, use temporary views where possible. Temporary views are lazily evaluated and not materialized, which can save on storage and processing costs.

[Vetrivel]

@Mounika_Tarigop Thanks for your reply. Do we have alternative to serverless DLT in databricks to build silver layer in which we don't build everything ourselves? I understand we can write customized code to perform merge operations but we have lot of tables and PK will not be same for all tables.