cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Optimal Cluster Selection for Continuous Delta Live Tables Pipelines: Bronze and Silver

MR07
New Contributor II

Hi,

I have two Delta Live Tables Pipelines. The first one is the Bronze pipeline, which handles bronze tables. These tables are defined as streaming tables, and this pipeline needs to be executed continuously. The second one is the Silver pipeline, which also needs to be executed continuously and handles tables defined as materialized views.

I would like to understand which type of cluster is best suited for each of these pipelines.

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @MR07When configuring Delta Live Tables pipelines, you’ll need to consider the type of cluster that best suits your requirements. 

  1. Bronze Pipeline (Streaming Tables):

  2. Silver Pipeline (Materialized Views):

    • Since the Silver pipeline handles materialized views, you might want to opt for the Pro edition of Delta Live Tables. This edition supports streaming ingest workloads as well as CDC workloads. It allows you to update tables based on changes in source data.
    • Additionally, if your materialized views require enforcing data quality constraints using Delta Live Tables expectations, consider the Advanced edition. It supports all features from Core and Pro editions, along with expectations enforcement1.

Remember that you can choose a different product edition for each pipeline, tailoring it to your specific needs. If any features in your pipeline are not supported by the selected edition, you’ll receive an error ...1.

Feel free to explore the Delta Live Tables product page for more details on these editions and their capabilities. If you have any further questions, feel free to ask! 😊

 
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!