Optimal Cluster Selection for Continuous Delta Live Tables Pipelines: Bronze and Silver

[MR07]

Hi,

I have two Delta Live Tables Pipelines. The first one is the Bronze pipeline, which handles bronze tables. These tables are defined as streaming tables, and this pipeline needs to be executed continuously. The second one is the Silver pipeline, which also needs to be executed continuously and handles tables defined as materialized views.

I would like to understand which type of cluster is best suited for each of these pipelines.

[Kaniz_Fatma]

Hi @MR07, When configuring Delta Live Tables pipelines, you’ll need to consider the type of cluster that best suits your requirements. 

1. Bronze Pipeline (Streaming Tables):

   • For the Bronze pipeline, which handles streaming tables, consider using the Core edition of Delta Live Tables. This edition is suitable for running streaming ingest workloads without advanced features like change data capture (CDC) or expectations.
   • You can choose to update the pipeline continuously or with manual triggers based on your use case¹.

2. Silver Pipeline (Materialized Views):

   • Since the Silver pipeline handles materialized views, you might want to opt for the Pro edition of Delta Live Tables. This edition supports streaming ingest workloads as well as CDC workloads. It allows you to update tables based on changes in source data.
   • Additionally, if your materialized views require enforcing data quality constraints using Delta Live Tables expectations, consider the Advanced edition. It supports all features from Core and Pro editions, along with expectations enforcement¹.

Remember that you can choose a different product edition for each pipeline, tailoring it to your specific needs. If any features in your pipeline are not supported by the selected edition, you’ll receive an error ...¹.

Feel free to explore the Delta Live Tables product page for more details on these editions and their capabilities. If you have any further questions, feel free to ask! 😊