Optimal Cluster Selection for Continuous Delta Live Tables Pipelines: Bronze and Silver

[MR07]

Hi,

I have two Delta Live Tables Pipelines. The first one is the Bronze pipeline, which handles bronze tables. These tables are defined as streaming tables, and this pipeline needs to be executed continuously. The second one is the Silver pipeline, which also needs to be executed continuously and handles tables defined as materialized views.

I would like to understand which type of cluster is best suited for each of these pipelines.

[Sidhant07]

Hi,

The best cluster type can depend on various factors, such as the specific requirements of your pipelines, the volume of data you're processing, and your budget. Therefore, it's always a good idea to test different cluster types and configurations to determine what works best for your use case.

I would recommend using a memory-optimised cluster if the volume of data is very big or a compute-optimised cluster if you are looking for reduced runtime for the pipeline.