topic Optimal Cluster Selection for Continuous Delta Live Tables Pipelines: Bronze and Silver in Data Engineering

Optimal Cluster Selection for Continuous Delta Live Tables Pipelines: Bronze and Silver

MR07 — Sat, 15 Jun 2024 03:22:00 GMT

Hi,

I have two Delta Live Tables Pipelines. The first one is the Bronze pipeline, which handles bronze tables. These tables are defined as streaming tables, and this pipeline needs to be executed continuously. The second one is the Silver pipeline, which also needs to be executed continuously and handles tables defined as materialized views.

I would like to understand which type of cluster is best suited for each of these pipelines.

Re: Optimal Cluster Selection for Continuous Delta Live Tables Pipelines: Bronze and Silver

Sidhant07 — Mon, 09 Dec 2024 07:25:32 GMT

Hi,

The best cluster type can depend on various factors, such as the specific requirements of your pipelines, the volume of data you're processing, and your budget. Therefore, it's always a good idea to test different cluster types and configurations to determine what works best for your use case.

I would recommend using a memory-optimised cluster if the volume of data is very big or a compute-optimised cluster if you are looking for reduced runtime for the pipeline.