Hi,

The best cluster type can depend on various factors, such as the specific requirements of your pipelines, the volume of data you're processing, and your budget. Therefore, it's always a good idea to test different cluster types and configurations to determine what works best for your use case.

I would recommend using a memory-optimised cluster if the volume of data is very big or a compute-optimised cluster if you are looking for reduced runtime for the pipeline.