topic Re: How do I setup the right cluster? in Data Engineering

How do I setup the right cluster?

Kaher — Tue, 28 Jun 2022 22:02:20 GMT

Re: How do I setup the right cluster?

Ybaselto — Wed, 29 Jun 2022 05:11:23 GMT

Personnaly, once my data processing is optimize, i benchmark different setups to find the one that respect my process time goal for the less dbu. (Sorry for my english)

Re: How do I setup the right cluster?

Rheiman — Thu, 30 Jun 2022 05:59:38 GMT

For general cluster decision making refer to this article https://docs.microsoft.com/en-gb/azure/databricks/clusters/cluster-config-best-practices

Once you've selected a cluster that makes sense, run it and check your ganglia metrics to see whether you need a compute, memory, or storage optimized cluster and then iterate from there.

To just see if your code works, starting with a small set of data on a single node is best practice.

Re: How do I setup the right cluster?

Hubert-Dudek — Mon, 11 Jul 2022 15:27:02 GMT

Great article. In the future serverless option will make it easier for newbies.