topic Re: Choosing the optimal cluster size/specs. in Data Engineering

Choosing the optimal cluster size/specs.

sage5616 — Wed, 03 Aug 2022 22:06:05 GMT

Hello everyone,

I am trying to determine the appropriate cluster specifications/sizing for my workload:

Run a PySpark task to transform a batch of input avro files to parquet files and create or re-create persistent views on these parquet files. This task runs every 5 mins and needs to complete within a minute.

The size of the batch of input parquet files ranges from a 100 KB to 100 MB per run.

It is important that the cluster supports creating and querying persistent views. I do not know yet how many processes will query the views, but estimating about 1 - 10 concurrent queries with simple select statements filtering data.

Big thank you 🙂

I have already researched Databricks manuals and guides and looking for an opinion/recommendation from the community.

Re: Choosing the optimal cluster size/specs.

sage5616 — Thu, 04 Aug 2022 13:40:57 GMT

Thanks? But please pay attention to what I wrote in my question above: "I have already researched Databricks manuals and guides and looking for an opinion/recommendation from the community."

Simply pointing to a manual is not helpful. @Debayan Mukherjee

Re: Choosing the optimal cluster size/specs.

Anonymous — Sun, 07 Aug 2022 20:25:11 GMT

If the data is 100MB, then I'd try a single node cluster, which will be the smallest and least expensive. You'll have more than enough memory to store it all. You can automate this and use a jobs cluster.

Re: Choosing the optimal cluster size/specs.

Debayan — Thu, 04 Aug 2022 06:34:43 GMT

Thanks for your question!

Please check https://docs.databricks.com/clusters/cluster-config-best-practices.html and let me know if it helps?