For general cluster decision making refer to this article https://docs.microsoft.com/en-gb/azure/databricks/clusters/cluster-config-best-practices

Once you've selected a cluster that makes sense, run it and check your ganglia metrics to see whether you need a compute, memory, or storage optimized cluster and then iterate from there.

To just see if your code works, starting with a small set of data on a single node is best practice.