I asked this question a while ago where I explain the cluster that my team uses on databricks. To save you some time, we use an all-purpose Standard D2ads v6 with 8 gigs of ram and 2 cores cluster. We are facing an issue with the memory, which is pin...