The best way is to test based on 
Data Size,
Job Complexity,
Frequency.

Test with small data set and then scale up based on needs.
for example
To configure a Databricks cluster for processing 100 GB of data, you must consider:

• Data size: 100 GB is moderate; it doesn’t require massive scale.

• Job complexity: Simple transformations need fewer resources; joins, shuffles, ML models may require more.

• Frequency: For scheduled jobs, use Job Clusters; for exploration, All-Purpose Clusters.

• Recommended setup:

  • 4–8 workers (each with 8–16 GB RAM)

  • Auto-scaling enabled (handles variable workload)

  • Use spot instances to reduce cost (if SLA allows)

 Always test with a smaller dataset and monitor performance using Spark UI before scaling up.