Hi All,
We are using the Azure Databricks platform for one of our Data Engg needs. Here's my setup -
1. Job compute that uses Cluster of size - 1 driver and 2 workers - all are of 'Standard_DS3_v2' type. (Photon is disabled).
2. The job compute takes the instances from the instance pool since we want to reduce the cluster start-up time. Instance pool uses "All spot" settings and keeps 3 instances idle.
How do I run the job?
1. The job is run via workflows every 30 minutes. It takes 7 to 8 minutes to complete.
The cost of this setup?
Based on my research, I have come up with the below cost estimation-
1. €0.233/hour/instance - For 7-8 mins during which my job is running thus utilizing both DBUs and VMs. (https://azure.microsoft.com/en-in/pricing/details/databricks/)
2. €0.0252/hour/instance - For the rest 22-23 minutes where my instances are idle but no active DBUs are consumed. (https://azure.microsoft.com/en-in/pricing/details/virtual-machines/linux/#pricing )
When calculating it at the monthly level there's a crazy difference between my estimated and actual costs.
Am I missing anything? One thing that I don't understand is the disk (storage) cost associated with the Azure VMs.
I am happy to share more information as needed on this, but can someone please help to understand the detailed cost?