topic Re: Cost Optimization in Data Engineering

Cost Optimization

christo_M — Wed, 28 Jun 2023 18:44:26 GMT

How can I optimize the cost on our Databricks platform ? Despite some optimization actions I've taken so far it's still difficult to lower the cost. I tried different technics like Vacuum , or shutting down a cluster running after 30 mins but still don't any significant improvement.

Re: Cost Optimization

RajeshReddy — Wed, 28 Jun 2023 21:53:10 GMT

you might want to take a look at job clusters for cost optimization

https://learn.microsoft.com/en-us/azure/databricks/clusters/cluster-config-best-practices

Re: Cost Optimization

JulioCastro — Wed, 28 Jun 2023 22:09:43 GMT

Give me more info on what you are doing. Is it something you can run once a day? Run on schedule. Are you cleaning data? Or is it inference? Really does depend

Re: Cost Optimization

rodrigosanchesz — Wed, 28 Jun 2023 22:13:27 GMT

Have you try to use Job Cluster?

You also can expurge no more use tables in data store and optimize table for querys can execute more fast.

Re: Cost Optimization

erigaud — Tue, 04 Jul 2023 06:30:20 GMT

Make sure you're using a cluster that is the right size for your workload. You can greatly reduce the costs by using smaller clusters.