Cost estimation before query execution similar to google cloud Big Query equivalent of --dry_run

NehaR — Wed, 20 Nov 2024 00:06:45 GMT

Hi ,

In databricks do we have a option to estimate cost of query before execution which is similar to Big Query equivalent of --dry_run.

Our use case is to estimate cost before execution and get alerted.

Regards

Neha

Re: Cost estimation before query execution similar to google cloud Big Query equivalent of --dry_run

Alberto_Umana — Mon, 25 Nov 2024 15:46:51 GMT

Hello @NehaR,

Currently, Databricks does not have a direct equivalent to BigQuery's --dry_run feature for estimating the cost of a query before execution. However, there are some mechanisms and ongoing projects that aim to provide similar functionality. There is no ETA yet, I will update over here if any update on its implementations.

For now, you can monitor the DBU consumption of your clusters and use historical data to estimate the cost of similar queries. Additionally, you can run smaller versions of your queries to get an idea of their cost and then extrapolate for larger datasets

topic Re: Cost estimation before query execution similar to google cloud Big Query equivalent of --dry_run in Data Engineering

Cost estimation before query execution similar to google cloud Big Query equivalent of --dry_run

Re: Cost estimation before query execution similar to google cloud Big Query equivalent of --dry_run