Cost estimation before query execution similar to google cloud Big Query equivalent of --dry_run

[NehaR]

Hi ,

In databricks do we have a option to estimate cost of query before execution which is similar to Big Query equivalent of --dry_run.

Our use case is to estimate cost before execution and get alerted.

Regards

Neha 

[Alberto_Umana]

Hello @NehaR,

Currently, Databricks does not have a direct equivalent to BigQuery's --dry_run feature for estimating the cost of a query before execution. However, there are some mechanisms and ongoing projects that aim to provide similar functionality. There is no ETA yet, I will update over here if any update on its implementations.

For now, you can monitor the DBU consumption of your clusters and use historical data to estimate the cost of similar queries. Additionally, you can run smaller versions of your queries to get an idea of their cost and then extrapolate for larger datasets