Predictive Optimization with multiple workspaces

KIRKQUINBAR — Wed, 16 Apr 2025 18:02:10 GMT

We currently have an older instance of Azure Databricks that i migrated to Unity Catalog. Unfortunately i ran into some weird issues that don't seem fixable so i created a new instance and pointed it to the same metastore. The setting at the metastore that turns on Predictive Optimization is currently disabled. If i turn it on will both Workspaces then try to do optimizations thus causing more compute usage than needed or is Predictive Optimization completely separate from the workspace?

Re: Predictive Optimization with multiple workspaces

Renu_ — Fri, 18 Apr 2025 10:35:13 GMT

Hi @KIRKQUINBAR, if you enable Predictive Optimization at the metastore level in Unity Catalog, it automatically applies to all Unity Catalog managed tables within that metastore, no matter which workspace is accessing them. PO runs centrally, so there’s no duplication of compute across different workspaces. However, compute usage is tied to table activity. If multiple workspaces are actively writing to the same tables, PO may kick in more frequently since it’s triggered by data changes. This could lead to an increase in overall compute usage and potentially higher costs.

To keep track of optimization activity and performance, you can use system tables like system.storage.predictive_optimization_operations_history. This helps you identify any unexpected behavior or spikes in compute usage related to predictive optimization runs.

topic Re: Predictive Optimization with multiple workspaces in Administration & Architecture

Predictive Optimization with multiple workspaces

Re: Predictive Optimization with multiple workspaces