serialized_dashboard

[egor]

I have a dashboard.json file, for example: {select * from ${{var.table_name}}}. I have job.yml and section serialized_dashboard there? bcs my job runs parallel with dashboard. Can I use variables in databrics.yml if I define the table_variable variable for the tst, prd, and dev environments?

[sarahbhord]

Hey Egor - Variable substitution inside dashboard SQL (e.g., dashboard.json or serialized_dashboard content) is not natively supported. This means that variables defined in databricks.yml will not be automatically replaced inside dashboard SQL queries. To achieve environment-specific SQL in dashboards, you need to preprocess or template the dashboard JSON file yourself before deployment as a workaround.

Fortunately, native support for this feature is planned. 

I would look into Databricks Asset Bundles. They fully support defining variables in databricks.yml for jobs and other YAML-based configs, including environment-specific overrides (for dev, tst, prd, etc.). In jobs (job.yml, YAML config), you can reference these variables directly using the ${var.variable_name} pattern, and these values will be correctly substituted when deploying to different environments.

[egor]

Hey Sarabhord!

Am I understanding correctly that if I have the following:

A dashboard.json file:

{
  "datasets": {
    "query_lines": [
      "SELECT * FROM ${var.table_name}"
    ]
  }
}

A job.yml with a serialized_dashboard section (bcs i need dashboard parallel with my job):

jobs:
  my_job:
    serialized_dashboard:
      table_name: ${var.table}

And a resources/vars.yml with environment-specific variables:

- variable: dev_table_name:
    value: dev_table
- variable: prd_table_name:
    value: prd_table

Along with a databricks.yml file that configures the entire deployment, then I cannot dynamically substitute the table names based on the environment?

Will I have to write env_dashboard.json configured for each environment and call it depending on the environment in which the job is running?

[szymon_dybczak]

Hi @egor , @sarahbhord 

You can check below issue at github. It seems that Databricks team is aware that this feature is needed by a lot of users and they're going to add this DAB.

DAB dashboards variable substitution in dataset queries · Issue #1915 · databricks/cli

As of now, you can also try to use workaround suggested by karoldegroot. I guess it's worth a try:

https://github.com/databricks/cli/issues/1915#issuecomment-3364978390

[Hubert-Dudek]

I currently use the parameter inside IDENTIFIER(:schema || 'my_table') and the 'bundle scripts' feature to perform substitutions, but I hope for better support soon.