cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks app with parameters from databricks asset bundle

Naga05
New Contributor II

Hellooo

I tried out setting up a Databricks App using asset bundle, where i was able to successfully parameterize the sql warehouse id which was specified on specific targets. However i was unable to get values of other variables from the targets, the variable does not get resolved on the app.

databricks.yml:

bundle:
  name: etl_deploy

include:
  - common/*.yml

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://11111111.4.gcp.databricks.com/
    variables:
      short_env_name: dev
      permission_level: CAN_MANAGE
      schedule_state: PAUSED
      db_connection: salesforce_dev
  qal:
    mode: production
    default: true
    workspace:
      host: https://1234567.1.gcp.databricks.com/
    variables:
      short_env_name: qal
      permission_level: CAN_RUN
      schedule_state: UNPAUSED
      db_connection: salesforce_uat

Streamlit app's app.yml file:

command:
  - "streamlit"
  - "run"
  - "app.py"

env:
  - name: SQL_WAREHOUSE_ID
    valueFrom: sql-warehouse
  - name: DB_CONNECTION
    value: ${var.db_connection}

My common_variables.yml file:

variables:
  # Database connections settings
  db_connection:
    default: "unknown"

 On my app.py file, i tried to access the variable as below:

connection = os.getenv("DB_CONNECTION")

 when i deploy the app, i can see from the Environment tab of the App as unresolved value:

"DB_CONNECTION=${var.db_connection}",

What would be the way to catch the variable values from the Bundle targets into the Databricks App?

4 REPLIES 4

saurabh18cs
Honored Contributor II

Hi @Naga05 

try this:

env:
  - name: DB_CONNECTION
    valueFrom: db_connection
 
OR 
 
env:
  - name: DB_CONNECTION
    value: salesforce_dev
 
Br
Saurabh

Naga05
New Contributor II
Hi Saurabh,

First approach there did not work for me. i see an error on the Databricks App logs:
[ERROR] Error resolving resource db_connection specified in app.yml. Please make sure to configured a resource with name db_connection
 
How to configure a variable on resources file? My current resources file is:
resources:
  apps:
    query-data:
      name: query-data
      description: 'To get sales data from the delta table'

      source_code_path: ../app

      resources:
        - name: "sql-warehouse"
          sql_warehouse:
            id: ${var.sql_warehouse_id}
            permission: "CAN_MANAGE"

      permissions:
        - level: CAN_MANAGE
          group_name: ${var.access_group}
​

 

 
Second approach always works since it is hardcoding the value for that variable. And this is not a suitable approach, i want to make it dynamic with values from different target variables of the bundle.

 

Thanks,
Naga

saurabh18cs
Honored Contributor II

Hi @Naga05 how about adding this variable under bundle’s resources file common_variables.yml?

1)
  variables:
    db_connection:
      default:
        value: ${var.db_connection}
 
OR 
variables:
    db_connection:
        default: ${var.db_connection}
 
2) reference this resource in app.yml
 
  - name: DB_CONNECTION
    valueFrom: db_connection
 

Naga05
New Contributor II

Found that this is an implementation in progress on the Databricks CLI. https://github.com/databricks/cli/issues/3679

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now