Databricks app with parameters from databricks asset bundle

[Naga05]

Hellooo

I tried out setting up a Databricks App using asset bundle, where i was able to successfully parameterize the sql warehouse id which was specified on specific targets. However i was unable to get values of other variables from the targets, the variable does not get resolved on the app.

databricks.yml:

bundle:
  name: etl_deploy

include:
  - common/*.yml

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://11111111.4.gcp.databricks.com/
    variables:
      short_env_name: dev
      permission_level: CAN_MANAGE
      schedule_state: PAUSED
      db_connection: salesforce_dev
  qal:
    mode: production
    default: true
    workspace:
      host: https://1234567.1.gcp.databricks.com/
    variables:
      short_env_name: qal
      permission_level: CAN_RUN
      schedule_state: UNPAUSED
      db_connection: salesforce_uat

Streamlit app's app.yml file:

command:
  - "streamlit"
  - "run"
  - "app.py"

env:
  - name: SQL_WAREHOUSE_ID
    valueFrom: sql-warehouse
  - name: DB_CONNECTION
    value: ${var.db_connection}

My common_variables.yml file:

variables:
  # Database connections settings
  db_connection:
    default: "unknown"

 On my app.py file, i tried to access the variable as below:

connection = os.getenv("DB_CONNECTION")

 when i deploy the app, i can see from the Environment tab of the App as unresolved value:

"DB_CONNECTION=${var.db_connection}",

What would be the way to catch the variable values from the Bundle targets into the Databricks App?

[saurabh18cs]

Hi @Naga05 

try this:

env:

  - name: DB_CONNECTION

    valueFrom: db_connection

OR 

env:

  - name: DB_CONNECTION

    value: salesforce_dev

Br

Saurabh

[Naga05]

Hi Saurabh,

First approach there did not work for me. i see an error on the Databricks App logs:

[ERROR] Error resolving resource db_connection specified in app.yml. Please make sure to configured a resource with name db_connection

How to configure a variable on resources file? My current resources file is:

resources:
  apps:
    query-data:
      name: query-data
      description: 'To get sales data from the delta table'

      source_code_path: ../app

      resources:
        - name: "sql-warehouse"
          sql_warehouse:
            id: ${var.sql_warehouse_id}
            permission: "CAN_MANAGE"

      permissions:
        - level: CAN_MANAGE
          group_name: ${var.access_group}
​

Second approach always works since it is hardcoding the value for that variable. And this is not a suitable approach, i want to make it dynamic with values from different target variables of the bundle.

Thanks,

Naga

[saurabh18cs]

Hi @Naga05 how about adding this variable under bundle’s resources file common_variables.yml?

1)

  variables:

    db_connection:

      default:

        value: ${var.db_connection}

OR 

variables:

    db_connection:

        default: ${var.db_connection}

2) reference this resource in app.yml

  - name: DB_CONNECTION

    valueFrom: db_connection

[Naga05]

Found that this is an implementation in progress on the Databricks CLI. https://github.com/databricks/cli/issues/3679