Streamlit app on Databricks doesn't recognise the DATABRICKS_WAREHOUSE_ID in the yaml file

[AnaMocanu]

Hey everyone,

So I managed to create a Streamlit app on Databricks, works fine deployed in the cloud.

However, when I try to run it locally, it complains about "assert os.getenv('DATABRICKS_WAREHOUSE_ID'), "DATABRICKS_WAREHOUSE_ID must be set in app.yaml."" It looks like 

valueFrom: "sql-warehouse" is not being picked up properly in the yaml file. In the app config, it created a service principal with the "sql-warehouse" key that should pick up my serverless warehouse that I want to use.

My workaround is to specify the value of the warehouse as an environment variable in the app.py and that seems to work. 
like this:

#assert os.getenv('DATABRICKS_WAREHOUSE_ID'), "DATABRICKS_WAREHOUSE_ID must be set in app.yaml."

os.environ['DATABRICKS_WAREHOUSE_ID'] = my_value

Any thoughts on how to enforce the yaml file to pick up the right values?

Many thanks!

Ana

[Alberto_Umana]

Hi @AnaMocanu,

How are you setting up: DATABRICKS_WAREHOUSE_ID?

When running the app locally, the environment variables set in the app.yaml file might not be picked up automatically. You can manually set the environment variable in your local environment before running the app. For example, in a Unix-based system, you can use

export DATABRICKS_WAREHOUSE_ID=<your_warehouse_id>
streamlit run app.py

Additionally, you might want to implement some prints to validate if information is correct:

import os
print("DATABRICKS_WAREHOUSE_ID:", os.getenv('DATABRICKS_WAREHOUSE_ID'))

[AnaMocanu]

Hi @Alberto_Umana 

Thanks for your reply!
DATABRICKS_WAREHOUSE_ID is set up in yaml file like this:

env:

- name: "DATABRICKS_WAREHOUSE_ID"

valueFrom: "sql-warehouse"

When the app is running on Databricks, I can see a Service principal in app resources that has the value set up.

Your suggestion is how I hack it at the moment, I'm setting up the id manually, but I don't want to do it like that, cause I'm using a serverless warehouse and that value could change?

[pdiamond]

I'm running into the same problem. Did you ever figure it out?

[AnaMocanu]

Unfortunately not 😞 the other way to avoid it is to not use serverless as the warehouse name won't change then.

[pdiamond]

I figured it out this morning. You have to use an underscore in the yaml file instead of a dash. This works for me:

  - name: "DATABRICKS_WAREHOUSE_ID"

    valueFrom: "sql_warehouse"

[AnaMocanu · Accepted solution]

thanks for that @pdiamond - the underscore didn't work for me, but trying again with the parameter set up it seems that it works now! 

I can see the key "sql-warehouse" under App resources. Yours must be set up "sql_warehouse" instead.

Glad we both sorted it.