Databricks Apps - Streamlit Performance Hangups

[brianr]

Hi All,

I have a Streamlit app running via Databricks Apps. The app is fairly simple and displays data from a small handful of lightweight database queries (each running in less than 1 second).

As of a few days ago, this app was running great. But as of Feb 12, the application hangs at seemingly random points. The app will start, display data from one or two queries, then hang for 2-3 minutes. Most of the time, after a few minutes the next query will execute and the data will be displayed in the app.

I have watched the warehouse query logs while the app is running, and this is not a problem with the query execution. The queries are still executing in under a second. It seems like the Streamlit server is taking a very long time to send the queries to the database.

1. How can I go about trouble shooting this? I'm unsure of where/how the Streamlit application server is hosted and if it's possible to output more verbose or detailed logs.

2. Could this be a problem with the database connection? I'm following the same connection and query pattern as the template app, although I am calling sqlQuery() multiple times in sequence.

def sqlQuery(query: str) -> pd.DataFrame:
    cfg = Config() # Pull environment variables for auth
    with sql.connect(
        server_hostname=cfg.host,
        http_path=f"/sql/1.0/warehouses/{os.getenv('DATABRICKS_WAREHOUSE_ID')}",
        credentials_provider=lambda: cfg.authenticate
    ) as connection:
        with connection.cursor() as cursor:
            cursor.execute(query)
            return cursor.fetchall_arrow().to_pandas()

[Alberto_Umana]

Hello @brianr,

Have you validated the SQL warehouse you are referring in your code? I would be useful to identify the delayed queries and check it's status from "Query History" in Databricks UI.

[brianr]

I have monitored the Query History while the app is running and there are no long running queries.

What I see is a long gap (2-3 minutes) between queries even reaching the warehouse to be executed.

Example of Query times and run length:

App Query 1:  8:30:00     500ms
App Query 2:  8:30:10     250ms
* 3 minute gap
App Query 3:  8:33:30     200ms
App Query 4:  8:33:25     300ms