I’m building a Databricks App that continuously queries a SQL Warehouse roughly every 30 seconds to retrieve updated data.

To avoid the overhead of repeatedly opening new connections, I’m currently caching the Databricks SQL connection using lru_cache.

from functools import lru_cache

from databricks import sql
from databricks.sql.client import Connection

from config import settings

cfg = Config()


@lru_cache(maxsize=1)
def get_connection() -> Connection:
    """Return a cached Databricks SQL connection using the configured warehouse."""
    return sql.connect(
        server_hostname=cfg.host,
        http_path=settings.sql_warehouse_http_path,
        credentials_provider=lambda: cfg.authenticate,
        use_cloud_fetch=False,
    )


def execute_query(query: str, params: dict | list | None = None) -> None:
    """Execute a SQL statement that returns no results (DDL / DML)."""
    conn = get_connection()
    with conn.cursor() as cursor:
        cursor.execute(query, parameters=params)

The issue

After running fine for a while (roughly ~10 hours, not exact), the app starts failing with this error:

2026-06-01 06:10:27,780 [INFO] databricks.sql.thrift_backend - Error during request to server:
{"method": "ExecuteStatement",
 "session-id": "...",
 "http-code": 200,
 "error-message": "",
 "original-exception": "ExecuteStatement command can only be retried for codes 429 and 503",
 "no-retry-reason": "non-retryable error",
 "attempt": "1/30"}

Once this happens, the app stops working reliably until I redeploy it. After redeploying, everything works again.

My assumption

Because the issue only appears after long runtime, I strongly suspect that the cached connection/session eventually becomes stale or invalid.

Given that I am reusing the same connection object for all queries, this seems like the most likely explanation.

Questions

1. Is it expected that a long-lived cached Databricks SQL connection eventually becomes invalid? 
2. What is the recommended approach for this kind of long-running polling app?
   • periodic reconnection (e.g. every 1 hour)?
   • retry + reconnect on failure?
   • or something like connection pooling provided by Databricks (if such exists, could not find it myself)?
3. Does Databricks have any built-in mechanism to handle stale sessions automatically for long-lived connections?

Thanks in advance! 🙂