SQLAlchemy dialect is a wrapper for the native databricks sql connector. You can try to pass the various authentication configuration supported by the underlying SQL connector directly into the connect_args dictionary parameter of the alchemy engine.

import os
from sqlalchemy import create_engine, text
from databricks.sql.auth import AuthType

# Workspace and app credentials
DATABRICKS_HOST = os.environ.get("DBX_HOST")
HTTP_PATH = os.environ.get("DBX_HTTP_PATH")
AZURE_CLIENT_ID = os.environ.get("AZURE_APP_ID")
AZURE_CLIENT_SECRET = os.environ.get("AZURE_APP_SECRET")

# Build the engine
engine = create_engine(
    f"databricks://token:not_used@{DATABRICKS_HOST}?http_path={HTTP_PATH}&catalog=main&schema=default",
    connect_args={
        "auth_type": AuthType.AZURE_SP_M2M.value,
        "azure_client_id": AZURE_CLIENT_ID,
        "azure_client_secret": AZURE_CLIENT_SECRET,
    }
)

Meanwhile, you can remove the cache lru_cache until you get the new approach fixed.