Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
4 weeks ago
You can expect Long lived cached SQL connections to become stale (due to idle and session timeouts) for better resource governance (warehouse auto scaling), security and optimizations (TLS drops, backend session expiration, routing). The underlying Thrift session is invalidated.
You can follow below
- Connection lifecycle management - You can implement a reconnect on failure wrapper or use SQLAlchemy with the databricks version. Its QueuePool provides various parameters - pool_pre_ping (True - validates connections before use) and pool_recycle (1800 - forces refresh every 30 mins) solving staleness. More details here
- Idempotent retries - You can catch session errors, discard the connection, instantiate a new one and implement retry with exponential backoff.
- Warehouse - You can disable auto stop in the warehouse if you require continuous availability.
You can plan for client side recycling as recycling is user responsibility.
