Databricks Community

saf_dbx · 3 weeks ago

I'm connecting to a Databricks catalog via a SQL Warehouse using the Databricks ODBC Driver (v2.12.0). During one of the fetches from my BI application, the transaction aborts with:

Database operation [SQLExecute] failed with error [[Databricks][Hardy] (124) A 503 response was returned but no Retry-After header was provided. Original error: Unknown]

I've seen references to a driver parameter called `EnableRetryWithoutRetryAfterHeader` that's supposed to make the driver retry a 503 even without a `Retry-After` header, but I can't find this documented in official Databricks ODBC docs.

This error doesn't come all the time, and most of the time it works okay.

Can anyone confirm:
If such a parameter exists, how can one configure this on the DSN settings?

Thanks in advance for any pointers.

ThomazNeto · 3 weeks ago

Hi,

I've hit this exact error in a few environments, so maybe this helps.

First, the 503 itself is coming from the Databricks side, not from your BI tool or the driver. In every case I've traced it, the timestamps lined up with the SQL Warehouse being in a transitional state: auto-stop / cold start, scaling up or down, or a node being replaced. The service answers 503 for a few seconds and then recovers on its own. Worth cross-checking your failure times against the warehouse events and Query History.

What makes it fatal is the driver behavior. Out of the box, the ODBC driver only retries a 503 when the server includes a Retry-After header, and Databricks doesn't always send one. Without the header the driver gives up immediately and you get the Hardy (124) error.

About the parameter: EnableRetryWithoutRetryAfterHeader does exist, but you won't find it in the Databricks docs. I couldn't find it in the docs.databricks.com ODBC pages or in the release notes from 2.9.1 up to 2.12.1. The only place I've seen it written down is a Qlik support article from May this year, which says it's available from driver 2.9.1 onward and defaults to 0. It's one of those "key name only" options that don't show up in the DSN dialog.

To set it:

Windows: open regedit, navigate to HKEY_LOCAL_MACHINE, then SOFTWARE, then ODBC, then ODBC.INI, then your DSN name, and add a string value named EnableRetryWithoutRetryAfterHeader with value 1. Or append it to the connection string if your BI tool lets you.

Linux/macOS: add the line EnableRetryWithoutRetryAfterHeader=1 to the DSN section in odbc.ini.

DSN-less connection string: add EnableRetryWithoutRetryAfterHeader=1 as one more semicolon-separated property.

You're on 2.12.0, so version isn't a problem.

On the warehouse side, if this happens a lot, look at the auto-stop setting (extend it or disable it during the refresh window), or move to Serverless, which starts in seconds and cuts down on the cold-start 503s considerably. If you want proof for a support ticket, turn on driver logging (LogLevel=4 plus LogPath) and you should see the raw 503 with no Retry-After in the trace.

One caveat: since the parameter isn't officially documented by Databricks, I'd test it in your environment before rolling it out broadly. It's been stable where I've used it, but I can't point you to an official Databricks reference for it.

Hope this helps.

Thomaz A. Rossito Neto
Principal Data Architect & AI Strategy — CI&T
thomazn@ciandt.com
linkedin.com/in/thomaz-antonio-rossito-neto

saf_dbx · 3 weeks ago

Thanks I will try setting the parameter and see if that will help. The databricks SQL Warehouse is already serverless in my case.

anuj_lathi · Thursday

Short answer: I can't confirm EnableRetryWithoutRetryAfterHeader as a supported Databricks ODBC setting. It isn't a key Databricks publishes for the driver, so I wouldn't build on it. Even if some build accepts it, I'd get that confirmed in writing before relying on it in production.

Where to look for authoritative driver keys

The Databricks ODBC Driver Guide is the source of truth for what the driver accepts. It's linked from the ODBC docs and also ships in the docs directory of your driver download [1]. The public docs cover the connection keys (Host, HTTPPath, AuthMech, etc.) [2] and the capability settings such as Schema, UseNativeQuery, ssp_query_tags, LogLevel and LogPath [3]. Search the guide for "retry" and "503" in your 2.12.0 copy. If the key isn't there, ask Databricks Support to confirm it for your build.

How to set any additional key

If you do get confirmation that a key exists for your driver version, you set it like any other:

Linux/macOS (odbc.ini😞 add it as another key=value line in the DSN section [4].

[Databricks]
Driver=<path-to-driver>
Host=<server-hostname>
Port=443
HTTPPath=<http-path>
SSL=1
ThriftTransport=2
<confirmed-key>=<value>

DSN-less: append <confirmed-key>=<value>; to the connection string [5]. This is often the easiest route if your BI tool lets you supply a raw connection string.
Windows: the DSN setup dialog only exposes the common fields (host, port, mechanism, HTTP options, SSL options) [4]. For extra keys, the usual ODBC approach is to add a string value under the DSN's registry key (...\ODBC\ODBC.INI\<your DSN>), or to use a DSN-less string.

What I'd do about the intermittent 503

A 503 that shows up occasionally and then works on the next attempt is a transient server or gateway-side condition. The driver's error says it received a 503 with no Retry-After hint, so it has nothing to base an automatic retry on and surfaces the failure instead.

Retry in the application. For read queries from a BI tool, wrap SQLExecute in a retry with exponential backoff and jitter, two or three attempts. That works regardless of what the driver exposes. Only do this for idempotent statements.
Capture driver logs around the failure. Set LogLevel (1–6) and LogPath to a writable folder [3]. You'll get the exact timestamp and the HTTP exchange, which is what Support will ask for.
Check the warehouse state at that time. A connection to a stopped warehouse triggers an auto-start [6], and warehouses scaling or starting under load are common places to see transient errors. Compare the failure timestamps against the warehouse's events and monitoring, and against query history.
Consider serverless. Databricks recommends serverless SQL warehouses where available, since capacity is handled for you and there's no startup wait [6].
Open a support ticket if it keeps happening. Include the driver version (2.12.0), the warehouse ID, the failure timestamps, and the driver log. A 503 from the service side is something they can trace.

I'd go with the application-level retry plus logging now, and treat any driver-level retry key as a bonus only if Support or the driver guide confirms it.