Issues making requests to Databricks from AWS Lambda

Sunioj — Fri, 13 Nov 2026 13:36:22 GMT

Hello,
I am getting acquainted with Databricks, of which I have signed up for a free account.
I have a Lambda function using Python 3.14 and the databricks-sql-connector library that runs on a schedule, to take data from a file in S3 and put the differential data into a workspace in Databricks. There are two errors that I am having:

1. RequestError: Error during request to server: BAD_REQUEST: Cannot create the resource, please try again later.

Here is a snippet:

from databricks import sql conn = sql.connect(server_hostname=databricks_host, http_path="/sql/1.0/warehouses/%s" % warehouse, access_token=databricks_token, )

This code works fine locally, but in AWS lambda seems to fail randomly without much explanation. I tried to consult with Genie, other LLMs, Google, etc. to figure out the issue. Genie recommended that I implement a retry loop, but I am not sure if that makes sense since subsequent invocations of the code also had the same error.

Instead, I implemented that Lambda makes a REST API call to Databricks before the "connection" takes place. It seemed to work for a bit, then I started getting the next error:

2. requests.exceptions.HTTPError: 400 Client Error: Bad Request for url:
Snippet:

response = requests.post( "%s/api/2.0/sql/warehouses/%s/start" % (databricks_host,warehouse), headers={ "Authorization": "Bearer %s" % databricks_token } ) response.raise_for_status()

AWS logs shows the URL which the request failed with, and that URL when consulting with Genie also seems fine. This code also works just fine locally as well.

I tried to search within this forum to see similar issues, but did not find any. If anyone could help it would be appreciated.

topic Issues making requests to Databricks from AWS Lambda in Databricks Free Edition Help

Issues making requests to Databricks from AWS Lambda