Hello,
I am getting acquainted with Databricks, of which I have signed up for a free account.
I have a Lambda function using Python 3.14 and the databricks-sql-connector library that runs on a schedule, to take data from a file in S3 and put the differential data into a workspace in Databricks. There are two errors that I am having:
1. RequestError: Error during request to server: BAD_REQUEST: Cannot create the resource, please try again later.
Here is a snippet:
from databricks import sql
conn = sql.connect(server_hostname=databricks_host,
http_path="/sql/1.0/warehouses/%s" % warehouse,
access_token=databricks_token,
)
This code works fine locally, but in AWS lambda seems to fail randomly without much explanation. I tried to consult with Genie, other LLMs, Google, etc. to figure out the issue. Genie recommended that I implement a retry loop, but I am not sure if that makes sense since subsequent invocations of the code also had the same error.
Instead, I implemented that Lambda makes a REST API call to Databricks before the "connection" takes place. It seemed to work for a bit, then I started getting the next error:
2. requests.exceptions.HTTPError: 400 Client Error: Bad Request for url:
Snippet:
response = requests.post(
"%s/api/2.0/sql/warehouses/%s/start" % (databricks_host,warehouse),
headers={
"Authorization": "Bearer %s" % databricks_token
}
)
response.raise_for_status()
AWS logs shows the URL which the request failed with, and that URL when consulting with Genie also seems fine. This code also works just fine locally as well.
I tried to search within this forum to see similar issues, but did not find any. If anyone could help it would be appreciated.
