" Token is expiring within 30 seconds." when running a job using Databricks SDK

dzmitry_tt — Tue, 02 Jan 2024 14:05:59 GMT

During attempt (in Azure DevOps environment) of running a job (and getting the result of the run) using Databricks SDK for Python, I've got this error:

databricks.sdk.core.DatabricksError: Token is expiring within 30 seconds. (token expired).

The stack trace is:

run_result = w.jobs.run_now(

File "/azp/agent/_work/r1/a/_cwp-dp-core/Databricks/s/venv/lib/python3.10/site-packages/databricks/sdk/service/_internal.py", line 45, in result return self._waiter(callback=callback, timeout=timeout, **kwargs)

File "/azp/agent/_work/r1/a/_cwp-dp-core/Databricks/s/venv/lib/python3.10/site-packages/databricks/sdk/service/jobs.py", line 2720, in wait_get_run_job_terminated_or_skipped poll = self.get_run(run_id=run_id)

File "/azp/agent/_work/r1/a/_cwp-dp-core/Databricks/s/venv/lib/python3.10/site-packages/databricks/sdk/service/jobs.py", line 3013, in get_run json = self._api.do('GET', '/api/2.1/jobs/runs/get', query=query)

File "/azp/agent/_work/r1/a/_cwp-dp-core/Databricks/s/venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 999, in do raise self._make_nicer_error(message=message) from None

The function used to run a job is databricks.sdk.service.jobs.run_now().result(), the 'timeout' parameter there is set to 'datetime.timedelta(days=1)'.

Also, I could not reproduce that in my local environment, I've got result of the job execution longer than 1 hour, without getting errors.

topic " Token is expiring within 30 seconds." when running a job using Databricks SDK in Data Engineering

" Token is expiring within 30 seconds." when running a job using Databricks SDK