Hello @Alex42 !

The error message indicates that access is forbidden due to an expired access token. This occurs when a notebook or job runs for an extended period, exceeding the default 48-hour threshold set for security reasons. The Databricks access token used by the MLflow Python client to communicate with the tracking server has a limited lifespan, typically expiring after 48 hours. If your ML tasks take longer than this to complete, the access token will expire, resulting in MLflow calls failing with a 403 Invalid access token error.

To resolve the issue, please follow these steps:

1. Create a Personal Access Token (PAT) by following the instructions outlined in this documentation: https://docs.databricks.com/en/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspa.... Please keep the current token’s default lifetime of 90 days.
2. Next, using the Databricks SDK, perform authentication at the beginning of your code by using the following commands:

from databricks.sdk import WorkspaceClient
import os

os.environ["DATABRICKS_TOKEN"] = "PAT-you-generated-in-step-one"
os.environ["DATABRICKS_HOST"] = "https://<YOUR_DATABRICKS_WORKSPACE_URL>;"

w = WorkspaceClient(
  host  = os.environ["DATABRICKS_HOST"],
  token = os.environ["DATABRICKS_TOKEN"]
)

Alternatively, you can store your PAT in a secret and perform this authentication in a more elegant and secure way, as described in steps 2 and 3 of this Knowledge Base article: https://kb.databricks.com/en_US/machine-learning/mlflow-invalid-access-token-error.

Best Regards,

Jéssica Santos