Hi
@Punit_Prajapati,
This is a common architectural pattern for IoT scenarios, and there are a few approaches to consider depending on exactly how your IoT devices are calling the Databricks App.
UNDERSTANDING THE ARCHITECTURE
First, it helps to clarify the two distinct authentication layers in Databricks Apps:
1. Inbound authentication (IoT device to your App): How external clients reach your FastAPI endpoints.
2. Outbound authentication (your App to Databricks APIs): How your app calls Databricks services like SQL warehouses or Unity Catalog.
For layer 2, Databricks Apps automatically inject DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET environment variables for the app's own service principal. The Databricks SDK handles token refresh transparently when you use it inside the app, so that part should not require manual token management.
The challenge you are describing is layer 1: how your IoT devices authenticate when calling your FastAPI app.
OPTION 1: USE A PERSONAL ACCESS TOKEN (PAT) FOR THE SERVICE PRINCIPAL
PATs can have a much longer lifetime than OAuth access tokens. The default maximum is 730 days (two years), and workspace admins can configure the max lifetime.
To create a PAT for a service principal:
Step 1: A workspace admin creates an initial PAT on behalf of the service principal using the Databricks CLI:
databricks token-management create-obo-token APPLICATION_ID --lifetime-seconds 63072000 -p PROFILE_NAME
(63072000 seconds = 730 days)
Step 2: The service principal can then create additional tokens itself:
databricks tokens create --lifetime-seconds 63072000 -p PROFILE_NAME
Your IoT devices would then include this PAT in their HTTP requests to your FastAPI app, for example in an Authorization header. Your FastAPI app validates the token by calling the Databricks token introspection API or by maintaining its own validation logic.
Reference:
https://docs.databricks.com/aws/en/dev-tools/auth/pat.htmlNote on OBO token UI: The "create-obo-token" functionality is available through the CLI and REST API (Token Management API), not through a workspace UI toggle. There is no separate UI setting to "enable" OBO tokens. You run the CLI command or call the API directly as a workspace admin.
OPTION 2: IMPLEMENT CLIENT-SIDE OAUTH TOKEN REFRESH
If you prefer to stick with the standard OAuth M2M flow, the IoT devices can hold the service principal's client_id and client_secret and request a fresh access token before each API call (or when the current token is close to expiring). The token endpoint is:
POST
https://YOUR_WORKSPACE.cloud.databricks.com/oidc/v1/token Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials&client_id=CLIENT_ID&client_secret=CLIENT_SECRET&scope=all-apis
Each token is valid for 1 hour. On the IoT device, you would implement a simple refresh loop or check token expiry before each request and re-fetch if needed. Many HTTP client libraries support this pattern natively (e.g., Python's requests-oauthlib or httpx with an auth handler).
This is the most secure approach because credentials are not stored as long-lived bearer tokens, but it does require the IoT devices to have outbound HTTPS connectivity to the workspace token endpoint.
OPTION 3: INTERMEDIARY TOKEN GATEWAY
If your IoT devices are too constrained to handle OAuth flows directly, consider placing a lightweight token gateway (for example, an AWS Lambda, Azure Function, or a small API service) between the devices and your Databricks App. The gateway:
1. Accepts requests from IoT devices authenticated with a long-lived API key or device certificate that you manage outside of Databricks.
2. Exchanges that credential for a short-lived Databricks OAuth token using the service principal's client credentials.
3. Forwards the request to your Databricks App with the fresh token.
This keeps the Databricks credential lifecycle separate from the device authentication lifecycle.
OPTION 4: APP-LEVEL CUSTOM AUTHENTICATION
Since your FastAPI app is the endpoint receiving IoT traffic, you can implement your own authentication layer within the app itself. For example:
- Generate your own long-lived API keys and store them in Databricks Secrets.
- Your FastAPI middleware validates incoming requests against those keys.
- Once validated, the app uses its own automatically managed service principal credentials (via the Databricks SDK) to interact with Databricks resources.
This decouples IoT device authentication from Databricks OAuth entirely. Your app handles device auth, and the Databricks SDK handles Databricks auth.
from databricks.sdk import WorkspaceClient
from fastapi import FastAPI, Depends, HTTPException, Header
app = FastAPI()
w = WorkspaceClient() # Automatically uses app service principal credentials
async def verify_device_key(x_api_key: str = Header(...)):
# Validate against keys stored in Databricks Secrets or a config
valid_keys = w.secrets.get_secret("iot-scope", "valid-api-keys")
if x_api_key not in valid_keys.value.split(","):
raise HTTPException(status_code=401, detail="Invalid API key")
@app.post("/ingest")
async def ingest_data(data: dict, _=Depends(verify_device_key)):
# Process IoT data using the app's service principal
# The SDK handles token refresh automatically
w.statement_execution.execute_statement(...)
RECOMMENDATION
For IoT use cases, Option 4 (custom app-level auth) or Option 1 (PATs) tend to be the most practical. Option 4 gives you full control over the device credential lifecycle without depending on Databricks token expiry at all. Option 1 gives you up to 2-year token validity with minimal code changes.
Key documentation references:
- Databricks Apps authorization:
https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth- OAuth M2M authentication:
https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m.html- Personal access tokens:
https://docs.databricks.com/aws/en/dev-tools/auth/pat.html- Token management for service principals:
https://docs.databricks.com/aws/en/admin/access-control/tokens.html* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.
If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.
