Databricks now deletes unused service principal secrets after 90 days

[LiresaFerizaj]

Here's a small change from this month's release notes that's easy to miss. Databricks now automatically deletes a service principal's OAuth client secret after 90 days without use. This matches how unused access tokens were already cleaned up.

Why this matters for data engineers

Most daily jobs won't notice, because their secrets are used all the time. The risk is in workloads that run rarely, for example:

• Quarterly or year-end reporting jobs
• Disaster recovery or backfill pipelines that only run when something goes wrong
• External tools (orchestrators, BI tools, ingestion services) connected once and then left idle
• Integrations for seasonal projects or audits

If one of these hasn't authenticated in over 90 days, its secret may be gone. The job then fails on authentication the next time it runs, often at the worst possible moment.

What to do

1. Take inventory of service principals. List the service principals in your account and note which ones back infrequent workloads.
2. Make sure each one has an owner. Every service principal should have a responsible team, so someone knows what to fix when it fails.
3. Plan for rotation instead of relying on long-lived secrets. Store secrets in a proper secret manager, and document how to create a new one and update the consumers that use it.
4. Add auth failures to your alerting. A clear alert for authentication errors turns a mysterious failure into a quick fix.
5. Know the recovery path. If a secret was deleted and is still needed, the fix is to create a new OAuth secret for that service principal and update wherever it's stored.

The bigger picture

Removing credentials nobody uses is good security practice: fewer old secrets lying around means less risk if one ever leaks. The trade-off is that "set it and forget it" integrations now need a bit of ownership. Treat it as a good reason to clean up.

How does your team track service principals for rarely-run jobs? I'd like to hear what's working.

[MadhupriyaDatEn]

Great point, Liresa. The 90-day cleanup policy highlights an important aspect of data engineering: authentication management for infrequently executed workloads can easily be overlooked.

I particularly agree with the importance of maintaining clear ownership, monitoring authentication failures, and having a documented credential-rotation and recovery process. Treating service principals as managed assets rather than one-time configurations can help prevent unexpected failures in critical reporting, DR, and backfill pipelines.

This is also a good reminder that security and reliability need to be considered together when designing data platforms.

[Khasim_1]

Great write-up, Liresa — this is a really important operational gotcha that's easy to overlook until it bites during a quarterly job or DR drill. A few thoughts to add to the discussion:

1. Automate tracking instead of manual lists Set up a scheduled check (using the Databricks API) that flags service principals nearing the 90-day idle mark — alert at day 60-75, so teams can rotate secrets before they get deleted, not after something breaks.
2. Tag service principals when you create them Add simple naming or tags showing the owner and purpose (e.g., sp-quarterly-report-finance). This makes it much easier to find and assign responsibility later, especially with many service principals.
3. Send "heartbeat" pings for critical rare-use integrations For DR/backfill pipelines that must stay ready but rarely run, schedule a simple monthly login (no real workload) just to keep the secret active — cheap insurance against failure during an actual emergency.
4. Automate secret rotation with Terraform If you use Terraform, script secret rotation to run on a schedule (e.g., quarterly) and auto-push the new secret to your vault. This removes reliance on someone remembering to do it manually.