Is it safe to expose JWT in Databricks Job?

[ayushpattnaik]

There is an usecase where I want to verify/extract the job caller identity in the job runtime of databricks. The job run_as and creator parameter seems to be fixed at the job configuration and doesn't reflect/update who actually triggered a job run.

Approaches tried till now: 

1. Notebook Context – dbutils.notebook.entry_point.getDbutils().notebook().getContext() - userName().get() → Returns the job owner identity (the user/SP that created the job), NOT the SP that called run-now.
2. Jobs API – GET /api/2.1/jobs/runs/get and /api/2.2/jobs/runs/get creator_user_name → Returns the job owner/creator, not the run trigger. I verified this by triggering a job owned by SPN A using a different SPN B. The creator_user_name remained SPN A in both cases. No trigger_info, triggered_by, submitter, initiator, or similar field exists in the response. Result: Triggering SP identity is NOT available via the Jobs API.
3. Audit tables from system.access.audit does return job caller details but there is a big delay(10mins+) for audit table to populate the job runs.

So I am thinking to pass the jwt in the job itself and then  verify this jwt for the job caller identity with the valid signature. Any better way to actually identify who has triggered the databricks job? If passing  jwt token is a no-go, only option left is a proxy service in between the databricks jobs and the clients just for this user identification usecase which is another system overhead.

[amirabedhiafi]

Hello !

What I can say is that your conclusion is correct since DBKS does not currently expose the identity of the principal that called run-now to the running job through the Jobs API or notebook context and the audit event is the authoritative source but it is not designed for synchronous runtime identity resolution.

I would not pass the Databricks/Entra OAuth JWT access token itself as a job parameter.

DBKS explicitly shows resolved parameters on the job run details page. https://docs.databricks.com/aws/en/jobs/dynamic-value-references more importantly the runNow audit event includes job_parameters, notebook_params, python_params... in its request params and you can check this doc link to better understand the situation https://docs.databricks.com/aws/en/admin/account-settings/audit-logs)

which means something like:

{
  "job_id": 123,
  "job_parameters": {
    "jwt": "eyJhbGciOi..."
  }
}

can potentially make a bearer credential persist in places where you don't want credentials to exist like in job run metadata, audit records, logs or parameter inspection.

So what you need to do is simply treat tokens as secrets and avoid putting credentials directly into notebooks or jobs or logging them.

If that JWT is the token SPN B used to call:

POST /api/2.2/jobs/run-now
Authorization: Bearer <token>

then anybody obtaining it before expiration could potentially replay it with the permissions of SPN B.

Interestingly, what I really find amazing is that does know the caller since the jobs audit event represents an on demand invocation and every audit record has:

user_identity

which DBKS defines as the identity of the user initiating the request. 

The problem is that DBKS does not propagate that audit user_identity into the job runtime and the documented dynamic values provide job.id, job.run_id, job.trigger.type, timestamps... but no triggered_by/caller identity https://docs.databricks.com/aws/en/jobs/dynamic-value-references

and unfortunately the system tables are explicitly not intended for realtime monitoring because in DBKS recent events may not appear immediately and Azure diagnostic log delivery isn't a reliable workaround either (auditable events typically arrive within about 15 minutes)

What I would recommend is if the caller identity is required inside the job immediately you can use a purpose built signed identity assertion rather than forwarding the OAuth access token.

for example SPN B (or preferably a trusted issuer) sends:

{
  "caller_assertion": "<signed-JWT>",
  "request_id": "8d6835..."
}

with claims such as:

{
  "sub": "SPN-B-application-id",
  "aud": "databricks-job-123",
  "iat": 1786352400,
  "exp": 1786352700,
  "jti": "8d6835..."
}

the job verifies:

signature
aud == expected job
exp < = 5 minutes
jti not previously used
issuer is trusted

so a JWT as a format is not inherently the problem. Passing a bearer/access JWT is.

For a prod security sensitive use case, I would go for :

SPN B
  │
  │ authenticated request
  ▼
thin trigger service or DBKS app
  │
  ├── validates caller = SPN B
  │
  ├── generates request_id
  │
  ├── creates short-lived signed assertion
  │
  ▼
DBKS run-now
  │
  │ caller_assertion + request_id
  ▼
Job running as SPN A
  │
  ├── verifies assertion
  └── knows caller = SPN B

this proxy does not need to be a large additional system since it can be a very thin authenticated endpoint whose only responsibility is identity propagation. A DBKS app is also worth considering because DBKS apps support OAuth for users or SP and explicitly recommend recording the original identity when performing actions on behalf of callers so if immediate caller identification is not required for authorization and is needed only for auditing or reporting, I would avoid all of this complexity and continue using:

SELECT
    event_time,
    user_identity,
    request_params['job_id'] AS job_id
FROM system.access.audit
WHERE service_name = 'jobs'
  AND action_name = 'runNow'

the audit table is actually the correct source of truth user_identity is the initiating identity while run_as is the execution identity.