There is an usecase where I want to verify/extract the job caller identity in the job runtime of databricks. The job run_as and creator parameter seems to be fixed at the job configuration and doesn't reflect/update who actually triggered a job run.
Approaches tried till now:
- Notebook Context – dbutils.notebook.entry_point.getDbutils().notebook().getContext() - userName().get() → Returns the job owner identity (the user/SP that created the job), NOT the SP that called run-now.
- Jobs API – GET /api/2.1/jobs/runs/get and /api/2.2/jobs/runs/get creator_user_name → Returns the job owner/creator, not the run trigger. I verified this by triggering a job owned by SPN A using a different SPN B. The creator_user_name remained SPN A in both cases. No trigger_info, triggered_by, submitter, initiator, or similar field exists in the response. Result: Triggering SP identity is NOT available via the Jobs API.
- Audit tables from system.access.audit does return job caller details but there is a big delay(10mins+) for audit table to populate the job runs.
So I am thinking to pass the jwt in the job itself and then verify this jwt for the job caller identity with the valid signature. Any better way to actually identify who has triggered the databricks job? If passing jwt token is a no-go, only option left is a proxy service in between the databricks jobs and the clients just for this user identification usecase which is another system overhead.
