topic Re: How to know, who started a job run? in Data Engineering

How to know, who started a job run?

MrJava — Tue, 07 Feb 2023 16:59:29 GMT

Hi there!

We have different jobs/workflows configured in our Databricks workspace running on AWS and would like to know who actually started the job run? Are they started by a user or a service principle using curl?

Currently one can only see, who is the owner and who created the job config but not who triggerd an actual run.

I tried to find something in the audit logs we have activated for the jobs but was not successful.

Any hints are highly welcome!

Re: How to know, who started a job run?

Aviral-Bhardwaj — Wed, 08 Feb 2023 04:08:04 GMT

check the event logs

and driver logs you will find that

Re: How to know, who started a job run?

MrJava — Wed, 08 Feb 2023 06:07:25 GMT

Hi!

I checked the cluster logs and looked under "Event log". Under the Event type CREATING the listed user is not who actually started the job. The name that is listed there is always the user that is given under "Run as" in the job config.

Had a look into the driver logs but did not see any user information. Could you please be more specific on where to look there or give me a search pattern I can look for?

I would like to check who started the job, not under which user the cluster will run.

Re: How to know, who started a job run?

Aviral-Bhardwaj — Wed, 08 Feb 2023 06:20:54 GMT

lets connect https://www.linkedin.com/in/aviralb/ to discuss more

Re: How to know, who started a job run?

MrJava — Wed, 08 Feb 2023 06:46:58 GMT

Please keep the discussion on this platform so the whole community may participate.

Re: How to know, who started a job run?

Anonymous — Sun, 09 Apr 2023 06:52:16 GMT

Hi @Matthias Mueller

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!

Re: How to know, who started a job run?

MrJava — Tue, 11 Apr 2023 06:01:40 GMT

Nope, still no luck

Re: How to know, who started a job run?

AntonioAzure — Thu, 11 May 2023 13:30:40 GMT

Trying to find exactly the same information. No luck and no entries in search engines so far. Looking for a solution for this.

Re: How to know, who started a job run?

JeffP — Wed, 04 Oct 2023 03:04:24 GMT

Hi, were you able to find the information on who started a job run? I'm also searching for this information but was not able to find it. Thanks!

Re: How to know, who started a job run?

leonorgrosso — Tue, 16 Apr 2024 12:58:26 GMT

I've just posted this idea on the Idea Portal of Databricks regarding this subject. Upvote it so it may get developed!

https://feedback.azure.com/d365community/idea/5d0fdbbf-eefb-ee11-a73c-0022485313bb

Re: How to know, who started a job run?

Harsha777 — Wed, 10 Jul 2024 09:36:25 GMT

Hi,
Any update on this feature?
With the info that is currently available, we cannot track the user who initiated the job run. The job run shows whether its scheduled or manual run but doesn't provide info on the user who triggered the manual run. "RUN AS" info is completely different which can be read from the job configuration itself as it is not a dynamic attribute.

Re: How to know, who started a job run?

ahsan_aj — Mon, 09 Sep 2024 08:22:19 GMT

It looks like there is no update on this, this is such a useful piece of information that ideally should have been available from the start.

Re: How to know, who started a job run?

Gianfranco — Fri, 27 Sep 2024 09:15:04 GMT

It is possible to retrieve this information through the system tables here is an example

select *,user_identity.email
from system.access.audit
where event_date = 'xxx'
workspace_id = 'xxx'
and service_name = 'jobs' and request_params.job_id = 'xxx'

Re: How to know, who started a job run?

ahsan_aj — Mon, 30 Sep 2024 06:25:03 GMT

I added an additional filter to your query: action_name = "runNow" to retrieve the details. However, I believe the system tables only log events performed through the UI, and there are delays in when these logs appear. It would be beneficial to have this information available through an API for instant access.

Re: How to know, who started a job run?

hodb — Wed, 09 Oct 2024 12:29:46 GMT

for some reason the user_identity.email includes only "unknown" or "Sustem-User"
any ideas how to repair to include the name of the person that triggered the job?

Re: How to know, who started a job run?

mcveyroosevelt — Wed, 27 Nov 2024 06:42:00 GMT

To determine who started a job run in Databricks, you can use the Audit Logs feature by enabling workspace-level events and analyzing the runStarted events. Look for the userIdentity field within these logs, which identifies whether the run was triggered by a user or a service principal (e.g., using curl). Ensure that the audit log retention period and permissions are properly configured in AWS CloudTrail or your logging system to capture these details. If userIdentity is unclear, cross-check with Databricks job execution details or enable job tags for easier tracking.

Re: How to know, who started a job run?

Ayush_Arora — Thu, 09 Jan 2025 12:40:27 GMT

The system table solution works only when the job is manually triggered each time. I have a job which is triggered using the job scheduler on databricks. So once someone resumes the trigger, the job goes into execution. After this, the audit tables display null values as the 'run by' and 'run as'.
I am unable to know who changed the trigger from 'Paused' to 'Resume'. How can we find out who started the job?

Re: How to know, who started a job run?

jeremy98 — Tue, 22 Apr 2025 14:08:03 GMT

News on this feature?