topic Re: Customize job run name when running jobs from adf in Data Engineering

Customize job run name when running jobs from adf

SG — Thu, 20 Jul 2023 20:01:53 GMT

Hi guys,

i am running my Databricks jobs on a cluster job from azure datafactory using a databricks Python activity

When I monitor my jobs in workflow-> job runs . I see that the run name is a concatenation of adf pipeline name , Databricks python activity name and a run id. I am wondering how can Customize this job run name to make it easier to select the a specific job ?

is there a way to do it in spark session or spark context properties ?

regards

hamza

Re: Customize job run name when running jobs from adf

databricksdev — Wed, 17 Apr 2024 13:44:13 GMT

Hi Hamza,

did u got any solution for this issue

Thanks

Re: Customize job run name when running jobs from adf

SG — Tue, 23 Apr 2024 13:58:18 GMT

Hi @databricksdev ,

unfortunatly, no one was able to help.

thanks

Re: Customize job run name when running jobs from adf

AmanSehgal — Tue, 23 Apr 2024 15:28:28 GMT

I don't think that level of customisation is provided. However, I can suggest some workarounds:

REST API: Create a job on the fly with desired name within ADF and trigger it using REST API in Web activity. This way you can track job completion status and decide what action to take if the job run was success or failure.
File trigger: Drop a file at a location using ADF, and set file arrival trigger in Databricks job. This will ensure that you're running the same job in Databricks.

Hope this helps.