cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Customize job run name when running jobs from adf

SG
New Contributor II

Hi guys,

 

i am running my Databricks jobs on a cluster job from azure datafactory using a databricks Python activity 

When I monitor my jobs in workflow-> job runs . I see that the run name is a concatenation of adf pipeline name , Databricks python activity name and a run id. I am wondering how can Customize this job run name to make it easier to select the a specific job ?

is there a way to do it in spark session or spark context properties ?

 

regards

hamza

3 REPLIES 3

databricksdev
New Contributor II

Hi Hamza,

did u got any solution for this issue

Thanks

 

SG
New Contributor II

Hi @databricksdev ,

unfortunatly, no one was able to help.

thanks

AmanSehgal
Honored Contributor III

I don't think that level of customisation is provided. However, I can suggest some workarounds:

  • REST API: Create a job on the fly with desired name within ADF and trigger it using REST API in Web activity. This way you can track job completion status and decide what action to take if the job run was success or failure.
  • File trigger: Drop a file at a location using ADF, and set file arrival trigger in Databricks job. This will ensure that you're running the same job in Databricks.

Hope this helps.

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group