cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

getting job_parameters object with sql

yairofek
New Contributor

Hey,

In order to create more meaningful monitoring or usage or few platformic jobs I am using I need to be able to access the job_parameters object of jon runs.
While job_parameters exists in system.workflow.job_run_timeline table, it is not populated for most job runs, even if they have parameters.
The data is saved somewhere as it does exist in the UI.
Can you please advise how to get the full object, preferably is sql, so I could complete my dashboards?

2 REPLIES 2

michelle653burk
New Contributor III

 


@yairofek wrote:

Hey,

In order to create more meaningful monitoring or usage or few platformic jobs I am using I need to be able to access the job_parameters object of jon runs.
While job_parameters exists in system.workflow.job_run_timeline table, it is not populated for most job runs, even if they have parameters.
The data is saved somewhere as it does exist in the UI.
Can you please advise how to get the full object, preferably is sql, so I could complete my dashboards?


Hello!

To access the full job_parameters object for job runs in the system.workflow.job_run_timeline table, you can use SQL queries to join this table with other relevant tables that might contain the missing data. Identify related tables that store job parameters or related information, then use SQL JOIN operations to combine data from these tables and extract the necessary fields to reconstruct the full job_parameters object. For example, you can join the job_run_timeline table with the job_parameters table on the job_run_id field and filter for a specific job ID. This approach helps in completing your dashboards with the required information.


@michelle653burk wrote:

 


@yairofek wrote:

Hey,

In order to create more meaningful monitoring or usage or few platformic jobs I am using I need to be able to access the job_parameters object of jon runs.
While job_parameters exists in system.workflow.job_run_timeline table, it is not populated for most job runs, even if they have parameters.
The data is saved somewhere mygreatlakes as it does exist in the UI.
Can you please advise how to get the full object, preferably is sql, so I could complete my dashboards?


Hello!

To access the full job_parameters object for job runs in the system.workflow.job_run_timeline table, you can use SQL queries to join this table with other relevant tables that might contain the missing data. Identify related tables that store job parameters or related information, then use SQL JOIN operations to combine data from these tables and extract the necessary fields to reconstruct the full job_parameters object. For example, you can join the job_run_timeline table with the job_parameters table on the job_run_id field and filter for a specific job ID. This approach helps in completing your dashboards with the required information.


Was that helpful to you?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group