workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

[chitrar]

Hi, 

I see with unity catalog we have the workflow and now the lakeflow schema.   I guess the intention is to capture audit logs of changes/ monitor runs but I wonder why we don't have all the  metadata  info on the jobs /tasks too 

for a given job   => the relevant metadata like the job details, job param, notifications, schedule etc...... i see only very few attributes in the table ..why not get all  

for a given  task   => the relevant task name, source,type, notebook, params , dependent libraries..... i see only very few attributes in the table ..why not get all  

The jobs filter search -> why does it not allow for multiple search conditions as of now i can only enter a string and it will search  the name/tag/task  but i cant search on cluster/trigger etc ....

[Brahmareddy]

Hi Chitrar,

How are you doing today?, As per my understanding, It looks like Unity Catalog's LakeFlow schema is focused on capturing audit logs and monitoring runs, but it doesn't include all metadata related to jobs and tasks. For a job, key details like job parameters, notifications, and schedules seem to be limited in the available attributes. Similarly, for a task, information such as task name, source, type, notebook, parameters, and dependent libraries appears to be incomplete.

Regarding the jobs filter search, it currently allows searching by name, tag, or task but doesn't support multiple search conditions, such as filtering by cluster or trigger. Expanding the search functionality could make it more useful for detailed job tracking. It would be great if more metadata and advanced filtering options were available to provide a more comprehensive view.

Regards,

Brahma

[chitrar]

Hi,

Thanks for your reply and agreeing with my ask to have more metadata info in these tables.  Guess my question/ask needs to be directed to Databricks product team and not sure how to go about it. any ideas?

[Brahmareddy]

Hi Chitrar,

Thanks for letting me know.  please contact Sujitha/Rishabh by tagging them in your reply using @.

Regards,

Brahma

[Sujitha]

@Brahmareddy thank you! 
Hi @chitrar , Let me share this with our internal teams. Thank you so much for raising it! 

[chitrar]

@Sujitha  Thank you.  Given the great features of Databricks some of the basic functionalities that are needed for maintaining seems primitive 🙂 in an an env with multiple teams and many many jobs, with constant change, i should be able to get metadata into easily  --with these tables that have been made available with UC, it seems easily doable and hope your internal team thinks so too. 

At the minimum, the search ability should be updated to allow for multiple criteria --e.g with a given tag value, i want to see all the jobs using a specific cluster or not using a specific,  or search jobs with a particular schedule etc ....  

[chitrar]

@Sujitha   GM.. any update/feedback  from your internal team ?

[Sujitha]

Hi @chitrar 
You are right — the current search functionality only supports single-string search across name, tag, and task. It doesn’t allow multi-condition searches (like cluster or trigger) yet. I’ll make sure to pass this feedback along to the team for consideration.
For jobs, it’s true that only a limited set of attributes (like job details, params, notifications, schedule, etc.) are currently available in the system tables. Expanding the metadata coverage is something the team has been working on, especially with the introduction of Unity Catalog and Lakeflow.

For tasks, similarly, only a minimal set of attributes (like task name, type, source, params, etc.) are available.

[chitrar]

@Sujitha  so, we can expect these enhancements in the "near" future ?

[Sujitha]

Of course! The team is aware and will be making the necessary changes. Thanks for following up @chitrar