topic Re: workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks in Data Engineering

workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

chitrar — Thu, 06 Mar 2025 16:02:41 GMT

Hi,

I see with unity catalog we have the workflow and now the lakeflow schema. I guess the intention is to capture audit logs of changes/ monitor runs but I wonder why we don't have all the metadata info on the jobs /tasks too

for a given job => the relevant metadata like the job details, job param, notifications, schedule etc...... i see only very few attributes in the table ..why not get all

for a given task => the relevant task name, source,type, notebook, params , dependent libraries..... i see only very few attributes in the table ..why not get all

The jobs filter search -> why does it not allow for multiple search conditions as of now i can only enter a string and it will search the name/tag/task but i cant search on cluster/trigger etc ....

Re: workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

Brahmareddy — Sat, 08 Mar 2025 22:01:16 GMT

Hi Chitrar,

How are you doing today?, As per my understanding, It looks like Unity Catalog's LakeFlow schema is focused on capturing audit logs and monitoring runs, but it doesn't include all metadata related to jobs and tasks. For a job, key details like job parameters, notifications, and schedules seem to be limited in the available attributes. Similarly, for a task, information such as task name, source, type, notebook, parameters, and dependent libraries appears to be incomplete.

Regarding the jobs filter search, it currently allows searching by name, tag, or task but doesn't support multiple search conditions, such as filtering by cluster or trigger. Expanding the search functionality could make it more useful for detailed job tracking. It would be great if more metadata and advanced filtering options were available to provide a more comprehensive view.

Regards,

Brahma

Re: workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

chitrar — Tue, 11 Mar 2025 13:19:09 GMT

Hi,

Thanks for your reply and agreeing with my ask to have more metadata info in these tables. Guess my question/ask needs to be directed to Databricks product team and not sure how to go about it. any ideas?

Re: workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

Brahmareddy — Tue, 11 Mar 2025 13:28:49 GMT

Hi Chitrar,

Thanks for letting me know. please contact Sujitha/Rishabh by tagging them in your reply using @.

Regards,

Brahma

Re: workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

Sujitha — Tue, 11 Mar 2025 13:31:49 GMT

@Brahmareddy thank you!
Hi @chitrar , Let me share this with our internal teams. Thank you so much for raising it!

Re: workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

chitrar — Tue, 11 Mar 2025 13:50:12 GMT

@Sujitha Thank you. Given the great features of Databricks some of the basic functionalities that are needed for maintaining seems primitive 🙂 in an an env with multiple teams and many many jobs, with constant change, i should be able to get metadata into easily --with these tables that have been made available with UC, it seems easily doable and hope your internal team thinks so too.

At the minimum, the search ability should be updated to allow for multiple criteria --e.g with a given tag value, i want to see all the jobs using a specific cluster or not using a specific, or search jobs with a particular schedule etc ....

Re: workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

chitrar — Fri, 14 Mar 2025 13:18:30 GMT

@Sujitha GM.. any update/feedback from your internal team ?

Re: workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

Sujitha — Fri, 14 Mar 2025 13:44:09 GMT

Hi @chitrar
You are right — the current search functionality only supports single-string search across name, tag, and task. It doesn’t allow multi-condition searches (like cluster or trigger) yet. I’ll make sure to pass this feedback along to the team for consideration.
For jobs, it’s true that only a limited set of attributes (like job details, params, notifications, schedule, etc.) are currently available in the system tables. Expanding the metadata coverage is something the team has been working on, especially with the introduction of Unity Catalog and Lakeflow.

For tasks, similarly, only a minimal set of attributes (like task name, type, source, params, etc.) are available.

Re: workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

chitrar — Fri, 14 Mar 2025 13:46:06 GMT

@Sujitha so, we can expect these enhancements in the "near" future ?

Re: workflow/lakeflow -why does it not capture all the metadata of the jobs/tasks

Sujitha — Fri, 14 Mar 2025 13:49:03 GMT

Of course! The team is aware and will be making the necessary changes. Thanks for following up @chitrar