Hey everyone,
Have you ever opened Databricks Catalog Explorer to audit a table, only to find the downstream job listed as "Untitled"? Databricks Unity Catalog is incredibly powerful for automated lineage, but it quietly breaks the moment you orchestrate pipelines using dbutils.notebook.run(). Because the platform executes child notebooks in an isolated, temporary "instance run" rather than the parent job run context, the link back to your scheduled workflow completely vanishes.
If this silent lineage gap is making production debugging a headache for your team, I just published a short write-up on how to easily fix it. By capturing the parent Job ID and passing it down into Deltaโs userMetadata configuration, you can embed a permanent audit trail directly into your table transaction logs.
Check out the quick step-by-step implementation and code snippets over on Medium: The Hidden Lineage in Databricks: The Mystery Behind โUntitledโ Lineage Entries.
