Databricks job calling DBT - persist job name

[noorbasha534]

Hello all,

Is it possible to persist Databricks job name into the Brooklyn audit tables data model when when a Databricks job calls DBT model?

Currently, my colleagues persist audit information into fact & dimensional tables of the Brooklyn data model. This data model has job run id but not the job name. So, it has become challenging for to build a pipeline health dashboard whose subject is the Databricks job (not DBT model).....

Appreciate any guidance here...

[Yogesh_378691]

Yes, it’s possible to include the Databricks job name in your Brooklyn audit tables, but it won’t happen automatically. Right now, only the job run ID is being logged, so you’d need to extend your audit logic a bit. One common approach is to pass the Databricks job name as a parameter or environment variable when the job triggers the dbt model, and then capture that along with the run ID in your audit insert. That way, both the run ID and the job name are stored in the Brooklyn fact/dim tables, and you can easily build dashboards that focus on Databricks jobs instead of just dbt models.

[noorbasha534]

@Yogesh_378691 thanks. engineers need to remember to pass the job name as parameter, right? else, we have the risk of not seeing the job name in the audit tables. can you please elaborate little bit more on environment variables? is there a way to not rely on engineers declaring the job name...