Accessing DBT Articafts, runs, tests from Databricks workflow using automated script.

data-enthu — Mon, 19 Aug 2024 15:58:06 GMT

I am running dbt on a databricks job. It saves all documentation: manifest.json, run_results.json, etc in "Download Artifacts" in a job. I am not able to find out a way to read those in codes, transform and save on databricks.

Tried job API. The artifacts API: api/2.1/jobs/runs/get-output returns a link to read or download these artifacts, but it requires task ID as an input. None of the APIs on databricks gives Task ID as output. There is no clear way for me to get these artifacts today.

Can someone pl help?

Recommendation for Databricks: Can we please create an option in the task to pass a path where the job can save artifacts?

Re: Accessing DBT Articafts, runs, tests from Databricks workflow using automated script.

rokata — Wed, 02 Jul 2025 08:12:42 GMT

I know it is a late thread, but did you solve this?
I am running into the same challange you have.

It seems you can use get/job, and expand the field "tasks" to get the task ids. and then you can use the get-output from that task id. If that helps
https://docs.databricks.com/api/workspace/jobs/getrun

topic Accessing DBT Articafts, runs, tests from Databricks workflow using automated script. in Data Engineering

Accessing DBT Articafts, runs, tests from Databricks workflow using automated script.

Re: Accessing DBT Articafts, runs, tests from Databricks workflow using automated script.