I am running dbt on a databricks job. It saves all documentation: manifest.json, run_results.json, etc in "Download Artifacts" in a job. I am not able to find out a way to read those in codes, transform and save on databricks.
Tried job API. The artifacts API: api/2.1/jobs/runs/get-output returns a link to read or download these artifacts, but it requires task ID as an input. None of the APIs on databricks gives Task ID as output. There is no clear way for me to get these artifacts today.
Can someone pl help?
Recommendation for Databricks: Can we please create an option in the task to pass a path where the job can save artifacts?
