Generate and export dbt documentation from the Workflow dbt task to S3

[Anonymous]

I'm testing the Databricks Jobs feature with a dbt task and wanted to know if you had any advice for me for managing dbt documentation.

I can use "dbt run" commands to run my models then "dbt docs generate" to generate the documentation. But is it possible to export the generated files to GitHub or to a File System like AWS S3 ?

[Kaniz]

Hi @Simon Auger​, Did you get a chance to look at this doc? To learn more about dbt Core, see the dbt documentation.

[Anonymous]

Hi,

Thanks for your answer. Actually, I used this documentation for generating my Databricks jobs but there is no mention about how to manage the dbt generated documentation. I do not know if that feature is already implemented by Databricks or if there is a work around.

[Jfoxyyc]

 Hi @Kaniz Fatma,

The documentation mentions:

• Automatic archiving of the artifacts from job runs, including logs, results, manifests, and configuration.

When a dbt task runs, do the logs, manifests and index.html automatically go back to the attached repo?

Is there a way to run slim ci with the dbt task? Can we use pre-commit? It would be good to be able to inspect manifest, capture models that have changed, shallow clone them, test their transforms, and if succeed, run those to prod.​

[Jfoxyyc]

You can use the jobs api and hit the jobs/runs/get-output/ and look at the dbt_output.artifacts_link to get an http link to download a tar.gz file that has all the artifacts in it. You can then unpack the tar.gz and store those files in adls or s3.