topic Re: Asset Bundles from Workspace for CI/CD in Data Engineering

Asset Bundles from Workspace for CI/CD

AxelM — Tue, 01 Oct 2024 10:22:14 GMT

Hello there,

I am exploring the possibilities for CI/CD from a DEV-Workspace to PROD. Besides the Notebooks (which can easily be handled by the GIT-provider) I am mainly interested in the Deployment o Jobs/Clusters/DDl...

I can nowhere find a tutorial how to use the existing Workspace and combine it into a bundle will all these assets. How can this be achieved?
My goal would be to still develop on the workspace > Wrap it all in a bundle (using VSC-Extension) and then deploy it on the prod-workspace.

Thanks for your help

Axel

Re: Asset Bundles from Workspace for CI/CD

datastones — Tue, 01 Oct 2024 12:54:15 GMT

i think the dab mlop stack template is pretty helpful re: how to bundle, schedule and trigger custom jobs

https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.html

you can bundle init it locally and it should give you the skeleton of how to bundle everything together.

re: ci/cd as long as you generate the access token from your databricks workspace and save it as an env secrets on github, you should be able to trigger a github actions of your dab.

hope this helps!