Migrate a notebook that reside in workspace using Databricks Asset Bundle

[niruban]

Hello Community Folks -

Did anyone implemented migration of notebooks that is in workspace to production databricks workspace using Databricks Asset Bundle? If so can you please help me with any documentation which I can refer? Thanks!!

Regards

Niruban M

[Kaniz]

Hi @niruban, Migrating notebooks from one Databricks workspace to another using Databricks Asset Bundles is a useful approach.

Let me guide you through the process and provide relevant documentation.

1. Databricks Asset Bundles Overview:

   • Databricks Asset Bundles (or simply “bundles”) allow you to programmatically validate, deploy, and run Databricks resources such as jobs. They can also manage Delta Live Tables pipelines and work with MLOps Stacks¹.
   • Bundles are particularly helpful when you want to move existing jobs from one workspace to another.

2. Requirements:

   • Ensure you have the Databricks CLI installed (version 0.218 or above). You can check your installed version by running databricks -v. If needed, install or update the Databricks CLI.
   • The remote workspace must have workspace files enabled¹.

3. Decision: Create the Bundle:

   • Decide whether you want to create the bundle using a template or manually:
     • Create the bundle by using a template:
       • Use the Databricks default bundle template for Python, which pairs a notebook or Python code with the definition of a job to run it.
       • Set up authentication between the Databricks CLI on your development machine and your Databricks workspace.
       • Deploy the bundle using the databricks bundle deploy command.
       • Validate, deploy, and run the job within your Databricks workspace.
     • Create the bundle manually:
       • Create a bundle configuration file that specifies the job definition, notebooks, and other sources.
       • Add an existing job definition to the bundle.
       • Configure the job to use serverless compute.
       • Deploy the bundle using the Databricks CLI¹.

4. Additional Tips:

   • For better compatibility with bundles, ensure that your notebooks use the IPython notebook format (.ipynb). You can export an existing notebook from a Databricks workspace into this format¹.
   • Consider putting your notebooks in Git and reusing Git in the new workspace as part of the CI/CD process. You can also use Terraform exporter with the -match flag to generate .tf scripts for re-deploying jo...².

5. Documentation:

   • To get started, refer to the official Databricks documentation on creating custom Databricks Asset B...³.
   • Additionally, explore the detailed guide on developing a job using Databricks Asset Bundles¹.

I hope this helps you successfully migrate your notebooks to the production Databricks workspace! If you have any further questions, feel free to ask. 😊