topic Re: How can I backup my Databricks instance? in Data Engineering

How can I backup my Databricks instance?

ChristianWuerdi — Tue, 21 Sep 2021 22:30:10 GMT

We have a Databricks instance on Azure that has somewhat organically grow with dozens of users and hundreds of notebooks. How do I conveniently backup this env so in case disaster strikes the notebooks aren't lost? The data itself is backed by Azure storage accounts so that's already taken care of.

Re: How can I backup my Databricks instance?

-werners- — Wed, 22 Sep 2021 08:35:28 GMT

I use the Databricks CLI for that and export the entire workspace on a regular basis.

Check it out, it has tons of handy functions.

https://docs.microsoft.com/en-us/azure/databricks/dev-tools/cli/

Re: How can I backup my Databricks instance?

ChristianWuerdi — Wed, 22 Sep 2021 23:59:21 GMT

Ok thanks, bit uncomfortable to rely on a tool that's labelled experimental to do backups but I guess it's the best option at the moment. You wouldn't happen to have some handy command line example to backup a whole workspace?

Re: How can I backup my Databricks instance?

-werners- — Thu, 23 Sep 2021 07:13:17 GMT

Totally understandable.

But why don't you use the new Repos functionality? Like that you always have your code in git.

(the cli command is:

databricks workspace export_dir SOURCE_PATH TARGET PATH

Sourcepath is "/" for the whole workspace.)

But Repos is a way better alternative, no idea why it did not pop into my head yesterday.

Re: How can I backup my Databricks instance?

ChristianWuerdi — Wed, 18 May 2022 22:28:19 GMT

@Kaniz Fatma All good thanks, combination of CLI + gradually migrating everything to git is a viable solution