topic Re: How can we do data copy from Databricks SQL using notebook? in Data Engineering

How can we do data copy from Databricks SQL using notebook?

Jayesh — Tue, 02 Aug 2022 10:19:22 GMT

Hi Team, we have a scenario where we have to connect to the DataBricks SQL instance 1 from another DataBricks instance 2 using notebook or Azure Data Factory. Can you please help?

Re: How can we do data copy from Databricks SQL using notebook?

User16764241763 — Tue, 02 Aug 2022 12:08:30 GMT

Hello @Jayesh Mehta

You can using Python connector from notebook

https://docs.databricks.com/dev-tools/python-sql-connector.html

Re: How can we do data copy from Databricks SQL using notebook?

Jayesh — Tue, 02 Aug 2022 12:59:20 GMT

Thanks a lot 🙂

Re: How can we do data copy from Databricks SQL using notebook?

Hubert-Dudek — Tue, 02 Aug 2022 13:32:44 GMT

You can also mount on both databricks workspaces the same storage folder and use COPY INTO for example,
In ADF you can use a databricks/delta connector to read from one delta and insert it into another,
If it is the one-time copy (like migration) we can copy using ADF just files between storage accounts, it is the fastest copy

Re: How can we do data copy from Databricks SQL using notebook?

artsheiko — Wed, 03 Aug 2022 17:01:09 GMT

Make sure that you need such an operation in principle : we have Clone use cases as well as Delta Sharing features that might be useful for you

Re: How can we do data copy from Databricks SQL using notebook?

Anonymous — Wed, 03 Aug 2022 18:07:58 GMT

Thanks for jumping in to help @Arvind Ravish @Hubert Dudek and @Artem Sheiko !