cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Create A data Pipeline between 2 databricks Instance one using unity catalog the Other Databrick not

Pratikmsbsvm
Contributor

I have 2 Databricks Instance. Databricks A and Databricks B.

The Application Hightouch is consuming Data from Databricks B through Unity Catalog.

I have to create a Data pipeline to push data from Databricks A to B Without using Delta sharing.

Diagram:

Pratikmsbsvm_1-1754562995401.png

 

Do I need to first store data in ADLS Gen2 of B than with Databricks add data to unity catalog. Is It Correct.

Or Do we have any better option.

Please share your thoughts.

Regards

 

1 ACCEPTED SOLUTION

Accepted Solutions

Stefan-Koch
Valued Contributor II

Hi @Pratikmsbsvm 

You could use Lakehouse Federation. Attach 1 instance to another instance as a connection. 

StefanKoch_0-1754627581179.png

https://docs.databricks.com/aws/en/query-federation/databricks

View solution in original post

1 REPLY 1

Stefan-Koch
Valued Contributor II

Hi @Pratikmsbsvm 

You could use Lakehouse Federation. Attach 1 instance to another instance as a connection. 

StefanKoch_0-1754627581179.png

https://docs.databricks.com/aws/en/query-federation/databricks

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now