Connect Databricks hosted on Azure, with RDS on AWS.

danial
New Contributor II

‎02-21-2023 04:30 AM

We have Databricks set up and running on Azure. Now we want to connect it with RDS (AWS) to transfer data from RDS to Azure DataLake using the Databricks.

I could find the documentation on how to do it within the same cloud (Either AWS or Azure) but not cross cloud.

So was wondering what would be the best approach. Also, the RDS has additional layer of basic http authentication for additional security.

Highly appreciate the help.

Labels:

Anonymous
Not applicable

‎03-31-2023 08:48 AM

@Danial Malik​ :

To connect Databricks running on Azure with RDS running on AWS, you can use JDBC or ODBC connectivity.

Here are the general steps:

  1. Set up a Virtual Private Network (VPN) or a peering connection between your Azure and AWS environments to ensure secure communication between them.
  2. Install the JDBC or ODBC driver for RDS on the Databricks cluster. You can find the driver download and installation instructions in the AWS RDS documentation.
  3. Configure the JDBC or ODBC connection to connect to the RDS database using the appropriate driver, database URL, username, and password. You may also need to configure additional parameters, such as SSL settings, depending on your RDS configuration.
  4. Write Spark code in Databricks to read data from RDS using the JDBC or ODBC connection, and then write the data to Azure DataLake.

Regarding the additional layer of basic http authentication on RDS, you will need to provide the appropriate credentials in the JDBC or ODBC connection configuration.

Keep in mind that cross-cloud data transfer can have performance and security implications, so you may want to consider optimizing your network and data transfer protocols to ensure efficient and secure data transfer.

Hope this helps!

0 Kudos

Subratpanigrahi
New Contributor II
In response to Anonymous

‎05-15-2024 05:32 AM - edited ‎05-15-2024 05:38 AM

Hello,

Can you please elaborate the step-1 with detailed information about how to set up VPN or Peering connection between Azure and AWS for secure connection. We can't use DELTA SHARE currently. 

I have a similar requirement to share data between cross- cloud.

Thanks

 

 

 

0 Kudos

Anonymous
Not applicable

‎04-21-2023 11:45 PM

Hi @Danial Malik​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

0 Kudos