topic Re: Reading data in Azure Databricks Delta Lake from AWS Redshift in Data Engineering

Reading data in Azure Databricks Delta Lake from AWS Redshift

playermanny2 — Tue, 25 Apr 2023 19:51:23 GMT

We have Databricks set up and running on Azure. Now we want to connect it with Redshift (AWS) to perform further downstream analysis for our redshift users.

I could find the documentation on how to do it within the same cloud (Either AWS or Azure) but not cross cloud.

So was wondering what would be the best approach to allow redshift to read the delta lake hosted in azure. I was hoping some sort of glue catalog could be set-up and that could allow reading from redshift as an external table

Highly appreciate the help.

Re: Reading data in Azure Databricks Delta Lake from AWS Redshift

Anonymous — Thu, 27 Apr 2023 04:52:21 GMT

@Manny Cato :

To allow Redshift to read data from Delta Lake hosted on Azure, you can use AWS Glue Data Catalog as an intermediary. The Glue Data Catalog is a fully managed metadata catalog that integrates with a variety of data sources, including Delta Lake and Redshift, to enable cross-cloud data integration.

Here are the high-level steps you can follow to set up this integration:

Create an AWS Glue Data Catalog in your AWS account. This will serve as the metadata repository for your data.
Set up a Glue Crawler to discover the schema and metadata for your Delta Lake table(s) hosted on Azure.
Configure a Glue ETL job to extract the data from your Delta Lake table(s) and load it into a Redshift cluster.
Define an external schema in Redshift that points to the Glue Data Catalog.
Create external tables in Redshift that reference the data in the Glue Data Catalog.
Query the data in Redshift as needed.

Note that there may be additional setup required for network connectivity between Azure and AWS, such as configuring VPC peering or VPN connections.

Overall, the approach of using AWS Glue Data Catalog as an intermediary allows you to seamlessly integrate data between cloud environments, while maintaining control over your data and maintaining a consistent metadata repository.

Re: Reading data in Azure Databricks Delta Lake from AWS Redshift

playermanny2 — Thu, 27 Apr 2023 13:31:00 GMT

Thank you -- would you happen to know the details on how to set-up at that crawler? There is an option for delta lake, but for the url it askes for an s3 location. Would i just plug in a azure data lake storage location, and how would authentication work?