Azure to AWS

priya9896 — Tue, 29 Sep 2026 19:17:54 GMT

Hi everyone,

We're evaluating an architecture pattern and would appreciate any guidance or recommendations.

Has anyone implemented a similar cross-cloud pattern? Specifically:

Can AWS Glue be used to access Azure Databricks data without copying it?
Are there recommended approaches such as federation, Delta Sharing, custom connectors, JDBC/SQL endpoints, or other patterns?

Thanks in advance!

Re: Azure to AWS

priya9896 — Tue, 29 Sep 2026 19:21:38 GMT

Hi everyone,

We're evaluating an architecture pattern and would appreciate any guidance or recommendations.

Current state:

Source data resides in Azure Databricks.
Consumers are in AWS .
The source team allows read-only access but does not allow data replication or ingestion into AWS.
Goal: We would like AWS consumers to access the data through Glue Catalog tables while keeping the data in Azure Databricks.

Has anyone implemented a similar cross-cloud pattern? Specifically:

Can AWS Glue be used to access Azure Databricks data without copying it?
Are there recommended approaches such as federation, Delta Sharing, custom connectors, JDBC/SQL endpoints, or other patterns?

Appreciate all your time and inputs regarding . Thanks in advance!

Re: Azure to AWS

Louis_Frolio — Fri, 02 Oct 2026 17:22:04 GMT

Hello @priya9896, I took a look at both internal and external documentation and here is what I found.

Straight answer: don't plan on the Glue Data Catalog as a federation layer over Azure Databricks. Glue's Delta integration and the Glue Catalog itself are built around tables in S3. Even Glue catalog federation to Unity Catalog's Iceberg REST endpoint works by having Lake Formation vend scoped credentials to data stored in S3, and AWS's walkthrough for it lists a Databricks workspace on AWS as a prerequisite. So: can a Glue Spark job read Azure Databricks data without copying it? Yes. Can you get Glue Catalog tables that Athena queries without a copy? No. Your two goals pull against each other, and one has to give.

With "no copy" as the hard constraint, the supported pattern is Delta Sharing, which Databricks renamed OpenSharing in June 2026. Same protocol; the package is still delta-sharing-spark and the format is still deltaSharing.

If your AWS consumers have a Unity Catalog-enabled Databricks workspace, use Databricks-to-Databricks sharing. The shared tables show up read-only in their catalog and there's no credential file to manage.

If Glue is the actual consumer, use Databricks-to-Open sharing. The source team creates a share with only the tables you need (partition filters or shared views if they want to narrow rows or columns), creates a recipient (bearer token or OIDC), and sends you a credential file. A Glue Spark job then reads it like this, with the credential file locked down since it is a bearer token:

df = spark.read.format("deltaSharing").load("s3://<secure-bucket>/config.share#<share>.<schema>.<table>")

The same credential file carries an icebergEndpoint, so Iceberg clients (Spark with an Iceberg REST catalog, PyIceberg, Trino, Snowflake) can read the share too.

Either way, Databricks returns the table's storage location with temporary cloud credentials, and your compute reads directly from Azure storage. Nothing persists in AWS. That's no replication, but it isn't zero network movement: rows cross the cloud boundary on every query. Prototype it from a Glue job, measure latency and throughput, and settle egress with the source team up front, because your cloud vendor may charge egress fees for sharing across clouds, and those land on whoever owns the storage, which is them. Databricks documents Cloudflare R2 as an egress-free option (a replica, so their call), and with SecureConnect, Databricks bills the data transfer rather than the cloud vendor; it's in Public Preview, so ask about it. Also pin down what "no ingestion into AWS" means. If it means no persisted copy, OpenSharing fits. If it means no bytes leave Azure, nothing here works and the consumers need to compute in Azure.

On the other patterns:

Unity Catalog's Iceberg REST catalog or Unity REST API with a service principal. Works from Spark, but the source team has to enable external data access on their metastore and manage a principal for you. OpenSharing is the purpose-built version for a separate consuming team: scoped to a share, revocable, audited.
JDBC to a Databricks SQL warehouse. Fine for targeted extracts, but every read runs on the source team's warehouse, rows move over JDBC rather than parallel Parquet reads, and the result still isn't a native Glue Catalog table Athena can query.
Direct storage access. AWS does publish an Azure Data Lake Storage connector for Glue that can read Delta tables straight from ADLS Gen2, but it runs on storage credentials, not Unity Catalog permissions, so it bypasses the source team's governance entirely. Most source teams won't agree, and I wouldn't ask.
Lakehouse Federation runs the other direction (Databricks querying external systems), so it doesn't help here.

If your consumers truly need glueContext.create_data_frame.from_catalog(...) or Athena, that's a different problem, and the only route is a governed, scheduled replica into S3 that Glue then catalogs. That's replication, which the source team has ruled out, so I'd have that conversation with them rather than engineer around it. First, though, confirm whether the consumers need Glue Catalog metadata or just governed read access. In my experience it's usually the latter, and OpenSharing covers it.

References:

OpenSharing announcement (Delta Sharing rename): https://www.databricks.com/blog/announcing-new-opensharing-and-marketplace-capabilities-ai-era
OpenSharing overview (Azure Databricks): https://learn.microsoft.com/en-us/azure/databricks/opensharing/
Databricks-to-Open sharing, provider side: https://learn.microsoft.com/azure/databricks/opensharing/share-data-open
Read shared data with Spark and Iceberg clients, recipient side: https://learn.microsoft.com/en-us/azure/databricks/opensharing/read-data-open
Monitor and manage OpenSharing egress costs: https://learn.microsoft.com/en-us/azure/databricks/opensharing/manage-egress
Unity Catalog Iceberg REST catalog for external engines: https://learn.microsoft.com/en-us/azure/databricks/external-access/iceberg
Delta Sharing open-source Spark connector: https://docs.delta.io/delta-sharing/
AWS Glue catalog federation to Unity Catalog: https://docs.aws.amazon.com/lake-formation/latest/dg/catalog-federation-databricks.html
AWS blog, Unity Catalog federation walkthrough: https://aws.amazon.com/blogs/big-data/access-databricks-unity-catalog-data-using-catalog-federation-in-the-aws-glue-data-catalog/
AWS Glue Delta Lake framework: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-delta-lake.html
AWS blog, reading Delta tables from ADLS Gen2 with the Glue connector: https://aws.amazon.com/blogs/big-data/migrate-delta-tables-from-azure-data-lake-storage-to-amazon-s3-using-aws-glue/

Regards, Louis.

Re: Azure to AWS

lmcorreahdb — Fri, 02 Oct 2026 22:43:41 GMT

Great explanation! I'd add one important consideration from a cross-cloud architecture and data governance perspective.

Before implementing OpenSharing with AWS Glue, I would clarify exactly what the source team means by "no replication or ingestion into AWS."

There is an important difference between avoiding a permanent replica in S3 and prohibiting any temporary data persistence or processing within AWS.

Even when using OpenSharing, the data still crosses cloud boundaries. Depending on the Glue job configuration, intermediate Spark data could also be written to temporary storage.

I'd suggest validating three things during a small proof of concept:

Governance: Confirm whether temporary processing and potential data spilling in AWS are permitted.

Performance: Measure cross-cloud transfer costs, query latency and the volume of data transferred.

Operations: Validate credentials, network connectivity and how shared-table schema changes affect downstream consumers.

If the requirement is strictly read-only access without a permanent replica, OpenSharing is worth evaluating.

However, if no data is permitted to leave Azure, I would consider keeping the processing in Azure and exposing only approved results.

One question: Is the Glue Catalog requirement driven by Athena integration, or could the AWS consumers work directly with shared datasets through Glue Spark?

topic Re: Azure to AWS in Data Engineering

Azure to AWS

Re: Azure to AWS

Re: Azure to AWS

Re: Azure to AWS