topic Re: Azure to AWS in Data Engineering

Azure to AWS

priya9896 — Tue, 29 Sep 2026 19:17:54 GMT

Hi everyone,

We're evaluating an architecture pattern and would appreciate any guidance or recommendations.

Has anyone implemented a similar cross-cloud pattern? Specifically:

Can AWS Glue be used to access Azure Databricks data without copying it?
Are there recommended approaches such as federation, Delta Sharing, custom connectors, JDBC/SQL endpoints, or other patterns?

Thanks in advance!

Re: Azure to AWS

priya9896 — Tue, 29 Sep 2026 19:21:38 GMT

Hi everyone,

We're evaluating an architecture pattern and would appreciate any guidance or recommendations.

Current state:

Source data resides in Azure Databricks.
Consumers are in AWS .
The source team allows read-only access but does not allow data replication or ingestion into AWS.
Goal: We would like AWS consumers to access the data through Glue Catalog tables while keeping the data in Azure Databricks.

Has anyone implemented a similar cross-cloud pattern? Specifically:

Can AWS Glue be used to access Azure Databricks data without copying it?
Are there recommended approaches such as federation, Delta Sharing, custom connectors, JDBC/SQL endpoints, or other patterns?

Appreciate all your time and inputs regarding . Thanks in advance!

Re: Azure to AWS

Louis_Frolio — Fri, 02 Oct 2026 17:22:04 GMT

Hello @priya9896, I took a look at both internal and external documentation and here is what I found.

Straight answer: don't plan on the Glue Data Catalog as a federation layer over Azure Databricks. Glue's Delta integration and the Glue Catalog itself are built around tables in S3. Even Glue catalog federation to Unity Catalog's Iceberg REST endpoint works by having Lake Formation vend scoped credentials to data stored in S3, and AWS's walkthrough for it lists a Databricks workspace on AWS as a prerequisite. So: can a Glue Spark job read Azure Databricks data without copying it? Yes. Can you get Glue Catalog tables that Athena queries without a copy? No. Your two goals pull against each other, and one has to give.

With "no copy" as the hard constraint, the supported pattern is Delta Sharing, which Databricks renamed OpenSharing in June 2026. Same protocol; the package is still delta-sharing-spark and the format is still deltaSharing.

If your AWS consumers have a Unity Catalog-enabled Databricks workspace, use Databricks-to-Databricks sharing. The shared tables show up read-only in their catalog and there's no credential file to manage.

If Glue is the actual consumer, use Databricks-to-Open sharing. The source team creates a share with only the tables you need (partition filters or shared views if they want to narrow rows or columns), creates a recipient (bearer token or OIDC), and sends you a credential file. A Glue Spark job then reads it like this, with the credential file locked down since it is a bearer token:

df = spark.read.format("deltaSharing").load("s3://<secure-bucket>/config.share#<share>.<schema>.<table>")

The same credential file carries an icebergEndpoint, so Iceberg clients (Spark with an Iceberg REST catalog, PyIceberg, Trino, Snowflake) can read the share too.

Either way, Databricks returns the table's storage location with temporary cloud credentials, and your compute reads directly from Azure storage. Nothing persists in AWS. That's no replication, but it isn't zero network movement: rows cross the cloud boundary on every query. Prototype it from a Glue job, measure latency and throughput, and settle egress with the source team up front, because your cloud vendor may charge egress fees for sharing across clouds, and those land on whoever owns the storage, which is them. Databricks documents Cloudflare R2 as an egress-free option (a replica, so their call), and with SecureConnect, Databricks bills the data transfer rather than the cloud vendor; it's in Public Preview, so ask about it. Also pin down what "no ingestion into AWS" means. If it means no persisted copy, OpenSharing fits. If it means no bytes leave Azure, nothing here works and the consumers need to compute in Azure.

On the other patterns:

Unity Catalog's Iceberg REST catalog or Unity REST API with a service principal. Works from Spark, but the source team has to enable external data access on their metastore and manage a principal for you. OpenSharing is the purpose-built version for a separate consuming team: scoped to a share, revocable, audited.
JDBC to a Databricks SQL warehouse. Fine for targeted extracts, but every read runs on the source team's warehouse, rows move over JDBC rather than parallel Parquet reads, and the result still isn't a native Glue Catalog table Athena can query.
Direct storage access. AWS does publish an Azure Data Lake Storage connector for Glue that can read Delta tables straight from ADLS Gen2, but it runs on storage credentials, not Unity Catalog permissions, so it bypasses the source team's governance entirely. Most source teams won't agree, and I wouldn't ask.
Lakehouse Federation runs the other direction (Databricks querying external systems), so it doesn't help here.

If your consumers truly need glueContext.create_data_frame.from_catalog(...) or Athena, that's a different problem, and the only route is a governed, scheduled replica into S3 that Glue then catalogs. That's replication, which the source team has ruled out, so I'd have that conversation with them rather than engineer around it. First, though, confirm whether the consumers need Glue Catalog metadata or just governed read access. In my experience it's usually the latter, and OpenSharing covers it.

References:

OpenSharing announcement (Delta Sharing rename): https://www.databricks.com/blog/announcing-new-opensharing-and-marketplace-capabilities-ai-era
OpenSharing overview (Azure Databricks): https://learn.microsoft.com/en-us/azure/databricks/opensharing/
Databricks-to-Open sharing, provider side: https://learn.microsoft.com/azure/databricks/opensharing/share-data-open
Read shared data with Spark and Iceberg clients, recipient side: https://learn.microsoft.com/en-us/azure/databricks/opensharing/read-data-open
Monitor and manage OpenSharing egress costs: https://learn.microsoft.com/en-us/azure/databricks/opensharing/manage-egress
Unity Catalog Iceberg REST catalog for external engines: https://learn.microsoft.com/en-us/azure/databricks/external-access/iceberg
Delta Sharing open-source Spark connector: https://docs.delta.io/delta-sharing/
AWS Glue catalog federation to Unity Catalog: https://docs.aws.amazon.com/lake-formation/latest/dg/catalog-federation-databricks.html
AWS blog, Unity Catalog federation walkthrough: https://aws.amazon.com/blogs/big-data/access-databricks-unity-catalog-data-using-catalog-federation-in-the-aws-glue-data-catalog/
AWS Glue Delta Lake framework: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-delta-lake.html
AWS blog, reading Delta tables from ADLS Gen2 with the Glue connector: https://aws.amazon.com/blogs/big-data/migrate-delta-tables-from-azure-data-lake-storage-to-amazon-s3-using-aws-glue/

Regards, Louis.

Re: Azure to AWS

lmcorreahdb — Fri, 02 Oct 2026 22:43:41 GMT

Great explanation! I'd add one important consideration from a cross-cloud architecture and data governance perspective.

Before implementing OpenSharing with AWS Glue, I would clarify exactly what the source team means by "no replication or ingestion into AWS."

There is an important difference between avoiding a permanent replica in S3 and prohibiting any temporary data persistence or processing within AWS.

Even when using OpenSharing, the data still crosses cloud boundaries. Depending on the Glue job configuration, intermediate Spark data could also be written to temporary storage.

I'd suggest validating three things during a small proof of concept:

Governance: Confirm whether temporary processing and potential data spilling in AWS are permitted.

Performance: Measure cross-cloud transfer costs, query latency and the volume of data transferred.

Operations: Validate credentials, network connectivity and how shared-table schema changes affect downstream consumers.

If the requirement is strictly read-only access without a permanent replica, OpenSharing is worth evaluating.

However, if no data is permitted to leave Azure, I would consider keeping the processing in Azure and exposing only approved results.

One question: Is the Glue Catalog requirement driven by Athena integration, or could the AWS consumers work directly with shared datasets through Glue Spark?

Re: Azure to AWS

Louis_Frolio — Wed, 07 Oct 2026 14:10:33 GMT

Greetings @priya9896, I looked into things a bit more and here is what I found. And thanks to @lmcorreahdb for the governance point; the definition question is the whole ballgame, so most of this puts specifics behind it.

First, a sharper version of the Glue federation point from my earlier reply. AWS now documents a Glue/Lake Formation federated catalog to Unity Catalog (connection type DATABRICKSICEBERGRESTCATALOG), and in principle you could point it at the Azure workspace and see the catalog's databases and tables in Glue. Queries would fail, though. AWS states that only Iceberg tables in Amazon S3 are queryable; any other storage lists fine and then fails at SELECT with "Object storage location is not supported." So the Glue Catalog door only opens if the tables live in S3 as Iceberg (or Delta with UniForm), which for an Azure source means a copy. Same conclusion as before, now with the exact error you'd hit.

Where bytes can land in AWS during a Glue job, even with OpenSharing and no explicit write:

Shuffle and spill. By default they go to the Glue workers' local disks and vanish with the job. If someone enables --write-shuffle-files-to-s3, shuffle files go to an S3 bucket, and AWS notes the shuffle manager doesn't clean them up afterward. Leave that flag off here.
--TempDir. Glue's S3 scratch location; some connectors and operations stage data there.
Logs. Continuous CloudWatch logging and Spark UI event logs (--enable-spark-ui, --spark-event-logs-path) carry driver and executor output and query plans. No rows unless somebody prints a DataFrame, so ban show() and collect() in production jobs.
Developer habits. cache() to disk, checkpoints, a "temporary" debug write. Policy catches these, tooling doesn't.

Write those down and ask the source team which are acceptable. That turns "no ingestion" from a slogan into a spec.

On the provider side, Databricks-to-Open sharing gives the source team more control than they probably expect:

Scope. The recipient sees only the tables and views in the share, with partition filters to narrow rows. Nothing else in the metastore is visible.
Token lifetime. Set at the metastore or per recipient, rotated on demand; Databricks recommends rotating or dropping a recipient promptly once its token expires.
IP access lists. An allow list on the recipient covering the sharing API, activation link and credential download. Without SecureConnect the short-lived storage URLs can still be used from any IP; with it the list applies to storage too. Run the Glue job through a VPC connection with a NAT gateway if you want stable egress IPs for that list.
Audit. Recipient queries against the share are logged in the provider's system.access.audit, so the source team sees who read what and when.
Schema changes. They show up at the recipient's next read. Select explicit columns rather than * and agree on a heads-up process with the provider.

On the Athena question @lmcorreahdb raised: if Glue Catalog really means Athena, close that door early. AWS's prebuilt Athena federated connectors don't include Databricks, the generic JDBC connector is deprecated, and the one Azure-oriented connector they publish (ADLS Gen2) routes through Azure Synapse rather than Databricks, so it sidesteps Unity Catalog, and like every Athena connector it spills results to an S3 bucket when a response outgrows Lambda's limits. Athena means a replica, plain and simple.

So the decision tree is short. Glue Spark (or EMR, SageMaker, anything running the sharing connector) with no permanent replica: OpenSharing, Databricks-to-Open. A Unity Catalog-enabled Databricks workspace on your side: Databricks-to-Databricks, simpler still. Glue Catalog or Athena mandatory: a governed S3 replica, and have that conversation now rather than after the POC. No bytes may leave Azure: compute in Azure and expose approved results only.

For the POC, one small table from a Glue job, shuffle-to-S3 off, nothing written, then have the source team find the reads in system.access.audit. Seeing their own audit trail does more for the governance conversation than any diagram. Please report back with the latency and egress numbers; plenty of folks will hit this same wall.

References:

AWS Lake Formation, federate to Databricks Unity Catalog (S3-only query note): https://docs.aws.amazon.com/lake-formation/latest/dg/catalog-federation-databricks.html
AWS Glue job parameters (--TempDir, Spark UI logs, continuous logging): https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
AWS Glue Spark shuffle plugin with Amazon S3: https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shuffle-manager.html
Create a bearer-token recipient, token lifetime and rotation (Azure Databricks): https://learn.microsoft.com/en-us/azure/databricks/opensharing/create-recipient-token
Restrict recipient access with IP access lists: https://learn.microsoft.com/azure/databricks/opensharing/access-list
Audit and monitor data sharing: https://learn.microsoft.com/en-us/azure/databricks/opensharing/audit-logs
Athena prebuilt data source connectors: https://docs.aws.amazon.com/athena/latest/ug/connectors-available.html
Athena ADLS Gen2 connector: https://docs.aws.amazon.com/athena/latest/ug/connectors-adls-gen2.html

Regards, Louis.

Re: Azure to AWS

vs4 — Wed, 07 Oct 2026 18:18:33 GMT

Hi @priya9896

Given the no-replication requirement, I would evaluate Delta Sharing/OpenSharing first rather than JDBC-based ingestion. It is designed for secure, read-only cross-platform data sharing without maintaining a replicated copy of the source data. Databricks Documentation

The pattern would be:

Azure Databricks / Unity Catalog → Delta Sharing/OpenSharing → AWS consumer

The important distinction is that Glue Catalog is a metadata/catalog layer, so I wouldn’t treat “creating a Glue table” itself as the mechanism for accessing the Azure Databricks data. The AWS-side engine still needs a supported way to consume the shared data.

I would therefore validate which AWS compute engine needs to query those Glue tables first, and then determine whether it can consume the OpenSharing share directly or requires an integration layer.

Also consider cross-cloud network egress cost and network controls, even though the architecture avoids maintaining another copy of the dataset.

Re: Azure to AWS

priya9896 — Wed, 07 Oct 2026 20:48:44 GMT

Thank you so much for all your help and guidance. I truly appreciate your responses and insights. I’ll definitely keep this in mind and follow up with the source team first.

Re: Azure to AWS

Aravind_Reddy — Thu, 08 Oct 2026 09:45:51 GMT

Thanks for the detailed explanation Louis. this really helps in understanding the concept more clearly. The distinction between “no replication” and “no data movement” is particularly important here.

If OpenSharing is used with AWS Glue, the data can remain in Azure while the AWS workload reads it on demand. One thing I would be interested in understanding is how this behaves for larger analytical workloads.

For example, if the AWS consumers frequently query large Delta tables, would it be better to expose curated/filtered shared views or partition-filtered tables through OpenSharing rather than allowing consumers to query the full datasets? That seems like it could help reduce cross-cloud data transfer, latency, and egress costs.

Re: Azure to AWS

Louis_Frolio — Thu, 08 Oct 2026 12:24:56 GMT

Hello @Aravind_Reddy,

Short answer: yes, share the smallest useful slice, but the mechanism matters, because OpenSharing serves tables and views very differently to an open recipient like Glue. And as @vs4 said above, settle which AWS engine is reading before you design the share.

Order of preference:

Partition-limited tables, when the access pattern has stable boundaries (date, region, tenant). The share carries a partition spec, static (PARTITION (year = '2026')) or recipient-driven (PARTITION (region = CURRENT_RECIPIENT().region)), and the server never lists files outside it. No provider compute, nothing for the consumer to get wrong. The consumer's own filters also travel to the server as predicate hints: partition columns prune well, other columns are best effort and file-level. Caveats: the table needs Hive-style partitions (you can't partition-filter a liquid-clustered table), a filter disqualifies the table from directory-based access (below), and a big table means a big file list per query with limits on file counts, so keep it compacted.
A curated Delta table, when the subset is more than a partition predicate. The source team maintains a narrow table in Azure (needed columns, rows, retention window) and shares that. It gets everything in point 1 with no per-query compute on their side.
A materialized view, for a stable reporting shape that's queried often. The heavy query runs on the refresh schedule instead of on every read; open recipients get the current snapshot.
Views, for governance (row filters or masking by recipient via CURRENT_RECIPIENT()) or for a slice the source team won't materialize. Don't assume a view is faster. For a Databricks-to-Open recipient, every query runs on the provider's serverless compute, the result is fully materialized regardless of the consumer's filters (no predicate or LIMIT pushdown), parked temporarily in the provider's storage and served through short-lived URLs. The provider pays, and can see what each recipient costs by joining system.sharing.materialization_history to system.billing.usage. A view as wide as the table is strictly worse than the table, plain and simple, and "frequently queried" means "frequently materialized."

Two things apply regardless. Consumer-side hygiene: select explicit columns (the connector fetches Parquet column chunks over range requests, so fewer columns means fewer bytes) and filter on partition columns that match the workload; a partition boundary the queries don't use buys you nothing. And the newer directory-based access mode: an unfiltered Delta table shared WITH HISTORY can be served with a short-lived, path-scoped credential, so the recipient's own Delta reader does the planning and data skipping, liquid clustering included. Check that your Glue connector version supports it (recent protocol addition, URL mode is the fallback), and know that the credential covers the Delta log, which includes commit history and deleted data not yet vacuumed.

For the POC, run the same representative queries three ways: unrestricted table, partition-limited share, curated or aggregated table. Measure bytes transferred, end-to-end latency, Glue runtime, provider compute (the materialization table above), and egress. If the workload keeps scanning most of the source data, cross-cloud reads stay expensive however you slice it; the honest fixes are upstream curation, compute in Azure, or the replica the source team has already ruled out.

References:

Create shares (partition specs, recipient-property filtering, views, cloud token eligibility): https://learn.microsoft.com/en-us/azure/databricks/opensharing/create-share
OpenSharing overview, including who pays for view materialization and table limitations: https://learn.microsoft.com/en-us/azure/databricks/opensharing/
Materialization history system table: https://learn.microsoft.com/azure/databricks/admin/system-tables/materialization
Monitor and manage OpenSharing egress costs: https://learn.microsoft.com/en-us/azure/databricks/opensharing/manage-egress
Delta Sharing protocol (predicate hints and access modes): https://github.com/delta-io/delta-sharing/blob/main/PROTOCOL.md

Regards, Louis.

Re: Azure to AWS

Louis_Frolio — Thu, 08 Oct 2026 12:25:38 GMT

Also, if you feel your question has been answered please "Accept as Solution" so that others can benefit.

Cheers, Lou.