Databricks on AWS Marketplace – Unity Catalog & S3... - Databricks Community

Administration & Architecture

Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.

Hi All,
I’m facing an issue accessing AWS S3 and Unity Catalog from a Databricks AWS Marketplace workspace.

Problem:

Whenever Databricks tries to access S3 or Unity Catalog, it fails with:

javax.net.ssl.SSLException: Connection reset

What works:

Spark jobs run fine

DNS resolution works

curl https://s3.us-east-1.amazonaws.com works from the cluster

NAT Gateway is configured

Outbound HTTPS (443) is allowed

What fails:

SHOW CATALOGS

dbutils.fs.ls("s3://...")

Creating schemas or tables in Unity Catalog

Environment:

AWS Marketplace Databricks (QuickLaunch)

Classic compute (Spark Connect)

Private subnets + NAT Gateway

Found an S3 Gateway VPC endpoint, removed its route table association and restarted cluster — issue still happens

Since basic network access works but Databricks-initiated calls fail, this looks like a Databricks / Unity Catalog connectivity issue, not IAM or general networking.

Has anyone faced this before?

Is S3 Gateway VPC endpoint supported with Unity Catalog in Marketplace workspaces?

Any guidance would really help. Thanks!