How to connect to AWS Custom VPC endpoint

alex_data
New Contributor

Hello,

Could somebody help me with the connection issue to my VPC endpoint

I have created a customer-managed VPC in AWS and setup new workspace with that VPC. There is an RDS  in other vpc that I want to connect from Databricks, and I have created a VPC endpoint service (in RDS VPC) and a VPC endpoint (in Databricks VPC).

But - i can't connect to this VPC endpoint from Databricks: 

%sh
nc -vz vpce-xxxx.vpce-svc-xxxx.eu-west-1.vpce.amazonaws.com 5432
 
vpce-xxxx.vpce-svc-xxxx.eu-west-1.vpce.amazonaws.com [10.0.xxx.xxx] 5432 (postgresql) : Connection refused
 
I have tried to create ec2 instance with the same security group as Databricks, and i can connect to vpc endpoint
 
What im doing wrong? Will be appreciated if somebody can share knowledge on usign custom VPC endpoint with databricks

nayan_wylde
Esteemed Contributor II
Connection refused” means the TCP handshake reached your endpoint ENI and the backend actively rejected the connection. That’s different from a timeout (routing/DNS), so your PrivateLink plumbing and DNS are mostly correct. 
 
Short fixes you can try:
1. Add your VPC Endpoint to ALL Databricks private subnets / AZs
In AWS Console → VPC → Endpoints → your endpoint → Edit subnets →
Add subnets for every AZ where Databricks private subnets exist.
 

2. Update SGs
On both:

the VPC Endpoint SG, and
the backend SG (RDS Proxy or EC2 proxy)

Allow inbound:

Source: Databricks Cluster Security Group
Port: 5432

3. Ensure backend is actually listening
If using RDS:
You cannot target the RDS instance directly — use RDS Proxy or an EC2 TCP proxy.


99% of cases are fixed by:
Adding endpoint ENIs to all AZs
Fixing backend SG to allow Databricks SG

View solution in original post