โ09-13-2021 08:01 AM
Hi,
We followed this document (https://docs.databricks.com/administration-guide/cloud-configurations/aws/vpc-peering.html) describing how to establish a connection between two (or more) VPC in AWS, but so far we haven't been able to communicate with the resources in the Databricks' VPC. The Databricks (requester) VPC and the accepter VPC are located both in the same account and in the same region. We don't have this issue with other VPC peering connections with VPC created and managed by us.
I was wondering if there's someone in the community that already stumble upon this issue, and could help us shed some light on any dark corners we might be missing checking since everything else - the route tables, the security groups, the DNS settings - is set as described in the guide.
Thanks,
-Martรญn
โ10-05-2021 05:24 PM
Hi Martin,
Sorry for the delayed response. Regarding the VPC peering connection, please check if the following is configured correctly:
1. On VPC management console, go to Peering Connections -> select the connection in question -> DNS, and make sure both "Allow accepter VPC to resolve DNS of hosts in requester VPC to private IP addresses" and "Allow requester VPC to resolve DNS of hosts in accepter VPC to private IP addresses" are enabled:
2. Under "Route tables" tab of VPC connection for both requester VPC and accepter VPC, please make sure it is associated with the correct route table. For example, for the requester (Databricks) VPC, make sure it is associated with Databricks workspace route table; for the accepter (your data source, say RDS) VPC, make sure it is associated with your data source route table.
Please give it a try and let us know if you are still experiencing any issues, and we will be happy to assist you further.
Thanks,
Huaming
โ10-05-2021 05:24 PM
Hi Martin,
Sorry for the delayed response. Regarding the VPC peering connection, please check if the following is configured correctly:
1. On VPC management console, go to Peering Connections -> select the connection in question -> DNS, and make sure both "Allow accepter VPC to resolve DNS of hosts in requester VPC to private IP addresses" and "Allow requester VPC to resolve DNS of hosts in accepter VPC to private IP addresses" are enabled:
2. Under "Route tables" tab of VPC connection for both requester VPC and accepter VPC, please make sure it is associated with the correct route table. For example, for the requester (Databricks) VPC, make sure it is associated with Databricks workspace route table; for the accepter (your data source, say RDS) VPC, make sure it is associated with your data source route table.
Please give it a try and let us know if you are still experiencing any issues, and we will be happy to assist you further.
Thanks,
Huaming
โ10-11-2021 11:56 PM
Hi Martin, were you able to fix the issue with the steps mentioned by Huaming?
โ10-20-2021 08:32 AM
Hi guys,
Thanks for stepping in (and apologies for the late reply, things have been a lot hectic around here lately).
Ultimately the issue was that the public subnets on the accepter VPC automatically inherit the PCX connection, but that does not hold true for the private subnets; after explicitly adding the PCX connection to the private subnets our issue was solved.
You may consider adding this extra bit of information to the wiki article ๐
โ10-23-2021 09:04 AM
Thats true @Martin Cigorragaโ we will try to explain a bit more details on that. Glad that the issue resolved.
โ10-29-2021 03:49 PM
Hi @Martin Cigorragaโ ,
If Huaming's fully answered your question, would you be happy to mark their answer as best so that others can quickly find the solution?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group