cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Allows to serveless compute to connect to postgres db

jeremy98
Contributor

Hi Community,


Is it possible to enable VNet peering between Databricks Serverless Compute and a private PostgreSQL database that is already configured with a VNet?

Currently, everything works fine when I create my personal cluster because I have set up the VNet peering. However, when I try to attach a notebook to a Serverless Compute instance, calls to my private database (hosted in Azure) fail, and I cannot determine the cause.

What is the procedure to enable VNet peering for Serverless Compute? How can I discover the IPs used by Serverless Compute to allow access to my database?

6 REPLIES 6

Rjdudley
Valued Contributor

Have you tried Lakehouse Federation for this request?  That might be a better way, depending on what you're doing.

Otherwise, I think you need to use an NCC to set up the trust between serverless and your PostgreSQL, see https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serv... (I know the URL says Private Link, it's not Private Link).

Hi,

Thanks for your answers, what is Lakehouse Federation? Considers that PostgresSQL is created inside Azure, so we need to enable essentially some VNet, I was reading your solution, I hope this will work, since It's important to migrate data from a current architecture to a new one in databricks.

Rjdudley
Valued Contributor

jeremy98
Contributor

Ah ok thanks, I already did it, but for us we need to migrate to databricks

Rjdudley
Valued Contributor

Is that PostgreSQL server going to go away after you migrate to Databricks, or is it going to continue to be used?  Either way, federation works for you.  If you're going to discontinue it, just do a full extract into an archive location and a one-time ETL from that.  If it will remain in service, you can treat it as a landing location and have extraction workbooks load bronze from it.

Hi, 
Thanks for your answer, actually we don't know if it will remains but since we have a medallion architecture having data in parquet files it's better because the data size is less rather than postgres.
But, at the end of our medallion architecture we have a new postgres database where we need to store our data for our portal. So, I could say we need to have serveless to catch faster data and don't wait for job compute up.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group