Allows to serveless compute to connect to postgres db
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Tuesday
Hi Community,
Is it possible to enable VNet peering between Databricks Serverless Compute and a private PostgreSQL database that is already configured with a VNet?
Currently, everything works fine when I create my personal cluster because I have set up the VNet peering. However, when I try to attach a notebook to a Serverless Compute instance, calls to my private database (hosted in Azure) fail, and I cannot determine the cause.
What is the procedure to enable VNet peering for Serverless Compute? How can I discover the IPs used by Serverless Compute to allow access to my database?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Tuesday
Have you tried Lakehouse Federation for this request? That might be a better way, depending on what you're doing.
Otherwise, I think you need to use an NCC to set up the trust between serverless and your PostgreSQL, see https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serv... (I know the URL says Private Link, it's not Private Link).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yesterday
Hi,
Thanks for your answers, what is Lakehouse Federation? Considers that PostgresSQL is created inside Azure, so we need to enable essentially some VNet, I was reading your solution, I hope this will work, since It's important to migrate data from a current architecture to a new one in databricks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yesterday
Lakehouse federation: What is Lakehouse Federation? - Azure Databricks | Microsoft Learn
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10 hours ago
Ah ok thanks, I already did it, but for us we need to migrate to databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
4 hours ago
Is that PostgreSQL server going to go away after you migrate to Databricks, or is it going to continue to be used? Either way, federation works for you. If you're going to discontinue it, just do a full extract into an archive location and a one-time ETL from that. If it will remain in service, you can treat it as a landing location and have extraction workbooks load bronze from it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 hours ago
Hi,
Thanks for your answer, actually we don't know if it will remains but since we have a medallion architecture having data in parquet files it's better because the data size is less rather than postgres.
But, at the end of our medallion architecture we have a new postgres database where we need to store our data for our portal. So, I could say we need to have serveless to catch faster data and don't wait for job compute up.