topic Re: Unity Catalog in Data Engineering

Unity Catalog

roshanjoebenny — Wed, 04 Dec 2024 09:01:10 GMT

When I try to connect my local postgres with databricks unity catalog I am facing issues. Could you please explain the steps in doing that

Re: Unity Catalog

ozaaditya — Wed, 04 Dec 2024 09:46:36 GMT

Hi,

You can take the help of the documentation shared below,

https://learn.microsoft.com/en-us/azure/databricks/query-federation/postgresql

Re: Unity Catalog

-werners- — Wed, 04 Dec 2024 09:58:19 GMT

Databricks does not have connectivity to your local network out of the box.
You should setup a VNet and VNet peering (and also firewall rules).
Connect your Azure Databricks workspace to your on-premises network - Azure Databricks | Microsoft Learn

Networking - Azure Databricks | Microsoft Learn

Re: Unity Catalog

roshanjoebenny — Wed, 04 Dec 2024 10:17:01 GMT

CREATE CONNECTION <connection-name> TYPE postgresql
OPTIONS (
host '<hostname>',
port '<port>',
user '<user>',
password '<password>'
);
When I am running the above code mentioned in the documentation I am getting errors

Re: Unity Catalog

roshanjoebenny — Wed, 04 Dec 2024 10:19:05 GMT

Re: Unity Catalog

-werners- — Wed, 04 Dec 2024 10:28:22 GMT

Again: is your local network and the database open for external connections?
Do you have a static ip (or dynDNS)?

Re: Unity Catalog

roshanjoebenny — Wed, 04 Dec 2024 10:49:44 GMT

yes I have opened the database for external connections and its static.

driver = "org.postgresql.Driver"

database_host = "<database-host-url>"
database_port = "5432" # update if you use a non-default port
database_name = "<database-name>"
table = "<table-name>"
user = "<username>"
password = "<password>"

url = f"jdbc:postgresql://{database_host}:{database_port}/{database_name}"

remote_table = (spark.read
.format("jdbc")
.option("driver", driver)
.option("url", url)
.option("dbtable", table)
.option("user", user)
.option("password", password)
.load()
)
I have also tried this code also and this also has the issues

Re: Unity Catalog

roshanjoebenny — Thu, 05 Dec 2024 05:29:42 GMT

Could you please suggest a solution to the above query.