topic Re: Connection Databricks Postgresql in Data Engineering

Connection Databricks Postgresql

nadia — Tue, 14 Jun 2022 14:33:55 GMT

I use Databricks and I try to connect to posgresql via the following code

"jdbcHostname = "xxxxxxx"

jdbcDatabase = "xxxxxxxxxxxx"

jdbcPort = "5432"

username = "xxxxxxx"

password = "xxxxxxxx"

jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jdbcPort, jdbcDatabase)

connectionProperties = {

"user" : username,

"password" : password,

"driver" : "org.postgresql.Driver"

}

df = spark.read.jdbc(url=jdbcUrl, table= "xxxxxxxxx" , properties=connectionProperties)"

I try to read a table that is 28 million rows and here is the error message;

"SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 6) (10.139.64.5 executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 150527 ms"

Could you help me please

Thanks

Re: Connection Databricks Postgresql

Prabakar — Thu, 16 Jun 2022 06:33:46 GMT

hi @Boumaza nadia Please check the Ganglia metrics for the cluster. This could be a scalability issue where cluster is overloading. This can happen due to a large partition not fitting into the given executor's memory. To fix this we recommend bumping up the worker node type. Switch to a bigger worker node instance to mitigate the issue.

Re: Connection Databricks Postgresql

santhosh11 — Wed, 04 Jun 2025 15:51:36 GMT

Can you tell me how you are able to connect postgres database from Databricks . Do we have to whitelist ips in postgres?