- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2022 07:33 AM
I use Databricks and I try to connect to posgresql via the following code
"jdbcHostname = "xxxxxxx"
jdbcDatabase = "xxxxxxxxxxxx"
jdbcPort = "5432"
username = "xxxxxxx"
password = "xxxxxxxx"
jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jdbcPort, jdbcDatabase)
connectionProperties = {
"user" : username,
"password" : password,
"driver" : "org.postgresql.Driver"
}
df = spark.read.jdbc(url=jdbcUrl, table= "xxxxxxxxx" , properties=connectionProperties)"
I try to read a table that is 28 million rows and here is the error message;
"SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 6) (10.139.64.5 executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 150527 ms"
Could you help me please
Thanks
- Labels:
-
Connection
-
Postgres
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-15-2022 11:33 PM
hi @Boumaza nadia Please check the Ganglia metrics for the cluster. This could be a scalability issue where cluster is overloading. This can happen due to a large partition not fitting into the given executor's memory. To fix this we recommend bumping up the worker node type. Switch to a bigger worker node instance to mitigate the issue.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2025 08:51 AM
Can you tell me how you are able to connect postgres database from Databricks . Do we have to whitelist ips in postgres?