display() function always return connection refused on tunneling despite successfully retrieving the schema

antoooks
New Contributor III

Hi everyone,

I am using SSH tunnelling with SSHTunnelForwarder to reach a target AWS RDS PostgreSQL database. The connection got through, however when I tried to display the retrieved data frame it always throws "connection refused" error. Please see screenshot below for clarity. I don't understand how this function works, it seems like it does not reuse the established connection from sparks.read command. Can somebody help me on this?

What I already tried:

  • SSH into the cluster, try tunnelling, and psql from there [works]
  • directly connect to db instance from proxy host [works]
  • Using panda to load the data frame instead of sparks [does not work]
  • Create new SSH key pair for Databricks [does not work]
  • Change postgresql.conf listen_address to ''*"

Screenshot:

image.png 

Thanks.

Regards,

Kurnianto