Hi @Rohit Rajendran​ and @charles okoh​ I was able to set up VPC peering for my customer's Databricks and Redshift account.

• I tested the connection via %sh host -t a <redshift hostname> and confirmed the IP address.
• I also ran a display(spark.sql("SELECT * FROM csv.`s3://<customer/s3bucketpath>`")) to confirm I can access the S3 bucket.

However when I attempt to run:

1. df = ( spark.read \
2. .format("jdbc") \
3. .option("url", "jdbc:redshift://<connection-string>:5439/database”)\
4. .option("dbtable", “<schema.table>“)\
5. .option("user", "customeruser")\
6. .option("password", “Pass*****”)
7. .load()
8. )

I get the error below:

Py4JJavaError: An error occurred while calling o486.load.

: java.sql.SQLException: The connection attempt failed.

at com.amazon.redshift.util.RedshiftException.getSQLException(RedshiftException.java:56)

at com.amazon.redshift.Driver.connect(Driver.java:339)

at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:46)

at com.databricks.spark.redshift.JDBCWrapper.getConnector(RedshiftJDBCWrapper.scala:355)

at com.databricks.spark.redshift.JDBCWrapper.getConnector(RedshiftJDBCWrapper.scala:376)

at com.databricks.spark.redshift.RedshiftRelation.$anonfun$schema$1(RedshiftRelation.scala:76)

at scala.Option.getOrElse(Option.scala:189)

at com.databricks.spark.redshift.RedshiftRelation.schema$lzycompute(RedshiftRelation.scala:73)

at com.databricks.spark.redshift.RedshiftRelation.schema(RedshiftRelation.scala:72)

at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:496)

at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:356)

at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:323)

at scala.Option.getOrElse(Option.scala:189)

at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:323)

at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:222)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)

at py4j.Gateway.invoke(Gateway.java:295)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.GatewayConnection.run(GatewayConnection.java:251)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.net.SocketTimeoutException: connect timed out

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:607)

at com.amazon.redshift.core.RedshiftStream.<init>(RedshiftStream.java:86)

at com.amazon.redshift.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:111)

at com.amazon.redshift.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:224)

at com.amazon.redshift.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)

at com.amazon.redshift.jdbc.RedshiftConnectionImpl.<init>(RedshiftConnectionImpl.java:322)

at com.amazon.redshift.Driver.makeConnection(Driver.java:502)

at com.amazon.redshift.Driver.connect(Driver.java:315)