Connecting to Serverless Redshift from a Databricks Notebook

arunak — Wed, 10 Apr 2024 00:50:37 GMT

Hello Experts,

A new databricks user here. I am trying to access an Redshift serverless table using a databricks notebook.
Here is what happens when I try the below code,

df = spark.read.format("redshift")\

.option("dbtable", "public.customer")\

.option("tempdir", "s3://BLAH/rs-temp/")\

.option("url", "jdbc:redshift://BLAH:5439/dev")\

.option("user", "user")\

.option("password", "password")\

.load()

df.show(10,False)

It fails with the below error

IllegalArgumentException: requirement failed: You must specify a method for authenticating Redshift's connection to S3 (aws_iam_role, forward_spark_s3_credentials, or temporary_aws_*. For a discussion of the differences between these options, please see the README.

If I edit the format to "jdbc", it works no issue. I am on 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)

I don't have an instance profile role. Why wouldn't the format("redshift") use the provided username and password and connect to redshift? What config should I be using?

Re: Connecting to Serverless Redshift from a Databricks Notebook

shan_chandra — Wed, 10 Apr 2024 20:10:24 GMT

@arunak - we need to specify forward_spark_s3_credentials to true during read. This will help spark detect the credentials used to authenticate to the S3 bucket and use these credentials to r read from redshift.

topic Connecting to Serverless Redshift from a Databricks Notebook in Data Engineering

Connecting to Serverless Redshift from a Databricks Notebook

Re: Connecting to Serverless Redshift from a Databricks Notebook