cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Is it possible to connect Databricks SQL with AWS Redshift DB?

LorenRD
Contributor

I would like to know if it's possible to connect Databricks SQL module with not just internal Metastore DB and tables from Data Science and Engineering module but also connect with an AWS Redshift DB to do queries and create alerts.

image 

1 ACCEPTED SOLUTION

Accepted Solutions

LorenRD
Contributor

Hi @Kaniz Fatmaโ€‹ I contacted Customer support explaining this issue, they told me that this feature is not implemented yet but it's in the roadmap with no ETA. It would be great if you ping me back when it's possible to access Redshift tables from SQL module in Databricks SQL

View solution in original post

11 REPLIES 11

Hubert-Dudek
Esteemed Contributor III

@Lorenzo Rondanโ€‹ , I agree with your vision as we can register parquet from some s3; why not as a data mesh register anything, for example, Redshift or Azure SQL database.

Anonymous
Not applicable

@Lorenzo Rondanโ€‹ , You can create a table pointing to redshift in a python notebook and register that table with the metastore.

LorenRD
Contributor

Hi @Joseph Kambourakisโ€‹  can you give an example or link docs to do it? Thanks!

Hubert-Dudek
Esteemed Contributor III

Yes sorry, indeed there is CREATE TABLE USING JDBC driver, never tried it.

CREATE TABLE example_table
USING com.databricks.spark.redshift
OPTIONS (
  dbtable '<your-table-name>',
  tempdir 's3a://<your-bucket>/<your-directory-path>',
  url 'jdbc:redshift://<the-rest-of-the-connection-string>'
);

LorenRD
Contributor

@Hubert Dudekโ€‹ @Joseph Kambourakisโ€‹ 

After testing this new table pointing to redshift it doesn't work because com.databricks.spark.redshift is not a supported data source in Databricks SQL (Not delta like a normal DB/Tabled stored in S3)

image

Hubert-Dudek
Esteemed Contributor III

@Lorenzo Rondanโ€‹ , So it seems that this syntax USING com.databricks.spark.redshift is working only in notebook/standard workspace.

Databricks SQL endpoints have a different runtime. So let's hope that the unity catalog mentioned in error will include external systems.

User16725394280
Contributor II

@Lorenzo Rondanโ€‹ , Hi You can check on the below content and please search for Redshift.

https://docs.databricks.com/data

Kaniz_Fatma
Community Manager
Community Manager

Hi @Lorenzo Rondanโ€‹  , Just a friendly follow-up. Do you still need help, or does @Hubert Dudek (Customer)โ€‹ 's and @Kiran Bharathiโ€‹ 's response help you to find the solution? Please let us know.

LorenRD
Contributor

Hi @Kaniz Fatmaโ€‹ I contacted Customer support explaining this issue, they told me that this feature is not implemented yet but it's in the roadmap with no ETA. It would be great if you ping me back when it's possible to access Redshift tables from SQL module in Databricks SQL

Hi @Lorenzo Rondanโ€‹ , Would you mind marking your answer as the best as you received the customer support and the latest update on the issue? We'll ping you once it's available.

Done, thanks. Looking forward for your update.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group