topic Re: Is it possible to connect Databricks SQL with AWS Redshift DB? in Data Engineering

Is it possible to connect Databricks SQL with AWS Redshift DB?

LorenRD — Mon, 04 Apr 2022 12:02:54 GMT

I would like to know if it's possible to connect Databricks SQL module with not just internal Metastore DB and tables from Data Science and Engineering module but also connect with an AWS Redshift DB to do queries and create alerts.

Re: Is it possible to connect Databricks SQL with AWS Redshift DB?

Hubert-Dudek — Mon, 04 Apr 2022 12:36:59 GMT

@Lorenzo Rondan , I agree with your vision as we can register parquet from some s3; why not as a data mesh register anything, for example, Redshift or Azure SQL database.

Re: Is it possible to connect Databricks SQL with AWS Redshift DB?

Anonymous — Mon, 04 Apr 2022 15:37:57 GMT

@Lorenzo Rondan , You can create a table pointing to redshift in a python notebook and register that table with the metastore.

Re: Is it possible to connect Databricks SQL with AWS Redshift DB?

LorenRD — Mon, 04 Apr 2022 15:40:45 GMT

Hi @Joseph Kambourakis can you give an example or link docs to do it? Thanks!

Re: Is it possible to connect Databricks SQL with AWS Redshift DB?

Hubert-Dudek — Mon, 04 Apr 2022 16:39:42 GMT

Yes sorry, indeed there is CREATE TABLE USING JDBC driver, never tried it.

CREATE TABLE example_table
USING com.databricks.spark.redshift
OPTIONS (
  dbtable '<your-table-name>',
  tempdir 's3a://<your-bucket>/<your-directory-path>',
  url 'jdbc:redshift://<the-rest-of-the-connection-string>'
);

Re: Is it possible to connect Databricks SQL with AWS Redshift DB?

LorenRD — Mon, 04 Apr 2022 21:35:12 GMT

@Hubert Dudek @Joseph Kambourakis

After testing this new table pointing to redshift it doesn't work because com.databricks.spark.redshift is not a supported data source in Databricks SQL (Not delta like a normal DB/Tabled stored in S3)

Re: Is it possible to connect Databricks SQL with AWS Redshift DB?

Hubert-Dudek — Tue, 05 Apr 2022 09:01:45 GMT

@Lorenzo Rondan , So it seems that this syntax USING com.databricks.spark.redshift is working only in notebook/standard workspace.

Databricks SQL endpoints have a different runtime. So let's hope that the unity catalog mentioned in error will include external systems.

Re: Is it possible to connect Databricks SQL with AWS Redshift DB?

User16725394280 — Fri, 08 Apr 2022 11:18:25 GMT

@Lorenzo Rondan , Hi You can check on the below content and please search for Redshift.

https://docs.databricks.com/data

Re: Is it possible to connect Databricks SQL with AWS Redshift DB?

LorenRD — Tue, 26 Apr 2022 13:20:33 GMT

Hi @Kaniz Fatma I contacted Customer support explaining this issue, they told me that this feature is not implemented yet but it's in the roadmap with no ETA. It would be great if you ping me back when it's possible to access Redshift tables from SQL module in Databricks SQL

Re: Is it possible to connect Databricks SQL with AWS Redshift DB?

LorenRD — Tue, 26 Apr 2022 15:00:53 GMT

Done, thanks. Looking forward for your update.