topic Re: SELECT * FROM delta doesn't work on Spark 3.2 in Data Engineering

SELECT * FROM delta doesn't work on Spark 3.2

CrisBerg_65149 — Thu, 14 Apr 2022 09:34:45 GMT

Using DBR 10 or later and I’m getting an error when running the following query

SELECT * FROM delta.`s3://some_path`

getting org.apache.spark.SparkException: Unable to fetch tables of db delta

For 3.2.0+ they recommend reading like this:

CREATE TEMPORARY VIEW parquetTable

USING org.apache.spark.sql.parquet

OPTIONS (

path "examples/src/main/resources/people.parquet"

)

SELECT * FROM parquetTable

Can you confirm this is the only way?

Re: SELECT * FROM delta doesn't work on Spark 3.2

-werners- — Thu, 14 Apr 2022 14:37:38 GMT

@Cristobal Berger , Databricks uses dbfs, so if you want to use a path to read the data, you should use the dbfs path.

Using a view works too, btw (or define it as a table).

Re: SELECT * FROM delta doesn't work on Spark 3.2

CrisBerg_65149 — Mon, 18 Apr 2022 09:02:41 GMT

Hi @Werner Stinckens, thanks for replying.

Actually, you can read directly from S3 on PySpark and Spark SQL. Amaz on S3 documentation can show you how to do it. Now, it looks from Spark 3.2 (DBR 10 or later), it's not possible to use syntactic sugar on the FROM statement. That's what I need to confirm.

Thanks

Re: SELECT * FROM delta doesn't work on Spark 3.2

Dan_Z — Wed, 04 May 2022 17:32:40 GMT

Hello @Cristobal Berger , - I could not reproduce this using DBR 10; I think you may be doing something wrong.

Re: SELECT * FROM delta doesn't work on Spark 3.2

CrisBerg_65149 — Wed, 11 May 2022 12:46:31 GMT

Got support from Databricks.

Unfortunately, someone created a DB called delta, so the query was done against that DB instead.

Issue was solved