SELECT * FROM delta doesn't work on Spark 3.2

[CrisBerg_65149]

Using DBR 10 or later and I’m getting an error when running the following query

SELECT * FROM delta.`s3://some_path`

getting org.apache.spark.SparkException: Unable to fetch tables of db delta

For 3.2.0+ they recommend reading like this:

CREATE TEMPORARY VIEW parquetTable

USING org.apache.spark.sql.parquet

OPTIONS (

path "examples/src/main/resources/people.parquet"

)

SELECT * FROM parquetTable

Can you confirm this is the only way?

[-werners-]

@Cristobal Berger​ , Databricks uses dbfs, so if you want to use a path to read the data, you should use the dbfs path.

Using a view works too, btw (or define it as a table).

[CrisBerg_65149]

Hi @Werner Stinckens​, thanks for replying.

Actually, you can read directly from S3 on PySpark and Spark SQL. Amaz on S3 documentation can show you how to do it. Now, it looks from Spark 3.2 (DBR 10 or later), it's not possible to use syntactic sugar on the FROM statement. That's what I need to confirm.

Thanks

[Dan_Z]

Hello @Cristobal Berger​ , - I could not reproduce this using DBR 10; I think you may be doing something wrong.

[Kaniz_Fatma]

Hi @Cristobal Berger​ , Just a friendly follow-up. Do you still need help? Please let us know.

[CrisBerg_65149 · Accepted solution]

Got support from Databricks.

Unfortunately, someone created a DB called delta, so the query was done against that DB instead.

Issue was solved

[Kaniz_Fatma]

Thank you for the update @Cristobal Berger​ !