โ04-14-2022 02:34 AM
Using DBR 10 or later and Iโm getting an error when running the following query
SELECT * FROM delta.`s3://some_path`
getting org.apache.spark.SparkException: Unable to fetch tables of db delta
For 3.2.0+ they recommend reading like this:
CREATE TEMPORARY VIEW parquetTable
USING org.apache.spark.sql.parquet
OPTIONS (
path "examples/src/main/resources/people.parquet"
)
SELECT * FROM parquetTable
Can you confirm this is the only way?
โ05-11-2022 05:46 AM
Got support from Databricks.
Unfortunately, someone created a DB called delta, so the query was done against that DB instead.
Issue was solved
โ04-14-2022 07:37 AM
@Cristobal Bergerโ , Databricks uses dbfs, so if you want to use a path to read the data, you should use the dbfs path.
Using a view works too, btw (or define it as a table).
โ04-18-2022 02:02 AM
Hi @Werner Stinckensโ, thanks for replying.
Actually, you can read directly from S3 on PySpark and Spark SQL. Amaz on S3 documentation can show you how to do it. Now, it looks from Spark 3.2 (DBR 10 or later), it's not possible to use syntactic sugar on the FROM statement. That's what I need to confirm.
Thanks
โ05-04-2022 10:32 AM
โ05-11-2022 04:30 AM
Hi @Cristobal Bergerโ , Just a friendly follow-up. Do you still need help? Please let us know.
โ05-11-2022 05:46 AM
Got support from Databricks.
Unfortunately, someone created a DB called delta, so the query was done against that DB instead.
Issue was solved
โ05-11-2022 08:13 AM
Thank you for the update @Cristobal Bergerโ !
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group