AnalysisException: [ErrorClass=INVALID_PARAMETER_VALUE] Missing cloud file system scheme

Madison
New Contributor II

‎09-01-2023 12:54 AM

I am trying to follow along Apache Spark Programming training module where the instructor creates events table from a parquet file like this:

%sql
CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "/mnt/training/ecommerce/events/events.parquet");

When I tried to run the above command, I got the following error message:

AnalysisException: [RequestId=... ErrorClass=INVALID_PARAMETER_VALUE] Missing cloud file system scheme
---------------------------------------------------------------------------
AnalysisException                         Traceback (most recent call last)
<command-644583705732552> in <cell line: 1>()
      5     display(df)
      6     return df
----> 7   _sqldf = ____databricks_percent_sql()
      8 finally:
      9   del ____databricks_percent_sql

<command-644583705732552> in ____databricks_percent_sql()
      2   def ____databricks_percent_sql():
      3     import base64
----> 4     df = spark.sql(base64.standard_b64decode("...=").decode())
      5     display(df)
      6     return df

/databricks/spark/python/pyspark/instrumentation_utils.py in wrapper(*args, **kwargs)
     46             start = time.perf_counter()
     47             try:
---> 48                 res = func(*args, **kwargs)
0 Kudos

Madison
New Contributor II

‎09-03-2023 08:06 AM - edited ‎09-03-2023 08:08 AM

@Retired_mod Thanks for your response. I didn't provide cloud file system scheme in the path while creating the table using DataFrame API, but I was still able to create the table. 

 

%python
# File location and type
file_location = "/mnt/training/ecommerce/users/users.parquet"
file_type = "parquet"

df = spark.read.format(file_type) \
  .load(file_location)

display(df)

temp_table_name = "test_catalog.test_schema.users"
df.createOrReplaceTempView(temp_table_name)

 

When I provided the scheme in SQL, I got the following error:

 

%sql
CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "s3://mnt/training/ecommerce/events/events.parquet");

AnalysisException: No parent external location found for path 's3://mnt/training/ecommerce/events/events.parquet'

 

 

0 Kudos