AnalysisException: [ErrorClass=INVALID_PARAMETER_VALUE] Missing cloud file system scheme

Madison — Fri, 01 Sep 2023 07:54:28 GMT

I am trying to follow along Apache Spark Programming training module where the instructor creates events table from a parquet file like this:

%sql CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "/mnt/training/ecommerce/events/events.parquet");

When I tried to run the above command, I got the following error message:

AnalysisException: [RequestId=... ErrorClass=INVALID_PARAMETER_VALUE] Missing cloud file system scheme --------------------------------------------------------------------------- AnalysisException Traceback (most recent call last) <command-644583705732552> in <cell line: 1>() 5 display(df) 6 return df ----> 7 _sqldf = ____databricks_percent_sql() 8 finally: 9 del ____databricks_percent_sql <command-644583705732552> in ____databricks_percent_sql() 2 def ____databricks_percent_sql(): 3 import base64 ----> 4 df = spark.sql(base64.standard_b64decode("...=").decode()) 5 display(df) 6 return df /databricks/spark/python/pyspark/instrumentation_utils.py in wrapper(*args, **kwargs) 46 start = time.perf_counter() 47 try: ---> 48 res = func(*args, **kwargs)

Re: AnalysisException: [ErrorClass=INVALID_PARAMETER_VALUE] Missing cloud file system scheme

Madison — Sun, 03 Sep 2023 15:08:42 GMT

@Retired_mod Thanks for your response. I didn't provide cloud file system scheme in the path while creating the table using DataFrame API, but I was still able to create the table.

%python # File location and type file_location = "/mnt/training/ecommerce/users/users.parquet" file_type = "parquet" df = spark.read.format(file_type) \ .load(file_location) display(df) temp_table_name = "test_catalog.test_schema.users" df.createOrReplaceTempView(temp_table_name)

When I provided the scheme in SQL, I got the following error:

%sql CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "s3://mnt/training/ecommerce/events/events.parquet"); AnalysisException: No parent external location found for path 's3://mnt/training/ecommerce/events/events.parquet'

topic Re: AnalysisException: [ErrorClass=INVALID_PARAMETER_VALUE] Missing cloud file system scheme in Data Engineering

AnalysisException: [ErrorClass=INVALID_PARAMETER_VALUE] Missing cloud file system scheme

Re: AnalysisException: [ErrorClass=INVALID_PARAMETER_VALUE] Missing cloud file system scheme