data_centers_q2_q3.snappy

William — Fri, 21 Mar 2025 11:42:25 GMT

I am writing to inquire about the following error:

om.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.AnalysisException: Unable to infer schema for Parquet. It must be specified manually.

at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$getOrInferFileFormatSchema$13(DataSource.scala:234)

at scala.Option.getOrElse(Option.scala:189)

at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:234)

at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:468)

at org.apache.spark.sql.execution.command.CreateDataSourceTableCommand.run(createDataSourceTables.scala:89)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)

at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:235)

at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3825)

at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$5(SQLExecution.scala:130)

at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:273)

at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:104)

at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:854)

at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:77)

at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:223)

at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3823)

at org.apache.spark.sql.Dataset.<init>(Dataset.scala:235)

at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:104)

at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:854)

Which is generated from the the following code (copied directly from the Databricks academy SQL training):

DROP TABLE IF EXISTS dc_data_raw;

CREATE TABLE dc_data_raw

USING parquet

OPTIONS (

PATH "/FileStore/Tables/data_centers_q2_q3.snappy.parquet"

);

I have done some investigation on my own but have turned up nothing that solves this. I have attached the downloaded version of the parquet file included for the course.

topic data_centers_q2_q3.snappy in Warehousing & Analytics

data_centers_q2_q3.snappy