I have been getting this error for the past few days while trying to create a table using parquet or csv options.
The error is :
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.io.IOException: s3a://AKIAJBRYNXGHORDHZB4A:a0BzE1bSegfydr3%2FGE3LSPM6uIV5A4hOUfpH8aFF@databricks-corp-training/common/online_retail/data-001/data.csv at com.databricks.backend.daemon.data.client.DbfsClient.send0(DbfsClient.scala:129) at com.databricks.backend.daemon.data.client.DbfsClient.sendIdempotent(DbfsClient.scala:69) at com.databricks.backend.daemon.data.client.DatabricksFileSystemV1.getFileStatus(DatabricksFileSystemV1.scala:272) at com.databricks.backend.daemon.data.client.DatabricksFileSystem.getFileStatus(DatabricksFileSystem.scala:212) at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1777) at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:60) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:407) at org.apache.spark.sql.execution.command.CreateDataSourceTableCommand.run(createDataSourceTables.scala:81) at org.apache.spark.sql.execution.command.ExecutedCommandExec.$anonfun$sideEffectResult$1(commands.scala:82) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:80) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:79) at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:91) at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$3(QueryExecution.scala:256)
My query is:
DROP TABLE IF EXISTS outdoorProductsRaw;
CREATE TABLE outdoorProductsRaw USING csv OPTIONS (
path "/mnt/training/online_retail/data-001/data.csv",
header "true"
)
