https://community.databricks.com/t5/get-started-discussions/com-databricks-backend-common-rpc-sparkdriverexceptions/m-p/76426#M7636 <P>I have been getting this error for the past few days while trying to create a table using parquet or csv options.&nbsp;</P><P>The error is :</P><DIV><DIV class="">IOException: <A class="" href="s3a://AKIAJBRYNXGHORDHZB4A:a0BzE1bSegfydr3%2FGE3LSPM6uIV5A4hOUfpH8aFF@databricks-corp-training/common/online_retail/data-001/data.csv" target="_blank" rel="noopener noreferrer">s3a://AKIAJBRYNXGHORDHZB4A:a0BzE1bSegfydr3%2FGE3LSPM6uIV5A4hOUfpH8aFF@databricks-corp-training/common/online_retail/data-001/data.csv</A></DIV></DIV><DIV class=""><DIV class=""><DIV class="">com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.io.IOException: s3a://AKIAJBRYNXGHORDHZB4A:a0BzE1bSegfydr3%2FGE3LSPM6uIV5A4hOUfpH8aFF@databricks-corp-training/common/online_retail/data-001/data.csv at com.databricks.backend.daemon.data.client.DbfsClient.send0(DbfsClient.scala:129) at com.databricks.backend.daemon.data.client.DbfsClient.sendIdempotent(DbfsClient.scala:69) at com.databricks.backend.daemon.data.client.DatabricksFileSystemV1.getFileStatus(DatabricksFileSystemV1.scala:272) at com.databricks.backend.daemon.data.client.DatabricksFileSystem.getFileStatus(DatabricksFileSystem.scala:212) at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1777) at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:60) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:407) at org.apache.spark.sql.execution.command.CreateDataSourceTableCommand.run(createDataSourceTables.scala:81) at org.apache.spark.sql.execution.command.ExecutedCommandExec.$anonfun$sideEffectResult$1(commands.scala:82) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:80) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:79) at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:91) at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$3(QueryExecution.scala:256)<BR /><BR />My query is:</DIV><DIV class=""><DIV><DIV><SPAN>DROP</SPAN> <SPAN>TABLE</SPAN> <SPAN>IF</SPAN> <SPAN>EXISTS</SPAN><SPAN> outdoorProductsRaw;</SPAN></DIV><DIV><SPAN>CREATE</SPAN> <SPAN>TABLE</SPAN><SPAN> outdoorProductsRaw </SPAN><SPAN>USING</SPAN><SPAN> csv </SPAN><SPAN>OPTIONS</SPAN><SPAN> (</SPAN></DIV><DIV><SPAN>path </SPAN><SPAN>"/mnt/training/online_retail/data-001/data.csv"</SPAN><SPAN>,</SPAN></DIV><DIV><SPAN>header </SPAN><SPAN>"true"</SPAN></DIV><DIV><SPAN>)</SPAN></DIV></DIV></DIV></DIV></DIV> Tue, 02 Jul 2024 04:33:55 GMT DMJ 2024-07-02T04:33:55Z https://community.databricks.com/t5/get-started-discussions/com-databricks-backend-common-rpc-sparkdriverexceptions/m-p/76426#M7636 <P>I have been getting this error for the past few days while trying to create a table using parquet or csv options.&nbsp;</P><P>The error is :</P><DIV><DIV class="">IOException: <A class="" href="s3a://AKIAJBRYNXGHORDHZB4A:a0BzE1bSegfydr3%2FGE3LSPM6uIV5A4hOUfpH8aFF@databricks-corp-training/common/online_retail/data-001/data.csv" target="_blank" rel="noopener noreferrer">s3a://AKIAJBRYNXGHORDHZB4A:a0BzE1bSegfydr3%2FGE3LSPM6uIV5A4hOUfpH8aFF@databricks-corp-training/common/online_retail/data-001/data.csv</A></DIV></DIV><DIV class=""><DIV class=""><DIV class="">com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.io.IOException: s3a://AKIAJBRYNXGHORDHZB4A:a0BzE1bSegfydr3%2FGE3LSPM6uIV5A4hOUfpH8aFF@databricks-corp-training/common/online_retail/data-001/data.csv at com.databricks.backend.daemon.data.client.DbfsClient.send0(DbfsClient.scala:129) at com.databricks.backend.daemon.data.client.DbfsClient.sendIdempotent(DbfsClient.scala:69) at com.databricks.backend.daemon.data.client.DatabricksFileSystemV1.getFileStatus(DatabricksFileSystemV1.scala:272) at com.databricks.backend.daemon.data.client.DatabricksFileSystem.getFileStatus(DatabricksFileSystem.scala:212) at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1777) at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:60) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:407) at org.apache.spark.sql.execution.command.CreateDataSourceTableCommand.run(createDataSourceTables.scala:81) at org.apache.spark.sql.execution.command.ExecutedCommandExec.$anonfun$sideEffectResult$1(commands.scala:82) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:80) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:79) at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:91) at org.apache.spark.sql.execution.QueryExecution$$anonfun$$nestedInanonfun$eagerlyExecuteCommands$1$1.$anonfun$applyOrElse$3(QueryExecution.scala:256)<BR /><BR />My query is:</DIV><DIV class=""><DIV><DIV><SPAN>DROP</SPAN> <SPAN>TABLE</SPAN> <SPAN>IF</SPAN> <SPAN>EXISTS</SPAN><SPAN> outdoorProductsRaw;</SPAN></DIV><DIV><SPAN>CREATE</SPAN> <SPAN>TABLE</SPAN><SPAN> outdoorProductsRaw </SPAN><SPAN>USING</SPAN><SPAN> csv </SPAN><SPAN>OPTIONS</SPAN><SPAN> (</SPAN></DIV><DIV><SPAN>path </SPAN><SPAN>"/mnt/training/online_retail/data-001/data.csv"</SPAN><SPAN>,</SPAN></DIV><DIV><SPAN>header </SPAN><SPAN>"true"</SPAN></DIV><DIV><SPAN>)</SPAN></DIV></DIV></DIV></DIV></DIV> Tue, 02 Jul 2024 04:33:55 GMT https://community.databricks.com/t5/get-started-discussions/com-databricks-backend-common-rpc-sparkdriverexceptions/m-p/76426#M7636 DMJ 2024-07-02T04:33:55Z