topic Re: Why does spark.read.csv come back with an error: com.databricks.sql.io.FileReadException: Error while reading file dbfs:/mnt/cntnr/demo/circuits.csv ? in Data Engineering

Why does spark.read.csv come back with an error: com.databricks.sql.io.FileReadException: Error while reading file dbfs:/mnt/cntnr/demo/circuits.csv ?

jch — Mon, 15 May 2023 21:48:30 GMT

I need help understanding why I can't open a file.

In a databricks notebook, I use this code:

%fs
 
ls /mnt/cntnr/demo

I get back dbfs:/mnt/cntnr/demo/circuits.csv as one of the path values.

When I use this code, I get an error:

circuits_df = spark.read.csv("dbfs:/mnt/cntnr/demo/circuits.csv")

Any suggestions?

ANy suggestion

daniel_sahal — Tue, 16 May 2023 05:38:54 GMT

@jch

Try spark.read.csv without dbfs:

You can learn more here:

jch — Thu, 18 May 2023 15:54:43 GMT

This didn't work either

circuits_df = spark.read.csv("/mnt/cntnr/demo/circuits.csv")

Shivanshu_ — Tue, 06 Jun 2023 06:14:50 GMT

Try this

circuits_df = spark.read.csv("/dbfs/mnt/cntnr/demo/circuits.csv")

jch — Wed, 21 Jun 2023 12:56:15 GMT

It turns out my spark config was wrong

#Set Spark configuration

configs = {"fs.azure.account.auth.type": "OAuth",

"fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",

"fs.azure.account.oauth2.client.id": client_ID,

"fs.azure.account.oauth2.client.secret": client_secret,

"fs.azure.account.oauth2.client.endpoint": f"https://login.microsoftonline.com/{tenant_ID}/oauth2/token"}