Why does spark.read.csv come back with an error: com.databricks.sql.io.FileReadException: Error while reading file dbfs:/mnt/cntnr/demo/circuits.csv ?

[jch]

I need help understanding why I can't open a file.

In a databricks notebook, I use this code:

%fs
 
ls /mnt/cntnr/demo

I get back dbfs:/mnt/cntnr/demo/circuits.csv as one of the path values.

When I use this code, I get an error:

circuits_df = spark.read.csv("dbfs:/mnt/cntnr/demo/circuits.csv")

Any suggestions?

ANy suggestion

[daniel_sahal]

@jch​ 

Try spark.read.csv without dbfs:

You can learn more here:

https://docs.databricks.com/files/index.html

[jch]

This didn't work either

circuits_df = spark.read.csv("/mnt/cntnr/demo/circuits.csv")

[Shivanshu_Tiwar · Accepted solution]

Try this

circuits_df = spark.read.csv("/dbfs/mnt/cntnr/demo/circuits.csv")

[jch]

It turns out my spark config was wrong

    #Set Spark configuration

    configs = {"fs.azure.account.auth.type": "OAuth",

          "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",

          "fs.azure.account.oauth2.client.id": client_ID,

          "fs.azure.account.oauth2.client.secret": client_secret,

          "fs.azure.account.oauth2.client.endpoint": f"https://login.microsoftonline.com/{tenant_ID}/oauth2/token"}