topic Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv' in Data Engineering

i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Venky — Tue, 26 Oct 2021 08:40:50 GMT

i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

-werners- — Tue, 26 Oct 2021 08:55:31 GMT

This means the path to the file you typed is not valid, the file is not there.

Can you check in the Data/DBFS page if the file is there (or via dbutils.fs.ls)?

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Venky — Tue, 26 Oct 2021 09:20:07 GMT

No, file path is correct

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Venky — Tue, 26 Oct 2021 09:21:34 GMT

Please help me on this error

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

-werners- — Tue, 26 Oct 2021 09:36:51 GMT

can you try without the dbfs part?

so /Filestore/tables/world_bank.csv

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Venky — Tue, 26 Oct 2021 10:46:56 GMT

it's not working

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Hubert-Dudek — Tue, 26 Oct 2021 10:57:43 GMT

I see that you are using databricks-course-cluster which have probably some limited functionality. Not sure where dbfs is mounted there. When you are using dbutils it display path for dbfs mount (dbfs file system).

Please use spark code instead of pandas so it will be executed properly:

df = spark.read.csv('dbfs:/FileStore/tables/world_bank.csv')
display(df)

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Alexis — Wed, 27 Oct 2021 14:17:28 GMT

ops I didn't see the other answers, anyway here you have how to use %fs magic to do the same that dbutils.fs.ls() utils.

Just before to create the spark data frame, check if the file exists in the mentioned path.

You can use the %fs magic like this:

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

klllmmm — Tue, 24 May 2022 16:13:47 GMT

Pls help,

I have the same problem

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

-werners- — Wed, 25 May 2022 06:33:10 GMT

I see you use pandas to read from dbfs.

But pandas will only read from local files,

see this topic also. It is about databricks-connect but the same principles apply.

So what you should do is first read the file using spark.read.csv and then converting the spark df to a pandas df.

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Alexis — Wed, 25 May 2022 06:52:02 GMT

you can try:

my_df = spark.read.format("csv")

.option("inferSchema","true") # to get the types from your data

.option("sep",",") # if your file is using "," as separator

.option("header","true") # if your file have the header in the first row

.load("/FileStore/tables/CREDIT_1.CSV")

display(my_df)

from above you can see that my_df is a spark dataframe and from there you can start with you code.

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

KumarPalle — Thu, 18 Aug 2022 16:54:39 GMT

I am wondering how it worked for you. I have tried the same steps and no

luck. using dbutils i am finding file but when i read using pandas its saying no such file or directory exists.

dbutils.fs.ls("FileStore/shared_uploads/kumarpalle/Covid19Europedata.csv")

Please help here. uploaded the same file that am working on.

Also, please check if you have applied any settings that we are missing.

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Alexis — Thu, 18 Aug 2022 17:40:08 GMT

Hi Kumar,

you're almost there, try this:

df1 = pd.read_csv("/dbfs/FileStore/shared_uploads/kumarpalle/Covid19Europedata.csv")

df1.describe()

Cheers!

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

KumarPalle — Fri, 19 Aug 2022 10:00:06 GMT

Hi @Alexis ,

Thanks for quick response. It above method didn't worked but reading using spark and appending to Pandas it worked.

Here are the steps that i followed.

df1 = spark.read.format("csv").option("header", "true").load("dbfs:/FileStore/shared_uploads/kumarpalle/Covid19Europedata-1.csv").toPandas()

df1.head()

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

KumarPalle — Fri, 19 Aug 2022 10:01:29 GMT

@Venky D Please follow the above steps to read using Spark as pandas doesn't work through dbfs.

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/world_bank.csv'

Tico23 — Fri, 03 Mar 2023 16:01:04 GMT

same issue here... .toPandas() was also my only solution... otherwise error!

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError:

SanchesSan — Wed, 30 Aug 2023 15:46:39 GMT

I did the same as you, but the reading does not work.

Can you help me? thanks

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError:

Eagle78 — Wed, 07 Aug 2024 14:02:44 GMT

I had the same issue: geopandas in Databricks notebooks does not open shapefiles from an Azure Storage mount.
I managed to copy the shapefile to the Databricks workspace using

dbutils.fs.cp(shapefile_path, f"file:{local_shapefile_copy_dest_path}")

The 'file:' prefix proved to be crucial here.

and then:

gdf = gpd.read_file(shapefile_path.replace('dbfs:', '')) gdf.display()

I copy the results back to the dbfs mount using

dbutils.fs.cp(f"file:{geoparquet_path}", f"{raw_path}{geoparquet_file_basename}.parquet")

@-werners- wrote:
I see you use pandas to read from dbfs.
But pandas will only read from local files,
see this topic also. It is about databricks-connect but the same principles apply.
So what you should do is first read the file using spark.read.csv and then converting the spark df to a pandas df.

Re: i am trying to read csv file using databricks, i am getting error like ......FileNotFoundError:

Eagle78 — Thu, 08 Aug 2024 13:30:31 GMT

Ik convert to parquet using

gdf.to_parquet(f"/dbfs{raw_path}/{file_name}.parquet")