cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databricks notebooks error

vivek_purbey
New Contributor II

I want to read a csv file using pandas library in python in Databricks Notebooks and I uploaded my csv file (employee_data) on adfs but it still shows no such file exists can anyone help me on this?

vivek_purbey_0-1749739088896.png

8 REPLIES 8

nikhilj0421
Databricks Employee
Databricks Employee

Hi @vivek_purbey, can you try reading with the path below? It takes '/dbfs' by default, you do not need to pass it explicitly. 

pd. read_csv('/FileStore/tables/employee_data.csv')

Still had the same issue.

vivek_purbey_0-1749746775458.png

 

Can you try uploading this file outside FileStore and give it a try? Maybe create a new folder in dbfs, and upload the files there?

can you please guide me how I can do that

You can create a new folder using like below:

 

dbutils.fs.mkdirs("/folder_name/")

 

Then, you can upload the file here manually the way you were uploading the FileStore. 
And then you can read the file using below code

 

pd.read_csv('/folder_name/employee_data.csv')

 

 

UmaMahesh1
Honored Contributor III

I think you are using databricks course cluster which might have some limitations.
Pandas reads only from local files. So try to read the file using spark instead of pandas.

df = spark.read.csv('dbfs:/FileStore/tables/employee_data.csv')
display(df)

You can then convert it to pandas df if it is absolutely needed using toPandas.

Uma Mahesh D

Sure, Thank You.

Alok0903
New Contributor II

Load it using PySpark and create a pandas data frame. Here is how you do it after uploading the data

file_path = "/FileStore/tables/your_file_name.csv"

# Load CSV as Spark DataFrame
df_spark = spark.read.option("header", "true").option("inferSchema", "true").csv(file_path)

# Convert to pandas DataFrame
df_pandas = df_spark.toPandas()

Alok K Pandey

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now