topic Re: Databricks notebooks error in Data Engineering

Databricks notebooks error

vivek_purbey — Thu, 12 Jun 2025 14:40:49 GMT

I want to read a csv file using pandas library in python in Databricks Notebooks and I uploaded my csv file (employee_data) on adfs but it still shows no such file exists can anyone help me on this?

Re: Databricks notebooks error

nikhilj0421 — Thu, 12 Jun 2025 16:15:37 GMT

Hi @vivek_purbey, can you try reading with the path below? It takes '/dbfs' by default, you do not need to pass it explicitly.

pd. read_csv('/FileStore/tables/employee_data.csv')

Re: Databricks notebooks error

vivek_purbey — Thu, 12 Jun 2025 16:46:50 GMT

Still had the same issue.

Re: Databricks notebooks error

nikhilj0421 — Thu, 12 Jun 2025 17:07:16 GMT

Can you try uploading this file outside FileStore and give it a try? Maybe create a new folder in dbfs, and upload the files there?

Re: Databricks notebooks error

vivek_purbey — Thu, 12 Jun 2025 17:17:43 GMT

can you please guide me how I can do that

Re: Databricks notebooks error

nikhilj0421 — Thu, 12 Jun 2025 17:33:09 GMT

You can create a new folder using like below:

dbutils.fs.mkdirs("/folder_name/")

Then, you can upload the file here manually the way you were uploading the FileStore.
And then you can read the file using below code

pd.read_csv('/folder_name/employee_data.csv')

Re: Databricks notebooks error

UmaMahesh1 — Thu, 12 Jun 2025 19:40:30 GMT

I think you are using databricks course cluster which might have some limitations.
Pandas reads only from local files. So try to read the file using spark instead of pandas.

df = spark.read.csv('dbfs:/FileStore/tables/employee_data.csv') display(df)

You can then convert it to pandas df if it is absolutely needed using toPandas.

Re: Databricks notebooks error

vivek_purbey — Fri, 13 Jun 2025 06:43:12 GMT

Sure, Thank You.

Re: Databricks notebooks error

Alok0903 — Fri, 13 Jun 2025 08:18:07 GMT

Load it using PySpark and create a pandas data frame. Here is how you do it after uploading the data

file_path = "/FileStore/tables/your_file_name.csv"

# Load CSV as Spark DataFrame
df_spark = spark.read.option("header", "true").option("inferSchema", "true").csv(file_path)

# Convert to pandas DataFrame
df_pandas = df_spark.toPandas()