topic Re: Can we run pandas dataframe inside databricks? in Data Engineering

Can we run pandas dataframe inside databricks?

tinendra — Fri, 28 Oct 2022 13:47:42 GMT

Hi, I want to run

df=pd.read_csv('/dbfs/FileStore/airlines1.csv') while trying to run getting error like

FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/airlines1.csv'

Could you please help me out how to run pandas dataframe inside databricks or we can not run pandas dataframe directly inside databricks?

Re: Can we run pandas dataframe inside databricks?

karthik_p — Fri, 28 Oct 2022 14:21:16 GMT

@Tinendra Kumar we have been seeing this issue, but as far as documentation i don't see any solid base why pandas read won't support. but we do have workaround 1. read your csv using spark and store in data frame 2. convert spark data frame to pandas . below links will help 1. CSV file | Databricks on AWS 2. Convert a spark DataFrame to pandas DF - Stack Overflow

Re: Can we run pandas dataframe inside databricks?

jose_gonzalez — Mon, 31 Oct 2022 18:24:12 GMT

Hi @Tinendra Kumar,

I would like to share the following documentation https://docs.databricks.com/languages/pandas-spark.html#pandas-api-on-spark This is Pandas APIs on Spark.

Pandas does not scale out to big data (runs on driver only). Pandas API on Spark fills this gap by providing pandas equivalent APIs that work on Apache Spark. Pandas API on Spark is useful not only for pandas users but also PySpark users, because pandas API on Spark supports many tasks that are difficult to do with PySpark, for example plotting data directly from a PySpark DataFrame.

Re: Can we run pandas dataframe inside databricks?

tinendra — Fri, 11 Nov 2022 03:48:39 GMT

Thank you Jose

Re: Can we run pandas dataframe inside databricks?

tinendra — Fri, 11 Nov 2022 04:01:14 GMT

Thanks, Karthik

Re: Can we run pandas dataframe inside databricks?

Anonymous — Mon, 21 Nov 2022 02:09:06 GMT

Hi,

I don't see any incorrect in your code, I think your code is correct. Could you check the exist of your file or enable dbfs in your workspace?

Re: Can we run pandas dataframe inside databricks?

Anonymous — Mon, 09 Jan 2023 05:10:36 GMT

Hi @Tinendra Kumar

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Can we run pandas dataframe inside databricks?

tinendra — Tue, 14 Feb 2023 12:10:07 GMT

Hi @Vidula Khanna

Yes, my query has been resolved and I got the solution. Thanks for your support.

Thanks,

Tinendra