topic pd.read_csv failed in Data Engineering

pd.read_csv failed

THIAM_HUATTAN — Wed, 22 Jun 2022 07:52:14 GMT

https://i.imgur.com/PAGzSr9.png

Re: pd.read_csv failed

Hubert-Dudek — Thu, 23 Jun 2022 20:12:34 GMT

Pandas will not see dbfs storage this way. You need to change to pandas on spark or use the classic path.

import pyspark.pandas as pd
pd.read_csv("dbfs:/your_path/your_file.csv")
 
import pandas as pd
pd.read_csv("/dbfs/your_path/your_file.csv")

Re: pd.read_csv failed

THIAM_HUATTAN — Fri, 24 Jun 2022 11:43:38 GMT

Thanks for your kind reply:

Below works for me:

https://imgur.com/BmMzatI

But why, as you mentioned, using the classic path, below does not work?

https://imgur.com/Ba1a4Iv

Re: pd.read_csv failed

jose_gonzalez — Fri, 08 Jul 2022 00:05:41 GMT

Hi @THIAM HUAT TAN ,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.