DE 2.2 - Providing Options for External Sources - Classroom setup error

[Psybelo]

Hi All,

I am unable to execute "Classroom-Setup-02.2" setup in Data Engineering Course. There is the following error: FileNotFoundError: [errno 2] no such file or directory: '/dbfs/mnt/dbacademy-datasets/data-engineer-learning-path/v01/ecommerce/raw/users-historical'

I am using the community edition of Databricks, with ressource's Runtime of 11.3, scala 2.12.

%run ../Includes/Classroom-Setup-02.2

 

May I know if someone has a solution for it.

Thanks

[Anonymous]

Hi @Mari B​ 

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

[Psybelo]

Hi Vidula,

I'm not sure they can help since it's a Classroom setup problem (Databricks config)... But thanks btw.

[Zhihong]

I had the same problem. There is another post with the same issue. And I got answer there. See below.

Problems with pandas.read_parquet() and path - Databricks - 19655

[Eagle78]

Inspired by 
https://stackoverflow.com/questions/58984925/pandas-missing-read-parquet-function-in-azure-databrick...

I changed 

df = pd.read_parquet(path = datasource_path.replace("dbfs:/", '/dbfs/')) # original, error!

into

df = spark.read.format("parquet").load(datasource_path)

and 

df.to_sql('users', conn, if_exists='replace', index = False)

into

df.toPandas().to_sql('users', conn, if_exists='replace', index = False)

Then it works!