Delta Lake Demo - Not working

[AJDJ]

Hi there,

I imported the delta lake demo notebook from databricks link and at command 12 it errors out. I tired other ways and path but couldnt get past the error. May be the notebook is outdated?

https://www.databricks.com/notebooks/Demo_Hub-Delta_Lake_Notebook.html?utm_source=youtube&utm_medium...

Thanks

[Prabakar]

hi @AJ DJ​ what is the error that you are getting?

[AJDJ]

Thanks Prabakar, i have replied with attachment below.

[Hubert-Dudek]

I've tested it, and the notebook is ok. The only error which I got is an error made in purpose:

[AJDJ]

Thanks for your reply Prabakar and Hubert. Please find the screenshot. Does it have to something to do with community edition? Please note all previous commands got completed without errors (i did not do any changes i just imported the script and clicked RUN ALL).

 

[Hubert-Dudek]

Sadly it seems that the %sh magic command doesn't work in the community edition (I think as it shared community servers there is no access to underlying file system)

[AJDJ]

Yes. I tried to browse dbfs files but couldnt visually see the files or the path. However when i try to read the file via python on the same path it works. So i was kind of confused. i had closed my windows. if you run python read from that temp path it works. its just for some reason cmd 12 doesnt pick it up.

[AJDJ]

i tired to by pass the demo shell script and try to write my own.. i guess i'm able to read but may be im not writing it. im missing something.

%python

import pandas as pd

url = 'https://pages.databricks.com/rs/094-YMS-629/images/SAISEU19-loan-risks.snappy.parquet'

df = pd.read_parquet(url)

df.to_parquet('/dbfs/delta/snappy.parquet')

#df.write.format('delta').mode('overwrite').save('/delta/snappy')

next cmd

SELECT * FROm parquet.'/dbfs/delta/snappy'​ 

btw is there a way to share my notebook? or do we both need to be in the same organization and only then we can share notebook.

[Hubert-Dudek]

I have included a screenshot with the working code. Pandas use dbfs as a folder, but in other places is used as a protocol.

[Anonymous]

Hi @AJ DJ​ 

Does @Hubert Dudek​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!