topic Re: Delta Lake Demo - Not working in Data Engineering

Delta Lake Demo - Not working

AJDJ — Fri, 30 Sep 2022 22:35:41 GMT

Hi there,

I imported the delta lake demo notebook from databricks link and at command 12 it errors out. I tired other ways and path but couldnt get past the error. May be the notebook is outdated?

https://www.databricks.com/notebooks/Demo_Hub-Delta_Lake_Notebook.html?utm_source=youtube&utm_medium=web&utm_campaign=7013f000000cVKYAA2

Thanks

Re: Delta Lake Demo - Not working

Prabakar — Sun, 02 Oct 2022 10:55:19 GMT

hi @AJ DJ what is the error that you are getting?

Re: Delta Lake Demo - Not working

Hubert-Dudek — Sun, 02 Oct 2022 18:30:30 GMT

I've tested it, and the notebook is ok. The only error which I got is an error made in purpose:

Re: Delta Lake Demo - Not working

AJDJ — Mon, 03 Oct 2022 18:02:33 GMT

Thanks for your reply Prabakar and Hubert. Please find the screenshot. Does it have to something to do with community edition? Please note all previous commands got completed without errors (i did not do any changes i just imported the script and clicked RUN ALL).

Re: Delta Lake Demo - Not working

AJDJ — Mon, 03 Oct 2022 18:03:37 GMT

Thanks Prabakar, i have replied with attachment below.

Re: Delta Lake Demo - Not working

Hubert-Dudek — Mon, 03 Oct 2022 20:04:07 GMT

Sadly it seems that the %sh magic command doesn't work in the community edition (I think as it shared community servers there is no access to underlying file system)

Re: Delta Lake Demo - Not working

AJDJ — Mon, 03 Oct 2022 20:09:45 GMT

Yes. I tried to browse dbfs files but couldnt visually see the files or the path. However when i try to read the file via python on the same path it works. So i was kind of confused. i had closed my windows. if you run python read from that temp path it works. its just for some reason cmd 12 doesnt pick it up.

Re: Delta Lake Demo - Not working

AJDJ — Mon, 03 Oct 2022 20:22:57 GMT

i tired to by pass the demo shell script and try to write my own.. i guess i'm able to read but may be im not writing it. im missing something.

%python

import pandas as pd

url = 'https://pages.databricks.com/rs/094-YMS-629/images/SAISEU19-loan-risks.snappy.parquet'

df = pd.read_parquet(url)

df.to_parquet('/dbfs/delta/snappy.parquet')

#df.write.format('delta').mode('overwrite').save('/delta/snappy')

next cmd

SELECT * FROm parquet.'/dbfs/delta/snappy'

btw is there a way to share my notebook? or do we both need to be in the same organization and only then we can share notebook.

Re: Delta Lake Demo - Not working

Hubert-Dudek — Tue, 11 Oct 2022 14:06:17 GMT

I have included a screenshot with the working code. Pandas use dbfs as a folder, but in other places is used as a protocol.

Re: Delta Lake Demo - Not working

Anonymous — Sun, 23 Oct 2022 06:11:37 GMT

Hi @AJ DJ

Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!