cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta Lake Demo - Not working

AJDJ
New Contributor III

Hi there,

I imported the delta lake demo notebook from databricks link and at command 12 it errors out. I tired other ways and path but couldnt get past the error. May be the notebook is outdated?

https://www.databricks.com/notebooks/Demo_Hub-Delta_Lake_Notebook.html?utm_source=youtube&utm_medium...

Thanks

9 REPLIES 9

Prabakar
Esteemed Contributor III

hi @AJ DJโ€‹ what is the error that you are getting?

AJDJ
New Contributor III

Thanks Prabakar, i have replied with attachment below.

Hubert-Dudek
Esteemed Contributor III

I've tested it, and the notebook is ok. The only error which I got is an error made in purpose:

image.png

AJDJ
New Contributor III

Thanks for your reply Prabakar and Hubert. Please find the screenshot. Does it have to something to do with community edition? Please note all previous commands got completed without errors (i did not do any changes i just imported the script and clicked RUN ALL).

dberror 

Hubert-Dudek
Esteemed Contributor III

Sadly it seems that the %sh magic command doesn't work in the community edition (I think as it shared community servers there is no access to underlying file system)

AJDJ
New Contributor III

Yes. I tried to browse dbfs files but couldnt visually see the files or the path. However when i try to read the file via python on the same path it works. So i was kind of confused. i had closed my windows. if you run python read from that temp path it works. its just for some reason cmd 12 doesnt pick it up.

AJDJ
New Contributor III

i tired to by pass the demo shell script and try to write my own.. i guess i'm able to read but may be im not writing it. im missing something.

%python

import pandas as pd

url = 'https://pages.databricks.com/rs/094-YMS-629/images/SAISEU19-loan-risks.snappy.parquet'

df = pd.read_parquet(url)

df.to_parquet('/dbfs/delta/snappy.parquet')

#df.write.format('delta').mode('overwrite').save('/delta/snappy')

next cmd

SELECT * FROm parquet.'/dbfs/delta/snappy'โ€‹ 

btw is there a way to share my notebook? or do we both need to be in the same organization and only then we can share notebook.

Hubert-Dudek
Esteemed Contributor III

I have included a screenshot with the working code. Pandas use dbfs as a folder, but in other places is used as a protocol.

image.png

Anonymous
Not applicable

Hi @AJ DJโ€‹ 

Does @Hubert Dudekโ€‹  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group