cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Downstream delta live table is unable to read data frame from upstream table

arw1070
New Contributor II

I have been trying to work on implementing delta live tables to a pre-existing workflow. Currently trying to create two tables: appointments_raw and notes_raw, where notes_raw is "downstream" of appointments_raw. Following this as a reference, I'm attempting to load the appointments_raw table using dlt.read (inside notes_raw), but the result of dlt.read("appointments_raw") appears to be an empty DataFrame. Appointments raw data frame does seem to be read correctly according to pipeline storage and hive metastore. we are following this example: https://docs.databricks.com/_extras/notebooks/source/dlt-wikipedia-python.html

Specifically, where โ€œtop referring pagesโ€ code is referencing dlt.read(โ€œclickstream_preparedโ€). We are trying to do the same but facing an error.

image.png

2 REPLIES 2

Anonymous
Not applicable

@Anna Wuestโ€‹ : Could you please send me the code snippet here? Thanks.

arw1070
New Contributor II

Do you mean this?

@dlt.table(

comment="Raw table of appointments from EDW",

)

def appointments_raw():

return fetch_data.fetch_appointments(spark=spark, secret_handler=SECRET_HANDLER)

@dlt.table(

comment="Raw table of notes from SOLR",

)

def notes_raw():

appointments = dlt.read("appointments_raw")

print(type(appointments))

print(appointments.head())

appointments = appointments.pandas_api()

mrns = fetch_data.select_mrns(

appointments, today=TIMESTAMP, days_ahead=APPOINTMENTS_DAYS_AHEAD

)

notes = fetch_data.fetch_notes(

mrns, cohort_id=COHORT_ID, secret_handler=SECRET_HANDLER, spark=spark

)

return notes

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.