DLT using the result of one view in another table with collect()

140015 — Mon, 19 Dec 2022 13:36:37 GMT

Hey,

Do you guys know, if there is an option to implement something like this in DLT:

@dlt.view()

def view_1():

# some calculations that return a small dataframe with around max 80 rows

@dlt.table()

def table_1():

result_df = dlt.read("view_1")

results = [row[0] for row in result_df.collect()]

for result in results:

# do some more calculations where results are used

I saw this point in the documentation https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-python-ref.html#limitations, and I guess it is why this is not working as supposed to (check it with logger and results during calculations is an empty list) I'm not sure what does it mean that it is possible to use collect() outside table or view function, what it gives? I can't do

result_df = dlt.read("view_1")

results = [row[0] for row in result_df.collect()]

outside the table/view function. Is there and solution to resolve something like that in DLT?

topic DLT using the result of one view in another table with collect() in Data Engineering