cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT using the result of one view in another table with collect()

140015
New Contributor III

Hey,

Do you guys know, if there is an option to implement something like this in DLT:

@dlt.view()

def view_1():

# some calculations that return a small dataframe with around max 80 rows

@dlt.table()

def table_1():

result_df = dlt.read("view_1")

results = [row[0] for row in result_df.collect()]

for result in results:

# do some more calculations where results are used

I saw this point in the documentation https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-python-ref.html#limitation..., and I guess it is why this is not working as supposed to (check it with logger and results during calculations is an empty list) I'm not sure what does it mean that it is possible to use collect() outside table or view function, what it gives? I can't do

result_df = dlt.read("view_1")

results = [row[0] for row in result_df.collect()]

outside the table/view function. Is there and solution to resolve something like that in DLT?

0 REPLIES 0
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!