cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

I have created a DLT pipeline which  reads data from json files which are stored in databricks volum

zero234
New Contributor III

I have created a DLT pipeline which  reads data from json files which are stored in databricks volume and puts data into streaming table 
This was working fine.
when i tried to read the data that is inserted into the table and compare the values with the precalculated ones in the same dlt pipline its failing.

is it because dlt is treating this as an initialization stage and executing these comparism before setting up tables or inserting data into tables

 

2 REPLIES 2

Palash01
Contributor

Hey @zero234 

Yes, your assumption looks aligned to mine your pipeline reads data from JSON files, inserts it into a streaming table, and then tries to compare values in the table with pre-calculated values before any data has been written. This leads to a comparison with an empty table, resulting in the error.

Possible Solution:

  1. Don't perform the comparison within the same notebook as table creation. Create a separate notebook or trigger that runs after the table has received data. This ensures comparison happens only when there's actual data to compare. You can also set this using a spark job which can help you trigger the dlt pipeline first and comparison afterwards.
  2. Before comparing, modify the comparison code to explicitly check if the streaming table has received any data. You can use table.isEmpty() or similar logic to confirm if there's data before proceeding.

 

Leave a like if this helps! Kudos,
Palash

AmanSehgal
Honored Contributor III

Keep your DLT code separate from your comparison code, and run your comparison code once your DLT data has been ingested.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.