cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

I have created a DLT pipeline whichย  reads data from json files which are stored in databricks volum

zero234
New Contributor III

I have created a DLT pipeline which  reads data from json files which are stored in databricks volume and puts data into streaming table 
This was working fine.
when i tried to read the data that is inserted into the table and compare the values with the precalculated ones in the same dlt pipline its failing.

is it because dlt is treating this as an initialization stage and executing these comparism before setting up tables or inserting data into tables

 

2 REPLIES 2

Palash01
Valued Contributor

Hey @zero234 

Yes, your assumption looks aligned to mine your pipeline reads data from JSON files, inserts it into a streaming table, and then tries to compare values in the table with pre-calculated values before any data has been written. This leads to a comparison with an empty table, resulting in the error.

Possible Solution:

  1. Don't perform the comparison within the same notebook as table creation. Create a separate notebook or trigger that runs after the table has received data. This ensures comparison happens only when there's actual data to compare. You can also set this using a spark job which can help you trigger the dlt pipeline first and comparison afterwards.
  2. Before comparing, modify the comparison code to explicitly check if the streaming table has received any data. You can use table.isEmpty() or similar logic to confirm if there's data before proceeding.

 

Leave a like if this helps! Kudos,
Palash

AmanSehgal
Honored Contributor III

Keep your DLT code separate from your comparison code, and run your comparison code once your DLT data has been ingested.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group