Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-18-2024 05:04 AM
Hi @Fatimah-Tariq ,
What about defining your DLT pipeline as below.
This way you will create a stream table that reads from your silver, apply all the needed changes and then write back to your silver.
%sql
-- Read from the streaming table in the silver schema
CREATE OR REFRESH STREAMING TABLE silver_stream
AS SELECT *
FROM STREAM(silver_schema.streaming_table);
-- Apply your logic update
-- ... perform necessary transformations ...
-- Delete faulty records
DELETE FROM silver_stream
WHERE <condition>;
-- Write the correct records back to the silver schema
CREATE OR REFRESH STREAMING TABLE silver_schema.streaming_table
AS SELECT *
FROM STREAM(silver_stream);