Data is not loaded when creating two different streaming table from one delta live table pipeline
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-21-2024 11:24 AM
i am trying to create 2 streaming tables in one DLT pipleine , both read json data from different locations and both have different schema , the pipeline executes but no data is inserted in both the tables.
where as when i try to run each table individually they execute perfectly
is it because DLT cannot process two different streaming table at once.?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-06-2024 09:24 PM
Delta Live Tables (DLT) can indeed process multiple streaming tables within a single pipeline.
Here are a few things to check:
1) Verify that each streaming table has a unique checkpoint location. Checkpointing is crucial for maintaining the state of streaming queries, and conflicts can arise if the same location is used for multiple streams.
2) Ensure that first_schema and second_schema are defined correctly and match the structure of the JSON data in first_source_path and second_source_path, respectively.
