topic Re: Data is not loaded when creating two different streaming table from one delta live table pipelin in Data Engineering

Data is not loaded when creating two different streaming table from one delta live table pipeline

zero234 — Wed, 21 Feb 2024 19:24:23 GMT

i am trying to create 2 streaming tables in one DLT pipleine , both read json data from different locations and both have different schema , the pipeline executes but no data is inserted in both the tables.
where as when i try to run each table individually they execute perfectly

is it because DLT cannot process two different streaming table at once.?

DF = spark.readStream.format("json") \

.schema(schema) \

.option("header", True) \

.option("nullValue", "") \

.load(source_path + "/*.json")

Re: Data is not loaded when creating two different streaming table from one delta live table pipelin

Mounika_Tarigop — Sat, 07 Dec 2024 05:24:58 GMT

Delta Live Tables (DLT) can indeed process multiple streaming tables within a single pipeline.

Here are a few things to check:

1) Verify that each streaming table has a unique checkpoint location. Checkpointing is crucial for maintaining the state of streaming queries, and conflicts can arise if the same location is used for multiple streams.

2) Ensure that first_schema and second_schema are defined correctly and match the structure of the JSON data in first_source_path and second_source_path, respectively.