Updating a streaming table in dlt

ashraf1395 — Mon, 16 Dec 2024 11:45:37 GMT

Can we update a streaming table in dlt my source and target will be same. That is the update should be made on the same table. If yes then can you guide me how.

I tried append_flow but it just appends data

CDC I am not sure whether we can have both target and source table same there or not

Re: Updating a streaming table in dlt

Walter_C — Mon, 16 Dec 2024 11:59:27 GMT

You can define the CDC flow to update the streaming table. This involves reading from the same table and applying changes.

@Dlt.table( name="my_streaming_table", comment="This table is updated using CDC", table_properties={"quality": "silver"} ) @dlt.expect_or_drop("valid_data", "column_name IS NOT NULL") def update_table(): source_df = dlt.read_stream("my_streaming_table") changes_df = source_df.filter("change_type = 'update'") return changes_df

Ensure your pipeline is configured to use the CDC functionality. This can be done in the pipeline settings.

topic Updating a streaming table in dlt in Data Engineering

Updating a streaming table in dlt

Re: Updating a streaming table in dlt