topic Re: DLT pipeline - reading from external tables in Data Engineering

DLT pipeline - reading from external tables

ksenija — Mon, 15 Jul 2024 08:43:09 GMT

Hello!

I created a DLT pipeline where my sources are external tables. I have to apply changes (stored_as_scd_type = 1). However, when I run my pipeline, I don't see any incremental uploads. The data remains in the same state as when I first created the pipeline. Do you have any suggestions?

Re: DLT pipeline - reading from external tables

lucasrocha — Mon, 15 Jul 2024 20:13:40 GMT

Hello @ksenija,

I hope this message finds you well.

Is your source table receiving new records? If so, are the fields (operation/sequenceNum) being filled?

If possible, please provide a sample of the code you are using to create your target table with apply changes.

Best regards,
Lucas Rocha

Re: DLT pipeline - reading from external tables

ksenija — Tue, 16 Jul 2024 07:40:57 GMT

Hello @lucasrocha ,

Yes, we are receiving new records in our tables and updated_at is always populated. We are using the same script to read data from foreign catalog and that DLT pipeline is working completely fine

dlt.apply_changes( target = target_table, source = source_table, keys = ["id"], sequence_by = col("updated_at"), stored_as_scd_type = "1" )

Thanks in advance!

Ksenija