topic Re: Limitation as of now in delta live table in Data Engineering

Limitation as of now in delta live table

User16826994223 — Thu, 17 Jun 2021 07:36:49 GMT

I am thinking of using delta live table, before that I want to be aware of the limitations it has as of now when it s announced on datasummit 2021

Re: Limitation as of now in delta live table

Anonymous — Fri, 18 Jun 2021 18:30:31 GMT

It currently supports Python and SQL. The following functionalities are currently not supported with Delta Live Tables:

updates/deletes (MERGE)
field level lineage
auto-scaling

Re: Limitation as of now in delta live table

PeteC — Mon, 28 Nov 2022 15:42:57 GMT

Also found out today that you can't delete any data from your delta live tables or else the process fails with an error:

......

This is currently not supported. If you'd like to ignore deletes, set the option 'ignoreDeletes' to 'true'.

It's not possible to set the ignoreDeletes to true either.

So, unless you're happy to keep the data in your Delta Live Tables forever, you've got a little problem. We're looking at workarounds but nothing pretty yet.

Re: Limitation as of now in delta live table

Zachary_Higgins — Thu, 02 Feb 2023 19:30:23 GMT

There doesn't appear to be a way to enforce a retention policy on source tables when defining a structured stream. Setting the options for "ignoreChanges" and "ignoreDeletes" doesn't seem to have any effect at all. CDC does not fill this role either, as it only lets you change sink operations based on source data.

The fact that this STILL doesn't support any sort of data retention/pruning strategy in structure stream sources has us investing a lot of time in figuring how to undo all of our investments into DLT and abandon it all together for regular old structure streams.