โ06-17-2021 12:36 AM
I am thinking of using delta live table, before that I want to be aware of the limitations it has as of now when it s announced on datasummit 2021
โ06-18-2021 11:30 AM
It currently supports Python and SQL. The following functionalities are currently not supported with Delta Live Tables:
โ06-18-2021 11:30 AM
It currently supports Python and SQL. The following functionalities are currently not supported with Delta Live Tables:
โ11-28-2022 07:42 AM
Also found out today that you can't delete any data from your delta live tables or else the process fails with an error:
......
This is currently not supported. If you'd like to ignore deletes, set the option 'ignoreDeletes' to 'true'.
It's not possible to set the ignoreDeletes to true either.
So, unless you're happy to keep the data in your Delta Live Tables forever, you've got a little problem. We're looking at workarounds but nothing pretty yet.
โ02-02-2023 11:30 AM
There doesn't appear to be a way to enforce a retention policy on source tables when defining a structured stream. Setting the options for "ignoreChanges" and "ignoreDeletes" doesn't seem to have any effect at all. CDC does not fill this role either, as it only lets you change sink operations based on source data.
The fact that this STILL doesn't support any sort of data retention/pruning strategy in structure stream sources has us investing a lot of time in figuring how to undo all of our investments into DLT and abandon it all together for regular old structure streams.
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!