cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Limitation as of now in delta live table

User16826994223
Honored Contributor III

I am thinking of using delta live table, before that I want to be aware of the limitations it has as of now when it s announced on datasummit 2021

1 ACCEPTED SOLUTION

Accepted Solutions

Anonymous
Not applicable

It currently supports Python and SQL. The following functionalities are currently not supported with Delta Live Tables: 

  • updates/deletes (MERGE)
  • field level lineage
  • auto-scaling

View solution in original post

3 REPLIES 3

Anonymous
Not applicable

It currently supports Python and SQL. The following functionalities are currently not supported with Delta Live Tables: 

  • updates/deletes (MERGE)
  • field level lineage
  • auto-scaling

PeteC
New Contributor III

Also found out today that you can't delete any data from your delta live tables or else the process fails with an error:

......

This is currently not supported. If you'd like to ignore deletes, set the option 'ignoreDeletes' to 'true'.

It's not possible to set the ignoreDeletes to true either.

So, unless you're happy to keep the data in your Delta Live Tables forever, you've got a little problem. We're looking at workarounds but nothing pretty yet.

Zachary_Higgins
Contributor

There doesn't appear to be a way to enforce a retention policy on source tables when defining a structured stream. Setting the options for "ignoreChanges" and "ignoreDeletes" doesn't seem to have any effect at all. CDC does not fill this role either, as it only lets you change sink operations based on source data.

The fact that this STILL doesn't support any sort of data retention/pruning strategy in structure stream sources has us investing a lot of time in figuring how to undo all of our investments into DLT and abandon it all together for regular old structure streams.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!