cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Limitation as of now in delta live table

User16826994223
Honored Contributor III

I am thinking of using delta live table, before that I want to be aware of the limitations it has as of now when it s announced on datasummit 2021

1 ACCEPTED SOLUTION

Accepted Solutions

Anonymous
Not applicable

It currently supports Python and SQL. The following functionalities are currently not supported with Delta Live Tables: 

  • updates/deletes (MERGE)
  • field level lineage
  • auto-scaling

View solution in original post

3 REPLIES 3

Anonymous
Not applicable

It currently supports Python and SQL. The following functionalities are currently not supported with Delta Live Tables: 

  • updates/deletes (MERGE)
  • field level lineage
  • auto-scaling

PeteC
New Contributor III

Also found out today that you can't delete any data from your delta live tables or else the process fails with an error:

......

This is currently not supported. If you'd like to ignore deletes, set the option 'ignoreDeletes' to 'true'.

It's not possible to set the ignoreDeletes to true either.

So, unless you're happy to keep the data in your Delta Live Tables forever, you've got a little problem. We're looking at workarounds but nothing pretty yet.

Zachary_Higgins
Contributor

There doesn't appear to be a way to enforce a retention policy on source tables when defining a structured stream. Setting the options for "ignoreChanges" and "ignoreDeletes" doesn't seem to have any effect at all. CDC does not fill this role either, as it only lets you change sink operations based on source data.

The fact that this STILL doesn't support any sort of data retention/pruning strategy in structure stream sources has us investing a lot of time in figuring how to undo all of our investments into DLT and abandon it all together for regular old structure streams.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.