topic I want to consolidate the delta tables historical versions and write to another table in append mode in Data Engineering

I want to consolidate the delta tables historical versions and write to another table in append mode

amitprasad01 — Thu, 17 Oct 2024 16:18:58 GMT

Hi Team,

I have a table let's say, employee and it has 5 versions .
version 0 and 1 has column A, column B,
from version 2 column A has been changed to column C and column B to column D.
I want to ingest version by version without manual intervention. Can you please suggest the best practice here?

Re: I want to consolidate the delta tables historical versions and write to another table in append

Lakshay — Thu, 17 Oct 2024 17:55:30 GMT

Hi @amitprasad01 , When you write to a new table, do you want to write the data from column A and column C to one single column A or you want to create two separate columns named A and C?

Re: I want to consolidate the delta tables historical versions and write to another table in append

Manoj43774 — Thu, 17 Oct 2024 18:04:05 GMT

Do you want to ingest column a, column b and column a as c at a same time or you want to ingest version 1 then version 2 and so on??

Re: I want to consolidate the delta tables historical versions and write to another table in append

uday_satapathy — Thu, 17 Oct 2024 19:17:51 GMT

Have you checked out deltaTable.cloneAtVersion()?

Re: I want to consolidate the delta tables historical versions and write to another table in append

amitprasad01 — Fri, 18 Oct 2024 05:59:02 GMT

@Lakshay I want to create everything into latest columns that means column a will become column c and column b will become column D