topic When does delta lake actually compute a table? in Data Engineering

When does delta lake actually compute a table?

zeta_load — Fri, 17 Feb 2023 14:41:15 GMT

Maybe I'm completely wrong, but from my understanding delta lake only calculates a table at certain points, for instance when you display your data. Before that point, operations are only written to the log file and are not executed (meaning no changes are made to your data). If that assumption is correct, how can I force an actual computation?

Please correct me here if I missunderstood anything!

Re: When does delta lake actually compute a table?

Lakshay — Fri, 17 Feb 2023 16:17:11 GMT

Hi @Lukas Goldschmied , The operations are calculated when you execute them on the delta table. So, to elaborate when you do an insert operation, a new data file in parquet format is created with new records and the operation metrics are captured in delta log in JSON format. Now, when you run the select command on this table, it will use the information in the delta log to collect the records from the data files and display them

Re: When does delta lake actually compute a table?

Anonymous — Mon, 20 Feb 2023 06:03:47 GMT

Hi @Lukas Goldschmied

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!