topic Re: Using Delta Tables without Time Travel features? in Data Engineering

Using Delta Tables without Time Travel features?

Mark1 — Wed, 13 Apr 2022 15:45:01 GMT

Hi Everyone / Experts,

is it possible to use Delta Tables without the Time Travel features? We are primarily interested in using the DML Features (delete, update, merge into, etc)

Thanks,

Mark

Re: Using Delta Tables without Time Travel features?

Hubert-Dudek — Wed, 13 Apr 2022 16:00:10 GMT

You can use the below code to delete all time travel features (older snapshot) older than 1 hour. Better not use 0 hours as you can delete current write and have a data consistency problem. If you don't need ACID features, including the ability to do concurrent writes, you can consider using parquet instead.

from delta.tables import *
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")
deltaTable = DeltaTable.forPath(spark, deltaPath)
deltaTable.vacuum(1)

Re: Using Delta Tables without Time Travel features?

Mark1 — Thu, 14 Apr 2022 11:43:59 GMT

Thank you Hubert