Getting started with Delta Lake basics

[rachelnguyen]

Hi all — I am working through the Delta Lake intro and wanted to share a short note on using OPTIMIZE after merge. Happy to hear better patterns.

[data_pulse]

Did you forgot to share the short note? 🙂

If you are referring to running optimize after Merge, its worth mentioning that it doesn't need to be run after every Merge.

MERGE can create or rewrite files, but whether OPTIMIZE is needed immediately depends on the table size, write frequency, file layout, and query pattern. For frequently updated tables, its better to avoid.

Better options are: Databricks can already reduce small-file issues with optimized writes and auto compaction.

ALTER TABLE catalog.schema.table_name
SET TBLPROPERTIES (
  delta.autoOptimize.optimizeWrite = true,
  delta.autoOptimize.autoCompact = true
);

Then use OPTIMIZE based on actual file/query behavior or let Predictive Optimization manage maintenance automatically for UC managed tables. Reference