topic Re: how to partition my Delta table? in Data Engineering

how to partition my Delta table?

jose_gonzalez — Fri, 04 Jun 2021 19:12:26 GMT

I would like to follow best practices to partition my Delta table. Should I partition by unique ID or date?

Re: how to partition my Delta table?

User16783854357 — Mon, 07 Jun 2021 18:09:45 GMT

Partitioning depends on the use cases associated with reading and writing data in the table. If the reads and writes are done for a limited set of dates , having the date as a partition would help speed up processing the operations.

Re: how to partition my Delta table?

RonanStokes_DB — Mon, 07 Jun 2021 18:21:37 GMT

Depending on the amount of data per partition - you may also want to consider partitioning by week, month or quarter.

The partitioning decision is often tied to the tiering model of data storage. For a Bronze ingest layer, the optimal partitioning is to partition by some time value so that all data for a particular ingest is in the same partition.

For serving data - such as provided by the Gold tier, the optimal partitioning strategy is to partition so that queries require retrieval of data from the fewest partitions.

Depending on your consumption patterns, this may be best achieved by partitioning by time, region or some other business criteria.