topic Re: Ingestion Time Clustering on initial load in Data Engineering

Ingestion Time Clustering on initial load

dvmentalmadess — Thu, 29 Jun 2023 00:03:32 GMT

We are migrating our data into Databricks and I was looking at the recommendations for partitioning here: https://docs.databricks.com/tables/partitions.html. This recommends not specifying partitioning and allowing "Ingestion Time Partitioning" (ITP) to optimize data files. However, details on this are slim. It seems like the data should be optimized by some timestamp but at the time of initial load the data all has the same ingest time. The data sources will mainly be change data capture data which will have an `updated_on` or other transaction timestamp that queries will use to get the latest copy of each record. I'd like to take advantage of ITP, but on the initial load it seems like I should indicate a specific column that should be used - like when I would call optimize. Should I just call optimize after my initial load? If I do will that disable ITP for the table? If not, what is the recommendation?

Re: Ingestion Time Clustering on initial load

Oliver_Angelil — Sat, 22 Jul 2023 18:14:23 GMT

@dvmentalmadess did you get anywhere with this? I'm having the same questions...

Feel free to reach out to me on linkedin: https://www.linkedin.com/in/oliver-angelil/

Re: Ingestion Time Clustering on initial load

Anonymous — Sun, 23 Jul 2023 04:08:50 GMT

Hi @dvmentalmadess

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Ingestion Time Clustering on initial load

dvmentalmadess — Tue, 25 Jul 2023 19:52:53 GMT

Thanks @Anonymous , I have not gotten any answers to this. If you have any information or clarification that would be helpful.