hkmodi
New Contributor II

I did have this approach in mind. But every 4 hours of data that I have is equivalent to 1.2 TB
I want to avoid writing all that into a table and then performing dedeuplication.

Is there a way I can create a temp view and do deduplication before writing that into a table?