cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

numersoz
by New Contributor III
  • 2094 Views
  • 5 replies
  • 10 kudos

Is ZORDER required after table overwrite?

Hi,After appending new values to a delta table, I need to delete duplicate rows.After deleting duplicate rows using PySpark, I overwrite the table (keeping the schema).My question is, do I have to do ZORDER again?Another question, is there another wa...

  • 2094 Views
  • 5 replies
  • 10 kudos
Latest Reply
DeepakMakwana74
New Contributor III
  • 10 kudos

Hii @Nurettin Ersoz​ try to use incremental load of data so it will avoid duplicate and you can use full load once if you have updation in your data

  • 10 kudos
4 More Replies
dataexplorer
by New Contributor III
  • 4601 Views
  • 6 replies
  • 5 kudos

Resolved! COPY INTO generating duplicate rows in Delta table

Hello Everyone,I'm trying to bulk load tables from a SQL server database into ADLS as parquet files and then loading these files into Delta tables (raw/bronze). I had done a one off history/base load but my subsequent incremental loads (which had a d...

  • 4601 Views
  • 6 replies
  • 5 kudos
Latest Reply
dataexplorer
New Contributor III
  • 5 kudos

thanks for the guidance!

  • 5 kudos
5 More Replies
Labels