- 7131 Views
- 8 replies
- 7 kudos
How to optimize exporting dataframe to delta file?
Scenario : I have a dataframe that have 5 billion records/rows and 100+ columns. Is there a way to write this in a delta format efficiently. I have tried to export it but cancelled it after 2 hours (write didnt finish) as this processing time is not ...
- 7131 Views
- 8 replies
- 7 kudos
Latest Reply
- 7 kudos
Hi @Franco Sia​ ,I will recommend to avoid to use the repartition(50), instead enable optimizes writes on your Delta table. You can find more details hereEnable optimized writes and auto compaction on your Delta table. Use AQE (docs here) to have eno...
- 7 kudos