Brahmareddy
Esteemed Contributor II

Hello @zed, May be you can try consider using partition pruning to limit the scope of the merge operation when calling write_table. Partitioning your feature table by daily or monthly increments will help reduce the data scanned during each merge, as only the relevant partitions are processed. Additionally, indexing frequently used columns or leveraging Z-order clustering on relevant columns can further optimize the read performance. You may also want to batch updates or focus only on new and changed records to reduce the load on large tables. Finally, monitor job performance metrics to track any bottlenecks and ensure resources are efficiently used during each write operation.