Background:I'm working on a data pipeline to insert JSON files as quickly as possible. Here are the details of my setup: File Size: 1.5 - 2 kB eachFile Volume: Approximately 30,000 files per hourPipeline: Using Databricks Delta Live Tables (DLT) in c...
