Databricks Community

MiBjorn

Background:I'm working on a data pipeline to insert JSON files as quickly as possible. Here are the details of my setup: File Size: 1.5 - 2 kB eachFile Volume: Approximately 30,000 files per hourPipeline: Using Databricks Delta Live Tables (DLT) in c...

MiBjorn

@Kaniz_Fatma The DLT pipeline use Core product edition, as there is no features used in the code that requires the pro/advanced edition. The pipeline runs without error, the issue, as mentioned above, is that it can take up to 30 min before a row is ...

Databricks Community

User Stats

User Activity

Optimizing Data Insertion Speed for JSON Files in DLT Pipeline

Re: Optimizing Data Insertion Speed for JSON Files in DLT Pipeline