Databricks Community

brian_zavareh · 04-02-2024

I'm currently facing challenges with optimizing the performance of a Delta Live Table pipeline in Azure Databricks. The task involves ingesting over 10 TB of raw JSON log files from an Azure Data Lake Storage account into a bronze Delta Live Table la...

brian_zavareh · 06-27-2023

1. Why am I here?I am present at the Databricks summit because I'm motivated to deepen my understanding of its groundbreaking technology. I'm passionate about learning how to leverage this advanced platform to better optimize and streamline the proc...

brian_zavareh · 04-09-2024

Thanks for the heads up @standup1. I agree I was using it for the purpose of POC and I will select bigger clusters for the main job. Do you know any good practice on how to select the number of clusters and their sizes for both worker and driver?

brian_zavareh · 04-06-2024

Hi @Kaniz_Fatma,Thank you for the insightful suggestions on optimizing our Delta Live Table pipeline. I'm excited to apply your recommendations, especially around partitioning, clustering, and Autoloader configurations. I'll implement these changes a...

Databricks Community

User Stats

User Activity

Optimizing Delta Live Table Ingestion Performance for Large JSON Datasets

Insights and Learnings from Databricks Summit: A Deep Dive into Data Engineering and Delta Live Tab

Re: Optimizing Delta Live Table Ingestion Performance for Large JSON Datasets

Re: Optimizing Delta Live Table Ingestion Performance for Large JSON Datasets