Databricks Community

Harun · 06-25-2024

Optimizing data storage and access is crucial for enhancing the performance of data processing systems. In Databricks, several optimization techniques can significantly improve query performance and reduce costs: Z-Order Optimize, Optimize Compaction...

Harun · 06-08-2024

Databricks is a popular unified data analytics platform known for its powerful data processing capabilities and seamless integration with Apache Spark. However, managing and optimizing costs in Databricks can be challenging, especially when it comes ...

Harun · 11-03-2023

Excited to invite you all to join me for a deep dive into Databricks Fundamentals! I'm thrilled to guide you through the essentials of the Databricks Lakehouse Platform. In this session, we'll explore:* What Databricks is and why it's pivotal in toda...

Harun · 03-22-2023

Hi Guys,I am working on streaming data movement from bronze to silver. My bronze table is having a entity_name column, based on the entity_name column i need to create multiple silver tables.I tried the below approach, But it is failing with error "'...

Harun · 03-15-2023

Hi All,I am working on a streaming data processing. As a intial step i have read the data from azure eventhub using readstream. now i want to writestream this into a delta table. My requirement is, The data should present in external location (adls g...

Harun · 03-16-2023

Hi @werners, i tried that option as well. But it will create a managed table. But i want a external table. So now i created a external table prior to the streaming part. What i understand from the research i did is, we cannot write to external table ...

Harun · 03-15-2023

Connection is fine, I need to know how to write the stream data to adls gen2 path and the same time have the delta table registered to the metastore as well.

Harun · 12-22-2022

No limits, but i faced some slowness when i had 100+ cells, so in that case better split the notebook and run it inside the other notebook using %run /02_notebook/

Harun · 12-18-2022

Hi @_ _ ,Under the job section, create your first task, then add dependent task (attached snapshot for your reference).

Harun · 12-15-2022

Thanks for sharing @Sujitha Ramamoorthy

Databricks Community

User Stats

User Activity

Comprehensive Guide to Databricks Optimization: Z-Order, Data Compaction, and Liquid Clustering

Optimizing Costs in Databricks by Dynamically Choosing Cluster Sizes

Databricks Fundamentals

Issue with Pyspark GroupBy GroupedData

how to load structured stream data into delta table whose location is in ADLS Gen2

Re: how to load structured stream data into delta table whose location is in ADLS Gen2

Re: how to load structured stream data into delta table whose location is in ADLS Gen2

Re: Notebooks cells limit

Re: Schedule job to run sequentially after another job

Re: Documentation Update Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data enginee...