Databricks Community

pooja_bhumandla

Hi Community,I encountered the following error: Failed to store executor broadcast spark_join_relation_1622863 (size = Some(67141632)) in BlockManager with storageLevel=StorageLevel(memory, deserialized, 1 replicas)in a Structured S...

pooja_bhumandla

Hi Community,I have a scenario where I’ve already calculated delta statistics for the first 32 columns after enabling the dataskipping property. Now, I need to include 10 more frequently used columns that were not part of the original 32.Goal:I want ...

pooja_bhumandla

Hi everyone,I’m evaluating whether Liquid Clustering would be beneficial for the tables based on the sizes. Below is the size distribution of tables in my environment:Size Bucket Table Count Large (> 1 TB)3Medium (10 GB – 1 TB)284Small (< 10 GB)17,26...

pooja_bhumandla

Hi all,I'm exploring Liquid Clustering (LC) and its effectiveness based on the size of the tables.Specifically, I’m interested in understanding how LC behaves with small, medium, and large tables and the best practices for each, along with size range...

pooja_bhumandla

Hi Databricks Community,I’m running a Structured Streaming job in Databricks with foreachBatch writing to a Delta table.Failed to store executor broadcast spark_join_relation_1622863(size = Some(67141632)) in BlockManager with storageLevel=StorageLev...

pooja_bhumandla

Hi @szymon_dybczak, thank you for your response. If I update the table property to include all 42 columns (new 10 + old 32 cols) in delta.dataSkippingStatsColumns,and then run ANALYZE TABLE only for the new 10 columns (col33–col42):My Questions:Will ...

pooja_bhumandla

Hi @bianca_unifeye , thank you for your response.My tables range in size from 1 KB to 5 TB. Given this, I’d love to hear your thoughts and experiences on whether Liquid Clustering (LC) would be a good fit in this scenario. Thanks in advance for shari...

pooja_bhumandla

Hi @mark_ott , thank you for your response.How Liquid clustering behaves with small, medium, and large tables and the best practices for each, along with size ranges for each category?Thanks in advance for sharing your knowledge!

pooja_bhumandla · 06-19-2025

What are the criterias based on which max and min files sizes vary from target file size?

pooja_bhumandla · 06-18-2025

Why file size bigger or smaller than the target is a better solution? What are the reasons for it?

Databricks Community

User Stats

User Activity

Error: Executor Memory Issue with Broadcast Joins in Structured Streaming – Unable to Store 69–80 MB

Best Practice for Updating Data Skipping Statistics for Additional Columns

Should I enable Liquid Clustering based on table size distribution?

Seeking Insights on Liquid Clustering (LC) Based on Table Sizes

Broadcast Join Failure in Streaming: Failed to store executor broadcast in BlockManager

Re: Best Practice for Updating Data Skipping Statistics for Additional Columns

Re: Seeking Insights on Liquid Clustering (LC) Based on Table Sizes

Re: When to Use and when Not to Use Liquid Clustering?

Re: data file size

Re: Auto tuning of file size