Databricks Community

ck7007

Reduced Monthly Databricks Bill from $47K to $12.7KThe Problem: We were scanning 2.3TB for queries needing only 8GB of data.Three Quick Wins1. Multi-dimensional Partitioning (30% savings)# Beforedf.write.partitionBy("date").parquet(path)# After-parti...

ck7007

Bloom Filters + Zonemaps: The Ultimate Query Optimization ComboAfter my zonemap post last week got great feedback, several of you asked about Bloom filter integration. Here's the complete implementation!Why Bloom Filters Changed EverythingZonemaps ar...

ck7007

Reduced Monthly Databricks Bill from $47K to $12.7KThe Problem: We were scanning 2.3TB for queries needing only 8GB of data.Three Quick Wins1. Multi-dimensional Partitioning (30% savings)# Beforedf.write.partitionBy("date").parquet(path)# After-parti...

ck7007

Problem: Queries on our 100M+ record Iceberg tables were taking 45+ seconds.Solution: Implemented lightweight zonemap indexing that tracks min/max values per file.Quick Implementationdef apply_zonemap_pruning(table_path, predicate_value):# Load zonem...

ck7007

@BSGood catch—totally my mistake! 臘Had multiple drafts open and posted the wrong one. Thanks for the heads-up!Just deleted the duplicate. What I meant to share was the Bloom filter follow-up that builds on that cost optimization:Quick update: Adding ...

ck7007

I see you're using dbutils.widgets. text and dropdown—perfect! You're already on the right track.Quick SolutionYour widgets are already dynamic! Just pass parameters in your job configuration:In your notebook (slight refactor of your code):# Define w...

ck7007

@DataBrickatorPerfect timing! I'm a heavy Databricks Notebooks user working on performance optimization and ML pipelines.My daily workflows involve:EDA on 100M+ record Iceberg tablesReal-time feature engineering with Spark StreamingCustom indexing st...

ck7007

Thanks, glad you found it helpful.Your detailed Bloom filter implementation above is fantastic—especially the RDD-based grouping approach for file-level filters. Have you measured the memory overhead in production?I'm curious about your experience wi...

Databricks Community

User Stats

User Activity

Advanced Technique

Technical Deep Dive

Cost

Achieved 87% Query Performance Improvement with Custom Zonemap Indexing

Re: Advanced Technique

Re: Pass Notebook parameters dynamically in Job task.

Re: [Databricks User Research] AI Assistance for Data Science Work

Re: Achieved 87% Query Performance Improvement with Custom Zonemap Indexing