Databricks Community

ck7007 · 09-07-2025

Zero-Downtime Unity Catalog Migration for 500TB Data LakeJust completed migrating 500TB to Unity Catalog without a single minute of downtime. Here's how:The Challenge500 TB across 12,000 tables200+ concurrent usersZero tolerance for downtimeMixed Hiv...

ck7007 · 09-02-2025

Maintain Zonemaps with Streaming Writes Challenge: Streaming breaks zonemaps due to constant micro-batches.Solution: Incremental Updatesdef write_streaming_with_zonemap(stream_df, table_path):def update_zonemap(batch_df, batch_id):# Write databatch_d...

ck7007 · 09-01-2025

Reduced Monthly Databricks Bill from $47K to $12.7KThe Problem: We were scanning 2.3TB for queries needing only 8GB of data.Three Quick Wins1. Multi-dimensional Partitioning (30% savings)# Beforedf.write.partitionBy("date").parquet(path)# After-parti...

ck7007 · 08-29-2025

Bloom Filters + Zonemaps: The Ultimate Query Optimization ComboAfter my zonemap post last week got great feedback, several of you asked about Bloom filter integration. Here's the complete implementation!Why Bloom Filters Changed EverythingZonemaps ar...

ck7007 · 08-28-2025

Reduced Monthly Databricks Bill from $47K to $12.7KThe Problem: We were scanning 2.3TB for queries needing only 8GB of data.Three Quick Wins1. Multi-dimensional Partitioning (30% savings)# Beforedf.write.partitionBy("date").parquet(path)# After-parti...

ck7007 · 09-08-2025

Thanks, @Khaja_Zaffer and @BS_THE_ANALYST!@Khaja_Zaffer:The toolkit has 5 main components:Pre-migration analyzer Compatibility scoringDrift monitor Real time consistency checksPermission migrator: Automated ACL copyingQuery rewriter: Hive→UC SQL con...

ck7007 · 09-04-2025

@szymon_dybczak @BS_THE_ANALYST @TheOC:The Actual Working SolutionThe pandas approach works (as @kmodelew confirmed), but here's the complete, tested solution for reading Excel from Volumes:# For Excel files in Unity Catalog Volumesimport pandas as p...

ck7007 · 09-04-2025

@ManojkMohanYour comparison table is solid. I have test data and production metrics for these approaches. Here's what I found:Test Data & Real-World ResultsI tested with 100M records streaming at 50K records/sec. Here's actual performance:1. Partitio...

ck7007 · 09-03-2025

@kmodelew The issue is that your volume path is missing the leading forward slash. Also, the library isn't loading properly.Quick Fix# Correct path format for Volumeslocation = '/Volumes/catalog/schema/volume_name/file.xlsx' ## Note the leading# For ...

ck7007 · 09-03-2025

@ManoramTaparia The issue is that current_timestamp() makes your MV non-deterministic, forcing complete recomputes. Here's how to fix it:Solution: Use the Source Table's Change TrackingOption 1: Leverage Source Table's Timestamp Column@Dlt.table(name...

Databricks Community

User Stats

User Activity

Unity Catalog Migration Strategy

Streaming Solution

Advanced Technique

Technical Deep Dive

Cost

Re: Unity Catalog Migration Strategy

Re: Unable to read excel file from Volume

Re: Streaming Solution

Re: Unable to read excel file from Volume

Re: Identify updated rows during incremental refresh in DLT Materialized Views