- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-02-2025 02:52 PM
Let me break down the bronze layer approach for your Medallion Architecture with detailed cost and architectural considerations.
Tables in Unity Catalog:
Why This is Better Than Raw Files:
- Schema Evolution - Handle SQL Server schema changes gracefully
- ACID Transactions - Consistent data during ingestion
- Time Travel - Debug data issues and recover from bad loads
- Performance - Z-ordering and file compaction
- Governance - Unity Catalog lineage and access control
Cost Breakdown Analysis:
FREE Components:
- Creating schemas/catalogs
- Registering tables (metadata only)
- Unity Catalog metadata storage
- Lineage tracking
- Access control policies
💰 PAID Components:
- DBU consumption during queries/operations
- Underlying storage (ADLS/S3)
- Compute for optimization operations
Detailed Cost Analysis
ADLS vs Databricks Charges:
ADLS Storage Costs:
- Standard tier: $0.0184/GB/month
- Cool tier: $0.01/GB/month (for archival bronze data)
- Transaction costs: ~$0.0004 per 10k operations
Databricks Costs:
- DBU consumption during:
* Data ingestion jobs
* Table optimization (OPTIMIZE, VACUUM)
* Query execution
* Cluster startup/shutdown
Cost Optimization Tips:
- Use spot instances: 60-70% savings
- Auto-terminate idle clusters
- Schedule optimization during off-peak hours
- Use smaller clusters for bronze layer
Final Recommendation
Use Unity Catalog Delta Tables for Bronze Layer because:
Cost-effective: Storage costs are similar to Parquet, but with better compression:
- Production-ready: ACID transactions, schema evolution, time travel
- Governance: Built-in lineage, access control, data discovery
- Performance: Better query performance for silver/gold transformations
- Maintenance: Automated optimization reduces long-term costs
The additional UC overhead is minimal compared to the operational benefits, especially in production environments where data quality and governance are crucial.