lingareddy_Alva
Esteemed Contributor

Hi @Mohan_Baabu1 

Let me break down the bronze layer approach for your Medallion Architecture with detailed cost and architectural considerations.

Tables in Unity Catalog:
Why This is Better Than Raw Files:
- Schema Evolution - Handle SQL Server schema changes gracefully
- ACID Transactions - Consistent data during ingestion
- Time Travel - Debug data issues and recover from bad loads
- Performance - Z-ordering and file compaction
- Governance - Unity Catalog lineage and access control

Cost Breakdown Analysis:
FREE Components:
- Creating schemas/catalogs
- Registering tables (metadata only)
- Unity Catalog metadata storage
- Lineage tracking
- Access control policies

💰 PAID Components:
- DBU consumption during queries/operations
- Underlying storage (ADLS/S3)
- Compute for optimization operations

Detailed Cost Analysis
ADLS vs Databricks Charges:
ADLS Storage Costs:
- Standard tier: $0.0184/GB/month
- Cool tier: $0.01/GB/month (for archival bronze data)
- Transaction costs: ~$0.0004 per 10k operations

Databricks Costs:
- DBU consumption during:
* Data ingestion jobs
* Table optimization (OPTIMIZE, VACUUM)
* Query execution
* Cluster startup/shutdown

Cost Optimization Tips:
- Use spot instances: 60-70% savings
- Auto-terminate idle clusters
- Schedule optimization during off-peak hours
- Use smaller clusters for bronze layer


Final Recommendation
Use Unity Catalog Delta Tables for Bronze Layer because:

Cost-effective: Storage costs are similar to Parquet, but with better compression:
 - Production-ready: ACID transactions, schema evolution, time travel
 - Governance: Built-in lineage, access control, data discovery
 - Performance: Better query performance for silver/gold transformations
 - Maintenance: Automated optimization reduces long-term costs

The additional UC overhead is minimal compared to the operational benefits, especially in production environments where data quality and governance are crucial.

 

LR