Hello @mbravonxp,
Create a separate catalog for each client to logically isolate their data. This helps in managing permissions and organizing data efficiently.
Within each catalog, create schemas for each environment (dev, pre, pro). This will help in managing the data lifecycle and access control for different stages of development
- Implement the medallion architecture by creating separate schemas or tables within each environment for the bronze, silver, and gold layers. This will help in organizing the data processing pipeline and maintaining data quality.
- Bronze: Raw data ingestion.
- Silver: Cleaned and enriched data.
- Gold: Aggregated and business-level data.
Use Databricks jobs or workflows to automate the data processing and updating of tables in the bronze, silver, and gold layers
Best Practices:
- Avoid giving direct storage-level access to users for Unity Catalog managed tables or volumes to maintain data security and governance.
- Co-locate your Databricks workspace, metastore, and storage in the same Azure region for optimal performance
