@cltj
This is a classic example of how an architecture can become difficult to scale over time. Initially, we onboarded only a few clients and maintained client-specific tables. However, as the number of clients grew, the solution resulted in a large proliferation of tables, increasing maintenance and operational complexity.
To address this, we redesigned the architecture using a metadata-driven framework powered by control/metadata tables. The control/metadata tables store column mappings and transformation rules, while a centralized transactional table stores common attributes along with sequence numbers and source identifiers (Like - sql, excel, datavserse).
This approach significantly improved scalability and maintainability. Onboarding a new client now requires only configuration changes in the metadata layer, without creating additional physical tables. Client-specific views can be exposed at the Gold layer, enabling easy access to curated data while keeping the underlying architecture standardized and scalable.
Naming convention should be like below:
department.Usecase.layer =
dep.finance.bronze_meta and dep.finance.bronze_transactional
dep.finance.silver_meta and dep.finance.silver_transactional
dep.finance.gold_meta and dep.finance.gold_transactional , if require client specific view on the basis of sources.