pradeep_singh
Contributor III

I am assuming you are talking about the job to load bronze and silver tables . Having one job per table seems like a bad idea since at scale you will most likely start hitting the limits of the workspace apart from operational overhead of maintaining/monitoring/deployment/compute wastage so many jobs . Typically you would use a metadata table/yaml file to define the configuration and then group your tables into diffrent pipelines based on various factors like ( business domains/trigger/schedule/volume/velocity etc ) .

Gold tables would have their own pipelines if they have complex dependencies but bronze and silver should be pretty straightforward metadata driven pipelines/jobs

Thank You
Pradeep Singh - https://www.linkedin.com/in/dbxdev