We typically organize our workloads with one job per catalog, and then use one or more pipelines to load tables into the appropriate schemas. As our data engineers ingest raw data, this structure is primarily applied in the Silver and Gold layers of our architecture.

For example, when loading Salesforce data, we might structure it like this:

• salesforce_silver (job)

  • sales (schema) → Pipeline

    • Sales-related tables (as needed within the schema)

  • procurement (schema) → Pipeline

    • Procurement-related tables (as needed within the schema)

This same job-and-pipeline pattern is carried into the Gold layer. However, the structure often evolves there, since Gold datasets may combine data across multiple catalogs and schemas.

Ultimately, your naming conventions and structure should reflect your specific design and use cases.

Larissa