We're building a lakehouse on Azure Databricks with Unity Catalog. Our data lands in Azure Data Lake Storage Gen2 (Hierarchical Namespace enabled) as JSON files. The challenge is multi-tenancy: each tenant's data is written to a separate container in the same storage account, following a naming convention like prefix-tenantA, prefix-tenantB, etc. We currently have a handful of tenants in dev but expect to scale to a few hundred.

We need to get all this data into Databricks, ideally into shared tables (all tenants in one table, with a tenant-name column to distinguish them).

What we've tried:

1. Autoloader with container-level wildcards (abfss://prefix-*@account.dfs.core.windows.net/common_path/...) — does not work. Wildcards are not supported in the container portion of the ABFSS path.
2. Single Autoloader with multiple paths (string-splitting or passing a list of container paths) — only reads from the first container.
3. One Autoloader per container — this works, but raises concerns about scale: can we run 100+ Autoloader pipelines efficiently? What are the compute cost and monitoring implications?

What we're considering:

• A config-driven approach where a configuration file lists all tenant names, and a deployment process creates/updates a DLT pipeline per tenant automatically.
• Alternatively, restructuring storage so tenants are directories inside one container instead of separate containers (but customers prefer container-level isolation).

Our questions:

1. Is there a recommended pattern for ingesting from many separate containers into Databricks? Are there limits on how many Autoloader streams can run concurrently?
2. Could Unity Catalog external locations or volumes be used to abstract over multiple containers without running separate Autoloader instances — for example, mounting all tenant containers as a single logical location?
3. For those running multi-tenant Databricks lakehouses at scale: how do you handle per-tenant ingestion? Separate pipelines, a single parameterized pipeline, or something else entirely?

Environment: Azure Databricks, Unity Catalog, ADLS Gen2 with HNS, DLT pipelines deployed via Databricks Asset Bundles, managed identity authentication.

Any guidance or experience reports appreciated. We'd especially like to hear from anyone running 50+ concurrent Autoloader streams.