Yes, having a generic, parameter-driven ingestion architecture is the standard enterprise design pattern in Databricks to avoid managing dozens of redundant pipelines.
However, how you achieve this depends on whether you are using Custom Spark/Delta Lake Pipelines or Databricks Lakeflow Connect (Managed Ingestion).
Use Case Breakdown & Feasibility
Use Case 1: Editing a pipeline to ingest new tables (table_3, table_4)
• Yes, this is fully supported. You do not need to build a new pipeline for new tables. You update your central configuration file (YAML/JSON) or configuration table to register table_3 and table_4.
• When the pipeline triggers, it dynamically reads the updated configuration, instantiates the flows for table_3 and table_4, and ingests them alongside or independently of existing tables.
Use Case 2: Using the same pipeline for 50 different Database Servers
• Yes, but with an architectural nuance:
o A single pipeline can loop through or accept dynamic connections if configured as a parameter-driven job.
o However, for scaling across 50 database servers, the best practice is One Generic Codebase/Template deployed to 50 Pipeline Jobs (via CI/CD or Databricks Asset Bundles) rather than forcing 50 servers sequentially into a single massive execution run.
Architecture Approaches in Databricks
Depending on how your ingestion is built, here are the two main ways to implement this:
Approach 1: Generic Databricks Declarative Pipeline / PySpark Framework (Best for Metadata-Driven Ingestion)
Rather than hardcoding table names or source servers in Python or SQL scripts, your code queries a central Metadata Config (YAML file or Delta Table).
1. Define your Configuration (ingestion_config.yaml):
YAML
sources:
- connection_name: "sql_server_east"
jdbc_url: "jdbc:sqlserver://server01.database.windows.net:1433;database=Sales"
secret_scope: "db_secrets"
secret_key: "server01_password"
tables:
- source_table: "dbo.Orders"
target_table: "bronze_orders"
primary_key: "OrderID"
- source_table: "dbo.Customers"
target_table: "bronze_customers"
primary_key: "CustomerID"
2. Dynamic Ingestion Engine (generic_ingest.py): Similar to below snap
Python
import yaml
from pyspark.sql import SparkSession

# Read YAML config from Workspace or DBFS/Volume
with open("/Volumes/main/default/configs/ingestion_config.yaml", "r") as f:
config = yaml.safe_load(f)

for source in config["sources"]:
db_password = dbutils.secrets.get(scope=source["secret_scope"], key=source["secret_key"])

for table_info in source["tables"]:
# Dynamic JDBC Read
df = (spark.read.format("jdbc")
.option("url", source["jdbc_url"])
.option("dbtable", table_info["source_table"])
.option("password", db_password)
.load())

# Write dynamically to Target Bronze Delta Table
df.write.format("delta").mode("append").saveAsTable(f"bronze.{table_info['target_table']}")
• To add table_3 and table_4: Simply append them to the YAML file. The next run will automatically pick them up without code changes.
• To handle multiple servers: Add a new entry under sources in the YAML, or pass the server_name as a Job Parameter (dbutils.widgets.get("server_name")) to reuse the single notebook for multiple runs.
Approach 2: Databricks Lakeflow Connect (Managed Ingestion)
If you are using Lakeflow Connect (Databricks' native managed change data capture / database connector feature):
1. Table Changes (Use Case 1): You can edit an existing Lakeflow managed ingestion pipeline directly via the UI, Databricks CLI, or Declarative Automation Bundles (DABs). Adding new tables to the pipeline config will trigger an initial snapshot for the new tables while keeping the existing tables running incrementally.
2. Multiple Servers (Use Case 2): Lakeflow managed ingestion requires a Unity Catalog Connection per database server.
o A single Lakeflow Ingestion Gateway & Pipeline is bound to one source connection (one database server).
o Recommendation: Use Databricks Asset Bundles (DABs) to define a single declarative template (in YAML). You then parameterize the connection name and deploy 50 small, isolated server pipelines automatically via CI/CD. This prevents one failing database server from crashing the ingestion of the other 49 servers.