stbjelcevic
Databricks Employee
Databricks Employee

@srikanthp24 ,

To your core question: multiple gateway pipelines against the same source database don't interfere with each other. Each gateway keeps its own independent state, its own staging volume, and its own CDC/change-tracking cursor, so a separate Dev, UAT, and Prod gateway pointing at the same SQL Server won't step on one another. As long as each pair uses distinct names, staging, and destination schemas (as noted above), your existing Dev pipeline is unaffected. Databricks even points at this pattern for resolving table-name conflicts: create multiple gateway-pipeline pairs writing to different destination schemas.

The one thing I'd watch that isn't obvious: the risk with several environments isn't on the Databricks side, it's the SQL Server CDC cleanup job. It runs on a retention schedule and isn't aware of individual consumers, so your least-active environment sets the floor. If a UAT or Dev gateway sits stopped past the CDC retention window, its cursor ends up pointing at change data that's already been purged, which forces a full refresh of the affected tables. That's why the gateway must run continuously, and why you should size source-side CDC retention for the slowest gateway rather than the typical one.

Two smaller notes. Adding more gateways doesn't consume extra CDC capture instances, since they all read the same capture instance rather than creating their own, so you won't hit SQL Server's two-instance-per-table limit just by adding environments. And since each environment adds a continuous reader against the source, change tracking is recommended over CDC for any table with a primary key to keep source load down. If you're greenfielding UAT/Prod, the newer Integrated CDC (Beta) mode also collapses the gateway and pipeline into one, worth a look.