How to Deploy Lakehouse Federation Using DABs from the Dev Environment to the Stage Environment

[shan-databricks]

We are using Lakehouse Federation to connect to a SQL Server database and are currently looking for a CI/CD deployment solution using DABs. Could you please advise on the available options for deploying from the Dev environment to the Stage environment?

[iyashk-DB]

There's a bit of a split here between what DABs can and can't manage for Lakehouse Federation.

The connection object itself, the one that holds the SQL Server host, port, and credentials, isn't a supported bundle resource type. DABs don't have a connection resource. That's actually fine in practice, since a connection is inherently environment specific (your dev SQL Server and stage SQL Server are different hosts with different credentials), so it's not something you'd really want to promote as code between environments the way you would a job or pipeline.

What DABs does support is the catalog resource, which is the Unity Catalog foreign catalog that points at a connection via connection_name. So the working pattern is:

1. Create the connection in each target workspace once, outside the bundle. You can do this with CREATE CONNECTION in a SQL editor/notebook, the REST API (POST /api/2.1/unity-catalog/connections), or the Databricks CLI. Something like:

CREATE CONNECTION stage_sqlserver_conn TYPE sqlserver
OPTIONS (
  host '<stage-sqlserver-host>',
  port '1433',
  user '<user>',
  password secret('<scope>', '<key>')
);

2. In your bundle, define the catalog resource and use bundle variables/targets so the connection_name (and catalog name, if you want it different per env) resolves differently for dev vs stage:

bundle:
  engine: direct    # catalog resources require the direct deployment engine

variables:
  connection_name:
    default: dev_sqlserver_conn

targets:
  dev:
    variables:
      connection_name: dev_sqlserver_conn

  stage:
    variables:
      connection_name: stage_sqlserver_conn

resources:
  catalogs:
    sqlserver_catalog:
      name: sqlserver_fed_catalog
      connection_name: ${var.connection_name}
      comment: "Lakehouse Federation catalog for SQL Server"

One thing to flag: the catalog resource type only works with bundle.engine: direct, not the default Terraform-based engine. If your bundle is already using Terraform under the hood, you'll need to opt into the direct engine for this to deploy.

So the short version: create the connection per environment by hand or via script/CLI as part of your pipeline setup, then let DABs own and promote the catalog resource that references it across dev and stage.