You've read it correctly. This is a service-side check, not a bundles/client issue, and right now column selection isn't available on the integrated (gateway-less) SQL Server pipeline.
Why the bundle model and docs don't match the error
- TableSpecificConfig in databricks-bundles is generated from the generic pipelines API schema, which every connector shares. It lists every field any connector could take, so include_columns / exclude_columns showing up there doesn't mean every connector or pipeline shape accepts them.
- The Some(SQLSERVER) in the message is a Scala Option leaking from the backend's validation step. The pipeline spec is checked against what that connector/mode supports, and the integrated SQL Server path rejects column selection outright. You'll see the same error with the Python SDK, the REST API or the UI. Only the server can change that.
- The SQL Server column-selection example in the docs was written for the gateway + ingestion-pipeline setup you migrated from, and that setup still supports it.
Options for now
- If any columns must not land in Databricks at all (PII, compliance), stay on the gateway + ingestion pipeline for those tables. It's the only option where the excluded columns never leave SQL Server. You can run both setups side by side: integrated for the tables where you want everything, gateway-based for the ones that need include_columns.
- If you just want a tidier schema or less downstream noise, keep the integrated pipeline, ingest everything, and project afterwards. Put a view (or a materialized view, if consumers need it to be a table) on top of the streaming table with only the columns you want, and grant consumers access to that object instead of the raw table. Don't drop or alter columns on the managed streaming table itself, because the next update will fight you on it.
- If raw-table access is the only concern, a column mask or a restricted view in Unity Catalog covers that without changing the ingestion design.
Worth doing either way
Raise it with your account team or through a support ticket and quote the exact error. The integrated pipeline is newer than the gateway-based one, and a feature gap like this is the kind of thing they track for parity. A support ticket will also get you a firm answer on whether it's on the roadmap, which nobody outside Databricks can give you.
Happy to look at your bundle YAML if you want to set up the split (the first option above) without duplicating the connection and catalog/schema config.