<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Re: Lakeflow Connect SQL Server (integrated CDC pipeline): no way to Include/exclude columns in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/lakeflow-connect-sql-server-integrated-cdc-pipeline-no-way-to/m-p/173097#M56699</link>
    <description>&lt;P&gt;&amp;nbsp;Hi&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/173462"&gt;@Oumeima&lt;/a&gt;&amp;nbsp;&lt;BR /&gt;&lt;BR /&gt;Just checking in to see whether the explanation above helped, and whether you were able to move forward, either by keeping the gateway-based pipeline for the tables that need include_columns or by projecting the columns downstream.&lt;BR /&gt;&lt;BR /&gt;&amp;nbsp;If it answered your question, I'd be really grateful if you could mark it as the accepted solution. It helps others who run into the same Some(SQLSERVER) error find the answer quickly.&lt;/P&gt;&lt;P&gt;&lt;BR /&gt;And if you're still stuck, or Databricks support came back with something different, please let me know. I'm happy to help further, for example with setting up the split in your bundle YAML.&lt;BR /&gt;&lt;BR /&gt;Thanks a lot!&lt;/P&gt;</description>
    <pubDate>Wed, 14 Oct 2026 00:47:36 GMT</pubDate>
    <dc:creator>SumeshKashyap</dc:creator>
    <dc:date>2026-10-14T00:47:36Z</dc:date>
    <item>
      <title>Lakeflow Connect SQL Server (integrated CDC pipeline): no way to Include/exclude columns</title>
      <link>https://community.databricks.com/t5/data-engineering/lakeflow-connect-sql-server-integrated-cdc-pipeline-no-way-to/m-p/172832#M56680</link>
      <description>&lt;P&gt;&lt;STRONG&gt;Setup:&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;- Lakeflow Connect, SQL Server source, `connector_type: CDC`, integrated pipeline (gateway-less — extraction and apply in one triggered pipeline on classic compute)&lt;BR /&gt;- Deployed with Declarative Automation Bundles, `databricks-bundles` 1.20.0, CLI current&lt;BR /&gt;- Migrated from the gateway + ingestion-pipeline pair, where `table_configuration.include_columns` was in use and working&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;Problem:&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;We don't need all the columns from the tables we're ingesting. So we need to include or exclude a selection.&lt;/P&gt;&lt;P&gt;When adding the parameter "include_columns" under "&lt;SPAN&gt;table_configuration"&amp;nbsp;&lt;/SPAN&gt;(the list that worked on the gateway-based pipeline) on the integrated pipeline, the bundle deployment fails with:&lt;BR /&gt;```&lt;BR /&gt;Column selection (include_columns / exclude_columns) is not supported for Some(SQLSERVER).&lt;BR /&gt;```&lt;/P&gt;&lt;P&gt;The "TableSpecificConfig" model in "databricks-bundles" documents both fields as "[Public Preview]" with no connector caveat, and the SQL Server connector docs show them in the example config, so the rejection looks like a service-side limitation specific to the integrated SQL Server connector rather than a client issue.&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;Questions:&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;-&amp;nbsp; Is column selection (`include_columns` / `exclude_columns`) expected to work on the integrated CDC pipeline for SQL Server?&lt;/P&gt;&lt;P&gt;Happy to share more details about this!&lt;/P&gt;</description>
      <pubDate>Mon, 12 Oct 2026 17:11:42 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/lakeflow-connect-sql-server-integrated-cdc-pipeline-no-way-to/m-p/172832#M56680</guid>
      <dc:creator>Oumeima</dc:creator>
      <dc:date>2026-10-12T17:11:42Z</dc:date>
    </item>
    <item>
      <title>Re: Lakeflow Connect SQL Server (integrated CDC pipeline): no way to Include/exclude columns</title>
      <link>https://community.databricks.com/t5/data-engineering/lakeflow-connect-sql-server-integrated-cdc-pipeline-no-way-to/m-p/172847#M56681</link>
      <description>&lt;P&gt;You've read it correctly. This is a service-side check, not a bundles/client issue, and right now column selection isn't available on the integrated (gateway-less) SQL Server pipeline.&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;Why the bundle model and docs don't match the error&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;- TableSpecificConfig in databricks-bundles is generated from the generic pipelines API schema, which every connector shares. It lists every field any connector could take, so include_columns / exclude_columns showing up there doesn't mean every connector or pipeline shape accepts them.&lt;/P&gt;&lt;P&gt;- The Some(SQLSERVER) in the message is a Scala Option leaking from the backend's validation step. The pipeline spec is checked against what that connector/mode supports, and the integrated SQL Server path rejects column selection outright. You'll see the same error with the Python SDK, the REST API or the UI. Only the server can change that.&lt;/P&gt;&lt;P&gt;- The SQL Server column-selection example in the docs was written for the gateway + ingestion-pipeline setup you migrated from, and that setup still supports it.&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;Options for now&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;- If any columns must not land in Databricks at all (PII, compliance), stay on the gateway + ingestion pipeline for those tables. It's the only option where the excluded columns never leave SQL Server. You can run both setups side by side: integrated for the tables where you want everything, gateway-based for the ones that need include_columns.&lt;/P&gt;&lt;P&gt;- If you just want a tidier schema or less downstream noise, keep the integrated pipeline, ingest everything, and project afterwards. Put a view (or a materialized view, if consumers need it to be a table) on top of the streaming table with only the columns you want, and grant consumers access to that object instead of the raw table. Don't drop or alter columns on the managed streaming table itself, because the next update will fight you on it.&lt;/P&gt;&lt;P&gt;- If raw-table access is the only concern, a column mask or a restricted view in Unity Catalog covers that without changing the ingestion design.&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;Worth doing either way&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;Raise it with your account team or through a support ticket and quote the exact error. The integrated pipeline is newer than the gateway-based one, and a feature gap like this is the kind of thing they track for parity. A support ticket will also get you a firm answer on whether it's on the roadmap, which nobody outside Databricks can give you.&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;Happy to look at your bundle YAML if you want to set up the split (the first option above) without duplicating the connection and catalog/schema config.&lt;/P&gt;</description>
      <pubDate>Mon, 12 Oct 2026 18:56:13 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/lakeflow-connect-sql-server-integrated-cdc-pipeline-no-way-to/m-p/172847#M56681</guid>
      <dc:creator>SumeshKashyap</dc:creator>
      <dc:date>2026-10-12T18:56:13Z</dc:date>
    </item>
    <item>
      <title>Re: Lakeflow Connect SQL Server (integrated CDC pipeline): no way to Include/exclude columns</title>
      <link>https://community.databricks.com/t5/data-engineering/lakeflow-connect-sql-server-integrated-cdc-pipeline-no-way-to/m-p/173097#M56699</link>
      <description>&lt;P&gt;&amp;nbsp;Hi&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/173462"&gt;@Oumeima&lt;/a&gt;&amp;nbsp;&lt;BR /&gt;&lt;BR /&gt;Just checking in to see whether the explanation above helped, and whether you were able to move forward, either by keeping the gateway-based pipeline for the tables that need include_columns or by projecting the columns downstream.&lt;BR /&gt;&lt;BR /&gt;&amp;nbsp;If it answered your question, I'd be really grateful if you could mark it as the accepted solution. It helps others who run into the same Some(SQLSERVER) error find the answer quickly.&lt;/P&gt;&lt;P&gt;&lt;BR /&gt;And if you're still stuck, or Databricks support came back with something different, please let me know. I'm happy to help further, for example with setting up the split in your bundle YAML.&lt;BR /&gt;&lt;BR /&gt;Thanks a lot!&lt;/P&gt;</description>
      <pubDate>Wed, 14 Oct 2026 00:47:36 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/lakeflow-connect-sql-server-integrated-cdc-pipeline-no-way-to/m-p/173097#M56699</guid>
      <dc:creator>SumeshKashyap</dc:creator>
      <dc:date>2026-10-14T00:47:36Z</dc:date>
    </item>
  </channel>
</rss>

