cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Opt-out of schema evolution with the Lakeflow connect for sql-server?

MrWick
New Contributor

I am trying to connect to setup lakeflow connect pulling from an on-prem sql server.  I get connected and choose the tables I want to pull data from.  Change tracking is setup on sql server, however, the dba's don't want to create the helper tables for schema evolution.  

I read I can opt-out of schema evolution but do not see anywhere to do this.  I see no toggle to opt out and I am unable to select specific columns to pull from (as some docs say you can do).

Anybody have info on this?

1 REPLY 1

Brahmareddy
Esteemed Contributor

Hi MrWick, How are you doing today?

This is a great question, and it’s understandable that your DBAs may be cautious about allowing schema evolution helper tables on the SQL Server side. As of now, in LakeFlow Connect, opting out of schema evolution is not exposed directly through a simple toggle in the UI, which can make it confusing. However, behind the scenes, LakeFlow expects those helper tables to manage schema changes over time—so if your setup skips that, you'll likely need to work with fixed schemas and rely on manual updates when source structures change. Unfortunately, column-level selection and schema evolution controls can be limited depending on connector version and LakeFlow’s current capabilities. I'd recommend reaching out to your Databricks support rep to confirm if column selection or schema evolution opt-out can be enabled via config flags or backend settings—they may be able to whitelist this or advise on best practices with minimal DBA overhead.

Regards,

Brahma

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now