cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Sql server setup for lakeflow sql server connector to create ingestion

ShivangiB1
New Contributor III

When i am executing below command change instance is getting created but without lakeflow as prefix, i read the documentation and it mentioned that to track schema evolution we need to have prefix, can I please get some assistance.

Command Used:

EXEC dbo.lakeflowSetupChangeDataCapture
    @Tables = 'dbo.test_table'
6 REPLIES 6

lingareddy_Alva
Esteemed Contributor

Hi @ShivangiB1 

The lakeflow prefix is not mandatory. Schema evolution is not tracked based on the CDC change instance name, but through Delta Lake metadata and Lakeflow pipeline configuration.

Your command is working correctly. The change instance being created without the lakeflow_ prefix is expected behavior for lakeflowSetupChangeDataCapture.

Schema evolution is handled at the Delta/Lakeflow layer (auto-merge, schema evolution settings), not by the CDC naming convention. The prefix mentioned in documentation is only a recommended or example convention, not a technical requirement.

LR

ShivangiB1
New Contributor III

Hey @lingareddy_Alva , thanks for the explanation, but when i checked the databricks documentation it mentioned below line :

Lakeflow Connect only manages capture instances that match this naming pattern(lakeflow_ prefix). Pre-existing capture instances with different names are preserved and remain unaffected by Lakeflow operations. 

Can you please help on this. 

ShivangiB1
New Contributor III

and when i altered the table got below warning : 

WARNING: Table [dbo].[test_table] has a pre-existing capture instance named 'dbo_test_table' that was not created by lakeflow. Lakeflow will preserve this instance and create its own instance alongside it. Settings (filegroup, role, supports_net_changes) will be copied from the pre-existing instance.

@ShivangiB1 

The documentation is correct. Lakeflow Connect only manages CDC capture instances with the lakeflow_ prefix. Any existing CDC instances without this prefix are treated as external and intentionally preserved.

the warning was, the original dbo_test_table instance existed prior to Lakeflow, so Lakeflow correctly created its own lakeflow_dbo_test_table instance alongside it. Schema evolution is handled at the Delta/Lakeflow pipeline level and is not dependent on the CDC instance naming convention. The warning confirms correct and expected behavior.

 

LR

Hey @lingareddy_Alva , but dbo_test_table was created by lakeflowSetupChangeDataCapture, how this instance will be monitored than, just bit confused related to instance creation.

Hi @ShivangiB1  Even though dbo_test_table was created by running lakeflowSetupChangeDataCapture, Lakeflow does NOT consider it a Lakeflow-managed instance unless it follows the lakeflow_ naming pattern.

So Created by lakeflowSetupChangeDataCapture ≠ Managed by Lakeflow
Managed by Lakeflow ⇒ Must have lakeflow_ prefix
If it does not have the prefix, Lakeflow will not monitor or control it, even if the stored procedure created it.

Even if dbo_test_table was created via lakeflowSetupChangeDataCapture, Lakeflow does not manage or monitor it unless the capture instance follows the lakeflow_ naming convention. The prefix is the sole mechanism Lakeflow uses to determine ownership and lifecycle management, which is why a second instance is created and monitored.

LR