I’ve been using the notebook style as there are some transformations that need to be done on destination_table names as the source_table names has special characters. My requirement is: for example, I have a schema with 5 tables, 10 individual tables from another schema. Since each table name needs to be transformed, I am getting all the tables and creating a list of objects. In the first run if the number of objects is 15 it creates pipeline with the name specified in the notebook. And by the second run of the pipeline, a new table gets created in the schema, we will have 16 objects which changes the ingestion definition of the pipeline. So, the second run creates a new pipeline with name “Managed Ingestion Pipeline - <original-pipeline_id>”. This has been the issue as we are not able to capture the history of the pipeline properly. I just want a way to update the ingestion definition of the pipeline but keep same pipeline name. Find the link to the documentation of notebook style Ingestoin Pipelines with Lakeflow Connect below.
SQL Server CDC Connector Setup

This is a code snippet I am working on:

Please advise if anyone has a solution to this issue.