Databricks Community

lrm_data

## ProblemWhen using Lakeflow Connect to ingest from SQL Server with SCD Type 2 enabled, any downstream Streaming Table (auto cdc flow) in a Spark Declarative pipeline will fail with the following error:"An error occurred because we detected an updat...

lrm_data

Has anyone else seen full refresh snapshots trigger outside of their configured refresh window in Lakeflow Connect?Here's our situation:- We have a full refresh window configured to restrict snapshot operations to off-hours- On at least one occasion,...

lrm_data

Has anyone else run into a situation where a breaking schema change on a SQL Server source table leaves their Lakeflow Connect pipeline in a state it can't recover from — even after destroying and recreating the pipeline?Here's what happened to us:- ...

lrm_data

Following up with a recommendation from Databricks:For tables that need incremental processing - SQL Server → Lakeflow Connect → Bronze SCD2 Streaming Table (CDF enabled → consume CDF, not base table using AUTO CDC → Silver SCD2 Streaming Table → Do...

lrm_data

Hello @Sumit_7 ,I have tested a few scenarios and logged a ticket with Databricks and discovered the following:Common MisconceptionThe start_window setting does not define a bounded time window during which full refreshesare contained. It is simply a...

lrm_data

Hey all,Following up.I was able to recover. The one step I was missing is resetting CDC in the source side. After that, I was able to destroy and recreate the bundle and successfully refresh all tables. Thanks!

lrm_data

Thank you ! I will continue monitoring and log internally if it continues happening.

lrm_data

Hello @emma_s + @abhi_dabhi ,Thank you so much! I had destroyed the bundle that included schema, ingestion pipeline and gateway. However, I did not clear out SQL Server CDC so that may have been the issue.I plan to leave the current gateway stopped a...

Databricks Community

User Stats

User Activity

Is Lakeflow Connect SCD Type 2 output is incompatible with Spark dec pipeline streaming tables?

Lakeflow Connect SQL Server — Snapshots Firing Outside Configured Full Refresh Window?

Lakeflow Connect - SQL Server - Issues restarting after failure

Re: Is Lakeflow Connect SCD Type 2 output is incompatible with Spark dec pipeline streaming tables?

Re: Lakeflow Connect SQL Server — Snapshots Firing Outside Configured Full Refresh Window?

Re: Lakeflow Connect - SQL Server - Issues restarting after failure

Re: Lakeflow Connect SQL Server — Snapshots Firing Outside Configured Full Refresh Window?

Re: Lakeflow Connect - SQL Server - Issues restarting after failure