Lakebridge reconciliation code keeps running continuously without Spark jobs or errors

[Akshay_Petkar]

Hi,

I am facing an issue while running the Lakebridge reconciliation code in Databricks using TriggerReconService.trigger_recon().

The code keeps running continuously without any output, error, or logs. Also, no Spark jobs are getting triggered or shown in the Spark UI.

I also added exception handling using ReconciliationException and generic Exception, but no exception is being thrown. The process seems to hang during the TriggerReconService.trigger_recon() execution itself.

The cluster is running properly and the notebook is attached correctly, so it looks like the process is getting stuck before Spark execution starts.

Has anyone faced a similar issue specifically during the Lakebridge reconciliation execution?

Any debugging suggestions or checks would be really helpful.

Thanks!

[amirabedhiafi · Accepted solution]

Hello  !

I had something similar and at that time I understood that it is an initialization issue and not a reconciliation performance issue.

Why ? because lakebridge reconciliation should eventually execute spark actions when it fetches schemas or data and writes reconciliation metadata. The flow runs TriggerReconService.trigger_recon(...) from the notebook and Lakebridge stores reconciliation output in its metadata catalog or schema after the run.

Check the package :

import databricks.labs.lakebridge as lb
print(lb.__version__)

The latest PyPI version currently shown is 0.12.2, released Feb 26, 2026. https://pypi.org/project/databricks-labs-lakebridge and this is relevant because recent releases improved reconciliation exception handling and logging and even serverless compatibility. https://github.com/databrickslabs/remorph/releases

I had to enable debug logging before calling trigger_recon() and test basic spark and workspace client separately and I found that it was hanging(so here the issue is not lakebridge reconciliation)

Lakebridge requires permissions to create or use reconciliation metadata objects such as USE CATALOG, CREATE SCHEMA and if using an existing schema, volume permissions.

What I discovered at that time that lakebridge uses UC volumes for intermediate persistence on serverless clusters and the volume must exist with write permissions.

[Akshay_Petkar]

Thanks! We are now able to see the debug logs and the code is progressing further than before.

Schema fetch is working successfully for both Synapse and Databricks. However, the Spark jobs are getting stuck with:

• 0 rows read

• 0 bytes read

• 0 bytes written

The notebook cell keeps running continuously without completing.

We are using:

• an all-purpose cluster

• report_type = row

• and the table contains only around 1000 rows

Has anyone seen Spark jobs remain in this state during Lakebridge reconciliation?

Any suggestions on possible causes or debugging steps would be very helpful.

Thanks!

[Akshay_Petkar]

We also tried multiple configurations including:

• report_type = row

• report_type = all

• report_type = schema

But in all cases, the Spark job eventually gets stuck at some point while the cell keeps running continuously without completing.

[Akshay_Petkar]

Could you please confirm whether Lakebridge reconciliation requires all connection properties to be stored in Key Vault/secrets, including:

• username

• password

• host

• port

• database

• encrypt

• trustServerCertificate

We are asking because reconciliation is generating queries with values like None.database_name.table_name, so we are trying to understand whether some connection properties are not being resolved correctly from secrets/configuration.

[amirabedhiafi]

Hi again @Akshay_Petkar  !

I don't think that all values must be stored in Azure Key Vault. it depends on which Lakebridge reconciliation config version or path you are using.

For the older notebook style config, Lakebridge has a secret_scope for source connection credentials and DatabaseConfig still needs the source or target catalog and schema values. If you check the doc secret_scope, source_schema, target_catalog, target_schema and optional source_catalog are part of the reconciliation config. https://databrickslabs.github.io/lakebridge/docs/reconcile/recon_notebook/

The important clue is this None.database_name.table_name whichnormally means one of the catalog values is being resolved as none not that Spark is slow.

[KrisJohannesen]

Could you give an example of the config files you have set up for running the reconciliation? The file determines most of the settings - so without it it is hard to assist you