Databricks Lakebridge: Azure SQL DB to Databricks (Error while import)

[shubham007]

Hi community experts,

I am getting error "cannot import name 'recon' from 'databricks.labs.lakebridge.reconcile.execute'" importing modules as shown in attached screenshot. I am follwing steps as mentioned in your partner training module "Lakebridge for SQL Source System Migration". Please help to resolve this issue.

[szymon_dybczak]

Hi @shubham007 ,

They made refactoring to that module in last month so that's why it stopped working. Probably Lakebridge for SQL Source System Migration module was recorded before that change.

And why they made change? It is explained here:

Split recon execute into smaller modules (#1892) · databrickslabs/lakebridge@7caba7a · GitHub

So now the correct way to use this module is like this:

lakebridge/docs/lakebridge/docs/reconcile/index.mdx at 7caba7a3d6d96850f7b9662307ebdaad392bfc80 · da...

from databricks.labs.lakebridge import __version__
from databricks.sdk import WorkspaceClient

from databricks.labs.lakebridge.reconcile.trigger_recon_service import TriggerReconService
from databricks.labs.lakebridge.reconcile.exception import ReconciliationException

ws = WorkspaceClient(product="lakebridge", product_version=__version__)


try:
  result = TriggerReconService.trigger_recon(
            ws = ws,
            spark = spark, # notebook spark session
            table_recon = table_recon, # previously created
            reconcile_config = reconcile_config # previously created
          )
  print(result.recon_id)
  print(result)
except ReconciliationException as e:
    recon_id = e.reconcile_output.recon_id
    print(f" Failed : {recon_id}")
    print(e)
except Exception as e:
    print(e.with_traceback)
    raise e
    print(f"Exception : {str(e)}")

[shubham007]

Hi community experts,

I am getting error while Data and Schema Validation with the Reconciler. As attached here screenshots. Please help resolve this issue.

Output:

[szymon_dybczak]

HI @shubham007 ,

Are you sure you configured it correctly?

[shubham007]

Regarding which configuration ? I have created scope and secrets correctly. You mean about “databricks labs Lakebridge configure-reconcile” ?

[szymon_dybczak]

I think you have mistake in your config file. Do you have a NONE catalog? I guess you don't have, so something went wrong during configuration phase:
NONE.SalesLT.customer

Did you follow this guide?

Reconcile Guide | Lakebridge

[shubham007]

I am following that guide only. My recon_config_file as below:

and given below parameters while configuring reconcile:

[szymon_dybczak]

Could you send me link to a guide you're using? I'll try to recreate your scenario then

[shubham007]

https://partner-academy.databricks.com/learn/courses/4326/lakebridge-for-sql-source-system-migration...

https://databrickslabs.github.io/lakebridge/docs/reconcile/

My Azure SQL DB:

connection string: jdbc:sqlserver://sqldb.database.windows.net:1433;database=sqldb;user=sqladmin@sqldb;password={your_password_here};encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;

Target-Databricks:

here need to migrate tables:

[shubham007]

Hi @szymon_dybczak , which type of .sql scripts are required for migration such as legacy metadata and INSERT scripts to populate data in respective tables ?