cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

jv_v
Contributor

I am currently working on a table migration project from a source Hive Metastore workspace to a target Unity Catalog workspace. After migrating the tables, I intend to write table validation scripts using the Remorph Reconcile tool. However, I am encountering an error while trying to install the tool with the following command:

Python Version: 3.12

Databricks CLI: installed

$ databricks labs install remorph

The error message I receive is as follows:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x80 pc=0x1a4982f]

goroutine 1 [running]:
github.com/databricks/cli/cmd/labs/project. (*installer).setupPythonVirtualEnvironment(0xc000112920, {0x21722c0, 0xc0001058f0}, 0x0)
github.com/databricks/cli/cmd/labs/project/installer. go:244 +0x54f
github.com/databricks/cli/cmd/labs/project. (*installer). Install (0xc000112920, {0x21722c0, 0xc0001058f0})
github.com/databricks/cli/cmd/labs/project/installer. go:108 +0x29e
github.com/databricks/cli/cmd/labs.New.newInstallCommand.func1(0xc00053e300? {0xc000107cb0? 0x4? 0x1f694d6?})
github.com/databricks/cli/cmd/labs/install. go:19 +0x74
github.com/spf13/cobra. (*Command). execute (0xc00053e300, {0xc000107c80, 0x1, 0x1})
github.com/spf13/cobra@v1.8.0/command. go:983 +0xabc
github.com/spf13/cobra. (*Command). Execute(0xc0001c5500)
github.com/spf13/cobra@v1.8.0/command. go:1115 +0x3ff
github.com/spf13/cobra. (*Command).ExecuteContextC(...)
github.com/spf13/cobra@v1.8.0/command. go:1048
github.com/databricks/cli/cmd/root. Execute(0x2172288?)
github.com/databricks/cli/cmd/root/root. go:99 +0x5b


I would greatly appreciate any guidance on resolving this installation issue. Additionally, I would like to confirm whether the Remorph Reconcile tool works for comparing tables between a source Hive Metastore workspace and a target Unity Catalog workspace, as I am attempting to use it in this context.

1 REPLY 1

Walter_C
Databricks Employee
Databricks Employee

The error you are encountering while trying to install the Remorph Reconcile tool with the command databricks labs install remorph seems to be related to an issue with the Python virtual environment setup. Based on the information we have internally, it appears that the Remorph tool currently supports Python versions 3.10 and 3.11. Since you are using Python 3.12, this might be causing the installation issue.

To resolve this, you should try using Python 3.10 or 3.11 instead.

As per the information we have this tool allows identifying discrepancies and variations in data when comparing the source with the Databricks target. We currently support Snowflake and Oracle platforms, and any table within Databricks.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group