Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

[jv_v]

I am currently working on a table migration project from a source Hive Metastore workspace to a target Unity Catalog workspace. After migrating the tables, I intend to write table validation scripts using the Remorph Reconcile tool. However, I am encountering an error while trying to install the tool with the following command:

Python Version: 3.12

Databricks CLI: installed

$ databricks labs install remorph

The error message I receive is as follows:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x80 pc=0x1a4982f]

goroutine 1 [running]:
github.com/databricks/cli/cmd/labs/project. (*installer).setupPythonVirtualEnvironment(0xc000112920, {0x21722c0, 0xc0001058f0}, 0x0)
github.com/databricks/cli/cmd/labs/project/installer. go:244 +0x54f
github.com/databricks/cli/cmd/labs/project. (*installer). Install (0xc000112920, {0x21722c0, 0xc0001058f0})
github.com/databricks/cli/cmd/labs/project/installer. go:108 +0x29e
github.com/databricks/cli/cmd/labs.New.newInstallCommand.func1(0xc00053e300? {0xc000107cb0? 0x4? 0x1f694d6?})
github.com/databricks/cli/cmd/labs/install. go:19 +0x74
github.com/spf13/cobra. (*Command). execute (0xc00053e300, {0xc000107c80, 0x1, 0x1})
github.com/spf13/cobra@v1.8.0/command. go:983 +0xabc
github.com/spf13/cobra. (*Command). Execute(0xc0001c5500)
github.com/spf13/cobra@v1.8.0/command. go:1115 +0x3ff
github.com/spf13/cobra. (*Command).ExecuteContextC(...)
github.com/spf13/cobra@v1.8.0/command. go:1048
github.com/databricks/cli/cmd/root. Execute(0x2172288?)
github.com/databricks/cli/cmd/root/root. go:99 +0x5b

I would greatly appreciate any guidance on resolving this installation issue. Additionally, I would like to confirm whether the Remorph Reconcile tool works for comparing tables between a source Hive Metastore workspace and a target Unity Catalog workspace, as I am attempting to use it in this context.

[Walter_C · Accepted solution]

The error you are encountering while trying to install the Remorph Reconcile tool with the command databricks labs install remorph seems to be related to an issue with the Python virtual environment setup. Based on the information we have internally, it appears that the Remorph tool currently supports Python versions 3.10 and 3.11. Since you are using Python 3.12, this might be causing the installation issue.

To resolve this, you should try using Python 3.10 or 3.11 instead.

As per the information we have this tool allows identifying discrepancies and variations in data when comparing the source with the Databricks target. We currently support Snowflake and Oracle platforms, and any table within Databricks.

[jv_v]

Thank you for your suggestion to try installing Remorph with Python 3.10. I followed your advice, but unfortunately, I am still encountering the same issue. Below are the steps I followed for clarity:

        1)Installed the Databricks CLI on my local machine.

        2)Configured the Databricks CLI for my workspace using the command:

            databricks configure --profile default

            This step successfully authenticated my Databricks workspace using the workspace URL and token.

        3)Attempted to install the Remorph package using the following command in Git Bash:

          databricks labs install remorph

Despite switching to Python 3.10, I encountered the same error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x80 pc=0x106982f]

goroutine 1 [running]:
github.com/databricks/cli/cmd/labs/project.(*installer).setupPythonVirtualEnvironment(0xc0003f7860, {0x17922c0, 0xc0001a55f0}, 0x0)
github.com/databricks/cli/cmd/labs/project/installer.go:244 +0x54f
github.com/databricks/cli/cmd/labs/project.(*installer).Install(0xc0003f7860, {0x17922c0, 0xc0001a55f0})
github.com/databricks/cli/cmd/labs/project/installer.go:108 +0x29e
github.com/databricks/cli/cmd/labs.New.newInstallCommand.func1(0xc0001aa300?, {0xc00008fcd0?, 0x4?, 0x15894d6?})
github.com/databricks/cli/cmd/labs/install.go:19 +0x74
github.com/spf13/cobra.(*Command).execute(0xc0001aa300, {0xc00008fca0, 0x1, 0x1})
github.com/spf13/cobra@v1.8.0/command.go:983 +0xabc
github.com/spf13/cobra.(*Command).ExecuteC(0xc00014d500)
github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).ExecuteContextC(...)
github.com/spf13/cobra@v1.8.0/command.go:1048
github.com/databricks/cli/cmd/root.Execute(0x1792288?)
github.com/databricks/cli/cmd/root/root.go:99 +0x5b
main.main()
github.com/databricks/cli/main.go:11 +0x2a

Could you please let me know if there are additional steps, configurations, or prerequisites I may have missed while trying to install Remorph?

I appreciate your assistance and look forward to your response.

[Walter_C]

You could try following steps in https://github.com/databrickslabs/remorph?tab=readme-ov-file#environment-setup 

[VINTER_S]

hi @jv_v were you able to solve the issue,facing same error inspite of using python 3.10?

[jv_v]

Hi @VINTER_S  , No Am unable to install Remorph still the same error with "databricks labs install remorph", somehow am able to install remorph with "pip install." but it is not working as expected, I followed the steps mentioned in the "https://github.com/databrickslabs/remorph?tab=readme-ov-file#environment-setup",

I feel it is better if there is any demo video using this tool and prerequisites setup video otherwise there will be definitely a confusion on how to use remorph