topic Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification in Data Engineering

Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

jv_v — Tue, 17 Dec 2024 11:34:35 GMT

I am currently working on a table migration project from a source Hive Metastore workspace to a target Unity Catalog workspace. After migrating the tables, I intend to write table validation scripts using the Remorph Reconcile tool. However, I am encountering an error while trying to install the tool with the following command:

Python Version: 3.12

Databricks CLI: installed

$ databricks labs install remorph

The error message I receive is as follows:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x80 pc=0x1a4982f]

goroutine 1 [running]:
github.com/databricks/cli/cmd/labs/project. (*installer).setupPythonVirtualEnvironment(0xc000112920, {0x21722c0, 0xc0001058f0}, 0x0)
github.com/databricks/cli/cmd/labs/project/installer. go:244 +0x54f
github.com/databricks/cli/cmd/labs/project. (*installer). Install (0xc000112920, {0x21722c0, 0xc0001058f0})
github.com/databricks/cli/cmd/labs/project/installer. go:108 +0x29e
github.com/databricks/cli/cmd/labs.New.newInstallCommand.func1(0xc00053e300? {0xc000107cb0? 0x4? 0x1f694d6?})
github.com/databricks/cli/cmd/labs/install. go:19 +0x74
github.com/spf13/cobra. (*Command). execute (0xc00053e300, {0xc000107c80, 0x1, 0x1})
github.com/spf13/cobra@v1.8.0/command. go:983 +0xabc
github.com/spf13/cobra. (*Command). Execute(0xc0001c5500)
github.com/spf13/cobra@v1.8.0/command. go:1115 +0x3ff
github.com/spf13/cobra. (*Command).ExecuteContextC(...)
github.com/spf13/cobra@v1.8.0/command. go:1048
github.com/databricks/cli/cmd/root. Execute(0x2172288?)
github.com/databricks/cli/cmd/root/root. go:99 +0x5b

I would greatly appreciate any guidance on resolving this installation issue. Additionally, I would like to confirm whether the Remorph Reconcile tool works for comparing tables between a source Hive Metastore workspace and a target Unity Catalog workspace, as I am attempting to use it in this context.

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

Walter_C — Tue, 17 Dec 2024 11:48:38 GMT

The error you are encountering while trying to install the Remorph Reconcile tool with the command databricks labs install remorph seems to be related to an issue with the Python virtual environment setup. Based on the information we have internally, it appears that the Remorph tool currently supports Python versions 3.10 and 3.11. Since you are using Python 3.12, this might be causing the installation issue.

To resolve this, you should try using Python 3.10 or 3.11 instead.

As per the information we have this tool allows identifying discrepancies and variations in data when comparing the source with the Databricks target. We currently support Snowflake and Oracle platforms, and any table within Databricks.

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

jv_v — Wed, 18 Dec 2024 11:37:50 GMT

Thank you for your suggestion to try installing Remorph with Python 3.10. I followed your advice, but unfortunately, I am still encountering the same issue. Below are the steps I followed for clarity:

1)Installed the Databricks CLI on my local machine.

2)Configured the Databricks CLI for my workspace using the command:

databricks configure --profile default

This step successfully authenticated my Databricks workspace using the workspace URL and token.

3)Attempted to install the Remorph package using the following command in Git Bash:

databricks labs install remorph

Despite switching to Python 3.10, I encountered the same error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x80 pc=0x106982f]

goroutine 1 [running]:
github.com/databricks/cli/cmd/labs/project.(*installer).setupPythonVirtualEnvironment(0xc0003f7860, {0x17922c0, 0xc0001a55f0}, 0x0)
github.com/databricks/cli/cmd/labs/project/installer.go:244 +0x54f
github.com/databricks/cli/cmd/labs/project.(*installer).Install(0xc0003f7860, {0x17922c0, 0xc0001a55f0})
github.com/databricks/cli/cmd/labs/project/installer.go:108 +0x29e
github.com/databricks/cli/cmd/labs.New.newInstallCommand.func1(0xc0001aa300?, {0xc00008fcd0?, 0x4?, 0x15894d6?})
github.com/databricks/cli/cmd/labs/install.go:19 +0x74
github.com/spf13/cobra.(*Command).execute(0xc0001aa300, {0xc00008fca0, 0x1, 0x1})
github.com/spf13/cobra@v1.8.0/command.go:983 +0xabc
github.com/spf13/cobra.(*Command).ExecuteC(0xc00014d500)
github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).ExecuteContextC(...)
github.com/spf13/cobra@v1.8.0/command.go:1048
github.com/databricks/cli/cmd/root.Execute(0x1792288?)
github.com/databricks/cli/cmd/root/root.go:99 +0x5b
main.main()
github.com/databricks/cli/main.go:11 +0x2a

Could you please let me know if there are additional steps, configurations, or prerequisites I may have missed while trying to install Remorph?

I appreciate your assistance and look forward to your response.

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

Walter_C — Wed, 18 Dec 2024 14:12:08 GMT

You could try following steps in https://github.com/databrickslabs/remorph?tab=readme-ov-file#environment-setup

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

VINTER_S — Fri, 20 Dec 2024 06:15:03 GMT

hi @jv_v were you able to solve the issue,facing same error inspite of using python 3.10?

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

jv_v — Fri, 20 Dec 2024 09:30:43 GMT

Hi @VINTER_S , No Am unable to install Remorph still the same error with "databricks labs install remorph", somehow am able to install remorph with "pip install." but it is not working as expected, I followed the steps mentioned in the "https://github.com/databrickslabs/remorph?tab=readme-ov-file#environment-setup",

I feel it is better if there is any demo video using this tool and prerequisites setup video otherwise there will be definitely a confusion on how to use remorph

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

varyan — Tue, 11 Feb 2025 06:42:32 GMT

I am facing similar issues while installing it my window machine. I am using the Python version 3.10. Can someone help here

Varya@Vik_Machine MINGW64 ~
$ python --versionPython 3.10.11

Varya@Vik_Machine MINGW64 ~
$ databricks --version
Databricks CLI v0.240.0

Varya@Vik_Machine MINGW64 ~
$ databricks labs install remorph
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0xc0 pc=0x1264262]

goroutine 1 [running]:
github.com/databricks/cli/cmd/labs/project.(*installer).setupPythonVirtualEnvironment(0xc0005085e0, {0x1a44730, 0xc0001e3110}, 0x0)
github.com/databricks/cli/cmd/labs/project/installer.go:243 +0x522
github.com/databricks/cli/cmd/labs/project.(*installer).Install(0xc0005085e0, {0x1a44730, 0xc0001e3110})
github.com/databricks/cli/cmd/labs/project/installer.go:107 +0x293
github.com/databricks/cli/cmd/labs.New.newInstallCommand.func1(0xc000155208, {0xc0001e5180?, 0x4?, 0x17fa8b8?})
github.com/databricks/cli/cmd/labs/install.go:19 +0x74
github.com/spf13/cobra.(*Command).execute(0xc000155208, {0xc0001e5150, 0x1, 0x1})
github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc000394608)
github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).ExecuteContextC(...)
github.com/spf13/cobra@v1.8.1/command.go:1050
github.com/databricks/cli/cmd/root.Execute({0x1a446f8?, 0x24c62a0?}, 0xc00007c000?)
github.com/databricks/cli/cmd/root/root.go:103 +0x65
main.main()
github.com/databricks/cli/main.go:13 +0x3b

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

VINTER_S — Tue, 11 Feb 2025 13:02:54 GMT

Try to see the python path during installation by using where python or similar command.it should show the correct folder installation path which is generally script folder for windows and bin folder for mac

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

Kvant — Thu, 12 Jun 2025 07:10:26 GMT

I would just like to mention that it might not be due to remorph or your python version that you encounter this error. I got a similar error message when trying to apply changes to the metastore grants through terraform.

It worked when I authenticated to a workspace instead of the account. The terraform provider documentation provided the useful hint: "Most of Unity Catalog APIs are only accessible via workspace-level APIs. ".
If you see this error, try authenticating to a workspace.

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

VINTER_S — Thu, 12 Jun 2025 09:21:39 GMT

Thanks @Kvant for your input. it will help other users following your usecase.I was using it in python virtual environment for databricks cdw migration accreditation.it was resolved after resolving python path.

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

Prashant_151 — Fri, 12 Sep 2025 07:31:38 GMT

hi @VINTER_S

Can you explain how you were able to fix error panic: runtime error: invalid memory address or nil pointer dereference..

I am also facing same error in edw migration accreditation. I have created virtual environment of 3.10 and databricks configuration in vs code shows no issues

Re: Issue with Installing Remorph Reconcile Tool and Compatibility Clarification

VINTER_S — Fri, 12 Sep 2025 08:27:23 GMT

Hey @Prashant_151

, it's been a while since I completed this accreditation, but could you verify which Python executable is being used in the terminal after activating a virtual environment (venv)? You can do this by running the which python command (on Mac/Linux) or where python (on Windows). It should point to the Scripts (Windows) or bin (Mac/Linux) folder of the active virtual environment. For me,it was getting changed to some other directory before installing remorph.