topic Re: Error in migration with UCX tool in Data Engineering

Error in migration with UCX tool

priyansh — Wed, 18 Sep 2024 06:15:52 GMT

Hey folks!

I am facing an issue while migration the tables from Hive to UC using UCX tool; after completely running the setup and getting the assessment overview, we ran the following commands i.e. "databricks labs ucx create-table-mapping" but after we ran the command "databricks labs ucx migrate-tables", it should be supposed to migrate the tables those which are in my dbfs and already in the parquet format. As after successfully running the command, I am not getting any error but the database that has been created in the UC, the following tables are not reflecting there in the UC. I also ran the SYNC command for the external tables too, but the issue is same in that case also

I would also like to know that in the assessment overview what does the UC readiness signifies?
For your reference I am attaching the screenshots, please refer to them...
Thanks,

Re: Error in migration with UCX tool

gchandra — Sun, 22 Sep 2024 03:55:05 GMT

After running this command databricks labs ucx create-table-mapping, you should see a text file with a list of tables. I hope you updated the respective Catalog names before running the migrate-tables command.

90% of the total objects can be readily migrated to UC.

Re: Error in migration with UCX tool

priyansh — Mon, 23 Sep 2024 04:07:19 GMT

@gchandra Yes we have updated the respective Catalog names accordingly but unfortunately the tables are not reflecting there in the UC Catalog
Can you also send me the proper steps, In case We are missing any of them.

Re: Error in migration with UCX tool

BasheerAhmed — Tue, 24 Sep 2024 07:41:21 GMT

Hi @priyansh ,

If your workflow run successfull without any error, you can check the logs why to find why your table is not migrated from hive metastore to UC

In above img you can see a log folder where you can see all your workflow logs,

and one query is can you able to create table in your catalog?? if not that will be an issue

Re: Error in migration with UCX tool

priyansh — Tue, 24 Sep 2024 08:19:29 GMT

Hii @BasheerAhmed ,
data bricks labs UCX create-catalogs-schemas
After running the above command, we were able to create the catalog and schemas,

But unfortunately, after running this command data bricks labs UCX migrate-tables, Tables are not reflecting in UC and on verifying the log files the error we found was 429 too many requests.
"Changes have been made in the commands to post "

Re: Error in migration with UCX tool

priyansh — Tue, 24 Sep 2024 08:21:53 GMT

@gchandra for your reference,

Re: Error in migration with UCX tool

gchandra — Tue, 24 Sep 2024 11:53:40 GMT

Please open an issue with the UCX team.

https://github.com/databrickslabs/ucx/issues

Re: Error in migration with UCX tool

Akash_Wadhankar — Wed, 05 Feb 2025 14:47:44 GMT

After running this tool I am getting error of authentication while running the tool

C:\Users\xyz>databricks labs ucx create-table-mapping
19:45:05 INFO [src/databricks/labs/ucx] Running cmd for workspace 2695741837056558
19:45:07 ERROR [src/databricks/labs/ucx.create-table-mapping] ValueError: Please run as account-admin: databricks labs ucx sync-workspace-info

Re: Error in migration with UCX tool

vidya_kothavale — Wed, 05 Feb 2025 17:12:25 GMT

Ensure that you are a Databricks account admin.

Before running the table mapping command, execute the following command:

databricks auth login https://accounts.cloud.databricks.com/

This command will prompt you to enter your Databricks account ID and PAT (Personal Access Token).

For more details on generating a Databricks PAT, refer to this link:
Databricks Personal Access Token Guide

Re: Error in migration with UCX tool

Akash_Wadhankar — Tue, 11 Feb 2025 11:30:12 GMT

While running the UCX tool in workspace we are not able to access the tables which are created in hive metastore on default schema. When we run the migrate-table workflow we get the error that the tables are not accessible.

We get the following error:

06:55:33 WARN [d.l.u.hive_metastore.mapping][checking_all_database_properties_1] Failed to get properties for Table hive_metastore.default.air_quality: [TABLE_OR_VIEW_NOT_FOUND] The table or view `default`.`air_quality` cannot be found. Verify the spelling and correctness of the schema and catalog. If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog. To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 19;

Since the tables are not accessible so migrations also not successful.

Re: Error in migration with UCX tool

Akash_Wadhankar — Tue, 11 Feb 2025 11:35:18 GMT

Also when we run the command - databricks labs ucx principal-prefix-access --aws-profile test-profile

we do not see aws_instance_profile_info.csv file instead of this uc_roles_access.csv is created. Not sure if it is related. Please help.