Anonymous
Not applicable
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-04-2023 06:17 PM
@vikashk84
The exception "RuntimeException: Caught Hive MetaException attempting to get partition metadata by filter from Hive" typically occurs when there is an issue with Hive metadata related to partitioning in Databricks. Here are a few steps you can try to resolve the issue:
- Check Hive Metastore Configuration: Ensure that the Hive Metastore is properly configured in Databricks. You can check the Hive Metastore configuration settings in the Databricks cluster's configuration or runtime environment, and verify that it is correctly pointing to the Hive Metastore service.
- Verify Partitioning: Double-check that the partitioning column "version" in the table "db.tbl" is correctly defined and matches the partitioning column used in the "MSCK REPAIR TABLE" command. Ensure that the partitioning column is of the correct data type and matches the data type of the "version" column in the Parquet dataframe "df".
- Check Table Metadata: Verify that the table metadata is correctly updated after running the "MSCK REPAIR TABLE" command. You can check the table metadata in the Hive Metastore to ensure that the partitions are correctly registered.
- Check View Definition: Review the view "db.tbl_v" definition to ensure that it is correctly referencing the table "db.tbl" and its partitioning column "version". Make sure that the view definition is correctly written and does not contain any errors.
- Verify Column Names: Check that the column names used in the "filter" and "drop" operations ("col1" and "col2") in the code are correct and match the column names in the table or view. Make sure there are no typographical errors or differences in column names between the table, view, and code.
If you have checked all of the above and the issue persists, it may be necessary to further investigate the specific details of your environment and data to identify the root cause of the exception.