01-17-2024 07:44 AM
Hi,
I have an external table which is created out of a S3 bucket. The first time I am creating the table I am using the following command :
01-17-2024 08:08 AM
Hi @Dp15, Yes, it is possible to refresh the metadata of an external table in Databricks. However, the MSCK REPAIR TABLE command is not supported in the Unity Catalog.
Instead, you can use the REFRESH TABLE command to invalidate the cached entries for data and metadat.... The invalidated cache is populated in a lazy manner when the cached table or the query associated wi....
Please note that you need to have the necessary privileges to execute these commands.
01-17-2024 08:08 AM
Hi @Dp15, Yes, it is possible to refresh the metadata of an external table in Databricks. However, the MSCK REPAIR TABLE command is not supported in the Unity Catalog.
Instead, you can use the REFRESH TABLE command to invalidate the cached entries for data and metadat.... The invalidated cache is populated in a lazy manner when the cached table or the query associated wi....
Please note that you need to have the necessary privileges to execute these commands.
01-17-2024 08:43 AM
Hi @Kaniz_Fatma thank you for reply, how can we handle the schema changes in the external location, what if there are additions or deletions on the schema, will the refresh table work then too?
01-18-2024 12:42 AM
Hi @Dp15, According to the documentation, you can update the schema of a Delta Lake table using DDL or DML. You can add, reorder, or rename columns in a table. However, you cannot change the data type or nullability of existing columns.
If you use an external location to store your data, you need to create a storage credential and an e.... You can alter the properties of an external location or rename it using the ALTER EXTERNAL LOCATION .... You can also refresh the metadata of an external table using the REFRESH TABLE command.
I hope this helps you understand how to handle schema changes in external locations in Databricks. 😊
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!