cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Refresh a External table metadata

Dp15
Contributor

Hi,

I have an external table which is created out of a S3 bucket. The first time I am creating the table I am using the following command :

 

query = """CREATE TABLE IF NOT EXISTS catalog.schema.external_table_s3
           USING PARQUET
           LOCATION '{}'""".format(bucket_location)
 
The first time the table is created the files in the 'bucket_location' are loaded into the table. Now, I have a use case where when new files are added in the S3, I would like the metadata of this external table to be refreshed. Is it possible to refresh the metadata of the external table? I tried using the MSCK Repair command 

MSCK REPAIR TABLE catalog.schema.external_table_s3

but my command failed with the following error:
 
AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): Repair Table are not supported in Unity Catalog. ;
RepairTableCommand catalog.schema.external_table_s3, true, false, MSCK REPAIR TABLE
 
1 REPLY 1

Dp15
Contributor

Hi @Retired_mod thank you for reply, how can we handle the schema changes in the external location, what if there are additions or deletions on the schema, will the refresh table work then too? 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group