topic unity catalog information schema columns metadata out of sync with table - cant refresh in Warehousing & Analytics

unity catalog information schema columns metadata out of sync with table - cant refresh

jakubk — Fri, 21 Jun 2024 03:36:16 GMT

I'm using unity catalog

I've changed the schema of my table by overwriting it with a newer file

df.write \ .format('delta') \ .partitionBy(partitionColumn) \ .mode("overwrite") \ .option("overwriteSchema", "true") \ .save(destinationPath)

queries to information_schema.columns within the catalog and from system.information_schema.columns both return the previous schema not the current one

The databricks gui catalog browser also shows the wrong columns

This is bad as many third party data viz and integration tools and code rely on querying information schema

How can i force databricks to show the correct metadata

Re: unity catalog information schema columns metadata out of sync with table - cant refresh

daniel_sahal — Fri, 21 Jun 2024 06:45:32 GMT

@jakubk Try running

REPAIR TABLE <table_name> SYNC METADATA

Re: unity catalog information schema columns metadata out of sync with table - cant refresh

jakubk — Tue, 25 Jun 2024 04:03:42 GMT

Thanks Daniel, that worked

i also had a workaround going with an alter table set tblproperties (optimizewrite = true) which I have set as a default anyway. But just touching the table triggered a metadata refresh

i prefer your command tho as that is its primary purpose