cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

unity catalog information schema columns metadata out of sync with table - cant refresh

jakubk
Contributor

I'm using unity catalog

I've changed the schema of my table by overwriting it with a newer file

    df.write \
        .format('delta') \
        .partitionBy(partitionColumn) \
        .mode("overwrite") \
        .option("overwriteSchema", "true") \
        .save(destinationPath)

queries to information_schema.columns within the catalog and from system.information_schema.columns both return the previous schema not the current one

The databricks gui catalog browser also shows the wrong columns

This is bad as many third party data viz and integration tools and code rely on querying information schema

How can i force databricks to show the correct metadata

1 ACCEPTED SOLUTION

Accepted Solutions

daniel_sahal
Esteemed Contributor

@jakubk Try running

REPAIR TABLE <table_name> SYNC METADATA

 

View solution in original post

2 REPLIES 2

daniel_sahal
Esteemed Contributor

@jakubk Try running

REPAIR TABLE <table_name> SYNC METADATA

 

Thanks Daniel, that worked

i also had a workaround going with an alter table set tblproperties (optimizewrite = true) which I have set as a default anyway. But just touching the table triggered a metadata refresh

i prefer your command tho as that is its primary purpose

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!