Unity catalog - external table lastUpdateversion

[rudyevers]

We are currently upgrading our Lakehouse to use the Unity Catalog benefits. We will mostly use external tables because alle our DETLA tables are already stored in Azure Storage. I try to figure out how to update the table property "delta.lastUpdateverion". Since table schema's can change over time you want the external tables have the last schema version. Looking at the documentation, nothing is mentioned about this table property as far as I have seen. The data in the table did change when we updated the delta table, but somehow you fully need to recreate the table if the schema has changed. 

What is a best practice is this case? Recreate the external table? Or did I overlooked something?

[rudyevers]

Hi @Retired_mod 

Thank you for your reponse. I am aware that you can change your DELTA with de DDL statements, but in our case we write directly to storage and not to the unity catalog. So when a external table is created referring to a external location it takes the version at that specific moment. When the schema is changed afterwards the external table is not update because it is still referring to a previous version. That's also what the tabel property delta.lastUpdateversion is saying. So in this case it looks like you have to drop and create the external table so the delta.lastUpdateversion is the correct one. 

[rudyevers]

Hi @Retired_mod,

OK! My assumption was right. So that is what we have to live with for now. But a sort of refresh table function would be nice for external DELTA tables 😉

The project team was quite early with adopting DELTA as storage format but is willing to adopt the Databricks capabilities more and more. But as early bird you sometimes suffer from the choices that are made in the past. Our whole data logistic process works in away that we are able to change it easly over night (manner of speaking). But we will get there over time.  

Thanks

[-werners-]

I am in the same boat.

That is the reason I opted to use managed tables instead.  OK; it means migrating tables and changing notebooks but besides not having to struggle with external tables, you also get something in return (liquid clustering f.e.).

[-werners-]

Liquid clustering ofc also exist for external tables, what I meant is all the upcoming AI-features, of which I doubt will be available for external tables.