Intro
Unlock Unity Catalog governance and performance by upgrading Hive Metastore (HMS) and AWS Glue foreign tables to Managed Tables using the new Upgrade Foreign Table workflow. Managed Tables provide automatic optimizations that can boost query performance by 20x and cut storage costs by 50%+.
This blog covers how the Upgrade Foreign Table workflow streamlines migrations by enabling customers to upgrade Foreign Tables in place using ALTER TABLE <foreign table> SET MANAGED | EXTERNAL. This workflow has saved customers months of migration planning by minimizing downtime, preserving table history, and maintaining existing table configurations, including the same name, settings, permissions, and views.
Simplifying Foreign Table Migration to Unity Catalog
Today, thousands of customers use Lakehouse Federation to query data stored in external systems. AWS and Glue federation help teams adopt UC in phases and avoid a single “big-bang” event, where every team has to migrate all at once.
For example, analysts may be onboard to UC first to take advantage of features like Genie, while the data engineering team continues to manage the same table in HMS during the transition. Without federation, this split can create schema drift because the same data is referenced in two systems. The Federation keeps the metadata for the table synchronized, allowing teams to adopt UC immediately and seamlessly.
The problem is that when customers are ready to upgrade foreign tables and complete their migration to UC, the process becomes difficult. Migrating a table to UC requires a customer to:
- Move workloads to a new namespace and update code
- Re-create access control lists
- Re-create foreign views that referenced federated tables
- Lose table history and time-travel capability
With ALTER TABLE <foreign table> SET MANAGED | EXTERNAL, all of that manual work goes away. You can now upgrade tables in the same namespace, with the same history, permissions, and settings - using a simple command.
In the next section, we’ll walk through how we’ve simplified the process of migrating foreign tables to UC with the new Upgrade Foreign Table workflow. At a high level, the process consists of three steps:
- Federate your HMS or AWS Glue catalog into Unity Catalog
- Upgrade tables using ALTER TABLE <foreign table> SET MANAGED | EXTERNAL
- Remove the federation connection once the migration is complete
“By converting foreign tables to UC with one click, we eliminated months of migration planning and accelerated our adoption of UC and Serverless.”
- Vipul Javeri, Sr. Staff Engineer, Outreach
How the Upgrade Foreign Table Workflow Works
Step 1: Set up Lakehouse Federation for your external catalog
If the Lakehouse Federation is not already configured, create a connection from your Databricks workspace to your external catalog and set up a foreign catalog (For more details, refer to the documentation). Unity Catalog will then crawl the external catalog and create foreign tables. Now, Databricks workloads can query source tables in HMS with UC governance.
With no disruption to your environment, your data becomes instantly accessible in Catalog Explorer and governable with UC.
Step 2: Migrate source tables to Unity Catalog
When you’re ready to migrate source tables to Unity Catalog, run:
ALTER TABLE <foreign table> SET MANAGED | EXTERNAL Choosing SET MANAGED unlocks UC’s Predictive Optimization and performance enhancements from day one:
- Self-maintenance with compaction, clustering, and vacuuming
- Storage and compute cost savings through intelligent optimization
- Faster queries across all clients, not just in Databricks
- Automatic upgrades with the latest features
- Learn more about Managed Tables and their benefits here.
Once converted, the table stops syncing with your external catalog, becomes writable from UC, and is no longer labeled as foreign, giving you a clear indication that it’s now fully governed under UC.
See Convert Foreign Table to Managed and Convert Foreign Table to External for more details.
Step 3: Drop the federation connection
Once your migrated tables and workloads are tested and operating smoothly, and you no longer rely on federation to your external catalog, you can remove the connection using:
ALTER CATALOG <foreign catalog> DROP CONNECTIONYou can now simplify your architecture with a fully unified governance layer in UC.
See documentation for Drop Connection for more details.
Next Steps: Move Forward with Unity Catalog
You can begin your Unity Catalog migration journey today, even if you have a complex catalog environment or want to avoid an all-at-once cutover. With Lakehouse Federation and the Upgrade Foreign Table workflow, you can migrate incrementally, reduce downtime, and move workloads at your own pace.
You can upgrade not only foreign tables but also external tables to Managed Tables! Learn more here
Try out the Upgrade Foreign Table workflow today! If you want tailored guidance, contact your Databricks account team, who can connect you with a specialist to support your rollout.
What's Coming Next?
- Ability to upgrade foreign Iceberg tables to managed Iceberg tables
- Ability to upgrade foreign tables from more Lakehouse Federation sources (Snowflake, Redshift, etc.) to Managed Tables
1 Comment
