Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
Showing results for 
Search instead for 
Did you mean: 

Column Data type miss-match while databricks catalog federation

New Contributor II

We are facing an issue related to the Data type mismatch when performing the federation queries (link).
Basic need is to perform union on these both tables from federated(system tables) and system tables (target/current workspace).

Struct and Map data type is getting converted into Varchar(255) while federating from another databricks account.

Temporary solution: Using of inbuild functions from_json() but again problem is the struct field are changing based on new resales which is breaking the queries. 
The federation is simple databricks-to-databricks.
Please visit these below links for more details: 

Below are the images showing the discrepancy in data-type mismatch.

image (8).pngimage (7).pngimage (6).png



New Contributor III
New Contributor III

System table data is available at the account level. If you are trying LH Federation for querying system tables isn't that already available in the source workspace?

Since system tables are governed by Unity Catalog, you need to have at least one Unity Catalog-enabled workspace in your account to enable and access system tables. System tables include data from all workspaces in your account but they can only be accessed from a Unity Catalog-enabled workspace.

System tables are enabled at the schema level. If you enable a system schema, you enable all the tables within that schema. When new schemas are released, an account admin needs to manually enable the schema.

@dkushari  correct, We have already enabled UC and system tables.
Federated Queries we are mainly using it for external data source.
For Ex:
One account is target(current account) say Azure another one which we want to federate is from Another account say AWS.
so its aws account/ system tables are federated into Azure is  Azure (system tables) <---- AWS (system tables)

Hope it makes clear. Please check the Images one is Federated(AWS/GCP) another is Target (Azure).     

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!