cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Column Data type miss-match while databricks catalog federation

vishal-kilaskar
New Contributor II

We are facing an issue related to the Data type mismatch when performing the federation queries (link).
Basic need is to perform union on these both tables from federated(system tables) and system tables (target/current workspace).

Problem
Struct and Map data type is getting converted into Varchar(255) while federating from another databricks account.

Temporary solution: Using of inbuild functions from_json() but again problem is the struct field are changing based on new resales which is breaking the queries. 
 
The federation is simple databricks-to-databricks.
Please visit these below links for more details: 
  * https://docs.databricks.com/en/query-federation/databricks.html
  * https://docs.databricks.com/en/query-federation/index.html

Below are the images showing the discrepancy in data-type mismatch.

image (8).pngimage (7).pngimage (6).png

 

2 REPLIES 2

dkushari
Contributor

System table data is available at the account level. If you are trying LH Federation for querying system tables isn't that already available in the source workspace? 

https://docs.databricks.com/en/admin/system-tables/index.html#what-are-system-tables

https://docs.databricks.com/en/admin/system-tables/index.html#enable-system-table-schemas

Since system tables are governed by Unity Catalog, you need to have at least one Unity Catalog-enabled workspace in your account to enable and access system tables. System tables include data from all workspaces in your account but they can only be accessed from a Unity Catalog-enabled workspace.

System tables are enabled at the schema level. If you enable a system schema, you enable all the tables within that schema. When new schemas are released, an account admin needs to manually enable the schema.

@dkushari  correct, We have already enabled UC and system tables.
Federated Queries we are mainly using it for external data source.
For Ex:
One account is target(current account) say Azure another one which we want to federate is from Another account say AWS.
so its aws account/ system tables are federated into Azure is  Azure (system tables) <---- AWS (system tables)

Hope it makes clear. Please check the Images one is Federated(AWS/GCP) another is Target (Azure).     

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group