Column Data type miss-match while databricks catalog federation
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-30-2024 01:34 AM
We are facing an issue related to the Data type mismatch when performing the federation queries (link).
Basic need is to perform union on these both tables from federated(system tables) and system tables (target/current workspace).
Problem: Struct and Map data type is getting converted into Varchar(255) while federating from another databricks account.
Temporary solution: Using of inbuild functions from_json() but again problem is the struct field are changing based on new resales which is breaking the queries.
The federation is simple databricks-to-databricks.
Please visit these below links for more details:
* https://docs.databricks.com/en/query-federation/databricks.html
* https://docs.databricks.com/en/query-federation/index.html
Below are the images showing the discrepancy in data-type mismatch.
- Labels:
-
Unity Catalog