Too long hive type string
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-22-2022 05:09 AM
Hello community!
I have a table with a column that is an array of a struct that has a very very long schema.
When the table is written, all works well. Though, when I create a view based on this table and I try to access the view I get the error:
rg.apache.spark.SparkException: Cannot recognize hive type string: array<struct<hitNumber:bigint....,latencyTracking:struct<pageLoadSample:bigint,pageLoadTime:bigint,pageDownloadTime:bigin, column: hits, db: test, table: test
as you see the actual schema has been truncated at the end, before the column name comes.
The question is: is there a configuration on the size of the hive type string I could play with?
Thank you in advance,
- Labels:
-
Azure databricks
-
Table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-25-2022 12:38 AM
what version of hive metastore do you use? Because there are issues with large metadata in metastore versions < 2.3.0
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-25-2022 05:11 AM
Hello @Werner Stinckens !
I see your point. Just checked and we are still in 0.13.0 (default one)
Is there any official documentation from #[Azure databricks] on how to do that?
Related to this, but not solved yet
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-25-2022 05:50 AM
https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore
That's the only doc I know about.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2022 12:08 AM
Thanks a lot @Werner Stinckens !
I also came across this kind of documentation.. The question is whether you can upgrade your current internal metastore, and nothing about that unfortunately 😑
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-22-2022 11:45 AM
@Pantelis Maroudis , were you able to solve this issue? Please advise if you got any tip. Thanks in advance
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-13-2022 09:17 AM
Hello Afzal,
Unfortunately no 😞 We have not been able to create a view on this table.

