โ07-22-2022 05:09 AM
Hello community!
I have a table with a column that is an array of a struct that has a very very long schema.
When the table is written, all works well. Though, when I create a view based on this table and I try to access the view I get the error:
rg.apache.spark.SparkException: Cannot recognize hive type string: array<struct<hitNumber:bigint....,latencyTracking:struct<pageLoadSample:bigint,pageLoadTime:bigint,pageDownloadTime:bigin, column: hits, db: test, table: test
as you see the actual schema has been truncated at the end, before the column name comes.
The question is: is there a configuration on the size of the hive type string I could play with?
Thank you in advance,
โ07-25-2022 12:38 AM
what version of hive metastore do you use? Because there are issues with large metadata in metastore versions < 2.3.0
โ07-25-2022 05:11 AM
Hello @Werner Stinckensโ !
I see your point. Just checked and we are still in 0.13.0 (default one)
Is there any official documentation from #[Azure databricks]โ on how to do that?
Related to this, but not solved yet
โ07-25-2022 05:50 AM
https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore
โ
That's the only doc I know about.
โ07-26-2022 12:08 AM
Thanks a lot @Werner Stinckensโ !
I also came across this kind of documentation.. The question is whether you can upgrade your current internal metastore, and nothing about that unfortunately ๐
โ08-22-2022 11:45 AM
@Pantelis Maroudisโ , were you able to solve this issue? Please advise if you got any tip. Thanks in advance
โ09-13-2022 09:17 AM
Hello Afzal,
Unfortunately no ๐ We have not been able to create a view on this table.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group