topic Too long hive type string in Data Engineering

Too long hive type string

pantelis_mare — Fri, 22 Jul 2022 12:09:45 GMT

Hello community!

I have a table with a column that is an array of a struct that has a very very long schema.

When the table is written, all works well. Though, when I create a view based on this table and I try to access the view I get the error:

rg.apache.spark.SparkException: Cannot recognize hive type string: array<struct<hitNumber:bigint....,latencyTracking:struct<pageLoadSample:bigint,pageLoadTime:bigint,pageDownloadTime:bigin, column: hits, db: test, table: test

as you see the actual schema has been truncated at the end, before the column name comes.

The question is: is there a configuration on the size of the hive type string I could play with?

Thank you in advance,

Re: Too long hive type string

-werners- — Mon, 25 Jul 2022 07:38:05 GMT

what version of hive metastore do you use? Because there are issues with large metadata in metastore versions < 2.3.0

Re: Too long hive type string

pantelis_mare — Mon, 25 Jul 2022 12:11:15 GMT

Hello @Werner Stinckens !

I see your point. Just checked and we are still in 0.13.0 (default one)

Is there any official documentation from #[Azure databricks] on how to do that?

Related to this, but not solved yet

Re: Too long hive type string

-werners- — Mon, 25 Jul 2022 12:50:44 GMT

https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore

That's the only doc I know about.

Re: Too long hive type string

pantelis_mare — Tue, 26 Jul 2022 07:08:55 GMT

Thanks a lot @Werner Stinckens !

I also came across this kind of documentation.. The question is whether you can upgrade your current internal metastore, and nothing about that unfortunately 😑

Re: Too long hive type string

Afzal — Mon, 22 Aug 2022 18:45:10 GMT

@Pantelis Maroudis , were you able to solve this issue? Please advise if you got any tip. Thanks in advance

Re: Too long hive type string

pantelis_mare — Tue, 13 Sep 2022 16:17:53 GMT

Hello Afzal,

Unfortunately no 😞 We have not been able to create a view on this table.