What are the best practices to store VectorsUDT in the databricks Feature Store?

145625 — Wed, 30 Mar 2022 16:14:19 GMT

Hello, I am having issues to save a table that has 2 columns containing VectorUDT data. I saw that the Feature Store of databricks cannot handle this data type but can handle arrays. So, I tried to convert the 2 columns into arrays.

When I tried to cast these columns as 'array<double>' using .cast('array<double>') or with the function vector_to_array from pyspark.ml.functions, I systematically got the same following error:

SparkException: Failed to execute user defined function(functions$$$Lambda$9020/747195126: (array<struct<type:tinyint,size:int,indices:array<int>,values:array<double>>>) => array<double>)

Caused by: IllegalArgumentException: function vector_to_array requires a non-null input argument and input type must be `org.apache.spark.ml.linalg.Vector` or `org.apache.spark.mllib.linalg.Vector`, but got scala.collection.mutable.WrappedArray$ofRef.

* What can I do to fix this error and convert my VectorUDT to arrays?

* Or, are there better practices to store VectorUDT in the databricks Feature Store?

Thank you for your help!

topic What are the best practices to store VectorsUDT in the databricks Feature Store? in Warehousing & Analytics

What are the best practices to store VectorsUDT in the databricks Feature Store?