topic Error with multiple FeatureLookup calls outside databricks in Data Engineering

Error with multiple FeatureLookup calls outside databricks

Nath — Fri, 09 Sep 2022 13:14:30 GMT

I access databricks feature store outside databricks with databricks-connect on my IDE pycharm.

The problem is just outside Databricks, not with a notebook inside Databricks.

I use FeatureLookup mecanism to pull data from Feature store tables in my customer Dataframe (class FeatureLookup, API Training_set and load_df).

I need to add several Features from several Feature store tables. So I call this mecanism several times.

It doesn't work from 6th call. If I pull the 6th feature only, it works.

I have attached the StackTraceError.

The problem is in the file databricks\feature_store\utils\feature_lookup_utils.py, line : plan = spark.sql("explain cost select * from view").collect()[0][0]).

Thanks.

Nath

Re: Error with multiple FeatureLookup calls outside databricks

shan_chandra — Wed, 12 Oct 2022 15:59:10 GMT

ProtoSerializer is known to be prone to StackOverflow when very deep query plans are being used. Could you please try increasing the stack size of the client JVM with the spark conf -spark.driver.extraJavaOptions which can be defined in ${spark_home}/conf/spark-defaults.conf (e.g,

spark.driver.extraJavaOptions -Xss4M

), where ${spark_home} can be found with databricks-connect get-spark-home

Re: Error with multiple FeatureLookup calls outside databricks

Harrison691 — Sat, 15 Oct 2022 04:50:23 GMT

I was looking for the same, Thank you for the suggestion.

GMGlobalConnect

Re: Error with multiple FeatureLookup calls outside databricks

shan_chandra — Thu, 27 Oct 2022 15:40:39 GMT

Also, Please refer to the below KB for additional resolution - https://learn.microsoft.com/en-us/azure/databricks/kb/dev-tools/dbconnect-protoserializer-stackoverflow