Error with multiple FeatureLookup calls outside databricks

[Nath]

I access databricks feature store outside databricks with databricks-connect on my IDE pycharm.

The problem is just outside Databricks, not with a notebook inside Databricks.

I use FeatureLookup mecanism to pull data from Feature store tables in my customer Dataframe (class FeatureLookup, API Training_set and load_df).

I need to add several Features from several Feature store tables. So I call this mecanism several times.

It doesn't work from 6th call. If I pull the 6th feature only, it works.

I have attached the StackTraceError.

The problem is in the file databricks\feature_store\utils\feature_lookup_utils.py, line : plan = spark.sql("explain cost select * from view").collect()[0][0]).

Thanks.

Nath

[shan_chandra · Accepted solution]

ProtoSerializer is known to be prone to StackOverflow when very deep query plans are being used. Could you please try increasing the stack size of the client JVM with the spark conf -spark.driver.extraJavaOptions which can be defined in ${spark_home}/conf/spark-defaults.conf  (e.g,

spark.driver.extraJavaOptions -Xss4M

), where ${spark_home}  can be found with databricks-connect get-spark-home

[Harrison691]

I was looking for the same, Thank you for the suggestion.

GMGlobalConnect

[shan_chandra]

Also, Please refer to the below KB for additional resolution - https://learn.microsoft.com/en-us/azure/databricks/kb/dev-tools/dbconnect-protoserializer-stackoverf...