- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-09-2022 06:14 AM
I access databricks feature store outside databricks with databricks-connect on my IDE pycharm.
The problem is just outside Databricks, not with a notebook inside Databricks.
I use FeatureLookup mecanism to pull data from Feature store tables in my customer Dataframe (class FeatureLookup, API Training_set and load_df).
I need to add several Features from several Feature store tables. So I call this mecanism several times.
It doesn't work from 6th call. If I pull the 6th feature only, it works.
I have attached the StackTraceError.
The problem is in the file databricks\feature_store\utils\feature_lookup_utils.py, line : plan = spark.sql("explain cost select * from view").collect()[0][0]).
Thanks.
Nath
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-12-2022 08:59 AM
ProtoSerializer is known to be prone to StackOverflow when very deep query plans are being used. Could you please try increasing the stack size of the client JVM with the spark conf -spark.driver.extraJavaOptions which can be defined in ${spark_home}/conf/spark-defaults.conf (e.g,
spark.driver.extraJavaOptions -Xss4M
), where ${spark_home} can be found with databricks-connect get-spark-home
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-12-2022 08:59 AM
ProtoSerializer is known to be prone to StackOverflow when very deep query plans are being used. Could you please try increasing the stack size of the client JVM with the spark conf -spark.driver.extraJavaOptions which can be defined in ${spark_home}/conf/spark-defaults.conf (e.g,
spark.driver.extraJavaOptions -Xss4M
), where ${spark_home} can be found with databricks-connect get-spark-home
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2022 09:50 PM
I was looking for the same, Thank you for the suggestion.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2022 08:40 AM
Also, Please refer to the below KB for additional resolution - https://learn.microsoft.com/en-us/azure/databricks/kb/dev-tools/dbconnect-protoserializer-stackoverf...