StackOverflow Error - FeatureLookup & fs.create_training_set

[lndlzy]

When trying to utilize feature_lookup on at least 2 feature tables and trying fs.create_training_set, I get a stackoverflow error. Can anyone help me understand why this happens? This hasn't happened before but now I get this error and I am unable to resolve this considering that my code works properly before. 

Thank you!

[Kaniz]

Hi @lndlzy, A StackOverflow error usually occurs when your program recurses too deeply.

In this case, it might be due to a problem with the FeatureStoreClient.create_training_set method or how the FeatureLookup objects are defined or used.

Here are a few things you could check:

• Make sure that the lookup_key and timestamp_lookup_key in each FeatureLookup object matches the actual keys in the corresponding feature tables. If there's a mismatch, it could cause an infinite loop, leading to a StackOverflow error.

• Check the data types of the lookup_key and timestamp_lookup_key in the feature tables and make sure they match with the data types of the corresponding keys in the DataFrame passed to FeatureStoreClient.create_training_set.

• Make sure that the feature_names in each FeatureLookup object match the actual feature names in the corresponding feature tables. If there's a mismatch, it could cause an error.

• If you're using many features from multiple feature tables, it might exceed the stack size limit. Try reducing the number of features or feature tables and see if the problem persists.

• Check if there's any recent update in Databricks or in the Feature Store library that might affect the FeatureStoreClient.create_training_set method.

You might need to update your code accordingly if that's the case.

If you've checked all these and still can't resolve the issue, it might be a bug in Databricks or the Feature Store library.

In that case, you should contact Databricks support by filing a support ticket for further assistance.

[lndlzy]

When I used one feature table with around 500 features, it worked. I was actually planning on just passing multiple feature tables for featurelookup and then somewhat do the selection and preprocessing from there however I think it does not work. Do you have any suggestion as to a workaround? Like maybe would it be just the same if I just join those feature tables?