cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Is it possible to create a feature store training set directly from a feature store table?

lewit
New Contributor II

Rather than joining features from different tables, I just wanted to use a single feature store table and select some of its features, but still log the model in the feature store. The problem I am facing is that I do not know how to create the training set without first creating another dataframe to join with features from the feature store.

2 REPLIES 2

Debayan
Databricks Employee
Databricks Employee

lewit
New Contributor II

I did check this but it does not help me sadly. My problem is not necessarily with the Feature Lookups part, but with the create_training_set() function. It demands a df with which one can join the features described by FeatureLookup().

In the example:

training_set = fs.create_training_set(

df=training_df,

feature_lookups = feature_lookups,

label = 'rating',

exclude_columns = ['customer_id', 'product_id'])

My problem is specifically with df. I have a feature store table that contains all the information I need, features and label, but as of now I need to create a separate dataframe with an index and my label of choice (which I am extracting from the feature store with read_table()), and then create a training_set joining it with the rest of the features I need from that table. This seems like an unnecessary step.

Isn't there a way to create the training set without having to create this "index" df first? If I use the multiple feature lookup trick I will still need to inform some df to the create_training_set() command for it to work since it is a mandatory parameter.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group