FeatureStoreClient speed up create_training_set

Kira — Thu, 07 Dec 2023 07:09:39 GMT

I am trying to create training set with 10 Feature Lookups (about 1200 features total).

# all args for create_training_set df = fs.create_training_set(args).load_df()

I must store this data to delta table for further analysis. Writing this returned data to delta table is taking up to 15 hours. How can I speed up this operation? Population size does not impact performance (same result for 200 rows)

Also, what is best practice for storing Feature Tables?

https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/time-series

Based on documentation:
"A time series feature table must have one timestamp key and cannot have any partition columns. The timestamp key column must be of TimestampType or DateType."

Is it good practice to store huge tables without partition columns?

P.S

I have tried different compute types, but still getting long processing time

topic FeatureStoreClient speed up create_training_set in Data Engineering

FeatureStoreClient speed up create_training_set