I am trying to create training set with 10 Feature Lookups (about 1200 features total).
# all args for create_training_set
df = fs.create_training_set(args).load_df()
I must store this data to delta table for further analysis. Writing this returned data to delta table is taking up to 15 hours. How can I speed up this operation? Population size does not impact performance (same result for 200 rows)
Also, what is best practice for storing Feature Tables?
https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/time-series
Based on documentation:
"A time series feature table must have one timestamp key and cannot have any partition columns. The timestamp key column must be of TimestampType or DateType."
Is it good practice to store huge tables without partition columns?
P.S
I have tried different compute types, but still getting long processing time
