How to drop single feature from feature store table

[deep_thought]

I have a feature store table and I would like to change one of the features from IntegerType to FloatType, I can't merge this change as it violates the schema. Is it possible to drop a single feature from the table and add the revised feature?

Currently I am dropping the whole table to solve this.

[artsheiko · Accepted solution]

you can use the feature tables API to update your table in a "overwrite" the existing one :

fs.write_table(
  name='recommender_system.customer_features',
  df = customer_features_df,
  mode = 'overwrite'
)

If this don't work for your use-case, each feature store table is represented by a traditional Delta Table under the hood. So, you can do the next :

df.write.option("mergeSchema", "true")
.format("delta")
.mode(mode="overwrite")
.saveAsTable(delta_table_name)

Finally, you can register an existing Delta table as a feature table :

fs.register_table(
  delta_table='recommender.customer_features',
  primary_keys='customer_id',
  description='Customer features'
)

[deep_thought]

hi @Artem Sheiko​ as the schema for the column has changed, mode='overwrite' does not work. When I encounter this problem with delta tables I can pass the overwriteSchema=True param to DataFrameWriter.saveAsTable.

It would be great if fs.write_table could take args for mergeSchema & overwriteSchema

[Vidula]

Hi there @_ _​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!