VivekWV
New Contributor II

Hi Mark, 

Thanks for your response. I followed the steps you suggested:

  1. Created the table and set primary key + time series key constraints.
  2. Enabled Change Data Feed.
  3. Created the feature table and deployed the online endpoint — this worked fine.
  4. Removed some columns from the offline table and updated it using:
    spark_df.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable(f"{table_name}")
  5. Updated the feature table using:
    fe.write_table(name=feature_store_name, df=df, mode="merge")
  6. Tried re-publishing to the online store using:
    fe.publish_table(source_table_name=feature_store_list, online_table_name="catlog_dev.abcd.fs_table_online", online_store=pg_store, mode="merge")
    — this step failed.

A LakeFlow pipeline was triggered and threw the following error:

org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] 
... [DELTA_SCHEMA_CHANGED_WITH_STARTING_OPTIONS] Detected schema change in version 7

It seems the schema change isn’t being handled during re-publication. I’ve attached the full error message. Let me know if you need more details or logs.