Feature Store - lookback_window does not work with primary keys of "date" type

Kjetil
Contributor

‎08-06-2024 03:39 AM

I just discovered what I believe is a bug in Feature Store. The expected value (of the "value" column) is 'NULL' but the actual value is "a". If I instead change the format to timestamp of the "date" column (i.e. removes the .date() in the generation of the date value in the feature table), the result is indeed 'NULL' as expected.

Databricks runtime: 14.3 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)

The code that re-creates the issue:

```python

import datetime as dt
from pyspark.sql import Row
from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup


feature_table_catalog_path = "catalog.schema.table" #insert your own unity catalog path
feature_table_data = [
Row(id=1, date=dt.datetime(2024, 1, 1).date(), value="a")
]
feature_table = spark.createDataFrame(feature_table_data)
fe = FeatureEngineeringClient()

fe.create_table(
name=feature_table_catalog_path,
primary_keys=["id", "date"],
schema=feature_table.schema,
timeseries_columns='date'
)
fe.write_table(
name=feature_table_catalog_path,
df=feature_table,
mode="merge"
)

dataset_with_target = [
Row(id=1, date=dt.datetime(2024, 1, 7), target=1),
]
dataset_with_target = spark.createDataFrame(dataset_with_target)
feature_lookup = FeatureLookup(table_name=feature_table_catalog_path, lookup_key="id", timestamp_lookup_key="date", lookback_window=dt.timedelta(days=3))
training_dataset = fe.create_training_set(df=dataset_with_target, feature_lookups=[feature_lookup], label='target')
training_dataset.load_df().show()

```

Labels:
0 Kudos