When testing this code (
fe.score_batch(
df=dataset.drop("Target").limit(10),
model_uri=f"models:/{model_name}/{mv.version}",
)
.select("prediction")
.limit(10)
.display()
) I get the error: “MlflowException: The...
I just discovered what I believe is a bug in Feature Store. The expected value (of the "value" column) is 'NULL' but the actual value is "a". If I instead change the format to timestamp of the "date" column (i.e. removes the .date() in the generation...
I've got a lot of large CSV files (> 1 GB) that updates regularly (stored in Data Lake Gen 2). The task is to concatenate these files into a single dataframe that is written to parquet format. However, since these files updates very often I get a rea...
I have multiple large CSV files. One or more of these files changes now and then (a few times a day). The changes in the CSV files are both of type update and append (so both new rows) and updates of old. I want to combine all CSV files into a datafr...
The feature store UI is lagging (Azure Databricks). It is extremely slow to update - takes several days from creating a feature table before it appears in the UI. Anyone share the same experience?
Thank you for answering. Yes, that is also what I figured out. In other words the lookback_window argument only works when using timestamp format for the primary key. I cannot see that this behavior is described in the documentation.
Anyone having the same experience here? The UI is still extremely slow to update - it takes days before the already registered feature tables appear in the UI. It makes the UI useless for browsing features! I really hope that Databricks is looking in...
Here is the code (forgot to add)(spark.readStream.format("cloudFiles").option("cloudFiles.format", "csv").option("header", "false") # Assuming the CSV files have headers.schema(schema) # Specify the schema here.option("cloudFiles.schemaLocation", che...
Thanks for replying.Just to be clear this is a problem with the Databricks UI. There is absolutely no issue with the job itself, everything runs smoothly and the feature tables are registered. This is purely an issue with the Databricks Feature Store...
