Lakehouse Monitoring of Inference Table

[grajee]

All,

I'm trying to setup a lakehouse monitoring process for the WineQuality model that is widely available. While setting up the Serving Endpoint, I enabled "Inference Table" option for which the inference table was created automatically. The columns in the winequality_payload table are as follows:

1. client_request_id
2. databricks_request_id
3. date
4. timestamp_ms
5. status_code
6. execution_time_ms
7. request
8. response
9. sampling_fraction
10. request_metadata (MAP type)

The request_metadata contains "model_name","endpoint_name" & "model_version".

While configuring the monitor against the inference table, I selected request_metadata as the model_id column but it is erroring out with the below error:

The given `DataMonitorInfo` is invalid for the following reason(s): - For table `dev_tst_mls.winequality_uc.winequality_payload`: The specified `model_id_col` (`request_metadata`) must be a groupable column, but instead it is a MAP type. Please check that all referenced columns exist in the table(s) and have compatible column type.

As you can see, there is no other column that I can pick for the model_id column. Why am I getting this error and what alternatives do I have?

Now, based on Databricks Assistant, I created a view adding the model_name and model_version and used this view to create the monitor. Things go through but the dashboard shows no data at all despite my making several scoring attempts.

I tried troubleshooting and found that the datetime in the window column of the profile_metrics table is way off:

profile_metrics table --> Column window  
start: "+057064-08-22T07:05:00.000Z" 
end: "+057064-08-22T07:10:00.000Z"

As you can see, I tried it today (02/03/2025), but the date is light years ahead!!.

Based on the link, the date column is "The UTC date on which the model serving request was received." and the timestamp_ms column is "The timestamp in epoch milliseconds on when the model serving request was received."

I checked the date column in the inference table and it correctly show 2025-02-03 but the timestamp_ms shows values like 1738620594270 which is "2024-12-31 23:09:54.270"

Am I doing something wrong? Has anyone experienced this before?

Thanks,

grajee