All,
I'm trying to setup a lakehouse monitoring process for the WineQuality model that is widely available. While setting up the Serving Endpoint, I enabled "Inference Table" option for which the inference table was created automatically. The columns in the winequality_payload table are as follows:
- client_request_id
- databricks_request_id
- date
- timestamp_ms
- status_code
- execution_time_ms
- request
- response
- sampling_fraction
- request_metadata (MAP type)
The request_metadata contains "model_name","endpoint_name" & "model_version".
While configuring the monitor against the inference table, I selected request_metadata as the model_id column but it is erroring out with the below error:
The given `DataMonitorInfo` is invalid for the following reason(s): - For table `dev_tst_mls.winequality_uc.winequality_payload`: The specified `model_id_col` (`request_metadata`) must be a groupable column, but instead it is a MAP type. Please check that all referenced columns exist in the table(s) and have compatible column type.
As you can see, there is no other column that I can pick for the model_id column. Why am I getting this error and what alternatives do I have?
Now, based on Databricks Assistant, I created a view adding the model_name and model_version and used this view to create the monitor. Things go through but the dashboard shows no data at all despite my making several scoring attempts.
I tried troubleshooting and found that the datetime in the window column of the profile_metrics table is way off:
profile_metrics table --> Column window
start: "+057064-08-22T07:05:00.000Z"
end: "+057064-08-22T07:10:00.000Z"
As you can see, I tried it today (02/03/2025), but the date is light years ahead!!.
Based on the link, the date column is "The UTC date on which the model serving request was received." and the timestamp_ms column is "The timestamp in epoch milliseconds on when the model serving request was received."
I checked the date column in the inference table and it correctly show 2025-02-03 but the timestamp_ms shows values like 1738620594270 which is "2024-12-31 23:09:54.270"
Am I doing something wrong? Has anyone experienced this before?
Thanks,
grajee