Louis_Frolio
Databricks Employee
Databricks Employee

Hello @grajee ,  I can see you're dealing with two separate issues here. Let me address both:

Issue 1: The model_id column (request_metadata MAP type)

You're correct that request_metadata is a MAP type and can't be directly used as the model_id column in Lakehouse Monitoring. Your approach of creating a view that extracts model_name and model_version from the request_metadata MAP is the right solution. You can create a view like this:

```sql
CREATE OR REPLACE VIEW your_catalog.your_schema.winequality_inference_view AS
SELECT
*,
request_metadata['model_name'] AS model_name,
request_metadata['model_version'] AS model_version,
CONCAT(request_metadata['model_name'], '_', request_metadata['model_version']) AS model_id
FROM your_catalog.your_schema.winequality_payload
```

Then create the monitor on this view using the model_id column.

Issue 2: The timestamp and profile_metrics date problem

I noticed something important in your post. The timestamp_ms value 1738620594270 actually converts to **2025-02-03 22:09:54** (not 2024-12-31 as you mentioned). So your timestamp_ms column is correct and matches your date column.

The real issue is that the profile_metrics table is showing dates in year 57064, which suggests Lakehouse Monitoring may be misinterpreting your timestamp_ms column. This could happen if:

1. The monitoring is treating timestamp_ms as seconds instead of milliseconds
2. There's a unit mismatch in how the timestamp column is being processed
3. The timestamp column being used for windowing isn't correctly specified

Troubleshooting steps:

1. Verify your monitor configuration explicitly specifies timestamp_ms as the timestamp column and that it's correctly formatted as epoch milliseconds (LONG type)
2. Check if there are any timezone configuration issues in your monitor setup
3. Try recreating the monitor and ensure you're using the InferenceLog profile type (not TimeSeries) for inference tables
4. Confirm your inference table schema matches the expected format with timestamp_ms as a LONG type

You might also want to run a query directly on your inference table to validate the timestamp_ms values are reasonable:

```sql
SELECT
date,
timestamp_ms,
FROM_UNIXTIME(timestamp_ms/1000) as converted_timestamp
FROM your_catalog.your_schema.winequality_payload
LIMIT 10
```

Hope this helps, Louis.