This is a known limitation of the Databricks Lakehouse Monitoring API. Here's what you need to know and the workarounds:
The Limitation:
"prediction_col" must be a double type (not a string, and not a list/array)
Reference : https://medium.com/marvelous-mlops/streamlining-ml-model-monitoring-with-databricks-lakehouse-and-in...
The API only accepts a single scalar column. There's no native support for multi-output predictions ...
e.g., for quantiles [0.1, 0.5, 0.9]
for quantile, col_name in [("q10", "pred_q10"), ("q50", "pred_q50"), ("q90", "pred_q90")]:
spark.sql(f"""
CREATE OR REPLACE TABLE {CATALOG}.{SCHEMA}.inference_log_{quantile}
AS SELECT *, {col_name} AS prediction FROM {TABLE_NAME}
""")
w.quality_monitors.create(
table_name=f"{CATALOG}.{SCHEMA}.inference_log_{quantile}",
inference_log=MonitorInferenceLog(
granularities=GRANULARITIES,
timestamp_col=TIMESTAMP_COL,
model_id_col=MODEL_ID_COL,
prediction_col="prediction",
problem_type=MonitorInferenceLogProblemType.PROBLEM_TYPE_REGRESSION,
label_col=LABEL_COL,
),
output_schema_name=f"{CATALOG}.{SCHEMA}",
assets_dir=f"{ASSETS_DIR}/{quantile}",
)
2. Use "custom_metrics" for the extra quantiles
Keep one quantile as the primary "prediction_col" (e.g., median), and track the others via custom metrics:
from databricks.sdk.service.catalog import MonitorMetric, MonitorMetricType
custom_metrics = [
MonitorMetric(
name="mean_pred_q10",
input_columns=["pred_q10"],
definition="avg(:pred_q10)",
output_data_type="double",
type=MonitorMetricType.CUSTOM_METRIC_TYPE_AGGREGATE,
),
MonitorMetric(
name="mean_pred_q90",
input_columns=["pred_q90"],
definition="avg(:pred_q90)",
output_data_type="double",
type=MonitorMetricType.CUSTOM_METRIC_TYPE_AGGREGATE,
),
]
w.quality_monitors.create(
table_name=TABLE_NAME,
inference_log=MonitorInferenceLog(
prediction_col="pred_q50", ]
...
),
custom_metrics=custom_metrics,
...
)
3. Use Snapshot analysis instead
If you don't need the inference-specific accuracy metrics (MSE, etc.), switch to a Snapshot monitor. It will profile all your quantile columns as regular numeric columns and still compute drift/distribution stats across all of them no "prediction_col" constraint.
