Model Serving Only Shows WARNING/ERROR Logs

[fede_bia]

Hi everyone,

I’m deploying a custom model using mlflow.pyfunc.PythonModel in Databricks Model Serving. Inside my wrapper code, I configured logging as follows:

logging.basicConfig(
    stream=sys.stdout,
    level=logging.INFO,
    format='%(asctime)s [%(levelname)s] %(name)s: %(message)s',
    force=True
)
logger = logging.getLogger()

However, in the Model Serving service logs I can only see logger.warning() and logger.error() messages.

I would like to understand:

• What is the default logging level for Model Serving endpoints?
• Is there a supported way to enable INFO level logs?
• If configurable, how can I ensure that all INFO logs (including those from modules used inside the wrapped mlflow.pyfunc.PythonModel) are visible?

Any guidance or documentation reference would be greatly appreciated.

Thanks in advance!

[SteveOstrowski · Accepted solution]

@fede_bia

This is worth walking through carefully. this is a common source of confusion when deploying custom models on Databricks Model Serving.

SHORT ANSWER

The default root logging level for Model Serving endpoints is set to WARNING. That is why you only see logger.warning() and logger.error() messages in the Logs tab -- your INFO-level messages are being filtered out before they reach the log output.

WHY THIS HAPPENS

Databricks Model Serving containers set the root Python logger to WARNING level by default. Even though your code calls logging.basicConfig() with level=logging.INFO, the serving infrastructure's own logging configuration can override or take precedence, especially since the container environment configures logging before your model code runs. The official documentation confirms this: it recommends using logging.warning(...) or logging.error(...) "for immediate display in the logs."

HOW TO ENABLE INFO-LEVEL LOGS

You can override the root logger level inside your model's load_context() method, which runs when the model is first loaded into the serving container. This is the supported approach:

import logging
import mlflow

class MyModel(mlflow.pyfunc.PythonModel):
def load_context(self, context):
root = logging.getLogger()
root.setLevel(logging.DEBUG)
for handler in root.handlers:
handler.setLevel(logging.DEBUG)

# Your other initialization code here
self.logger = logging.getLogger(__name__)
self.logger.info("Model loaded successfully -- this should now appear!")

def predict(self, context, model_input, params=None):
self.logger.info("Received inference request")
# your prediction logic
return result

The key points here are:

1. Use load_context(), not module-level code or __init__, to configure logging. This method runs after the serving container has set up its environment, so your overrides take effect.

2. Reset BOTH the root logger level AND each handler's level. The container may attach handlers that have their own level filters set to WARNING, so just changing the logger level alone may not be enough.

3. Set the level to DEBUG or INFO depending on your needs. Setting it to DEBUG will give you the most verbose output, which is helpful during initial deployment and troubleshooting.

VIEWING THE LOGS

Once you have reconfigured the logger as shown above, your INFO messages will appear in:

- The "Logs" tab in the Serving UI (ephemeral service logs that capture stdout/stderr in real time)
- Via the REST API at GET /api/2.0/serving-endpoints/{name}/served-models/{served-model-name}/logs

PERSISTENT LOGGING WITH OPENTELEMETRY (OPTIONAL)

If you need long-term log retention beyond what the ephemeral Logs tab provides, Databricks supports persisting logs to Unity Catalog Delta tables using OpenTelemetry. When enabled, your logs (including INFO and DEBUG if you set the level as described above) are written to a <prefix>_otel_logs table with columns like timestamp, severity_text, body, trace_id, and span_id. This is configured during endpoint creation through the Serving UI or REST API. This is especially useful for production debugging and compliance.

A NOTE ON print() STATEMENTS

Standard print() calls write to stdout and will also appear in the ephemeral service logs. However, using the Python logging module is recommended over print() because it gives you structured output with timestamps, log levels, and logger names, making it much easier to filter and debug.

DOCUMENTATION REFERENCES

- Monitor and diagnose serving endpoints (covers ephemeral logs, build logs, and OpenTelemetry): https://docs.databricks.com/en/machine-learning/model-serving/monitor-diagnose-endpoints.html
- Persist logs to Unity Catalog with OpenTelemetry: https://docs.databricks.com/en/machine-learning/model-serving/custom-model-serving-uc-logs.html
- Debug model serving endpoints: https://docs.databricks.com/en/machine-learning/model-serving/model-serving-debug.html
- Serving endpoints logs API reference: https://docs.databricks.com/api/workspace/servingendpoints/logs

Hope this helps! Let us know if you run into any issues after making the change.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.