Hi @d_szepietowska,

Thank you for the detailed investigation, especially the side-by-side comparison between the legacy online table (MySQL) and the Lakebase-backed Online Feature Store. That is very helpful for narrowing down the behavior.

UNDERSTANDING THE TWO ENVIRONMENT VARIABLES

These two flags serve different purposes:

1. ENABLE_FEATURE_TRACING = "true" controls whether the automatic feature lookup augmented DataFrame (the looked-up feature values joined with your input) gets logged to the inference table. This is the one documented specifically for feature store tracing.

2. ENABLE_MLFLOW_TRACING = "true" controls MLflow trace logging, which captures the full span-level trace data (inputs, outputs, intermediate steps) and writes it to MLflow experiments and/or inference tables. This was originally designed for GenAI/agent observability, but it also captures feature store spans when the model uses automatic feature lookup.

Your test results confirm an important interaction: when only ENABLE_FEATURE_TRACING is set, neither scenario returned feature values in the response body. When both variables were set, the legacy online table path produced the full trace with feature lookup spans, but the Lakebase path did not include the detailed feature values in the trace spans.

WHAT YOUR RESULTS SHOW

In Scenario 1 (legacy online table), the trace includes a "feature_lookup" span with mlflow.spanOutputs containing the full augmented DataFrame with sepal_length, sepal_width, petal_length, petal_width values. The trace also includes an "online_feature_store" span showing the raw response from the online store.

In Scenario 2 (Lakebase), the trace only has a single "databricks_feature_store" span at the top level with no nested feature_lookup or online_feature_store child spans. The feature values are absent from the trace output.

This indicates that the Lakebase code path for automatic feature lookup does not yet emit the same granular trace spans as the legacy online table path. The underlying feature lookup still works (your predictions are correct, meaning features were looked up successfully), but the tracing instrumentation for the Lakebase path may not be fully producing the same span detail.

RECOMMENDED NEXT STEPS

1. Verify your databricks-feature-engineering library version. You mentioned 0.13.01, which should support tracing. Confirm it is the latest available version on your runtime, as newer patch releases may include improvements to Lakebase trace instrumentation.

2. Confirm the environment variable values are lowercase strings. Both must be exactly "true" (not "True" or a boolean). You can verify via the API:

GET /api/2.0/serving-endpoints/{endpoint_name}

Check that served_entities[].environment_vars shows exactly:
"ENABLE_FEATURE_TRACING": "true"
"ENABLE_MLFLOW_TRACING": "true"

3. Try recreating the endpoint. If it was created before February 2025, augmented DataFrame logging to inference tables requires a newly created endpoint. Even if you believe it is new, recreating it ensures you pick up the latest platform-side instrumentation.

4. Check the inference table directly (not just the response body). The augmented DataFrame may be logged to the inference table even when it does not appear in the synchronous response payload. Query your inference table and look for a column containing trace or feature data:

SELECT * FROM your_catalog.your_schema.your_inference_table
ORDER BY timestamp DESC
LIMIT 10

5. Consider opening a support ticket. Your comparison is solid evidence that the Lakebase code path behaves differently from the legacy path regarding trace span generation. A support ticket with your two-scenario comparison would help the product team investigate whether this is a gap in the current Lakebase tracing instrumentation that will be addressed.

RELEVANT DOCUMENTATION

- Configure environment variables on serving endpoints:
https://docs.databricks.com/aws/en/machine-learning/model-serving/store-env-variable-model-serving

- Automatic feature lookup:
https://docs.databricks.com/aws/en/machine-learning/feature-store/automatic-feature-lookup

- Databricks Online Feature Store (Lakebase):
https://docs.databricks.com/aws/en/machine-learning/feature-store/online-tables

- MLflow Tracing:
https://docs.databricks.com/aws/en/mlflow/mlflow-tracing

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.