All Machine Learning posts

Re: Docker image with libraries + MLFlow Experiments

Joshleo — Mon, 02 Nov 2026 09:20:39 GMT

Proper handling keeps GHK-Cu stable throughout a project. Researchers should protect the lyophilised powder from moisture, label vials clearly, and keep records of storage conditions. Good handling habits protect both the quality of GHK-Cu and the reliability of the data.

Click Here

Re: Possible Incorrect Answer Key: Spark ML Estimators — OneHotEncoder, Imputer, and StandardScaler

NinaThoni — Wed, 28 Oct 2026 17:25:07 GMT

Hi @amittian,

Thanks for flagging this, you're right.
Since Spark 3.0, OneHotEncoder is an Estimator. It learns the category sizes during fit() and returns a OneHotEncoderModel, so three of the four options are Estimators.

The likely cause is that the question was written for Spark 2.x. In that version, OneHotEncoder was a Transformer and the learning version was a separate class called OneHotEncoderEstimator. Spark 3.0 merged the two under the OneHotEncoder name, and the answer key doesn't seem to have been updated.

I checked with our certification team. They asked for a ticket here so the question can be formally reviewed: https://help.databricks.com/s/contact-us?ReqType=training

If you'd rather I file it for you, just send me a bit more context, such as which exam or assessment it was and roughly when you took it, and I'll take care of it.

Thanks again for raising it!
Nina

Possible Incorrect Answer Key: Spark ML Estimators — OneHotEncoder, Imputer, and StandardScaler

amittian — Tue, 27 Oct 2026 22:56:11 GMT

Hello Databricks Community,

While preparing for the Databricks Machine Learning Associate certification, I came across a question in a practice assessment that I believe may have an incorrect or incomplete answer key.

Question:

"In a Spark ML Pipeline, which two of the following are Estimators — meaning they learn parameters from the training data during fit()? (Select two)"

Options:

OneHotEncoder
VectorAssembler
Imputer
StandardScaler

My selected answers:

OneHotEncoder
StandardScaler

Assessment result: Partially correct — 2.5 out of 5 marks.

Answers provided by the assessment:

Imputer
StandardScaler

My understanding:

According to the Apache Spark ML API, three of the listed components are Estimators:

OneHotEncoder: An Estimator that learns category sizes during fit() and produces a OneHotEncoderModel.
Imputer: An Estimator that learns imputation statistics and produces an ImputerModel.
StandardScaler: An Estimator that learns feature statistics and produces a StandardScalerModel.

VectorAssembler is a Transformer because it combines feature columns without learning parameters through fit().

Therefore, technically, the question appears to have three correct answers rather than two.

Could someone from the Databricks training or certification team clarify whether the question or its answer key needs correction?

I would appreciate clarification so that learners can align their understanding with the actual Spark ML API.

Thank you!

Why I started evaluating the pipeline instead of choosing the best model at each stage

FrankAzzollini — Tue, 27 Oct 2026 19:42:25 GMT

I recently worked on a multi-label classification pipeline on Databricks, using embeddings, scikit-learn classifiers, Delta Lake, and MLflow to track experiments.

One result caught my attention: the embedding model that ranked fourth in my initial screening ended up in the best-performing combination when evaluated with the actual classifier.

The experiments also surfaced a few less obvious issues:

The best decision threshold varied significantly across model combinations.
Rebuilding the label vocabulary at inference time could silently associate predictions with the wrong labels.
Differences between training and inference inputs could go unnoticed if you only evaluate individual stages.

I documented the experiments, results, code, and lessons learned here:

My Medium Article

I'm curious how others approach this in Databricks: when you have multiple interacting components, do you run a joint search over the pipeline, or narrow down candidates using stage-level metrics first?

Re: Document AI is a pipeline, not a model

DoTA — Sun, 18 Oct 2026 01:41:56 GMT

On your closing question, the rule I'd use for the line: anything with a checkable ground truth, either inside the document or in a system of record, goes to deterministic logic, and the model only proposes. Amount in figures vs words, date validity, account number checksums, required fields. The model extracts, a rule verifies, and a mismatch routes to a human instead of the model arguing its way out of it.

The model keeps what is genuinely ambiguous and has no oracle, like messy handwriting or whether a signature plausibly matches. For those I'd have it return a confidence and set the thresholds per field by risk, since a wrong amount should cost more review than a wrong memo line.

On the Databricks side, what makes this governable is writing each field's extracted value, confidence, the rule that fired and the final decision to one Delta table. Then every rule-vs-model disagreement becomes a labelled example you can use to retune thresholds or fine-tune the signature model, and an auditor can replay any decision later.

Re: auto.arima in R keeps running without end

arkobose — Fri, 16 Oct 2026 12:36:13 GMT

Okay, this is resolved. Windows doesn't allow forking, so parallel = TRUE was having no effect. But Databricks runtime runs on Linux, which enables forking. So keeping parallel = TRUE seems to be having the opposite effect than the anticipated one: due to process overheads, and the fact that parallel only works when stepwise = FALSE, thereby forcing the model to perform a brute-force search.

Tl;dr: I turned off parallel and the execution finished fairly quickly.

auto.arima in R keeps running without end

arkobose — Fri, 16 Oct 2026 10:41:59 GMT

I have a time-series that I am trying to model using auto.arima in R. Upon execution on a cluster runtime with 96GB memory and 48 cores, I observed that the code kept running without any end in sight. It ran for over a day without finishing.

Curious, I ran the same code on the same time-series on my modest laptop (32GB RAM, 10 cores), the code finished execution in under 5 minutes.

I am at a complete loss to understand what is taking the code I am running on my Databricks cluster runtime take so long!

For completeness, the code I am trying to execute is simply this:

baselines_daily <- auto.arima(daily_network_efficiency,

xreg = F,

seasonal = FALSE,

stepwise = FALSE,

parallel = TRUE,

approximation = FALSE)

where daily_network_efficiency is my time-series.

Any help in shedding light on this mystery will be greatly appreciated.

Thank you!

Re: Getting error when running databricks deploy bundle command

Louis_Frolio — Wed, 14 Oct 2026 08:49:19 GMT

Greetings @jitenjha11, I did some digging and here is what I found.

@emma_s is on the right track. Make sure the identity running the deploy has workspace access and cluster creation rights, and that the workspace is attached to a Unity Catalog metastore. This thread is about nine months old, so you may have sorted it already, but here's the full picture for anyone who lands here.

Your deploy is failing for two independent reasons, plus one warning:

No metastore assigned for the current workspace is the real blocker. The MLOps Stacks template registers models in Unity Catalog (your log shows dev.my_mlops_project.my_mlops_project-model), so an account admin needs to assign a metastore to the workspace. Then create the dev catalog and my_mlops_project schema, and grant the deploying user or service principal USE CATALOG, USE SCHEMA, CREATE MODEL, and CREATE TABLE on them.
ACLs for job/mlflowExperiment are disabled or not available in this tier comes from the permissions blocks the template adds to your jobs and experiment. Confirm the workspace in your .databrickscfg is really the Premium one, and that workspace access controls are enabled. Heads up: Azure "role-based access controls" and workspace ACLs are related but not the same thing. If it's just a dev workspace and you want to keep moving, remove the permissions: blocks for the jobs and experiment and redeploy. Leave the Unity Catalog model grants alone unless you're deliberately changing model governance.
The unknown field: description warning isn't causing the failure, but it's an easy cleanup. Remove it from resources/ml-artifacts-resource.yml or move the note into an MLflow experiment tag.

One more thing: stop deleting terraform.tfstate. It won't fix any of this, and since some jobs were already created, wiping state can leave you with duplicates on the next deploy. Once the workspace is fixed, run:

databricks bundle validate -t dev
databricks bundle deploy -t dev --debug

For GitLab, good news: it's a supported option. Pick GitLab for input_cicd_platform during databricks bundle init mlops-stacks and the template generates the pipeline config. A few tips:

Use separate service principals for staging and production.
Store OAuth credentials as protected GitLab variables, not a committed PAT.
Shape the pipeline as unit tests, then bundle validate, then staging deploy, then integration tests, then a manual production deploy from a protected release branch.

References:

MLOps Stacks repo: https://github.com/databricks/mlops-stacks
MLOps Stacks docs: https://docs.databricks.com/aws/en/dev-tools/bundles/mlops-stacks
Create a Unity Catalog metastore (Azure): https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/create-metastore
Unity Catalog privileges reference: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/access-control/privileges-reference
Access control lists: https://learn.microsoft.com/en-us/azure/databricks/security/auth/access-control/
Bundle resource reference: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/reference

Bottom line: assign the metastore, set up the catalog, schema, and grants, sort out the ACL blocks, and redeploy. That should clear it.

Regards, Louis.

Re: MLflow traces accepted by StartTraceV3 (200 OK) but never stored: GetTrace returns NOT_FOUND

anuj_lathi — Thu, 08 Oct 2026 09:05:42 GMT

Short version: your new observation rules out a client-side drop. If a retried StartTraceV3 with the same id returns "already exists" while GetTrace and SearchTraces return NOT_FOUND, the server accepted and recorded something for that id, but the read path can't see it. That points to a server-side persistence or visibility inconsistency, and it needs a support ticket with server-side investigation. I'm not aware of a public known issue matching this, and this isn't behavior Databricks guarantees.

What the evidence says

The request reached the backend and was acknowledged, and the id is now reserved or recorded somewhere.
The same id is invisible to both point lookups and search, for hours. That rules out ordinary eventual-consistency lag.
Your retry workaround will not work for these traces. Re-sending the same id gets "already exists", so you'd need to regenerate the trace id (or treat the trace as lost) instead of resending.

Plausible server-side shapes, which only Databricks can tell apart: an id or idempotency record written but the main record never committed or later rolled back, a record written but not indexed or visible to reads, or a record written and then removed. I can't say which from the outside.

Your questions

1. Known behavior / ingestion pipeline? Experiment-backed traces live under the experiment [1], and success from StartTraceV3 should mean the trace info is durably stored. A success followed by permanent NOT_FOUND is a defect, not expected behavior. Whether there's an async stage behind the endpoint is internal, and I wouldn't guess at it. Support can answer that.

2. Server-side visibility? There's nothing customer-facing I'd point you to for per-trace ingestion status. As you found, system.access.audit isn't the place. Support or engineering can search service-side logs if you give them a request id, workspace id, experiment id, trace id and a UTC timestamp. You already have all of those except the request id.

3. Would Unity Catalog storage avoid it? I wouldn't assume so, and I wouldn't migrate on that hope. It's a different storage backend, so it may behave differently, but I can't tell you whether it shares the failing component. If you want to test it, run a parallel experiment with UC-backed traces under the same load and compare loss rates. Ask support the same question about the ingestion path.

4. Request id? Databricks REST responses carry an x-request-id header. Capture it on every StartTraceV3 call, and log it next to the trace id. One way is to wrap the MLflow REST helper, which returns a requests.Response:

import logging
from mlflow.utils import rest_utils

log = logging.getLogger("trace-audit")
_orig = rest_utils.http_request

def _wrapped(*args, **kwargs):
    resp = _orig(*args, **kwargs)
    log.info(
        "endpoint=%s status=%s x-request-id=%s",
        kwargs.get("endpoint") or (args if len(args) > 2 else None),
        resp.status_code,
        resp.headers.get("x-request-id"),
    )
    return resp

rest_utils.http_request = _wrapped

The patch point may differ slightly between client versions, since modules import http_request by name. If it doesn't take effect, patch the reference where the tracing REST store imports it, or enable DEBUG logging for urllib3 and capture headers there.

What I'd collect before opening the ticket

For each lost trace: the trace id, x-request-id from the StartTraceV3 response, UTC timestamps, the experiment id, the workspace id and the run id.
The x-request-id from the retried call that returned "already exists", plus the exact error body. It's strong evidence that the record exists server-side.
The size of the StartTraceV3 payload for lost versus good traces. Check the trace info fields (tags, metadata, previews), not the span data. Your 300 KB to few MB traces are mostly span data going through the upload path, so a size correlation would be informative either way.
An isolation test:
- Give each xdist worker its own run, or don't link traces to a run at all.
- Run a single worker.
If the loss rate drops, the shared-run write contention across 8 processes is a strong lead. If it doesn't, that's useful to tell support too.
Whether the lost traces' ids ever show up through a direct trace-info fetch, as opposed to search_traces and get_trace, which would help separate "metadata missing" from "data missing".

Interim approach

Until there's a root cause, make your writes safe against this failure. Verify each trace with get_trace after a delay. On a miss, don't resend under the same id. Re-emit the trace under a new id, with the lost id recorded as a tag so you can reconcile later. Keep logging the lost ids and request ids, since each one is another data point for support.

References

[1] Organize training runs with MLflow experiments | Databricks on AWS — https://docs.databricks.com/aws/en/mlflow/experiments

Document AI is a pipeline, not a model

FrankAzzollini — Thu, 01 Oct 2026 13:17:21 GMT

When a financial document needs to be approved or rejected, there usually isn't one model making the decision.

In the check / payment-slip systems I worked on, the pipeline looked more like this:

1. Multimodal models + OCR to extract fields
2. YOLO to detect regions and crop signatures
3. Fine-tuned signature verification models, outperforming generic models on bank-specific documents
4. A business-rule engine for decisions the model should not be making

Some lessons were particularly clear:

- A model can read the amount. A rule should verify that the amount in numbers matches the amount in words.

- A model can detect a signature. A specialized model can verify whether it matches an authorized signer.

- Front/back consistency, required fields, thresholds, and institution-specific requirements belong in deterministic logic.

- For critical fields, specialized crops + specialized models can outperform an end-to-end VLM.

And when document formats differ significantly between institutions, having separate rule branches isn't necessarily technical debt. Sometimes it's simply the correct representation of the business.

The interesting part starts when you scale this.

On Databricks, I would treat document understanding, model serving, monitoring, and business rules as one governed production path?not as a notebook demo connected to a webhook.

If you're designing IDP on Databricks / Mosaic AI, I'm interested in how you're drawing the line between what the model should decide and what should remain deterministic.

Re: MLflow traces accepted by StartTraceV3 (200 OK) but never stored: GetTrace returns NOT_FOUND

holyone2 — Thu, 01 Oct 2026 11:03:18 GMT

I have since discovered a useful observation:

1. StartTraceV3 succeeds;

2. later GetTrace and SearchTraces return NOT_FOUND;

3. a retried StartTraceV3 with the same id returns 'already exists'!

MLflow traces accepted by StartTraceV3 (200 OK) but never stored: GetTrace returns NOT_FOUND

holyone2 — Wed, 30 Sep 2026 16:15:56 GMT

Hi all,

We're seeing cases where traces go missing in an experiment-backed MLflow trace store (not Unity Catalog). The client gets a successful response for both trace writes, and the server echoes back the same trace id, but the trace is never stored. `GetTrace` returns `NOT_FOUND` indefinitely, and the trace never appears in `search_traces` for the experiment or its run.

**Setup**
- MLflow Python client 3.15.1, databricks-sdk 0.102.0, Python 3.13
- Tracking URI `databricks`, traces stored in a regular MLflow experiment (about 2.8k traces total, well under the 100k cap)
- AWS workspace, ap-northeast-1. Client machine in a different region (UK).
- pytest with pytest-xdist: 8 worker processes, each writing one trace every 1–2 minutes, all attached to one shared run. That's far below the documented 50 trace creations/sec limit.
- Each trace is about 300 KB to a few MB (one root span plus N child spans)
- Tried with both async (default) and synchronous export (`MLFLOW_ENABLE_ASYNC_TRACE_LOGGING=false`); the problem happens with both

**What happens**

Roughly 1 in 37 traces per run disappears, on a different test each time. We wrapped `TracingClient.start_trace` and `TracingClient._upload_trace_data` to log every call. For a lost trace, we see:

```
16:00:51 UTC start_trace tr-0eb74c3a913e0f24188d56595ee45ca6 (experiment <id>) returned tr-0eb74c3a913e0f24188d56595ee45ca6 (experiment <id>) in 1.5s
16:00:54 UTC upload_trace_data tr-0eb74c3a913e0f24188d56595ee45ca6 done in 3.0s
16:01:56 UTC mlflow.get_trace(...) still None after 10 attempts (~60s, exponential backoff)
```

Hours later, `mlflow.get_trace("tr-0eb74c3a913e0f24188d56595ee45ca6")` still returns `NOT_FOUND`, and the trace isn't in a timestamp-window search of the experiment. Every other trace written by the same process in the same window is present and readable.

The MLflow exporter raised no error or warning, and the local trace was exported normally: it wasn't deferred and wasn't left in the in-memory buffer. As far as the client can tell, the write succeeded.

Other examples: `tr-1d335ca4b3ebd64db0b31c6d8025b297` (about 15:27:29 UTC), `tr-aed91304847d49e1674d7fa9eaf632c4` and `tr-f9ef235585cdaa7dafc2796bed560239`, all on 2026-09-30.

**Questions**
1. Is it known that `StartTraceV3` can return success without the trace being persisted for experiment-backed trace storage? Is there an ingestion pipeline behind it that can drop records?
2. Is there any server-side log or system table where we could see what happened to a specific trace id? `system.access.audit` doesn't seem to cover trace API calls.
3. Would migrating traces to Unity Catalog storage avoid this, or does it share the same ingestion path?
4. Which response header carries a request id, so we can capture it for support?

For now our workaround is to keep a copy of each trace client-side, re-send it if `get_trace` still misses after about a minute, and then check again. We'd much rather understand the root cause.

Thanks!

Re: How do you train forecasting models for events that never happened?

niteshm — Fri, 25 Sep 2026 05:19:10 GMT

Hey @ThiamLee , great question, and one I've wrestled with too. Haven't tried Remix Labs specifically, tho.

My take, after going back and forth between pure historical data and augmentation: real data first, augment second, but only where augmentation earns its place.

Three things that helped, in order:

Fix censored demand before generating anything.
When you're out of stock, your sales numbers stop measuring demand. They just measure what was on the shelf. So your data quietly under-reports demand exactly when it spiked. I learned this the hard way. Flag those periods and fill in your best guess of true demand. A simple rolling average of nearby non-stockout days works fine. On one of my datasets this one step beat every augmentation trick I tried.
Forecast the range, not a single number.
For rare events, one predicted number is nearly useless. What you need is the range of what could happen. Train your model to predict quantiles (P10/P50/P90) instead of a single value. If you want intervals that hold up under regime shifts, add conformalized intervals calibrated on your latest backtest fold.
Augment like a skeptic.
When you do generate synthetic data, start simple.The fancy generators like TimeGAN make data that looks real, but they smooth out the extremes, which is the very thing you are after. Keep them for scenario stress-testing. Three rules I never break: fit the generator inside each training fold only, never on the whole dataset. Judge the result on real held-out spikes, never on realism scores. And check the synthetic data still respects your real relationships.

Implementation on Databricks:

Bronze: raw sales + inventory/stockout flags + covariates (promo calendar, price, holidays, weather) as Delta tables.
Silver: de-censored latent-demand table + causal lag/rolling features, computed in Spark with no future leakage.
Baselines: StatsForecast seasonal-naive, then MLForecast + LightGBM quantiles trained globally. Rolling-origin backtests, spike-slice metrics reported separately.
Augmentation branch: cheap generators first, one MLflow run per (model × fold), with vs. without augmentation. Promote only on real held-out spike wins.
Champion model to Unity Catalog registry, conformal intervals on top for regime-shift coverage.
Gold: scheduled scoring writes forecasts here, joined with actuals. This is where monitoring lives. When spike-period error or interval coverage drifts, it triggers a retrain.

Give it a shot and report back. I'm curious what works and what breaks on your data. Good luck!!

Re: How do you train forecasting models for events that never happened?

ivanvyd — Tue, 22 Sep 2026 20:52:20 GMT

One other thing I'd probably validate is whether the synthetic spikes preserve the relationships with things like promotions, price, holidays, and inventory. A spike can look realistic on its own but still teach the model the wrong context if those relationships get distorted.

Re: How do you train forecasting models for events that never happened?

ivanvyd — Tue, 22 Sep 2026 20:46:42 GMT

@ThiamLee thanks for sharing, this is a really interesting problem and useful topic to discuss.

I'd be open to trying augmentation here. I'd compare the same model with and without synthetic data on identical rolling backtests, keeping the test data real and fitting the generator only on each training split. I'd also check real spike periods separately, so the overall score doesn't hide weaker results there.

For events you've never observed, realistic, domain-reviewed scenarios could help with stress testing.Those results are useful, but they still depend on the assumptions behind the scenarios.

One wrinkle with stockouts: sales can understate demand when inventory runsout, so I'd account for that before augmentation.

Have you had a chance to test whether the improvements carry over to held-out real events?

How do you train forecasting models for events that never happened?

ThiamLee — Tue, 22 Sep 2026 08:27:23 GMT

Most of my time-series work hits the same wall. The models only learn from scenarios that already happened.

Rare events barely show up in the training data. Demand spikes, stockouts, and extreme regimes are exactly the cases I care about. They are also the ones history is thin on.

Lately I have been testing synthetic time-series data as a way around this. The idea is to generate new scenarios from patterns already in your data, then train or stress-test against them.

One tool I came across is Remix Labs. It synthesizes new time-series datasets from data you already have. The pipeline is no-code, and it runs models like N-BEATS, NHITS, LSTM, and GRU under the hood. No new data collection needed.

Curious how others handle this. Do you augment with synthetic data, or stick to real historical records only? What has worked for you?

Re: Docker image with libraries + MLFlow Experiments

jackleo — Fri, 18 Sep 2026 11:04:48 GMT

Mojang's commitment to regularly releasing a new Minecraft Patch Download demonstrates their deep respect and genuine appreciation for the passionate global community that has made Minecraft the most beloved game in history.
Click Here

Re: Image Annotation

ThiamLee — Wed, 16 Sep 2026 14:13:26 GMT

Great overview of image annotation! The sections on different techniques, real-world use cases, and pricing factors are especially helpful for anyone getting started with AI/ML. Looking forward to exploring the future trends in image annotation!

Re: Difference between Workspace and Unity Catalog experiments when using MLflow autologging?

ThiamLee — Wed, 16 Sep 2026 14:10:51 GMT

Great breakdown! The distinction between autologging and experiment governance is really helpful, especially for teams moving toward production ML workflows. 🚀

Re: Difference between Workspace and Unity Catalog experiments when using MLflow autologging?

ThiamLee — Tue, 15 Sep 2026 11:00:38 GMT

Great question! I’d also be interested to know if autologging has any behavioral or permission differences with Unity Catalog experiments, especially around governance and model lineage.