- 189 Views
- 2 replies
- 0 kudos
Why I started evaluating the pipeline instead of choosing the best model at each stage
I recently worked on a multi-label classification pipeline on Databricks, using embeddings, scikit-learn classifiers, Delta Lake, and MLflow to track experiments.One result caught my attention: the embedding model that ranked fourth in my initial scr...
- 189 Views
- 2 replies
- 0 kudos
- 0 kudos
Frank, here’s a third answer to your joint-search vs. prune-first question, building on @Isaac_18’s.Instead of running a new search every time, I build an offline evaluation suite from historical run data. I replay candidate pipelines against those c...
- 0 kudos
- 5586 Views
- 4 replies
- 1 kudos
Resolved! Docker image with libraries + MLFlow Experiments
Hi everybody,I have a scenario where we have multiple teams working with Python and R, and this teams uses a lot of different libraries. Because of this dozen of libraries, the cluster start took much time. Then I created a Docker image, where I can ...
- 5586 Views
- 4 replies
- 1 kudos
- 1 kudos
Mojang's commitment to regularly releasing a new Minecraft Patch Download demonstrates their deep respect and genuine appreciation for the passionate global community that has made Minecraft the most beloved game in history.Click Here
- 1 kudos
- 185 Views
- 1 replies
- 0 kudos
Possible Incorrect Answer Key: Spark ML Estimators — OneHotEncoder, Imputer, and StandardScaler
Hello Databricks Community,While preparing for the Databricks Machine Learning Associate certification, I came across a question in a practice assessment that I believe may have an incorrect or incomplete answer key.Question:"In a Spark ML Pipeline, ...
- 185 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @amittian, Thanks for flagging this, you're right. Since Spark 3.0, OneHotEncoder is an Estimator. It learns the category sizes during fit() and returns a OneHotEncoderModel, so three of the four options are Estimators. The likely cause is that th...
- 0 kudos
- 348 Views
- 1 replies
- 1 kudos
Document AI is a pipeline, not a model
When a financial document needs to be approved or rejected, there usually isn't one model making the decision.In the check / payment-slip systems I worked on, the pipeline looked more like this:1. Multimodal models + OCR to extract fields2. YOLO to d...
- 348 Views
- 1 replies
- 1 kudos
- 1 kudos
On your closing question, the rule I'd use for the line: anything with a checkable ground truth, either inside the document or in a system of record, goes to deterministic logic, and the model only proposes. Amount in figures vs words, date validity,...
- 1 kudos
- 229 Views
- 1 replies
- 0 kudos
Resolved! auto.arima in R keeps running without end
I have a time-series that I am trying to model using auto.arima in R. Upon execution on a cluster runtime with 96GB memory and 48 cores, I observed that the code kept running without any end in sight. It ran for over a day without finishing.Curious, ...
- 229 Views
- 1 replies
- 0 kudos
- 0 kudos
Okay, this is resolved. Windows doesn't allow forking, so parallel = TRUE was having no effect. But Databricks runtime runs on Linux, which enables forking. So keeping parallel = TRUE seems to be having the opposite effect than the anticipated one: d...
- 0 kudos
- 2592 Views
- 2 replies
- 0 kudos
Getting error when running databricks deploy bundle command
HI all,I am trying to implement MLOps project using https://github.com/databricks/mlops-stacks repo.I have created azure databricks with Premium (+ Role-based access controls) (Click to change) and following bundle creation and deploy using uRL: http...
- 2592 Views
- 2 replies
- 0 kudos
- 0 kudos
Greetings @jitenjha11, I did some digging and here is what I found. @emma_s is on the right track. Make sure the identity running the deploy has workspace access and cluster creation rights, and that the workspace is attached to a Unity Catalog metas...
- 0 kudos
- 397 Views
- 2 replies
- 0 kudos
MLflow traces accepted by StartTraceV3 (200 OK) but never stored: GetTrace returns NOT_FOUND
Hi all,We're seeing cases where traces go missing in an experiment-backed MLflow trace store (not Unity Catalog). The client gets a successful response for both trace writes, and the server echoes back the same trace id, but the trace is never stored...
- 397 Views
- 2 replies
- 0 kudos
- 0 kudos
Short version: your new observation rules out a client-side drop. If a retried StartTraceV3 with the same id returns "already exists" while GetTrace and SearchTraces return NOT_FOUND, the server accepted and recorded something for that id, but the r...
- 0 kudos
- 887 Views
- 3 replies
- 1 kudos
Resolved! How do you train forecasting models for events that never happened?
Most of my time-series work hits the same wall. The models only learn from scenarios that already happened.Rare events barely show up in the training data. Demand spikes, stockouts, and extreme regimes are exactly the cases I care about. They are als...
- 887 Views
- 3 replies
- 1 kudos
- 1 kudos
Hey @ThiamLee , great question, and one I've wrestled with too. Haven't tried Remix Labs specifically, tho.My take, after going back and forth between pure historical data and augmentation: real data first, augment second, but only where augmentation...
- 1 kudos
- 438 Views
- 2 replies
- 2 kudos
Image Annotation
What is Image Annotation?What are the Steps of Image Annotation?What are the Different Techniques of Image Annotation?Types Used in Image AnnotationHow are Companies Handling Image Annotation?Features to Look for in Image Annotation Service Providers...
- 438 Views
- 2 replies
- 2 kudos
- 2 kudos
Great overview of image annotation! The sections on different techniques, real-world use cases, and pricing factors are especially helpful for anyone getting started with AI/ML. Looking forward to exploring the future trends in image annotation!
- 2 kudos
- 1512 Views
- 5 replies
- 6 kudos
Resolved! Difference between Workspace and Unity Catalog experiments when using MLflow autologging?
Hi everyone,I am trying to understand the exact differences between using Workspace experiments versus Unity Catalog experiments, specifically in the context of MLflow autologging (mlflow.autolog()).Does autologging behave differently depending on wh...
- 1512 Views
- 5 replies
- 6 kudos
- 6 kudos
Great question! I’d also be interested to know if autologging has any behavioral or permission differences with Unity Catalog experiments, especially around governance and model lineage.
- 6 kudos
- 1287 Views
- 2 replies
- 1 kudos
Data practitioner in AI Era
As the AI revolution takes off in 2025, there is a renewed emphasis on adopting a Data-First approach. Organizations are increasingly recognizing the need to establish a robust data foundation while preparing a skilled fleet of Data Engineers to tack...
- 1287 Views
- 2 replies
- 1 kudos
- 1 kudos
Absolutely agree! AI is only as powerful as the data behind it. The evolving role of data engineers in building that foundation is definitely worth discussing.
- 1 kudos
- 783 Views
- 1 replies
- 1 kudos
Resolved! How are you combining forecasting models for time-series work?
Most time-series forecasting jobs do not come down to one model. Teams run N-BEATS, NHITS, LSTM, and GRU, then struggle to combine them well.I want to open a discussion on model synthesis. Not data cleaning. Not generative AI. I mean merging several ...
- 783 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi,Good topic. A few things that have held up for us.On weighting across horizons, fit weights per horizon step rather than one global set, and derive them from rolling origin backtests so the errors are genuinely out of sample. Models tend to trade ...
- 1 kudos
- 1549 Views
- 8 replies
- 4 kudos
Resolved! fastai import in databricks broken
Has anyone else had a problem today (2026-09-04) with imports from fastai?The error message points to a problem with the underlying fastcore package:AttributeError: 'Function' object attribute '__doc__' is read-onlyI'm getting the same error using fa...
- 1549 Views
- 8 replies
- 4 kudos
- 4 kudos
I’m seeing the same issue today—seems like the recent compute runtime update may be related. Hopefully someone from the fastai/Databricks side can confirm.
- 4 kudos
- 2085 Views
- 2 replies
- 1 kudos
Learn Databricks AI medium article series for fellow learners.
When it comes to machine learning, the platform plays a pivotal role in successful implementation. Databricks offers a best-in-class machine learning platform with cutting-edge features such as MLflow, Model Registry, Feature Store, and MLOps, which ...
- 2085 Views
- 2 replies
- 1 kudos
- 1 kudos
Great share! Databricks really has built something special on the ML side — MLflow alone has become almost a default for experiment tracking, and pairing it with Model Registry and Feature Store makes the whole path from experimentation to productio...
- 1 kudos
- 620 Views
- 2 replies
- 1 kudos
Tuning with Optuna and MlflowSparkStudy
I am following the guide for tuning a model with Optuna and MlflowSparkStudy. My compute is configured with autoscaling enabled, with 1–2 Spark workers, each with 8 cores and 32 GB of memory. I set n_jobs=2 and trials=100, in mlflow_study.optimize()....
- 620 Views
- 2 replies
- 1 kudos
- 1 kudos
Great questions—especially the distinction between Optuna’s trial-level parallelism and LightGBM’s intra-trial threading. The interaction with Spark autoscaling and data locality is also something I’d love to see documented with a concrete example. C...
- 1 kudos
