Machine Learning topics

Tuning with Optuna and MlflowSparkStudy

AdamIH123 — Mon, 07 Sep 2026 19:24:05 GMT

I am following the guide for tuning a model with Optuna and MlflowSparkStudy. My compute is configured with autoscaling enabled, with 1–2 Spark workers, each with 8 cores and 32 GB of memory. I set n_jobs=2 and trials=100, in mlflow_study.optimize().

I have a few questions about how MlflowSparkStudy distributes the workload:

How should I think about n_jobs in mlflow_study.optimize() vs. num_threads/n_jobs in LightGBM?
For example, with 1–2 Spark workers and 8 cores per worker, would it make sense to set mlflow_study.optimize(n_jobs=2) and LightGBM n_jobs=7? My dataset is large, so ideally I would like to run only one model-training job per Spark worker and use the remaining cores on that worker for the LightGBM training.
Does MlflowSparkStudy automatically trigger or make use of Spark autoscaling?
If I configure my cluster with a minimum of 1 worker and a maximum of 2 workers, will MlflowSparkStudy cause Spark to scale up to 2 workers as needed when running multiple trials in parallel?
What is the recommended way to make a large DataFrame available to each Spark worker?
Currently, I have a Pandas DataFrame that I pass to the objective function used by mlflow_study.optimize(). Should I broadcast the DataFrame, cache it in Spark, or use another approach to avoid repeatedly transferring the data to each worker for every trial?

Any guidance on the recommended configuration or best practices would be greatly appreciated.

#mlflow #optuna #MlflowSparkStudy

https://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/optuna

fastai import in databricks broken

barnabywalker — Fri, 04 Sep 2026 14:47:47 GMT

Has anyone else had a problem today (2026-09-04) with imports from fastai?

The error message points to a problem with the underlying fastcore package:

AttributeError: 'Function' object attribute '__doc__' is read-only

I'm getting the same error using fastai versions 2.8.5 and 2.8.8.

The only change I can see is in the compute event logs, my 'Compute is using release 18.3.6' compared to 18.3.5 yesterday.

Difference between Workspace and Unity Catalog experiments when using MLflow autologging?

kunduruanil — Wed, 02 Sep 2026 08:44:13 GMT

Hi everyone,

I am trying to understand the exact differences between using Workspace experiments versus Unity Catalog experiments, specifically in the context of MLflow autologging (mlflow.autolog()).

Does autologging behave differently depending on whether the experiment is registered in the Workspace or in Unity Catalog (using a 3-level namespace)? Are there any limitations, best practices, or specific configurations I should be aware of when using autologging with Unity Catalog compared to the traditional Workspace setup?

Any insights or documentation links would be greatly appreciated. Thanks!

Model Serving An internal error occurred during feature store lookup all deploys failing

ticusss — Thu, 27 Aug 2026 12:57:49 GMT

We can't deploy models to Model Serving. Failures started around 2026-08-16 and were intermittent at first — our current production config deployed cleanly on 08-20 — but since then every attempt fails at the feature store lookup setup step, with no diagnostics we can find anywhere. Hoping someone has seen this.

Failure mode 1 — fails after linking everything

Endpoint events look like this every time:

Endpoint updated by <user>
[25-34x] Linked with Databricks Online Store table ...
System service principal creation with ID `<uuid>` succeeded
System role ... created successfully for Online Feature Store `<store>/production`
An internal error occurred during feature store lookup setup <-- ~31-126s later
Served entity `<model>` entered DEPLOYMENT_FAILED

Served entity message: Online feature store setup failed. Please see event logs for...

It never reaches "Provisioning resources". serving-endpoints logs returns "There are currently no replicas in a running state", and build logs show a clean container build. So no user code ever runs and there is no traceback on our side.

The same model on the same 25 tables also showt fails. Nothing changed on our side betweenthese two deploys:
- 08-20, succeeded: update to system service principal created in 40 seconds, DEPLOYMENT_READY after 4m49s total.
- 08-27, failed: update to system service principal created in 8m27s, internal error at 9m52s.

Failure mode 2 — update hangs in pending with zero events
Separately, some updates never fail or progres a pending_config in DEPLOYMENT_CREATING, served entity message Creating resources for served entity, and no events are emitted at all — not even a single Linked with Databricks Online Store table line. We've left these running for 4h48m and 2h18m before cancelling by hand. Six consecutive attempts on one endpoint, zero tables linked on any of them.

Two things about this that may be relevant:

- One variant involved an online store whose Lakebase project was created through the Lakebase UI rather than create_online_store. publish_table against it succeeded, the online tables are ACTIVE and syncing, but no endpoint has ever linked a single one of them. The project does not appear in fe.list_online_stores(), which the docs say is expected for UI-created projects.
- A wedged update degrades everything else. With two stuck updates still pending, table linking on unrelated endpoints took 49–76 minutes; on a clear queue the same work takes under 90 seconds. This makes failures look far more random than they are, so we now cancel every failed or stuck update before measuring anything.

Setup

- ~34 feature tables with FeatureLookup, logged via FeatureEngineeringClient.log_model
- Online Feature Store created 2025-10-20 as Lakebase Provisioned, CU_1. Per the upgrade docs, all Provisioned instances were migrated to the Autoscaling platform in July 2026; the Database instance API still reports ours as capacity: CU_1, PG_VERSION_16 - Online tables are SYNCED_TABLE_ONLINE_NO_PENnity_catalog_provisioning_state: ACTIVE, syncingnormally
- Container resolves databricks-feature-lookup==1.* → 1.14.0, with mlflow==3.0.0 - databricks-feature-engineering==0.13.0 for p

Already ruled out (please don't suggest these)

- FEATURE_SERVING_CONNECTION_POOL_SIZE does not fix this. We tried 2, 3, 4 and 10. Deploys failed at every value, and the values that once looked promising were jusmade no difference.
- Not table count — a 25-table model that deployed cleanly on 08-20 now fails identically. 34-table models fail the same way.
- Not the model artifact — the same model version deployed fine on one endpoint while failing on another, with byte-identical build logs.
- Not tables unpublished / source_table_id mismatch — the two documented causes of "No suitable online store found for feature tables". We get a different, unnamed error and no table is ever named in it.
- Not concurrent sync — we checked every sync pipeline against a failure window. The sync wave started two minutes after the failure; nothing was syncing during the deploy.
- Not catalog/database name mismatch — catalog name and effective_logical_database_name match.
- Not connection capacity — max_connections 1802, ~124 in use, role rolconnlimit -1.
- Not scale-to-zero — docs confirm it isn't enabled by default on upgraded instances.
- Not grants, schemas, or keys — offline and online schemas match, PKs non-null, timeseries keys set, grants identical between tables that work and tables that don't.
- Not a documented limit — 50 tables allowed per model, we use 34. FeatureSpec well under the 1000-feature cap.
- Not DBR version, workload size, or container libraries.

The only two failures that left an audit trace

Across dozens of failures, exactly two show anything in system.access.audit (action_name='getOnlineFeatureTables', non-200 status):

RESOURCE_EXHAUSTED: You've exceeded the rate limit
INTERNAL_ERROR: Failed to fetch database instance with ID <online-store-instance-uuid> Every other failure produces no audit row at areTables call surrounding them returns 200. Sowhatever fails does not write to the audit table.

Questions
1. The one traced error we have is Failed to fetch database instance with ID ..., which is the UUID of our online stor— an instance that per the docs was migrated faling in July. We are not claiming the upgradebroke this, since deploys worked fine for week error expected against an upgraded instance,and could the lookup setup path resolve it inconsistently?
2. Is An internal error occurred during feature store lookup setup a catch-all? Is there any log surface that cits real cause, given it produces no replica l3. For failure mode 2 — is there any way to sewhen it emits no events? And is there asupported way to detect a wedged update, short of waiting hours and cancelling manually? 4. We see 289 orphaned Postgres roles in the omed role per deploy attempt, never reclaimedwhen a served entity or endpoint config is tor Lakebase Autoscaling has a 500-roles-per-branch limit that Provisioned did not. Is this accumulation expected, and is manual cleanup safe while an endpoint is serving?
5. Has anyone recovered from this? Or is standonline store with create_online_store andrepublishing everything the only route?

Image Annotation

ben_ai — Mon, 24 Aug 2026 09:22:11 GMT

What is Image Annotation?

What are the Steps of Image Annotation?

What are the Different Techniques of Image Annotation?

Types Used in Image Annotation

How are Companies Handling Image Annotation?

Features to Look for in Image Annotation Service Providers

Common Image Annotation Use Cases

High‑Performance Image Annotation Tools

Factors Influencing Pricing of Image Annotation for AI/ML Projects

Top 10 Image Annotation & Labeling Service Providers in 2026

Future Trends in Image Annotation

Vector Store update stale in Syncing status even the actual sync task is done.

airbots — Wed, 19 Aug 2026 18:06:47 GMT

VS status remains "syncing" indefinitely — rechecked after 12 hours, status still "syncing". The Databricks job completes successfully and the new data is queryable:

ML Training low File I/O and Throughout

aswinkks — Mon, 17 Aug 2026 18:55:08 GMT

Hi everyone,

I have an image-based deep learning workload running on Azure Databricks, while the training dataset must remain in AWS S3 due to some constraints. We cannot move or replicate the dataset to Azure.
Our current architecture is roughly:
AWS S3 (images) → Unity Catalog Volume → Azure Databricks GPU compute → PyTorch training

The main issue we're seeing is very high file I/O latency and relatively low training throughput. Since the dataset contains a large number of individual image files, reading the images through the UC Volume appears to involve significant network overhead and many individual file reads.
I initially expected Mosaic Streaming / StreamingDataset to improve this because the dataset can be converted into MDS shards and the shards can be downloaded progressively to local storage while training continues.

However, interestingly, in our testing:
Direct image loading from the UC Volume is currently faster than Mosaic Streaming.
I'm trying to understand whether we are missing an important configuration or whether the cross-cloud architecture itself is the primary bottleneck.

A few questions:
1. What is the recommended architecture for training on S3 data from Azure Databricks when the data cannot be moved to Azure?

2. For Mosaic Streaming, what are the recommended values/strategies for:
shard size
num_workers
predownload
cache_limit
shuffle configuration

3. Is there an optimal MDS shard size for image datasets to minimize S3/network overhead?

4. Would increasing DataLoader workers and prefetching significantly improve throughput in this cross-cloud scenario?

5. Would it be better to use local caching of UC Volume files instead of Mosaic Streaming for a multi-epoch image training workload?

6. Are there any Databricks-recommended approaches for measuring whether the bottleneck is S3 → Azure network bandwidth, file-level latency, CPU image decoding, or GPU starvation?

Our main objective is to maximize GPU utilization and training throughput without moving the source dataset out of S3.
Any recommendations, benchmarks, or reference architectures for this type of cross-cloud training setup would be greatly appreciated

End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

kartheek_rao — Thu, 13 Aug 2026 16:31:55 GMT

I have been working on a project to understand Databricks end to end, rather than just loading some data and training a model.

I picked GDELT news data and the use case is to identify supply chain disruption related news and eventually predict which events could escalate.

I started by using ADF to pull GDELT GKG files and store the original ZIP files in ADLS.

Initially I did this with a single hardcoded file. Later I parameterized the pipeline, started reading the latest GDELT file information dynamically and added checks so the same file doesn't get ingested again.

One thing I intentionally did was to keep ADF mostly for data movement, instead of doing transformations there.

Once the files were available in ADLS, I connected the storage with Databricks using Managed Identity + Access Connector + Unity Catalog external locations/volumes.

Then came the part which I really wanted to learn — Auto Loader and Structured Streaming.

ADLS landing files are streamed into a Bronze Delta table. Bronze is intentionally almost raw and also stores things like source filename, ingestion timestamp and source path.

I created a separate checkpoint location for this stream.

This was probably one of the most useful things I learnt because earlier checkpoint was just another Spark term for me. After actually stopping the stream, starting it again and seeing that previously processed files were not processed again, it made much more sense.

From Bronze I created another stream going into Silver.

Here I parse the GDELT records and create proper article level fields like title, published time, URL, domain, themes, organisations, locations, tone etc.

So now if a new file arrives in ADLS, it can move through:

Auto Loader → Bronze → Silver incrementally.

For the ML side, I created a versioned dataset from Silver.

I am starting with TF-IDF + Logistic Regression as a baseline, but I don't want the NLP part to stop there. The plan is to compare it with transformer models like DistilBERT/DeBERTa, embeddings, NER and eventually story clustering.

For expensive training/HPO I am also experimenting with cheaper GPU compute outside Databricks instead of unnecessarily keeping Databricks compute running for hours.

But the model comes back to Databricks.

That is where I want to use MLflow, Unity Catalog Model Registry, model versions and production inference.

The final goal is that whenever a new GDELT file comes:

new article → Bronze → Silver → registered model → prediction.

Predictions will be stored in Delta/Gold tables and also written back to an ADLS backed location so other systems outside Databricks can consume them.

Still a lot left to build, specially around embeddings, clustering and escalation prediction, but this project has already changed how I look at ML projects.

Earlier I mostly thought:

data → model → prediction

Now I'm thinking more about ingestion, checkpoints, replayability, governance, model lifecycle and how the prediction actually runs when new data comes in.

Would love to hear from people working with Databricks — anything you would design differently in this architecture?

#Databricks #Azure #ADF #ADLS #PySpark #StructuredStreaming #AutoLoader #DeltaLake #UnityCatalog #MLflow #NLP #MLOps #GDELT

End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

kartheek_rao — Thu, 13 Aug 2026 16:10:17 GMT

Building an End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

I recently worked on an end-to-end streaming NLP project using GDELT news data, Azure Data Factory, ADLS Gen2 and Azure Databricks.

The goal was not just to train an NLP model. I wanted to understand the complete lifecycle:

external data ingestion → durable cloud storage → streaming ingestion → Bronze/Silver processing → NLP training and HPO → model registration → streaming inference → predictions → ADLS

The architecture evolved into:

GDELT
  ↓
Azure Data Factory
  ↓
ADLS Gen2
  ├── raw/compressed
  └── landing/extracted
  ↓
Unity Catalog External Location / Volume
  ↓
Databricks Auto Loader
  ↓
Bronze Delta
  ↓
Structured Streaming
  ↓
Silver Delta
  ↓
NLP Training + HPO
  ↓
MLflow / Unity Catalog Model Registry
  ↓
Streaming Model Inference
  ↓
Prediction Delta Tables
  ↓
ADLS Gen2

1. Acquiring GDELT data

GDELT publishes multiple datasets at regular intervals, including:

Events
Event Mentions
Global Knowledge Graph — GKG

For the NLP component, I started with the GKG dataset, because it provides useful article-level metadata such as:

article URL
themes
organizations
people
locations
tone
article title through the GKG Extras field

Instead of making Databricks responsible for downloading data from the internet, I separated the acquisition layer from the processing layer.

I used Azure Data Factory for acquisition.

The initial flow was:

GDELT HTTP endpoint
       ↓
ADF Binary Copy
       ↓
ADLS Gen2

I first tested the architecture with one GKG file.

For example:

20260802103000.gkg.csv.zip

ADF copied the original compressed file into:

raw/compressed/gkg/
ingestion_date=YYYY-MM-DD/

The original ZIP was deliberately preserved.

This became important later because it gave the pipeline:

replayability
traceability
debugging capability
separation between acquisition and transformation

ADF then decompressed the ZIP into:

landing/extracted/gkg/
ingestion_date=YYYY-MM-DD/

The guiding principle was:

ADF = acquire and land
ADLS = preserve
Databricks = process and model

2. Making ADF ingestion dynamic

The first pipeline used a fixed GDELT filename.

After proving that one file could move successfully from GDELT to ADLS, I parameterized the pipeline.

I added parameters for:

file_name
ingestion_date
expected_size
checksum

Then I used GDELT's latest-file information to automatically identify the newest GKG file.

ADF performed:

Lookup latest GDELT metadata
        ↓
Filter GKG entry
        ↓
Extract filename
        ↓
Derive ingestion date
        ↓
Check whether file already exists
        ↓
Ingest only if new

This introduced an important engineering property:

idempotency.

Rerunning the pipeline would not continually ingest the same file.

3. Simulating streaming without running 24×7

Because this was a learning project, I did not want to keep cloud compute running continuously.

Instead, I created controlled streaming sessions.

The ADF pipeline was configured to collect approximately three consecutive GDELT files.

Conceptually:

File 1
  ↓
Wait
  ↓
File 2
  ↓
Wait
  ↓
File 3
  ↓
Stop

The session used:

an Until loop
a target-file count
Wait activities
file-existence checks
a maximum session timeout

This allowed me to learn near-real-time ingestion while keeping infrastructure costs under control.

4. Connecting ADLS securely to Databricks

Instead of using storage keys or legacy DBFS mounts, I connected ADLS using:

Databricks Access Connector
        ↓
Managed Identity
        ↓
Unity Catalog Storage Credential
        ↓
External Location
        ↓
External Volume

This exposed the ADLS landing data through a governed path such as:

/Volumes/gdelt_dev/raw/gdelt_landing/

One of the most useful concepts I learned here was that /Volumes/... is a Databricks governed view over the underlying cloud storage.

The physical data still lives in ADLS.

5. Streaming ADLS files into Bronze using Auto Loader

The next layer was Databricks Auto Loader.

ADLS landing
      ↓
Unity Catalog Volume
      ↓
cloudFiles
      ↓
Structured Streaming
      ↓
Bronze Delta

I deliberately kept Bronze close to the source.

Instead of doing all business parsing immediately, the Bronze table stored:

raw_record
source_file_name
source_file_path
source_file_size
source_file_modification_time
ingested_at
ingestion_date

Auto Loader used a dedicated checkpoint:

checkpoints/bronze_gkg/

and a separate schema location.

The checkpoint became one of the most important concepts in the project.

It allows the stream to remember:

Which source files have already been successfully processed?

I could stop the Databricks cluster, restart it later and use the same checkpoint.

The stream continued from its previous progress rather than starting from scratch.

6. Bronze to Silver streaming

Once the raw GKG records were reliably landing in Bronze, I created another Structured Streaming pipeline:

Bronze Delta
    ↓
readStream
    ↓
GKG parsing
    ↓
cleaning
    ↓
Silver Delta

The GKG records were tab-delimited.

In Silver I parsed the useful fields and produced article-level features such as:

article_id
published_at
article_url
source_domain
title
tone
themes
organizations
persons
locations

The article title was extracted from the GKG Extras field.

Silver also performed operations such as:

timestamp conversion
URL/domain extraction
HTML title decoding
theme preparation
exact duplicate checks
malformed-record filtering

Importantly, Silver used a different checkpoint from Bronze.

So the architecture had independent state:

ADLS → Bronze checkpoint

Bronze → Silver checkpoint

At this stage I had a genuine multi-hop streaming architecture:

ADF lands file
      ↓
Auto Loader detects it
      ↓
Bronze updates
      ↓
Silver streaming query detects new Bronze rows
      ↓
Silver updates

7. Preparing the NLP dataset

From Silver I created an NLP-oriented profile dataset containing fields such as:

article_id
published_at
article_url
source_domain
title
title_normalized
tone
themes_clean
organizations
persons
locations

A versioned snapshot was then created for reproducibility.

For example:

article_nlp_profile_v001

I exported this snapshot as Parquet into a dedicated ADLS ML exchange area:

ml_exchange/
└── relevance/
    ├── datasets/
    └── models/

This provided a clean boundary between the governed Databricks data platform and the training environment.

8. NLP modelling strategy

I deliberately did not jump directly to BERT.

I structured the modelling as a progression.

Baseline

Title
 ↓
TF-IDF
 ↓
Logistic Regression

The purpose of the baseline was not to create the final model.

It established a benchmark.

Any more complicated transformer model needed to demonstrate that the additional complexity provided measurable value.

Deep NLP

The next models used pretrained transformer encoders for supply-chain relevance classification.

The progression included experiments such as:

TF-IDF
   ↓
DistilBERT
   ↓
DeBERTa / modern encoder models

The classification target was conceptually:

1 = supply-chain disruption
0 = not a supply-chain disruption

The project was designed to eventually extend beyond binary classification into:

disruption-category classification
named entity recognition
semantic embeddings
duplicate detection
story clustering
escalation prediction

9. Training and Hyperparameter Optimization

An important architectural decision was not to make the expensive training environment the centre of the platform.

Databricks remained the system of record.

Training/HPO could use lower-cost external GPU compute where appropriate.

The pattern became:

Databricks
 ↓
Versioned training dataset
 ↓
ADLS ML Exchange
 ↓
GPU training/HPO
 ↓
Best model artefact
 ↓
ADLS
 ↓
Databricks

For HPO I focused on parameters that materially affect transformer performance, for example:

learning rate
batch size
epochs
weight decay
warmup ratio
max sequence length

Model selection used more than accuracy.

For an early-warning problem, metrics such as these are more useful:

Precision
Recall
F1
PR-AUC
Precision@K
Recall@K

10. Bringing the model back into Databricks

The best model was not left in the training notebook.

The model artefact, configuration and evaluation metadata were returned to Databricks.

The model bundle included information such as:

model
best parameters
validation metrics
dataset version
feature schema
library requirements
training notes

Then Databricks became responsible again for the production lifecycle:

Best model
   ↓
MLflow
   ↓
Model evaluation
   ↓
Unity Catalog Model Registry
   ↓
Champion model

This separation was intentional.

Training compute could be disposable.

The governed model lifecycle remained in Databricks.

11. Driving streaming predictions

This was the point where the data-engineering and ML parts of the project came together.

The production inference path became:

New GDELT file
      ↓
ADF
      ↓
ADLS
      ↓
Auto Loader
      ↓
Bronze
      ↓
Silver
      ↓
Registered ML model
      ↓
Prediction

For every newly processed article, the model generates something conceptually similar to:

article_id
prediction_timestamp
model_version
relevance_probability
predicted_label

For example:

Title:
"Port workers announce nationwide strike"

Supply-chain disruption probability:
0.94

Prediction:
Relevant

Only relevant articles need to continue into more expensive downstream NLP:

Relevant article
      ↓
Disruption category
      ↓
Entity extraction
      ↓
Embeddings
      ↓
Semantic duplicate detection
      ↓
Story clustering

This is also useful for cost optimization because expensive NLP is performed only on the subset of records that passes the relevance model.

12. Persisting predictions back into ADLS

Predictions are first written into governed Delta tables.

For example:

gdelt_dev.ml.article_relevance_predictions

or later:

gdelt_dev.gold.active_alerts

The output contains both prediction data and lineage:

article_id
source_file
prediction_timestamp
model_name
model_version
probability
predicted_class
processing_run_id

Where external Azure consumers require the prediction output, the data can then be exported through a Unity Catalog External Volume backed by ADLS:

Databricks Delta prediction
        ↓
Gold/serving transformation
        ↓
External Volume
        ↓
ADLS

For example:

abfss://gdelt@<storage-account>.dfs.core.windows.net/
predictions/relevance/

This completes the round trip:

GDELT
 ↓
Azure
 ↓
Databricks
 ↓
Machine Learning
 ↓
Databricks predictions
 ↓
Azure

13. What I learned from this architecture

The biggest learning for me was that an ML project is much larger than model.fit().

I had to think about:

Data engineering

external acquisition
ADF
ADLS Gen2
immutable raw storage
idempotency
replayability
parameterized pipelines

Streaming

Auto Loader
Structured Streaming
processing-time triggers
checkpoints
restart behaviour
Bronze → Silver incremental processing

Governance

managed identities
Unity Catalog
external locations
Volumes
lineage

NLP

TF-IDF baselines
transformer fine-tuning
contextual embeddings
classification
semantic similarity

Machine learning

labelled datasets
time-aware evaluation
HPO
class imbalance
PR-AUC
threshold selection

MLOps

dataset versioning
MLflow
Model Registry
model versions
production inference
prediction lineage

Cost engineering

I also intentionally avoided running everything 24×7.

The same architectural concepts can be learned using controlled streaming sessions:

start resources
→ process several GDELT intervals
→ observe streaming behaviour
→ validate output
→ gracefully stop streams
→ terminate compute

That allowed me to learn the architecture without turning a personal learning project into an unnecessarily expensive cloud workload.

What comes next

The next stages are focused on making the intelligence layer deeper:

Relevance classification
        ↓
Disruption-category classification
        ↓
NER / entity normalization
        ↓
Sentence embeddings
        ↓
Semantic duplicate detection
        ↓
Story clustering
        ↓
Temporal feature engineering
        ↓
Predict whether a story will escalate
        ↓
Ranked supply-chain disruption alerts

For me, the most valuable part of this project has been connecting all the pieces rather than treating data engineering, NLP and MLOps as separate subjects.

ADF acquires the data.

ADLS preserves it.

Databricks streams and governs it.

ML models extract intelligence from it.

MLflow governs the model lifecycle.

And the prediction pipeline turns continuously arriving data into actionable outputs.

#Azure #AzureDataFactory #ADLS #Databricks #ApacheSpark #PySpark #StructuredStreaming #AutoLoader #DeltaLake #UnityCatalog #MLflow #NLP #BERT #MachineLearning #MLOps #DataEngineering #GDELT

Traffic split behavior when traffic_percentage values across served entities sum to more than 100%

MageshS — Wed, 29 Jul 2026 10:02:33 GMT

Hi all,

I'm configuring a Model Serving endpoint with two served entities and ran into some unexpected behavior while testing different traffic_config splits.

When I set traffic_percentage to 100 for each of the two served entities (so the total sums to 200%, not 100%), the configuration is accepted and saves successfully — both through the Serving UI and the REST API. I expected this to be rejected, since traffic percentages across routes should normally sum to 100.

Config used:

json

"traffic_config": {
  "routes": [
    { "served_model_name": "model-A", "traffic_percentage": 100 },
    { "served_model_name": "model-B", "traffic_percentage": 100 }
  ]
}

What I'm trying to understand:

Is there backend validation that should reject a traffic_config where the values don't sum to 100? If so, why does this particular config get accepted?
If it is accepted, how does routing actually work here — are the values treated as relative weights and normalized (e.g., 100/100 effectively becomes 50/50), or is there some other fallback logic?
Is this documented anywhere, or is it more of an edge case / bug that hasn't been addressed?

I ran a batch of test requests afterward and saw roughly a 50/50 split across the two entities, which lines up with the "weights get normalized" theory — but I'd rather get confirmation from someone who knows the internal routing logic than rely purely on my own testing.

Environment: AWS Databricks

Snowflake OR Databricks

suryaprayaga — Mon, 20 Jul 2026 16:56:09 GMT

I am doing a case choice analysis on whether I should have my data in Snowflake to support my Customer workloads or have them migrated to Databricks to do the same.

What has been your experience especially in handling large volumes of data especially when dealing with real-time data coming to into your DWH?

My customers would like to see real-time insights almost in real-time with some of the most complex Data Science workloads.

Using Machine Learning to Improve Emulator Compatibility Predictions

lucas1147 — Wed, 15 Jul 2026 06:18:47 GMT

Hi everyone,

I'm working on a personal project and would appreciate some advice from people who have experience with machine learning on Databricks.

Imagine having a dataset containing thousands of game compatibility records collected from different emulator versions, graphics backends, hardware configurations, frame rates, and user-reported issues. The goal is to predict whether a game will run correctly on a specific setup and identify the settings most likely to improve performance.

Would this problem be better approached as a classification task or a recommendation system? Which ML algorithms would you start with for this type of structured dataset?

I also wonder whether Databricks AutoML is a good starting point before building custom models. Has anyone here used AutoML for a project with a large number of categorical features and configuration variables?

For context, the dataset is related to retro gaming and PlayStation 2 emulator compatibility, similar to the type of information available on https://ps2biosonline.com/.

I'd appreciate any suggestions on feature engineering, model selection, or best practices for organizing this kind of project in Databricks.

Serving endpoint - automatic system updates

d_szepietowska — Thu, 09 Jul 2026 09:57:13 GMT

Dear Community 🙂

I need some support with investigation related to Serving Endpoints. Recently some of endpoints with deployed ML models display message:

This endpoint is out of compliance because it is too old and automatic system updates have failed.
Please update this endpoint to bring it into compliance.

The endpoint exist approximately 2 month and is used in business processes.

My question are:
1. What type of system updated are performed on existing endpoints?
2. Are there any log files which are used for tracking this type of updates?
3. What action should be performed to fix it?

Can't select Serverless GPU on notebooks

LPBigot — Thu, 09 Jul 2026 08:20:11 GMT

When trying to use serverless gpu on a notebook, I get this error. Serverless CPU works fine though.

Can you help me understand why I get this?

Free Edition CrossValidator not working because of internal caching

pjvi — Tue, 07 Jul 2026 21:05:51 GMT

Hi all,

I am getting the following error when using CrossValidator in Databricks Free Edition v5 (see attachment):

In shared or serverless cluster, SPARK_ML_TMP_DFS_PATH environmental variable must be set to a UC volume path like '/Volumes/...' in order to support Saprk DataFrame caching

Could you please check if this the expected behaviour, and if there a workaround or something we can try?

Thanks

Notebooks for Labs

ShaRKS — Sun, 05 Jul 2026 05:21:21 GMT

Hello can someone tell me where do I find the notebooks in the demo sessions of Associate ML pathways?

what is the GPU quota limit for free edition?

rakeshdey — Wed, 01 Jul 2026 14:58:04 GMT

what is the GPU quota limit for free edition? I used 5 minutes few days back now can not attach GPU cluster, it showing quota exceeded

Serverless ML

Daniele-T — Wed, 24 Jun 2026 09:48:58 GMT

Hello,

I'm trying to set up a DAB job that runs an ML job. For this it would be useful to use a serverless ML environment, that I can select in notebooks. Anyway, I do not find a meaningful way to define the base environment as ML.

I do not want to give the requirements-ML.txt, as I think there would be a larger start-up time. I could not find any useful documentation for it.

I tried something like

environments: - environment_key: default spec: environment_version: "5" base_environment: "ML" dependencies: - ... light dependencies

but it does expect a yml file on base environment.

Does anybody has a tip?

Thank you,

Daniele

AutoML on Azure Databricks as of June 2026

nepiskopos — Tue, 23 Jun 2026 12:29:41 GMT

Hello everybody,

My team and I are facing a sudden and unexpected issue with training Forecasting models utilizing AutoML on Azure Databricks.

As of two weeks ago, we privatized our Azure Databricks environment behind a private network, accessible only by using a VPN.

Since then, Serverless training of Forecasting ML models fails for every type of forecasting ML framework, including ARIMA, Prophet and DeepAR. To be more precise:

ARIMA training experiments complete. However, the deployment of trained ARIMA models in Endpoints fails. The respective Service logs are available here.
Prophet training experiments fail to complete, being cancelled at the Tuning stage. The only output in the run indicating that something is wrong is "databricks.automl.shared.databricks_utils: No host name to create absolute URL".
DeepAR training experiments also fail to complete, being cancelled at the Tuning stage. The only output in the run indicating that something is wrong is "databricks.automl.shared.databricks_utils: No host name to create absolute URL".

On the other hand, using a compute cluster to train ARIMA and Prophet models, the training completes successfully. The compute runtime is 17.3 LTS for Machine Learning. However, when trying to use these models for inference, we get the following issues.

For the ARIMA models, the deployment fails, indicating that the failure reason is "mlflowserving.scoring_server.diagnostic_classifier._PhaseTaggedError: [mlflow_parse] ModuleNotFoundError: No module named 'databricks.automl_runtime'"
For the Prophet models, the deployment completes. However, when trying to use any Prophet model for inference, the procedure fails, indicating that the failure reason is "Inference failed: Encountered an unexpected error while evaluating the model. Verify that the input is compatible with the model for inference. Error ''Prophet' object has no attribute 'stan_backend'' (Support code: INFERENCE_FAILED-9430702a5a9f)"

The serverless training of Forecasting ML models worked perfectly end-to-end a few weeks back and so did the deployment and the inference.

Could we get some help on how to resolve this issues? Is this a temporary bug? Do we need to take any actions?

I have also opened a respective GitHub issue.

Thank you in advance.

Can Databricks Jobs Run on Kubernetes Clusters?

ASH1243434 — Tue, 23 Jun 2026 11:33:54 GMT

Context: We're exploring using Kubernetes (EKS) as our compute infrastructure instead of Databricks managed clusters. We want to understand if Databricks can orchestrate, deploy, and monitor jobs that run on a Kubernetes cluster.

Questions:

Is it possible to configure Databricks Jobs to run workloads on an external Kubernetes cluster?
Can Databricks manage the job lifecycle (submit, monitor, track logs) on Kubernetes pods?
What's the recommended way to use Databricks as an orchestration layer for Kubernetes workloads?

What we want to do:

Write Databricks jobs/notebooks
Have those jobs execute as pods in our Kubernetes cluster (not Databricks-managed clusters)
Monitor job execution and logs from the Databricks UI
Keep using Databricks features (MLflow, Delta Lake) for data and model management

Why this is important:

Scalability: Kubernetes provides better horizontal scaling for large workloads
Faster cluster initialization: Pre-running EKS cluster eliminates spin-up delays
Cost efficiency: EKS with spot instances is more cost-effective than always-on Databricks clusters
Multi-tenant support: Share compute infrastructure across different teams and workloads