rss.livelink.threads-in-node

ML Training low File I/O and Throughout

aswinkks — Mon, 17 Aug 2026 18:55:08 GMT

Hi everyone,

I have an image-based deep learning workload running on Azure Databricks, while the training dataset must remain in AWS S3 due to some constraints. We cannot move or replicate the dataset to Azure.
Our current architecture is roughly:
AWS S3 (images) → Unity Catalog Volume → Azure Databricks GPU compute → PyTorch training

The main issue we're seeing is very high file I/O latency and relatively low training throughput. Since the dataset contains a large number of individual image files, reading the images through the UC Volume appears to involve significant network overhead and many individual file reads.
I initially expected Mosaic Streaming / StreamingDataset to improve this because the dataset can be converted into MDS shards and the shards can be downloaded progressively to local storage while training continues.

However, interestingly, in our testing:
Direct image loading from the UC Volume is currently faster than Mosaic Streaming.
I'm trying to understand whether we are missing an important configuration or whether the cross-cloud architecture itself is the primary bottleneck.

A few questions:
1. What is the recommended architecture for training on S3 data from Azure Databricks when the data cannot be moved to Azure?

2. For Mosaic Streaming, what are the recommended values/strategies for:
shard size
num_workers
predownload
cache_limit
shuffle configuration

3. Is there an optimal MDS shard size for image datasets to minimize S3/network overhead?

4. Would increasing DataLoader workers and prefetching significantly improve throughput in this cross-cloud scenario?

5. Would it be better to use local caching of UC Volume files instead of Mosaic Streaming for a multi-epoch image training workload?

6. Are there any Databricks-recommended approaches for measuring whether the bottleneck is S3 → Azure network bandwidth, file-level latency, CPU image decoding, or GPU starvation?

Our main objective is to maximize GPU utilization and training throughput without moving the source dataset out of S3.
Any recommendations, benchmarks, or reference architectures for this type of cross-cloud training setup would be greatly appreciated

Azure Databricks Serverless Compute Unable to Connect to Azure SQL MI Using Failover Group FQDN via

LokeshChikuru — Mon, 17 Aug 2026 14:16:58 GMT

We are using Azure Databricks Serverless Compute to connect to Azure SQL Managed Instance (SQL MI) through Network Connectivity Configuration (NCC) with Private Endpoint Rules. Connectivity works successfully when using the primary SQL MI FQDN. However, connectivity fails when the SQL Managed Instance is configured behind a Failover Group and the connection is attempted using the Failover Group listener FQDN. As a result, Databricks Serverless Compute is currently unable to establish a connection to Azure SQL MI via the Failover Group FQDN, while direct connections to the primary SQL MI endpoint continue to work as expected.

I want to help me with registration key

Kayunga — Mon, 17 Aug 2026 11:18:27 GMT

If statement in DAB YAML file support

QuanDo1 — Mon, 17 Aug 2026 08:17:59 GMT

Hi,
Is there a plan to have if statement support in DAB YAML file?

For example, I would like to have different schedule for jobs based on the environment (higher frequency in PROD, lower frequency in DEV, QA). How can I do it in Databricks via workflow yml files?

Unity Catalog Lineage APIs – Are these REST APIs publicly supported?

faruk — Mon, 17 Aug 2026 07:11:04 GMT

Hi everyone,

I’m currently working on a PoC to build a data dictionary in Databricks, and one of the requirements is to retrieve lineage information programmatically.

I was advised that Databricks provides Unity Catalog Lineage APIs that can retrieve table-level and column-level lineage, following the Unity Catalog permission model and without requiring direct access to the system.access schema.

I was provided with an example endpoint:

/api/2.1/unity-catalog/lineage/table-lineages

However, after searching the current Databricks documentation, I cannot find this endpoint or any official documentation for a table-lineages REST API.

The situation is that our parent company does not allow us to have direct access to the system.access schema for security and governance reasons. Therefore, querying system.access.table_lineage is not an option for our PoC.

My goal is to build a data dictionary and enrich it with lineage information, so I’m looking for a supported REST API that would allow me to retrieve table-level lineage without accessing system.access.table_lineage directly.

Could anyone clarify:

Is /api/2.1/unity-catalog/lineage/table-lineages a supported/public Databricks REST API?
If so, where can I find the official documentation?
Is there another supported REST API to retrieve table-level lineage without querying system.access.table_lineage?
Is column-level lineage also available through a supported API?

Any guidance or documentation would be greatly appreciated.

Thanks!

Is the Semantic Layer the Real Challenge With Databricks Genie?

kartikchoudhary — Mon, 17 Aug 2026 05:49:16 GMT

I've been thinking about this while looking at different Genie use cases.

It seems like the hardest part of implementing Genie isn't getting users to ask questions in natural language. It's making sure Genie understands what those questions actually mean in the context of the business.

For example, if someone asks:

“Why did our performance drop this week?”

Genie still needs to know:

Which metric are we talking about?
Which data is authoritative?
How are the tables related?
What does “performance” actually mean for that business?
Are there business rules that should affect the answer?

This becomes even more important for operational use cases. Questions around NPT, ROP, equipment performance, or cost can require data from multiple systems.

That makes me wonder if we're putting too much emphasis on the conversational interface and not enough on what sits underneath it.

Maybe the real Genie implementation project isn't building the Genie space. It's building the context that allows Genie to give a trustworthy answer.

Curious what others are seeing:

What has been the bigger challenge in your Genie projects — configuring Genie itself, or getting the underlying data, metrics, and business definitions ready for it?

Oracle CDC Ingestion Pipeline - Schema Exploration finds 0 tables when DB_DOMAIN

karuppusamy — Sun, 16 Aug 2026 13:01:36 GMT

Hi,

I'm setting up an Oracle Integrated CDC Managed Ingestion Pipeline (LakeFlow Connect) on Azure Databricks. The pipeline connects to Oracle 19c successfully but Schema Exploration always returns 0 tables.

Error:

Schema Exploration has COMPLETED in 5 seconds. Found 0 table(s) in connection.
Extractor FAILED. [DLT ERROR CODE: INGESTION_GATEWAY_INTERNAL_ERROR]
Timed out waiting for all replicated tables.

Environment:

- Azure Databricks (Workspace with Unity Catalog)
- Oracle 19c Standard Edition 2 (Single-Tenant, non-CDB, non-RAC)
- Previews enabled: "Direct Cdc Managed Ingestion Pipeline" = ON, "Integrated Cdc Oracle Connector" = ON
- Pipeline type: MANAGED_INGESTION
- Channel: PREVIEW

Oracle Configuration (all verified by direct connection):

- ARCHIVELOG mode: Enabled
- Supplemental logging: MIN=YES, PK=YES (DB level) + ALL COLUMN at table level
- Table has Primary Key
- DB_DOMAIN = example.com
- Registered service name: `mydb.example.com` (with domain)
- No service registered without domain (connecting to just `MYDB` gives ORA-12514)

User Privileges (all confirmed):

- CREATE SESSION, SELECT ANY TABLE, SELECT ANY DICTIONARY, LOGMINING, SELECT ANY TRANSACTION, FLASHBACK ANY TABLE
- EXECUTE_CATALOG_ROLE granted
- Explicit SELECT grants on all target tables
- EXECUTE on DBMS_LOGMNR, DBMS_LOGMNR_D

CDC Validation (run by the pipeline itself): All 10 checks PASSED

The Problem:

My Oracle DB has `DB_DOMAIN` set, so the only valid service name is `mydb.example.com` (contains dots). But the Databricks documentation for INGESTION_GATEWAY_ORACLE_MISSING_CONFIGURATION states:

> The 'source_catalog' parameter must contain only printable ASCII characters such as letters, numbers, and underscores.

This creates a conflict:

- `source_catalog: MYDB` → Connection fails (ORA-12514, service doesn't exist without domain)
- `source_catalog: mydb.example.com` → Connection succeeds, TLS established, DB version detected, CDC validation passes, but Schema Exploration returns 0 tables

What I've tried (all failed with 0 tables):

- source_catalog: mydb (lowercase)
- source_catalog: MYDB (uppercase)
- source_catalog: mydb.example.com (full service name)
- channel: CURRENT and PREVIEW
- Full refresh

Connection config:

- Host: 10.x.x.x
- Port: 1521
- Service Name: mydb.example.com
- Encryption: native_network_encryption

Questions:

1. Does the Oracle CDC connector support databases with DB_DOMAIN set?
2. How should source_catalog be configured when the only valid service name contains dots?
3. Is adding a static listener registration (without domain) the recommended workaround?

Any help appreciated. Thanks!

Exam issues due to Webassessor maintenance

Leito1903 — Sat, 15 Aug 2026 20:19:58 GMT

Good afternoon,
To the Databricks team,
I am writing to inform you that I was scheduled to take my exam Databricks Certified Generative AI Engineer on August 11th at 11:00 PM, but unfortunately, I was unable to do so due to maintenance on the Webassessor website.

Therefore, I would like to request the opportunity to reschedule the exam, as the unavailability of the Webassessor testing site prevented me from taking it. I submitted my request on August 12th but have not yet received a response; I only have the assigned ticket number, which is:

#00989390

Issues with Custom Agents on Free Edition

taniumalloy — Sat, 15 Aug 2026 19:03:59 GMT

I have ran into a few issues while learning how to use Custom Agents in Databricks Apps that I would like to call out - both to help others and hopefully motivate platform improvements.

The only model that seems to work is databricks-meta-llama-3-3-70b-instruct, but I only found that out through a DataExpert.io YouTube video. It would be nice if the app templates would default to this or at least provide some guidance on the Free Edition limitations page.
Recent changes to MCP have caused issues with the default OpenAI Agents SDK app template. You have to manually add "mcp<2.0.0" to the dependencies in pyproject.toml to get it up and running.
The second message from the user starts to cause errors. I haven't figured out how to solve this, but it is something related to the format of the message chain. For example:

{ "detail": "Unhandled item type or structure: {'status': None, 'content': [{'text': \"Hello, it's nice to meet you. Is there something I can help you with or would you like to chat?\", 'type': 'output_text'}], 'role': 'assistant', 'type': 'message'}" }

Are We Entering the Context Engineering Era?

Brahmareddy — Sat, 15 Aug 2026 11:36:20 GMT

I’ve been thinking about how enterprise AI is changing, especially after attending the Data + AI Summit 2026 and hearing Ali Ghodsi emphasize the importance of enterprise context. It made the direction much clearer to me. We already have very capable AI models. The bigger challenge now is giving those models the right understanding of the business.

What does this data mean?

Which metric should the AI trust?

Who can access it?

What business rules should it follow?

And what action should it take next?

That also made me think more about the 4Cs: Context, Control, Choice, and Cost. Context helps AI understand the business. Control helps enterprises govern what AI can access and do. Choice gives organizations flexibility across models and tools instead of locking everything into one path. Cost becomes critical when AI moves from a few experiments to thousands or even millions of agent actions.

This is why I find the direction Databricks is taking with Genie, Genie Ontology, Unity Catalog, and agents very interesting. We spent years building platforms that make enterprise data usable for people. Now we are entering a phase where that same data, meaning, governance, and business knowledge need to become usable by AI agents.

For me, the 4Cs are becoming a simple way to think about enterprise AI: give AI the right Context, keep the right Control, preserve Choice, and understand the Cost.

Would love to hear how others in the Databricks Community are thinking about this.

SQL Query billing

anmolhhns — Fri, 14 Aug 2026 15:51:29 GMT

Is there any way to find a cost of one single query? i can't find any table in system catalog to connect a query with usage to find the exact cost of one query run in sql warehouse?

Billing

kosarchik — Fri, 14 Aug 2026 13:54:28 GMT

Hello, where can I find a breakdown of usage from the monitoring.azure_costs.costs_details table? I filtered MeterCategory = 'Foundry Models' to find out who is using which models, but the data there is only aggregated.

API Get Metadata Registerd Models

vinhdv4 — Fri, 14 Aug 2026 07:58:09 GMT

Hi all,

I has been call DBX API for get all metadata model
My user have no EXECUTE privileges on registered model, it only have USE CATALOG / USE SCHEMA on parent catalog/schema
I still can get all metadata of Model

Here is my python code for call API:
___

import json

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

list_models = w.registered_models.list(

catalog_name="system",

schema_name="ai",

include_browse=True)

i=0

for model in list_models:

print(json.dumps(model.as_dict(), indent=4, ensure_ascii=False))

i+=1

print(i)

___

But this documentation required EXECUTE privilege on the registered model (https://docs.databricks.com/api/workspace/registeredmodels/list)

Can anyone explain for me, please? T___T

How to use AI for photo filter during registration like OwnMates?

C_45 — Thu, 13 Aug 2026 22:25:31 GMT

I am creating a website where user can upload the pic but it will be automatically filter if the picture created by AI.

Some social website are using it

#ai #artificial

facing issue in llm models

shrm80 — Thu, 13 Aug 2026 18:03:22 GMT

i have signed up to databricks using aws but when i try to access model i m getting error Error (403): PERMISSION_DENIED: The endpoint is temporarily disabled due to a Databricks-set rate limit of 0.

Excel Add-in Regression - Sign-in opens external browser window instead of embedding in task pane, r

data_lover — Thu, 13 Aug 2026 17:46:56 GMT

Issue: The Databricks Excel Add-in launches authentication in an external browser window rather than within the Excel task pane. The Unity Catalog does display in the external browser window, but after sign-in, Excel remains stuck spinning/loading indefinitely.
Affected Platforms: Reproduced on both Windows and macOS using Microsoft 365 Excel.
Deployment Method: The add-in is side-loaded through a custom XML manifest rather than deployed tenant-wide from the Microsoft Marketplace.
Troubleshooting Completed:
- Verified and modified manifest trust and authentication settings.
- Cleared Excel Web Extensions Framework (WEF) caches and reloaded the manifest.
- Tested browser settings, pop-up permissions, cookies, and documented authentication workarounds.
- Reproduced on multiple independent Windows and Mac installations.
Business Impact: Users cannot refresh a recurring internal report through the add-in and must perform manual exports from the Databricks UI as a temporary workaround.
Suspected Cause: The issue may be tied to an August 2026 update to the Databricks Excel Add-in that introduced new features such as cascading filters and Genie One integration. The timing aligns with the appearance of the authentication problem. https://docs.databricks.com/aws/en/release-notes/product/2026/august
Request to Support: Investigate the regression and provide any known workarounds or fixes.

why micro-batching matters so much in Databricks Auto Loader and Structured Streaming

kartheek_rao — Thu, 13 Aug 2026 16:48:18 GMT

I recently spent some time understanding why micro-batching matters so much in Databricks Auto Loader and Structured Streaming, and it changed the way I look at streaming pipelines.

Earlier, I used to think streaming meant processing every record the instant it arrived.

But in most analytical data pipelines, that is not really the requirement.

What matters more is processing new data continuously, reliably, and with a latency that matches the business need.

This is where micro-batching becomes very practical.

Instead of processing every record individually, Spark groups newly available data into small batches and processes them using its distributed engine.

That gives us a useful balance between batch efficiency and streaming behaviour.

For Auto Loader, this becomes even more important because the source is usually cloud object storage such as ADLS, S3, or GCS.

New files arrive over time, and Auto Loader identifies only the files that have not already been processed.

Those new files are then picked up in the next micro-batch.

What I found especially useful is how naturally this works with checkpoints.

A checkpoint keeps track of the progress of the streaming query.

So if a stream successfully processes some data and the cluster later stops or fails, Spark can restart from the last known state instead of reprocessing everything from the beginning.

That made checkpoints much easier for me to understand.

They are not just folders that we create because Structured Streaming asks for them.

They represent the state and progress of the stream.

Micro-batching also gives us a way to control how much data is processed at a time.

If a large number of files suddenly arrive, we do not necessarily want the pipeline to consume everything at once and put unnecessary pressure on the compute.

Instead, the workload can be distributed across multiple micro-batches.

This also creates an important cost and latency trade-off.

For example, in my GDELT project, I do not need a news article prediction within a few milliseconds.

If the data is processed within 30 seconds or one minute, that is still perfectly acceptable for the business use case.

So trying to build an ultra-low-latency architecture would add complexity without providing much additional value.

Another thing I learned is that micro-batching fits very naturally with the Medallion Architecture.

My Bronze ingestion has its own streaming state, while the Bronze-to-Silver processing has a separate checkpoint and recovery boundary.

That makes each stage easier to reason about independently.

If Bronze stops, I can restart Bronze from its checkpoint.

If Silver stops, Silver can continue independently from its own checkpoint.

This separation makes the overall pipeline much more manageable.

I also started seeing AvailableNow differently.

For many workloads, I may not need to keep a streaming query running continuously.

I can start the job, process all currently available data incrementally, update the checkpoint, and then allow the compute to stop.

For cost-sensitive workloads, that can be a very useful pattern.

The biggest change in my understanding was realizing that streaming is not simply a choice between “batch” and “real time.”

There is a spectrum.

Traditional batch processing works for large periodic workloads.

AvailableNow works well when data should be processed incrementally whenever a scheduled job runs.

Micro-batch streaming works well when seconds- or minutes-level latency is required.

Ultra-low-latency streaming is valuable only when the business genuinely needs millisecond-level responses.

For most analytical pipelines I am working with, especially Auto Loader with Bronze, Silver, and Gold layers, micro-batching feels like the practical middle ground.

It gives good throughput, checkpoint-based recovery, incremental processing, controlled workload size, and reasonable latency without forcing the system into unnecessary operational complexity.

That is why I now see micro-batching as one of the most important concepts to understand when learning Databricks Structured Streaming.

I would be interested to know how others decide between ProcessingTime, AvailableNow, and lower-latency streaming approaches in production systems.

End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

kartheek_rao — Thu, 13 Aug 2026 16:31:55 GMT

I have been working on a project to understand Databricks end to end, rather than just loading some data and training a model.

I picked GDELT news data and the use case is to identify supply chain disruption related news and eventually predict which events could escalate.

I started by using ADF to pull GDELT GKG files and store the original ZIP files in ADLS.

Initially I did this with a single hardcoded file. Later I parameterized the pipeline, started reading the latest GDELT file information dynamically and added checks so the same file doesn't get ingested again.

One thing I intentionally did was to keep ADF mostly for data movement, instead of doing transformations there.

Once the files were available in ADLS, I connected the storage with Databricks using Managed Identity + Access Connector + Unity Catalog external locations/volumes.

Then came the part which I really wanted to learn — Auto Loader and Structured Streaming.

ADLS landing files are streamed into a Bronze Delta table. Bronze is intentionally almost raw and also stores things like source filename, ingestion timestamp and source path.

I created a separate checkpoint location for this stream.

This was probably one of the most useful things I learnt because earlier checkpoint was just another Spark term for me. After actually stopping the stream, starting it again and seeing that previously processed files were not processed again, it made much more sense.

From Bronze I created another stream going into Silver.

Here I parse the GDELT records and create proper article level fields like title, published time, URL, domain, themes, organisations, locations, tone etc.

So now if a new file arrives in ADLS, it can move through:

Auto Loader → Bronze → Silver incrementally.

For the ML side, I created a versioned dataset from Silver.

I am starting with TF-IDF + Logistic Regression as a baseline, but I don't want the NLP part to stop there. The plan is to compare it with transformer models like DistilBERT/DeBERTa, embeddings, NER and eventually story clustering.

For expensive training/HPO I am also experimenting with cheaper GPU compute outside Databricks instead of unnecessarily keeping Databricks compute running for hours.

But the model comes back to Databricks.

That is where I want to use MLflow, Unity Catalog Model Registry, model versions and production inference.

The final goal is that whenever a new GDELT file comes:

new article → Bronze → Silver → registered model → prediction.

Predictions will be stored in Delta/Gold tables and also written back to an ADLS backed location so other systems outside Databricks can consume them.

Still a lot left to build, specially around embeddings, clustering and escalation prediction, but this project has already changed how I look at ML projects.

Earlier I mostly thought:

data → model → prediction

Now I'm thinking more about ingestion, checkpoints, replayability, governance, model lifecycle and how the prediction actually runs when new data comes in.

Would love to hear from people working with Databricks — anything you would design differently in this architecture?

#Databricks #Azure #ADF #ADLS #PySpark #StructuredStreaming #AutoLoader #DeltaLake #UnityCatalog #MLflow #NLP #MLOps #GDELT

How to extract data from SAP to Databricks?

RaulTovar — Thu, 13 Aug 2026 16:25:01 GMT

Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.

A few things I'm trying to figure out:

How are you actually pulling the data out? CDS views, ODP extractors, some third party connector, or a custom build?
Are you going real time (Kafka/CDC style) or is scheduled batch working fine for your use case?
What's bitten you the most? Schema changes breaking things, SAP getting cranky under load, latency, something else entirely?

We're dealing with decent volume on our end and really don't want to hammer the SAP source system, so reliability matters more to us than raw speed. If you've been through this, I'd really appreciate hearing what worked (and what didn't).

Thanks in advance!

End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

kartheek_rao — Thu, 13 Aug 2026 16:10:17 GMT

Building an End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

I recently worked on an end-to-end streaming NLP project using GDELT news data, Azure Data Factory, ADLS Gen2 and Azure Databricks.

The goal was not just to train an NLP model. I wanted to understand the complete lifecycle:

external data ingestion → durable cloud storage → streaming ingestion → Bronze/Silver processing → NLP training and HPO → model registration → streaming inference → predictions → ADLS

The architecture evolved into:

GDELT
  ↓
Azure Data Factory
  ↓
ADLS Gen2
  ├── raw/compressed
  └── landing/extracted
  ↓
Unity Catalog External Location / Volume
  ↓
Databricks Auto Loader
  ↓
Bronze Delta
  ↓
Structured Streaming
  ↓
Silver Delta
  ↓
NLP Training + HPO
  ↓
MLflow / Unity Catalog Model Registry
  ↓
Streaming Model Inference
  ↓
Prediction Delta Tables
  ↓
ADLS Gen2

1. Acquiring GDELT data

GDELT publishes multiple datasets at regular intervals, including:

Events
Event Mentions
Global Knowledge Graph — GKG

For the NLP component, I started with the GKG dataset, because it provides useful article-level metadata such as:

article URL
themes
organizations
people
locations
tone
article title through the GKG Extras field

Instead of making Databricks responsible for downloading data from the internet, I separated the acquisition layer from the processing layer.

I used Azure Data Factory for acquisition.

The initial flow was:

GDELT HTTP endpoint
       ↓
ADF Binary Copy
       ↓
ADLS Gen2

I first tested the architecture with one GKG file.

For example:

20260802103000.gkg.csv.zip

ADF copied the original compressed file into:

raw/compressed/gkg/
ingestion_date=YYYY-MM-DD/

The original ZIP was deliberately preserved.

This became important later because it gave the pipeline:

replayability
traceability
debugging capability
separation between acquisition and transformation

ADF then decompressed the ZIP into:

landing/extracted/gkg/
ingestion_date=YYYY-MM-DD/

The guiding principle was:

ADF = acquire and land
ADLS = preserve
Databricks = process and model

2. Making ADF ingestion dynamic

The first pipeline used a fixed GDELT filename.

After proving that one file could move successfully from GDELT to ADLS, I parameterized the pipeline.

I added parameters for:

file_name
ingestion_date
expected_size
checksum

Then I used GDELT's latest-file information to automatically identify the newest GKG file.

ADF performed:

Lookup latest GDELT metadata
        ↓
Filter GKG entry
        ↓
Extract filename
        ↓
Derive ingestion date
        ↓
Check whether file already exists
        ↓
Ingest only if new

This introduced an important engineering property:

idempotency.

Rerunning the pipeline would not continually ingest the same file.

3. Simulating streaming without running 24×7

Because this was a learning project, I did not want to keep cloud compute running continuously.

Instead, I created controlled streaming sessions.

The ADF pipeline was configured to collect approximately three consecutive GDELT files.

Conceptually:

File 1
  ↓
Wait
  ↓
File 2
  ↓
Wait
  ↓
File 3
  ↓
Stop

The session used:

an Until loop
a target-file count
Wait activities
file-existence checks
a maximum session timeout

This allowed me to learn near-real-time ingestion while keeping infrastructure costs under control.

4. Connecting ADLS securely to Databricks

Instead of using storage keys or legacy DBFS mounts, I connected ADLS using:

Databricks Access Connector
        ↓
Managed Identity
        ↓
Unity Catalog Storage Credential
        ↓
External Location
        ↓
External Volume

This exposed the ADLS landing data through a governed path such as:

/Volumes/gdelt_dev/raw/gdelt_landing/

One of the most useful concepts I learned here was that /Volumes/... is a Databricks governed view over the underlying cloud storage.

The physical data still lives in ADLS.

5. Streaming ADLS files into Bronze using Auto Loader

The next layer was Databricks Auto Loader.

ADLS landing
      ↓
Unity Catalog Volume
      ↓
cloudFiles
      ↓
Structured Streaming
      ↓
Bronze Delta

I deliberately kept Bronze close to the source.

Instead of doing all business parsing immediately, the Bronze table stored:

raw_record
source_file_name
source_file_path
source_file_size
source_file_modification_time
ingested_at
ingestion_date

Auto Loader used a dedicated checkpoint:

checkpoints/bronze_gkg/

and a separate schema location.

The checkpoint became one of the most important concepts in the project.

It allows the stream to remember:

Which source files have already been successfully processed?

I could stop the Databricks cluster, restart it later and use the same checkpoint.

The stream continued from its previous progress rather than starting from scratch.

6. Bronze to Silver streaming

Once the raw GKG records were reliably landing in Bronze, I created another Structured Streaming pipeline:

Bronze Delta
    ↓
readStream
    ↓
GKG parsing
    ↓
cleaning
    ↓
Silver Delta

The GKG records were tab-delimited.

In Silver I parsed the useful fields and produced article-level features such as:

article_id
published_at
article_url
source_domain
title
tone
themes
organizations
persons
locations

The article title was extracted from the GKG Extras field.

Silver also performed operations such as:

timestamp conversion
URL/domain extraction
HTML title decoding
theme preparation
exact duplicate checks
malformed-record filtering

Importantly, Silver used a different checkpoint from Bronze.

So the architecture had independent state:

ADLS → Bronze checkpoint

Bronze → Silver checkpoint

At this stage I had a genuine multi-hop streaming architecture:

ADF lands file
      ↓
Auto Loader detects it
      ↓
Bronze updates
      ↓
Silver streaming query detects new Bronze rows
      ↓
Silver updates

7. Preparing the NLP dataset

From Silver I created an NLP-oriented profile dataset containing fields such as:

article_id
published_at
article_url
source_domain
title
title_normalized
tone
themes_clean
organizations
persons
locations

A versioned snapshot was then created for reproducibility.

For example:

article_nlp_profile_v001

I exported this snapshot as Parquet into a dedicated ADLS ML exchange area:

ml_exchange/
└── relevance/
    ├── datasets/
    └── models/

This provided a clean boundary between the governed Databricks data platform and the training environment.

8. NLP modelling strategy

I deliberately did not jump directly to BERT.

I structured the modelling as a progression.

Baseline

Title
 ↓
TF-IDF
 ↓
Logistic Regression

The purpose of the baseline was not to create the final model.

It established a benchmark.

Any more complicated transformer model needed to demonstrate that the additional complexity provided measurable value.

Deep NLP

The next models used pretrained transformer encoders for supply-chain relevance classification.

The progression included experiments such as:

TF-IDF
   ↓
DistilBERT
   ↓
DeBERTa / modern encoder models

The classification target was conceptually:

1 = supply-chain disruption
0 = not a supply-chain disruption

The project was designed to eventually extend beyond binary classification into:

disruption-category classification
named entity recognition
semantic embeddings
duplicate detection
story clustering
escalation prediction

9. Training and Hyperparameter Optimization

An important architectural decision was not to make the expensive training environment the centre of the platform.

Databricks remained the system of record.

Training/HPO could use lower-cost external GPU compute where appropriate.

The pattern became:

Databricks
 ↓
Versioned training dataset
 ↓
ADLS ML Exchange
 ↓
GPU training/HPO
 ↓
Best model artefact
 ↓
ADLS
 ↓
Databricks

For HPO I focused on parameters that materially affect transformer performance, for example:

learning rate
batch size
epochs
weight decay
warmup ratio
max sequence length

Model selection used more than accuracy.

For an early-warning problem, metrics such as these are more useful:

Precision
Recall
F1
PR-AUC
Precision@K
Recall@K

10. Bringing the model back into Databricks

The best model was not left in the training notebook.

The model artefact, configuration and evaluation metadata were returned to Databricks.

The model bundle included information such as:

model
best parameters
validation metrics
dataset version
feature schema
library requirements
training notes

Then Databricks became responsible again for the production lifecycle:

Best model
   ↓
MLflow
   ↓
Model evaluation
   ↓
Unity Catalog Model Registry
   ↓
Champion model

This separation was intentional.

Training compute could be disposable.

The governed model lifecycle remained in Databricks.

11. Driving streaming predictions

This was the point where the data-engineering and ML parts of the project came together.

The production inference path became:

New GDELT file
      ↓
ADF
      ↓
ADLS
      ↓
Auto Loader
      ↓
Bronze
      ↓
Silver
      ↓
Registered ML model
      ↓
Prediction

For every newly processed article, the model generates something conceptually similar to:

article_id
prediction_timestamp
model_version
relevance_probability
predicted_label

For example:

Title:
"Port workers announce nationwide strike"

Supply-chain disruption probability:
0.94

Prediction:
Relevant

Only relevant articles need to continue into more expensive downstream NLP:

Relevant article
      ↓
Disruption category
      ↓
Entity extraction
      ↓
Embeddings
      ↓
Semantic duplicate detection
      ↓
Story clustering

This is also useful for cost optimization because expensive NLP is performed only on the subset of records that passes the relevance model.

12. Persisting predictions back into ADLS

Predictions are first written into governed Delta tables.

For example:

gdelt_dev.ml.article_relevance_predictions

or later:

gdelt_dev.gold.active_alerts

The output contains both prediction data and lineage:

article_id
source_file
prediction_timestamp
model_name
model_version
probability
predicted_class
processing_run_id

Where external Azure consumers require the prediction output, the data can then be exported through a Unity Catalog External Volume backed by ADLS:

Databricks Delta prediction
        ↓
Gold/serving transformation
        ↓
External Volume
        ↓
ADLS

For example:

abfss://gdelt@<storage-account>.dfs.core.windows.net/
predictions/relevance/

This completes the round trip:

GDELT
 ↓
Azure
 ↓
Databricks
 ↓
Machine Learning
 ↓
Databricks predictions
 ↓
Azure

13. What I learned from this architecture

The biggest learning for me was that an ML project is much larger than model.fit().

I had to think about:

Data engineering

external acquisition
ADF
ADLS Gen2
immutable raw storage
idempotency
replayability
parameterized pipelines

Streaming

Auto Loader
Structured Streaming
processing-time triggers
checkpoints
restart behaviour
Bronze → Silver incremental processing

Governance

managed identities
Unity Catalog
external locations
Volumes
lineage

NLP

TF-IDF baselines
transformer fine-tuning
contextual embeddings
classification
semantic similarity

Machine learning

labelled datasets
time-aware evaluation
HPO
class imbalance
PR-AUC
threshold selection

MLOps

dataset versioning
MLflow
Model Registry
model versions
production inference
prediction lineage

Cost engineering

I also intentionally avoided running everything 24×7.

The same architectural concepts can be learned using controlled streaming sessions:

start resources
→ process several GDELT intervals
→ observe streaming behaviour
→ validate output
→ gracefully stop streams
→ terminate compute

That allowed me to learn the architecture without turning a personal learning project into an unnecessarily expensive cloud workload.

What comes next

The next stages are focused on making the intelligence layer deeper:

Relevance classification
        ↓
Disruption-category classification
        ↓
NER / entity normalization
        ↓
Sentence embeddings
        ↓
Semantic duplicate detection
        ↓
Story clustering
        ↓
Temporal feature engineering
        ↓
Predict whether a story will escalate
        ↓
Ranked supply-chain disruption alerts

For me, the most valuable part of this project has been connecting all the pieces rather than treating data engineering, NLP and MLOps as separate subjects.

ADF acquires the data.

ADLS preserves it.

Databricks streams and governs it.

ML models extract intelligence from it.

MLflow governs the model lifecycle.

And the prediction pipeline turns continuously arriving data into actionable outputs.

#Azure #AzureDataFactory #ADLS #Databricks #ApacheSpark #PySpark #StructuredStreaming #AutoLoader #DeltaLake #UnityCatalog #MLflow #NLP #BERT #MachineLearning #MLOps #DataEngineering #GDELT