End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

kartheek_rao — Thu, 13 Aug 2026 16:10:17 GMT

Building an End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

I recently worked on an end-to-end streaming NLP project using GDELT news data, Azure Data Factory, ADLS Gen2 and Azure Databricks.

The goal was not just to train an NLP model. I wanted to understand the complete lifecycle:

external data ingestion → durable cloud storage → streaming ingestion → Bronze/Silver processing → NLP training and HPO → model registration → streaming inference → predictions → ADLS

The architecture evolved into:

GDELT
  ↓
Azure Data Factory
  ↓
ADLS Gen2
  ├── raw/compressed
  └── landing/extracted
  ↓
Unity Catalog External Location / Volume
  ↓
Databricks Auto Loader
  ↓
Bronze Delta
  ↓
Structured Streaming
  ↓
Silver Delta
  ↓
NLP Training + HPO
  ↓
MLflow / Unity Catalog Model Registry
  ↓
Streaming Model Inference
  ↓
Prediction Delta Tables
  ↓
ADLS Gen2

1. Acquiring GDELT data

GDELT publishes multiple datasets at regular intervals, including:

Events
Event Mentions
Global Knowledge Graph — GKG

For the NLP component, I started with the GKG dataset, because it provides useful article-level metadata such as:

article URL
themes
organizations
people
locations
tone
article title through the GKG Extras field

Instead of making Databricks responsible for downloading data from the internet, I separated the acquisition layer from the processing layer.

I used Azure Data Factory for acquisition.

The initial flow was:

GDELT HTTP endpoint
       ↓
ADF Binary Copy
       ↓
ADLS Gen2

I first tested the architecture with one GKG file.

For example:

20260802103000.gkg.csv.zip

ADF copied the original compressed file into:

raw/compressed/gkg/
ingestion_date=YYYY-MM-DD/

The original ZIP was deliberately preserved.

This became important later because it gave the pipeline:

replayability
traceability
debugging capability
separation between acquisition and transformation

ADF then decompressed the ZIP into:

landing/extracted/gkg/
ingestion_date=YYYY-MM-DD/

The guiding principle was:

ADF = acquire and land
ADLS = preserve
Databricks = process and model

2. Making ADF ingestion dynamic

The first pipeline used a fixed GDELT filename.

After proving that one file could move successfully from GDELT to ADLS, I parameterized the pipeline.

I added parameters for:

file_name
ingestion_date
expected_size
checksum

Then I used GDELT's latest-file information to automatically identify the newest GKG file.

ADF performed:

Lookup latest GDELT metadata
        ↓
Filter GKG entry
        ↓
Extract filename
        ↓
Derive ingestion date
        ↓
Check whether file already exists
        ↓
Ingest only if new

This introduced an important engineering property:

idempotency.

Rerunning the pipeline would not continually ingest the same file.

3. Simulating streaming without running 24×7

Because this was a learning project, I did not want to keep cloud compute running continuously.

Instead, I created controlled streaming sessions.

The ADF pipeline was configured to collect approximately three consecutive GDELT files.

Conceptually:

File 1
  ↓
Wait
  ↓
File 2
  ↓
Wait
  ↓
File 3
  ↓
Stop

The session used:

an Until loop
a target-file count
Wait activities
file-existence checks
a maximum session timeout

This allowed me to learn near-real-time ingestion while keeping infrastructure costs under control.

4. Connecting ADLS securely to Databricks

Instead of using storage keys or legacy DBFS mounts, I connected ADLS using:

Databricks Access Connector
        ↓
Managed Identity
        ↓
Unity Catalog Storage Credential
        ↓
External Location
        ↓
External Volume

This exposed the ADLS landing data through a governed path such as:

/Volumes/gdelt_dev/raw/gdelt_landing/

One of the most useful concepts I learned here was that /Volumes/... is a Databricks governed view over the underlying cloud storage.

The physical data still lives in ADLS.

5. Streaming ADLS files into Bronze using Auto Loader

The next layer was Databricks Auto Loader.

ADLS landing
      ↓
Unity Catalog Volume
      ↓
cloudFiles
      ↓
Structured Streaming
      ↓
Bronze Delta

I deliberately kept Bronze close to the source.

Instead of doing all business parsing immediately, the Bronze table stored:

raw_record
source_file_name
source_file_path
source_file_size
source_file_modification_time
ingested_at
ingestion_date

Auto Loader used a dedicated checkpoint:

checkpoints/bronze_gkg/

and a separate schema location.

The checkpoint became one of the most important concepts in the project.

It allows the stream to remember:

Which source files have already been successfully processed?

I could stop the Databricks cluster, restart it later and use the same checkpoint.

The stream continued from its previous progress rather than starting from scratch.

6. Bronze to Silver streaming

Once the raw GKG records were reliably landing in Bronze, I created another Structured Streaming pipeline:

Bronze Delta
    ↓
readStream
    ↓
GKG parsing
    ↓
cleaning
    ↓
Silver Delta

The GKG records were tab-delimited.

In Silver I parsed the useful fields and produced article-level features such as:

article_id
published_at
article_url
source_domain
title
tone
themes
organizations
persons
locations

The article title was extracted from the GKG Extras field.

Silver also performed operations such as:

timestamp conversion
URL/domain extraction
HTML title decoding
theme preparation
exact duplicate checks
malformed-record filtering

Importantly, Silver used a different checkpoint from Bronze.

So the architecture had independent state:

ADLS → Bronze checkpoint

Bronze → Silver checkpoint

At this stage I had a genuine multi-hop streaming architecture:

ADF lands file
      ↓
Auto Loader detects it
      ↓
Bronze updates
      ↓
Silver streaming query detects new Bronze rows
      ↓
Silver updates

7. Preparing the NLP dataset

From Silver I created an NLP-oriented profile dataset containing fields such as:

article_id
published_at
article_url
source_domain
title
title_normalized
tone
themes_clean
organizations
persons
locations

A versioned snapshot was then created for reproducibility.

For example:

article_nlp_profile_v001

I exported this snapshot as Parquet into a dedicated ADLS ML exchange area:

ml_exchange/
└── relevance/
    ├── datasets/
    └── models/

This provided a clean boundary between the governed Databricks data platform and the training environment.

8. NLP modelling strategy

I deliberately did not jump directly to BERT.

I structured the modelling as a progression.

Baseline

Title
 ↓
TF-IDF
 ↓
Logistic Regression

The purpose of the baseline was not to create the final model.

It established a benchmark.

Any more complicated transformer model needed to demonstrate that the additional complexity provided measurable value.

Deep NLP

The next models used pretrained transformer encoders for supply-chain relevance classification.

The progression included experiments such as:

TF-IDF
   ↓
DistilBERT
   ↓
DeBERTa / modern encoder models

The classification target was conceptually:

1 = supply-chain disruption
0 = not a supply-chain disruption

The project was designed to eventually extend beyond binary classification into:

disruption-category classification
named entity recognition
semantic embeddings
duplicate detection
story clustering
escalation prediction

9. Training and Hyperparameter Optimization

An important architectural decision was not to make the expensive training environment the centre of the platform.

Databricks remained the system of record.

Training/HPO could use lower-cost external GPU compute where appropriate.

The pattern became:

Databricks
 ↓
Versioned training dataset
 ↓
ADLS ML Exchange
 ↓
GPU training/HPO
 ↓
Best model artefact
 ↓
ADLS
 ↓
Databricks

For HPO I focused on parameters that materially affect transformer performance, for example:

learning rate
batch size
epochs
weight decay
warmup ratio
max sequence length

Model selection used more than accuracy.

For an early-warning problem, metrics such as these are more useful:

Precision
Recall
F1
PR-AUC
Precision@K
Recall@K

10. Bringing the model back into Databricks

The best model was not left in the training notebook.

The model artefact, configuration and evaluation metadata were returned to Databricks.

The model bundle included information such as:

model
best parameters
validation metrics
dataset version
feature schema
library requirements
training notes

Then Databricks became responsible again for the production lifecycle:

Best model
   ↓
MLflow
   ↓
Model evaluation
   ↓
Unity Catalog Model Registry
   ↓
Champion model

This separation was intentional.

Training compute could be disposable.

The governed model lifecycle remained in Databricks.

11. Driving streaming predictions

This was the point where the data-engineering and ML parts of the project came together.

The production inference path became:

New GDELT file
      ↓
ADF
      ↓
ADLS
      ↓
Auto Loader
      ↓
Bronze
      ↓
Silver
      ↓
Registered ML model
      ↓
Prediction

For every newly processed article, the model generates something conceptually similar to:

article_id
prediction_timestamp
model_version
relevance_probability
predicted_label

For example:

Title:
"Port workers announce nationwide strike"

Supply-chain disruption probability:
0.94

Prediction:
Relevant

Only relevant articles need to continue into more expensive downstream NLP:

Relevant article
      ↓
Disruption category
      ↓
Entity extraction
      ↓
Embeddings
      ↓
Semantic duplicate detection
      ↓
Story clustering

This is also useful for cost optimization because expensive NLP is performed only on the subset of records that passes the relevance model.

12. Persisting predictions back into ADLS

Predictions are first written into governed Delta tables.

For example:

gdelt_dev.ml.article_relevance_predictions

or later:

gdelt_dev.gold.active_alerts

The output contains both prediction data and lineage:

article_id
source_file
prediction_timestamp
model_name
model_version
probability
predicted_class
processing_run_id

Where external Azure consumers require the prediction output, the data can then be exported through a Unity Catalog External Volume backed by ADLS:

Databricks Delta prediction
        ↓
Gold/serving transformation
        ↓
External Volume
        ↓
ADLS

For example:

abfss://gdelt@<storage-account>.dfs.core.windows.net/
predictions/relevance/

This completes the round trip:

GDELT
 ↓
Azure
 ↓
Databricks
 ↓
Machine Learning
 ↓
Databricks predictions
 ↓
Azure

13. What I learned from this architecture

The biggest learning for me was that an ML project is much larger than model.fit().

I had to think about:

Data engineering

external acquisition
ADF
ADLS Gen2
immutable raw storage
idempotency
replayability
parameterized pipelines

Streaming

Auto Loader
Structured Streaming
processing-time triggers
checkpoints
restart behaviour
Bronze → Silver incremental processing

Governance

managed identities
Unity Catalog
external locations
Volumes
lineage

NLP

TF-IDF baselines
transformer fine-tuning
contextual embeddings
classification
semantic similarity

Machine learning

labelled datasets
time-aware evaluation
HPO
class imbalance
PR-AUC
threshold selection

MLOps

dataset versioning
MLflow
Model Registry
model versions
production inference
prediction lineage

Cost engineering

I also intentionally avoided running everything 24×7.

The same architectural concepts can be learned using controlled streaming sessions:

start resources
→ process several GDELT intervals
→ observe streaming behaviour
→ validate output
→ gracefully stop streams
→ terminate compute

That allowed me to learn the architecture without turning a personal learning project into an unnecessarily expensive cloud workload.

What comes next

The next stages are focused on making the intelligence layer deeper:

Relevance classification
        ↓
Disruption-category classification
        ↓
NER / entity normalization
        ↓
Sentence embeddings
        ↓
Semantic duplicate detection
        ↓
Story clustering
        ↓
Temporal feature engineering
        ↓
Predict whether a story will escalate
        ↓
Ranked supply-chain disruption alerts

For me, the most valuable part of this project has been connecting all the pieces rather than treating data engineering, NLP and MLOps as separate subjects.

ADF acquires the data.

ADLS preserves it.

Databricks streams and governs it.

ML models extract intelligence from it.

MLflow governs the model lifecycle.

And the prediction pipeline turns continuously arriving data into actionable outputs.

#Azure #AzureDataFactory #ADLS #Databricks #ApacheSpark #PySpark #StructuredStreaming #AutoLoader #DeltaLake #UnityCatalog #MLflow #NLP #BERT #MachineLearning #MLOps #DataEngineering #GDELT

topic End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks in Machine Learning

End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

Building an End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

1. Acquiring GDELT data

2. Making ADF ingestion dynamic

3. Simulating streaming without running 24×7

4. Connecting ADLS securely to Databricks

5. Streaming ADLS files into Bronze using Auto Loader

6. Bronze to Silver streaming

7. Preparing the NLP dataset

8. NLP modelling strategy

Baseline

Deep NLP

9. Training and Hyperparameter Optimization

10. Bringing the model back into Databricks

11. Driving streaming predictions

12. Persisting predictions back into ADLS

13. What I learned from this architecture

What comes next

Re: End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks