Building an End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks
I recently worked on an end-to-end streaming NLP project using GDELT news data, Azure Data Factory, ADLS Gen2 and Azure Databricks.
The goal was not just to train an NLP model. I wanted to understand the complete lifecycle:
external data ingestion → durable cloud storage → streaming ingestion → Bronze/Silver processing → NLP training and HPO → model registration → streaming inference → predictions → ADLS
The architecture evolved into:
GDELT
↓
Azure Data Factory
↓
ADLS Gen2
├── raw/compressed
└── landing/extracted
↓
Unity Catalog External Location / Volume
↓
Databricks Auto Loader
↓
Bronze Delta
↓
Structured Streaming
↓
Silver Delta
↓
NLP Training + HPO
↓
MLflow / Unity Catalog Model Registry
↓
Streaming Model Inference
↓
Prediction Delta Tables
↓
ADLS Gen2
1. Acquiring GDELT data
GDELT publishes multiple datasets at regular intervals, including:
For the NLP component, I started with the GKG dataset, because it provides useful article-level metadata such as:
Instead of making Databricks responsible for downloading data from the internet, I separated the acquisition layer from the processing layer.
I used Azure Data Factory for acquisition.
The initial flow was:
GDELT HTTP endpoint
↓
ADF Binary Copy
↓
ADLS Gen2I first tested the architecture with one GKG file.
For example:
20260802103000.gkg.csv.zip
ADF copied the original compressed file into:
raw/compressed/gkg/
ingestion_date=YYYY-MM-DD/
The original ZIP was deliberately preserved.
This became important later because it gave the pipeline:
ADF then decompressed the ZIP into:
landing/extracted/gkg/
ingestion_date=YYYY-MM-DD/
The guiding principle was:
ADF = acquire and land
ADLS = preserve
Databricks = process and model
2. Making ADF ingestion dynamic
The first pipeline used a fixed GDELT filename.
After proving that one file could move successfully from GDELT to ADLS, I parameterized the pipeline.
I added parameters for:
file_name
ingestion_date
expected_size
checksum
Then I used GDELT's latest-file information to automatically identify the newest GKG file.
ADF performed:
Lookup latest GDELT metadata
↓
Filter GKG entry
↓
Extract filename
↓
Derive ingestion date
↓
Check whether file already exists
↓
Ingest only if newThis introduced an important engineering property:
idempotency.
Rerunning the pipeline would not continually ingest the same file.
3. Simulating streaming without running 24×7
Because this was a learning project, I did not want to keep cloud compute running continuously.
Instead, I created controlled streaming sessions.
The ADF pipeline was configured to collect approximately three consecutive GDELT files.
Conceptually:
File 1
↓
Wait
↓
File 2
↓
Wait
↓
File 3
↓
Stop
The session used:
This allowed me to learn near-real-time ingestion while keeping infrastructure costs under control.
4. Connecting ADLS securely to Databricks
Instead of using storage keys or legacy DBFS mounts, I connected ADLS using:
Databricks Access Connector
↓
Managed Identity
↓
Unity Catalog Storage Credential
↓
External Location
↓
External VolumeThis exposed the ADLS landing data through a governed path such as:
/Volumes/gdelt_dev/raw/gdelt_landing/
One of the most useful concepts I learned here was that /Volumes/... is a Databricks governed view over the underlying cloud storage.
The physical data still lives in ADLS.
5. Streaming ADLS files into Bronze using Auto Loader
The next layer was Databricks Auto Loader.
ADLS landing
↓
Unity Catalog Volume
↓
cloudFiles
↓
Structured Streaming
↓
Bronze DeltaI deliberately kept Bronze close to the source.
Instead of doing all business parsing immediately, the Bronze table stored:
raw_record
source_file_name
source_file_path
source_file_size
source_file_modification_time
ingested_at
ingestion_date
Auto Loader used a dedicated checkpoint:
checkpoints/bronze_gkg/
and a separate schema location.
The checkpoint became one of the most important concepts in the project.
It allows the stream to remember:
Which source files have already been successfully processed?
I could stop the Databricks cluster, restart it later and use the same checkpoint.
The stream continued from its previous progress rather than starting from scratch.
6. Bronze to Silver streaming
Once the raw GKG records were reliably landing in Bronze, I created another Structured Streaming pipeline:
Bronze Delta
↓
readStream
↓
GKG parsing
↓
cleaning
↓
Silver DeltaThe GKG records were tab-delimited.
In Silver I parsed the useful fields and produced article-level features such as:
article_id
published_at
article_url
source_domain
title
tone
themes
organizations
persons
locations
The article title was extracted from the GKG Extras field.
Silver also performed operations such as:
Importantly, Silver used a different checkpoint from Bronze.
So the architecture had independent state:
ADLS → Bronze checkpoint
Bronze → Silver checkpoint
At this stage I had a genuine multi-hop streaming architecture:
ADF lands file
↓
Auto Loader detects it
↓
Bronze updates
↓
Silver streaming query detects new Bronze rows
↓
Silver updates7. Preparing the NLP dataset
From Silver I created an NLP-oriented profile dataset containing fields such as:
article_id
published_at
article_url
source_domain
title
title_normalized
tone
themes_clean
organizations
persons
locations
A versioned snapshot was then created for reproducibility.
For example:
article_nlp_profile_v001
I exported this snapshot as Parquet into a dedicated ADLS ML exchange area:
ml_exchange/
└── relevance/
├── datasets/
└── models/This provided a clean boundary between the governed Databricks data platform and the training environment.
8. NLP modelling strategy
I deliberately did not jump directly to BERT.
I structured the modelling as a progression.
Baseline
Title
↓
TF-IDF
↓
Logistic Regression
The purpose of the baseline was not to create the final model.
It established a benchmark.
Any more complicated transformer model needed to demonstrate that the additional complexity provided measurable value.
Deep NLP
The next models used pretrained transformer encoders for supply-chain relevance classification.
The progression included experiments such as:
TF-IDF
↓
DistilBERT
↓
DeBERTa / modern encoder models
The classification target was conceptually:
1 = supply-chain disruption
0 = not a supply-chain disruption
The project was designed to eventually extend beyond binary classification into:
9. Training and Hyperparameter Optimization
An important architectural decision was not to make the expensive training environment the centre of the platform.
Databricks remained the system of record.
Training/HPO could use lower-cost external GPU compute where appropriate.
The pattern became:
Databricks
↓
Versioned training dataset
↓
ADLS ML Exchange
↓
GPU training/HPO
↓
Best model artefact
↓
ADLS
↓
Databricks
For HPO I focused on parameters that materially affect transformer performance, for example:
learning rate
batch size
epochs
weight decay
warmup ratio
max sequence length
Model selection used more than accuracy.
For an early-warning problem, metrics such as these are more useful:
Precision
Recall
F1
PR-AUC
Precision@K
Recall@K
10. Bringing the model back into Databricks
The best model was not left in the training notebook.
The model artefact, configuration and evaluation metadata were returned to Databricks.
The model bundle included information such as:
model
best parameters
validation metrics
dataset version
feature schema
library requirements
training notes
Then Databricks became responsible again for the production lifecycle:
Best model
↓
MLflow
↓
Model evaluation
↓
Unity Catalog Model Registry
↓
Champion model
This separation was intentional.
Training compute could be disposable.
The governed model lifecycle remained in Databricks.
11. Driving streaming predictions
This was the point where the data-engineering and ML parts of the project came together.
The production inference path became:
New GDELT file
↓
ADF
↓
ADLS
↓
Auto Loader
↓
Bronze
↓
Silver
↓
Registered ML model
↓
PredictionFor every newly processed article, the model generates something conceptually similar to:
article_id
prediction_timestamp
model_version
relevance_probability
predicted_label
For example:
Title:
"Port workers announce nationwide strike"
Supply-chain disruption probability:
0.94
Prediction:
Relevant
Only relevant articles need to continue into more expensive downstream NLP:
Relevant article
↓
Disruption category
↓
Entity extraction
↓
Embeddings
↓
Semantic duplicate detection
↓
Story clusteringThis is also useful for cost optimization because expensive NLP is performed only on the subset of records that passes the relevance model.
12. Persisting predictions back into ADLS
Predictions are first written into governed Delta tables.
For example:
gdelt_dev.ml.article_relevance_predictions
or later:
gdelt_dev.gold.active_alerts
The output contains both prediction data and lineage:
article_id
source_file
prediction_timestamp
model_name
model_version
probability
predicted_class
processing_run_id
Where external Azure consumers require the prediction output, the data can then be exported through a Unity Catalog External Volume backed by ADLS:
Databricks Delta prediction
↓
Gold/serving transformation
↓
External Volume
↓
ADLSFor example:
abfss://gdelt@<storage-account>.dfs.core.windows.net/
predictions/relevance/
This completes the round trip:
GDELT
↓
Azure
↓
Databricks
↓
Machine Learning
↓
Databricks predictions
↓
Azure
13. What I learned from this architecture
The biggest learning for me was that an ML project is much larger than model.fit().
I had to think about:
Data engineering
external acquisition
ADF
ADLS Gen2
immutable raw storage
idempotency
replayability
parameterized pipelines
Streaming
Governance
managed identities
Unity Catalog
external locations
Volumes
lineage
NLP
TF-IDF baselines
transformer fine-tuning
contextual embeddings
classification
semantic similarity
Machine learning
labelled datasets
time-aware evaluation
HPO
class imbalance
PR-AUC
threshold selection
MLOps
dataset versioning
MLflow
Model Registry
model versions
production inference
prediction lineage
Cost engineering
I also intentionally avoided running everything 24×7.
The same architectural concepts can be learned using controlled streaming sessions:
start resources
→ process several GDELT intervals
→ observe streaming behaviour
→ validate output
→ gracefully stop streams
→ terminate compute
That allowed me to learn the architecture without turning a personal learning project into an unnecessarily expensive cloud workload.
What comes next
The next stages are focused on making the intelligence layer deeper:
Relevance classification
↓
Disruption-category classification
↓
NER / entity normalization
↓
Sentence embeddings
↓
Semantic duplicate detection
↓
Story clustering
↓
Temporal feature engineering
↓
Predict whether a story will escalate
↓
Ranked supply-chain disruption alertsFor me, the most valuable part of this project has been connecting all the pieces rather than treating data engineering, NLP and MLOps as separate subjects.
ADF acquires the data.
ADLS preserves it.
Databricks streams and governs it.
ML models extract intelligence from it.
MLflow governs the model lifecycle.
And the prediction pipeline turns continuously arriving data into actionable outputs.
#Azure #AzureDataFactory #ADLS #Databricks #ApacheSpark #PySpark #StructuredStreaming #AutoLoader #DeltaLake #UnityCatalog #MLflow #NLP #BERT #MachineLearning #MLOps #DataEngineering #GDELT
