End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks

[kartheek_rao]

I have been working on a project to understand Databricks end to end, rather than just loading some data and training a model.

I picked GDELT news data and the use case is to identify supply chain disruption related news and eventually predict which events could escalate.

I started by using ADF to pull GDELT GKG files and store the original ZIP files in ADLS.

Initially I did this with a single hardcoded file. Later I parameterized the pipeline, started reading the latest GDELT file information dynamically and added checks so the same file doesn't get ingested again.

One thing I intentionally did was to keep ADF mostly for data movement, instead of doing transformations there.

Once the files were available in ADLS, I connected the storage with Databricks using Managed Identity + Access Connector + Unity Catalog external locations/volumes.

Then came the part which I really wanted to learn — Auto Loader and Structured Streaming.

ADLS landing files are streamed into a Bronze Delta table. Bronze is intentionally almost raw and also stores things like source filename, ingestion timestamp and source path.

I created a separate checkpoint location for this stream.

This was probably one of the most useful things I learnt because earlier checkpoint was just another Spark term for me. After actually stopping the stream, starting it again and seeing that previously processed files were not processed again, it made much more sense.

From Bronze I created another stream going into Silver.

Here I parse the GDELT records and create proper article level fields like title, published time, URL, domain, themes, organisations, locations, tone etc.

So now if a new file arrives in ADLS, it can move through:

Auto Loader → Bronze → Silver incrementally.

For the ML side, I created a versioned dataset from Silver.

I am starting with TF-IDF + Logistic Regression as a baseline, but I don't want the NLP part to stop there. The plan is to compare it with transformer models like DistilBERT/DeBERTa, embeddings, NER and eventually story clustering.

For expensive training/HPO I am also experimenting with cheaper GPU compute outside Databricks instead of unnecessarily keeping Databricks compute running for hours.

But the model comes back to Databricks.

That is where I want to use MLflow, Unity Catalog Model Registry, model versions and production inference.

The final goal is that whenever a new GDELT file comes:

new article → Bronze → Silver → registered model → prediction.

Predictions will be stored in Delta/Gold tables and also written back to an ADLS backed location so other systems outside Databricks can consume them.

Still a lot left to build, specially around embeddings, clustering and escalation prediction, but this project has already changed how I look at ML projects.

Earlier I mostly thought:

data → model → prediction

Now I'm thinking more about ingestion, checkpoints, replayability, governance, model lifecycle and how the prediction actually runs when new data comes in.

Would love to hear from people working with Databricks — anything you would design differently in this architecture?

#Databricks #Azure #ADF #ADLS #PySpark #StructuredStreaming #AutoLoader #DeltaLake #UnityCatalog #MLflow #NLP #MLOps #GDELT

[rdokala]

This is great, how did you design your Bronze and Silver? does the source always send only incremental data?

[krishgarikipati]

Thanks for the overview. Since you are using auto loader for files ingestion,Can we say is this pipeline brings real time data or near real time data feed to the ML model?

[ThiamLee]

Really like this approach. The shift from just “data → model” to thinking about ingestion, replayability, and model lifecycle is what makes this feel like a real-world ML project.

[kunduruanil]

@kartheek_rao, you are on the right track.

Since you are doing clustering of new articles, it's unsupervised learning; you need to understand the feature engineering part more and the EDA part with MLFlow experiments. 
You have model monitoring as well, which you can explore more, and model explainability as well. It's LLM age now; since you are starting with machine learning its fine, if you are using LLM model you dont need all above.