It is Friday afternoon. An upstream team ships a harmless-looking change: they add a region field to the events your service already emits, and somewhere along the way, a sensor starts reporting temp...
Introduction In Part 1 of this series, Integration Testing for LakeFlow Jobs with Pytest and Databricks Connect, we built a blueprint for testing LakeFlow Jobs: deploy the job, trigger it from pytest,...
The Problem: One Too Many Warehouses Your Tableau dashboards hit a cloud data warehouse. Your data engineering team builds ETL pipelines in Databricks. And somewhere in between, you're paying to move,...
If you have ever tried to make a large, complex structured data estate available to AI agents and assistants, you have likely run into the same wall we did: agents are only as good as the contex...
In a modern manufacturing facility, high-resolution cameras and precision sensors continuously inspect products as they move along the assembly line. Every image, measurement, and signal is analyzed w...
TL;DR Until recently, a Spark Declarative Pipeline’s (SDP) jurisdiction ended where the lakehouse ended. You could land, clean, and curate data into governed tables in Unity Catalog, and that was the ...
Does your “open source” architecture actually deliver freedom? For an enterprise, lock-in isn’t a binary choice; it is a spectrum of exit costs that must be weighed against total TCO—specifically the ...
The Lakeflow Spark Declarative Pipelines (SDP) Kafka sink is now generally available, turning your declarative pipeline into a unified engine for both data ingestion and real-time egress. You can now ...
Taking the proven MLOps deploy-code pattern and adapting it for GenAI agents: evaluation-driven CI/CD, a native observability loop, and open OpenTelemetry integration. From MLOps to AgentOps Most data...
Most teams I work with don't struggle with the AI part of MLOps. They struggle with the boring parts: getting code from a notebook into production without breaking it, knowing when a model has gone st...
