- 3122 Views
- 1 replies
- 4 kudos
From SSIS to Databricks: Accelerating ETL Modernization with AI-Powered Utility
As enterprises race toward cloud-native data platforms, modernising legacy ETL pipelines remains one of the most persistent bottlenecks. For organizations that have relied on SQL Server Integration Services (SSIS) for years, rewriting hundreds of pac...
- 3122 Views
- 1 replies
- 4 kudos
- 414 Views
- 0 replies
- 2 kudos
From Data Quality to Context Quality
High Quality Data is no longer enough for AI based decisions. Governance, Lineage and Observability helped creation of Trusted Data Products to allow consumers to use the trusted information delivered by the Data Platforms.Generative AI and Agentic A...
- 414 Views
- 0 replies
- 2 kudos
- 557 Views
- 0 replies
- 2 kudos
Learn Databricks Apache Spark Declarative Pipelines | From Associate to Professional
Databricks Training & Certifications Learn Databricks Apache Spark Declarative Pipelines Declare what you want. Let the engine build and run the pipeline for you. Declarative ETL ◆ Streaming tables & MVs ✓ Data quality With Apache Spark™ Decl...
- 557 Views
- 0 replies
- 2 kudos
- 362 Views
- 0 replies
- 2 kudos
WordPress to Databricks: Building a Practical Data Pipeline
WordPress is often used as the front end for publishing, ecommerce, forms, memberships, and other online activities. Behind that website, useful information can accumulate quickly. Posts, users, orders, comments, form submissions, and other records c...
- 362 Views
- 0 replies
- 2 kudos
- 871 Views
- 1 replies
- 2 kudos
Preventing Duplicate Records When Reprocessing Data in Databricks
A pipeline can finish successfully and still produce the wrong result after a retry. Imagine an orders load that writes its data, then fails during a later task. Repeating the load with an append can add the same orders again. Replacing existing rows...
- 871 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Islam_hoti ,The Core Pattern: Identity + OrderingThe article argues that idempotency is not achieved by simple "Appends." Instead, it requires two specific definitions:Identity: A business key (e.g., order_id) that tells you which entity the reco...
- 2 kudos
- 382 Views
- 0 replies
- 0 kudos
Deletion Vectors Don’t Delete Data the Way You Think They Do: A Deep Dive into Delta Lake Maintenanc
Hi everyone! In my previous post, I discussed how enabling Deletion Vectors helped reduce our Delta MERGE runtime from 22 minutes down to 6 minutes by eliminating write amplification.However, deferring file rewrites introduces an important architectu...
- 382 Views
- 0 replies
- 0 kudos
- 515 Views
- 0 replies
- 0 kudos
Lakehouse vs data warehouse vs data lake, drawn step by step: the one-line interview answer
A follow-up to my Databricks architecture post. The next question I kept getting was "what actually is a lakehouse, and why not just use a warehouse?", so I drew that one too.The crisp answer: a data warehouse gives you ACID transactions, enforced sc...
- 515 Views
- 0 replies
- 0 kudos
- 418 Views
- 0 replies
- 0 kudos
Databricks SQL 2026: From SQL Analytics to an AI-Ready Data Platform
atabricks SQL 2026: From SQL Analytics to an AI-Ready Data PlatformThe Databricks SQL 2026 release cycle is not just about adding more SQL functions.It reflects a much bigger shift:Databricks SQL is evolving from a query and reporting layer into a fo...
- 418 Views
- 0 replies
- 0 kudos
- 506 Views
- 0 replies
- 0 kudos
A Metadata-Driven Backdated Extraction Pattern for IoT/Sensor Pipelines Using Zerobus and Databricks
IntroductionIoT devices generate a large amount of data every day. In a perfect scenario, all this data arrives on time and our ingestion pipeline processes it using incremental loads. But in real-world IoT systems, this does not always happen.A devi...
- 506 Views
- 0 replies
- 0 kudos
- 574 Views
- 0 replies
- 2 kudos
Solution Accelerator Series | Customer Entity Resolution
Building a customer 360 requires connecting customer records across disparate data sets and establishing common customer identities. The Customer Entity Resolution Solution Accelerator shows how to build that foundation by translating customer attrib...
- 574 Views
- 0 replies
- 2 kudos
- 777 Views
- 0 replies
- 1 kudos
SAP Medallion Architecture Patterns: What Bronze, Silver and Gold Have to Do Differently
Answer firstA medallion architecture over SAP data is the same three layers everyone draws, with three jobs that are specific to the source. Bronze has to preserve deletes. Silver has to rebuild a business document that lives in five tables. Gold has...
- 777 Views
- 0 replies
- 1 kudos
- 472 Views
- 0 replies
- 0 kudos
The Missing Correlation Layer in Databricks ModelOps | Building an AI Health Control Plane for DBx
SummaryThis post walks through the architecture of The Third Eye, a continuous AI health and governed ModelOps control plane built entirely on native Databricks capabilities. The core constraint: it never recomputes a metric Databricks already comput...
- 472 Views
- 0 replies
- 0 kudos
- 457 Views
- 0 replies
- 1 kudos
Task Success is not Routing Health: a Databricks Smart Routing Case Study
I ran the same 24 coding tasks through three arms: fixed Haiku, fixed Opus and Databricks-managed selection. The managed-selection arm passed all 24 tasks. Its 24 routing records also said task profiling was unavailable.That is not a contradiction. T...
- 457 Views
- 0 replies
- 1 kudos
- 319 Views
- 0 replies
- 0 kudos
Building for Failure: Implementing Data Quality Firewalls in Petabyte-Scale Medallion Architectures
In my 13 years of architecting data platforms—from Retail to Industrial Gas Power—I’ve learned one universal truth: A fast pipeline that delivers bad data is just a liability.As we move toward Declarative Pipelines (DLT), the role of the Architect sh...
- 319 Views
- 0 replies
- 0 kudos
- 633 Views
- 0 replies
- 1 kudos
Where Omnigent Fits Alongside ChatGPT Work, Claude Cowork and Cursor Projects
If you use ChatGPT Work, Claude Cowork or Cursor Projects, much of Omnigent will look familiar.All of these products now organize work around projects, files and persistent context. They can run tasks for longer periods, connect to external tools and...
- 633 Views
- 0 replies
- 1 kudos