Dear Databricks Community,I would like to share a few thoughts on how the role of data platforms may evolve in the agentic AI era. Inspired by Satya Nadella’s recent article on the “Reverse Information Paradox,” I started thinking about a deeper ques...
I came across this checklist for hiring a Databricks implementation partner, and it turned out to be the best in-house build checklist I've read in a while. 27 Questions to Choose a Databricks Implementation Partner looks like procurement filler at f...
I Built a Custom MCP Server on Databricks — Here’s What the Managed Version Can’t DoPart 2 of the MCP on Databricks seriesIn Part 1, I covered what Genie MCP and DBSQL MCP give you out of the box — and where they stop.The short version: both are stat...
Databricks Academy offers the free Automated Deployment with Declarative Automation Bundles course to help data engineers apply DevOps practices to Databricks projects. As the fourth course in the Advanced Data Engineering with Databricks series, it ...
AI-assisted coding is changing how data engineers build on Databricks.A developer can now generate Spark SQL, PySpark transformations, Delta table DDL, pipeline logic, data-quality checks, documentation, and test scenarios much faster than before. W...
Fraud detection becomes more effective when organizations can understand where and how customers normally spend. The Identify Fraud With Geospatial Analytics and AI Solution Accelerator shows how geospatial data, machine learning and a lakehouse arch...
Hi Databricks Community,I wrote an article sharing my thoughts on where data engineering is heading in the AI era.For a long time, data engineering has been about pipelines, transformations, validations, trusted layers, and dashboards. That foundatio...
Databricks Academy offers the free Databricks Performance Optimization course to help data engineers improve workload performance on the Databricks Data Intelligence Platform. As part of the Advanced Data Engineering with Databricks series, it focuse...
Advances in deep learning have made computer vision a practical approach for product quality inspection. The Product Quality Inspection Solution Accelerator shows how organizations can implement an end-to-end computer vision pipeline, build and train...
Real time capabilities in healthcare is a critical factor in care outcomes and operational efficiency. From streaming continuous vitals from IoT-enabled care monitors to routing instantaneous telemetry from wearable medical devices, modern healthcare...
Interview Question:Many people start with the row count.I would start with the architecture.Billions of records are not new in enterprise data engineering. The real challenge is designing a pipeline that runs predictably, efficiently, and within SLA....
Fair point - if it's genuinely append-only with no corrections, you're right, you're already touching the minimum amount of data each run. So, the problem really is what you originally said: how do you process a genuinely huge incoming batch efficien...
I'm currently working with a banking customer migrating from Hive Metastore to Unity Catalog. While planning their catalog layout, one of their platform engineers asked the question that prompted this post: for their schemas and volumes, should th...
Hi Ashwin,Really clean write-up - the schema override demo with the actual DESCRIBE DETAIL output is what makes it click. Most posts on this topic stop at "here's the theory," this one actually shows it happening.Something similar came up on a SAP HA...
Part 5 - the final post in my enterprise data platform series - is up. This one is about what actually happened when we connected BI tools to the Views layer.We had two tools: Tableau for the finance team's analytical workbooks, and Databricks Dashbo...
Here’s What Nobody Tells You.A hands-on look at Genie MCP and DBSQL MCP — what works, what doesn’t, and when to skip MCP entirely.Let me start with something most MCP content skips. MCP servers don’t just expose tools. They expose three things — tool...
For years, enterprise data architecture has followed a familiar pattern.An application writes customer orders, account updates, inventory changes, or transactions into an operational database.Then data engineering takes over.We capture changes throug...
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 72 | |
| 60 | |
| 43 |
