Hi Databricks Community,I wrote an article sharing my thoughts on where data engineering is heading in the AI era.For a long time, data engineering has been about pipelines, transformations, validations, trusted layers, and dashboards. That foundatio...
Databricks Academy offers the free Databricks Performance Optimization course to help data engineers improve workload performance on the Databricks Data Intelligence Platform. As part of the Advanced Data Engineering with Databricks series, it focuse...
Advances in deep learning have made computer vision a practical approach for product quality inspection. The Product Quality Inspection Solution Accelerator shows how organizations can implement an end-to-end computer vision pipeline, build and train...
Real time capabilities in healthcare is a critical factor in care outcomes and operational efficiency. From streaming continuous vitals from IoT-enabled care monitors to routing instantaneous telemetry from wearable medical devices, modern healthcare...
Interview Question:Many people start with the row count.I would start with the architecture.Billions of records are not new in enterprise data engineering. The real challenge is designing a pipeline that runs predictably, efficiently, and within SLA....
Fair point - if it's genuinely append-only with no corrections, you're right, you're already touching the minimum amount of data each run. So, the problem really is what you originally said: how do you process a genuinely huge incoming batch efficien...
I'm currently working with a banking customer migrating from Hive Metastore to Unity Catalog. While planning their catalog layout, one of their platform engineers asked the question that prompted this post: for their schemas and volumes, should th...
Hi Ashwin,Really clean write-up - the schema override demo with the actual DESCRIBE DETAIL output is what makes it click. Most posts on this topic stop at "here's the theory," this one actually shows it happening.Something similar came up on a SAP HA...
Part 5 - the final post in my enterprise data platform series - is up. This one is about what actually happened when we connected BI tools to the Views layer.We had two tools: Tableau for the finance team's analytical workbooks, and Databricks Dashbo...
I just earned my Databricks Certified Generative AI Engineer Associate Certification, and in this post, I’m sharing the key tips, resources, and including what confused me, what actually worked, and the traps I nearly fell into. Why I Took This ExamI...
Here’s What Nobody Tells You.A hands-on look at Genie MCP and DBSQL MCP — what works, what doesn’t, and when to skip MCP entirely.Let me start with something most MCP content skips. MCP servers don’t just expose tools. They expose three things — tool...
For years, enterprise data architecture has followed a familiar pattern.An application writes customer orders, account updates, inventory changes, or transactions into an operational database.Then data engineering takes over.We capture changes throug...
IntroductionDifferent organizations use different names for these artifacts: business requirements, mapping specifications, source-to-target mappings, data contracts, transformation rules, or semantic definitions. The name matters less than the goal:...
Overview Prompted by a customer question, I wanted to see what was possible in terms of MCP integration into Genie Code, in order to try this out I decided to look at Azure Dev Ops, as it's a common workflow to want to see your tickets alongside the ...
Hi Emma how are you? is there any solution to make this agent avaliable for multiple users? the main restriction is that the path will be linked to a single user/service princpal.thanks!
A question came up in the community recently that I thought deserved more than a short answer. The question was around how to build a reliable investigation sequence for slow Spark jobs, specifically when symptoms overlap. A long-running stage with ...
Thanks for this! Very insightful and detailed. Your sequence for diagnosing slow Spark jobs when symptoms overlap is exactly what I needed. Bookmarked this for my team.
One Question Can Light the SparkMy Databricks journey started in 2022 with simple interest, curiosity, and a dream to learn more.At that time, I was just trying to understand the platform, follow the updates, learn from others, and slowly build my co...
My turn! (couldn't let this disappear from the homepage that quickly ).This one is officially going down in Community history now, @Brahmareddy .
If you've ever worked on an insurance data warehouse, or really any warehouse where data arrives from different systems at different times, you know the pain of keeping things in sync. I spent years building data warehouses for a property and casual...
I know I'm reading this a bit late, but it's absolutely worth the read. Excellent write-up!Please share link for part-2
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 68 | |
| 63 | |
| 44 |
