Databricks Academy offers the free Automated Deployment with Declarative Automation Bundles course to help data engineers apply DevOps practices to Databricks projects. As the fourth course in the Advanced Data Engineering with Databricks series, it ...
AI-assisted coding is changing how data engineers build on Databricks.A developer can now generate Spark SQL, PySpark transformations, Delta table DDL, pipeline logic, data-quality checks, documentation, and test scenarios much faster than before. W...
Fraud detection becomes more effective when organizations can understand where and how customers normally spend. The Identify Fraud With Geospatial Analytics and AI Solution Accelerator shows how geospatial data, machine learning and a lakehouse arch...
Hi Databricks Community,I wrote an article sharing my thoughts on where data engineering is heading in the AI era.For a long time, data engineering has been about pipelines, transformations, validations, trusted layers, and dashboards. That foundatio...
Databricks Academy offers the free Databricks Performance Optimization course to help data engineers improve workload performance on the Databricks Data Intelligence Platform. As part of the Advanced Data Engineering with Databricks series, it focuse...
Advances in deep learning have made computer vision a practical approach for product quality inspection. The Product Quality Inspection Solution Accelerator shows how organizations can implement an end-to-end computer vision pipeline, build and train...
Real time capabilities in healthcare is a critical factor in care outcomes and operational efficiency. From streaming continuous vitals from IoT-enabled care monitors to routing instantaneous telemetry from wearable medical devices, modern healthcare...
Interview Question:Many people start with the row count.I would start with the architecture.Billions of records are not new in enterprise data engineering. The real challenge is designing a pipeline that runs predictably, efficiently, and within SLA....
Fair point - if it's genuinely append-only with no corrections, you're right, you're already touching the minimum amount of data each run. So, the problem really is what you originally said: how do you process a genuinely huge incoming batch efficien...
I'm currently working with a banking customer migrating from Hive Metastore to Unity Catalog. While planning their catalog layout, one of their platform engineers asked the question that prompted this post: for their schemas and volumes, should th...
Hi Ashwin,Really clean write-up - the schema override demo with the actual DESCRIBE DETAIL output is what makes it click. Most posts on this topic stop at "here's the theory," this one actually shows it happening.Something similar came up on a SAP HA...
Part 5 - the final post in my enterprise data platform series - is up. This one is about what actually happened when we connected BI tools to the Views layer.We had two tools: Tableau for the finance team's analytical workbooks, and Databricks Dashbo...
Here’s What Nobody Tells You.A hands-on look at Genie MCP and DBSQL MCP — what works, what doesn’t, and when to skip MCP entirely.Let me start with something most MCP content skips. MCP servers don’t just expose tools. They expose three things — tool...
For years, enterprise data architecture has followed a familiar pattern.An application writes customer orders, account updates, inventory changes, or transactions into an operational database.Then data engineering takes over.We capture changes throug...
IntroductionDifferent organizations use different names for these artifacts: business requirements, mapping specifications, source-to-target mappings, data contracts, transformation rules, or semantic definitions. The name matters less than the goal:...
Overview Prompted by a customer question, I wanted to see what was possible in terms of MCP integration into Genie Code, in order to try this out I decided to look at Azure Dev Ops, as it's a common workflow to want to see your tickets alongside the ...
Hi Emma how are you? is there any solution to make this agent avaliable for multiple users? the main restriction is that the path will be linked to a single user/service princpal.thanks!
A question came up in the community recently that I thought deserved more than a short answer. The question was around how to build a reliable investigation sequence for slow Spark jobs, specifically when symptoms overlap. A long-running stage with ...
Thanks for this! Very insightful and detailed. Your sequence for diagnosing slow Spark jobs when symptoms overlap is exactly what I needed. Bookmarked this for my team.
| User | Count |
|---|---|
| 85 | |
| 76 | |
| 73 | |
| 60 | |
| 43 |
