Data platforms are getting smarter — but are we asking the right questions about what that means for data engineering?I wrote about how Databricks Predictive Optimization is shifting the role of data engineers from reactive maintenance to autonomous ...
A practical guide from 12 years of building Azure data platforms.If you've spent any time working on Azure data platforms, you've probably hit this question at some point — should I build this in Azure Data Factory or just do it in Databricks?I've be...
Hi everyone,I recently published a technical blog on Z-Ordering vs Liquid Clustering in Delta Lake, covering the internals of both techniques in detail.Rather than focusing only on the syntax, the blog goes deeper into: The origin of Z-Ordering — tra...
Honestly, I should have paid attention to this sooner.You know that feeling when you build a dashboard, you're proud of it, and then five minutes into the meeting someone says "can you just add one more thing?" And then another thing. And another. Un...
Hello Everyone! I've been spending a lot of time lately thinking about something that keeps coming up in almost every GenAI project I touch — how do you actually know if your model is working well? Not just in demos, but in production, day after day....
Hello Everyone, I have jotted down some of the points on getting started with model serving. What is it about?, How we used to do things earlier? And why to use model serving?This blog focused on the fundamentals of Databricks Model Serving and its r...
This post is more focused for Databricks Platform Admin setting up for GCP Cloud Platform https://medium.com/@mogaxer871/from-zero-to-governed-setting-up-databricks-on-gcp-the-right-way-341d4b5afac1
Sharing my hands-on experience with Lakeflow Connect for anyone evaluating it for database ingestion. I recently moved data from PostgreSQL on AWS RDS into Databricks, and it replaced a painful legacy pipeline. Keeping this simple and practical.What ...
Great article!
Part 2 of my series on building an enterprise data platform on Databricks — this one's about Silver.Part 1 covered why we ran two ingestion paths in parallel (GoldenGate CDC + JDBC batch) and kept them as separate bronze tables. If you missed it:http...
One of my learning from a project.After migrating from an on-premises environment to the cloud, the data engineering team began noticing seemingly random failures in workflow-scheduled Databricks jobs.The failures appeared intermittent and often succ...
Spent some time on a Snowflake → Databricks federation issue that didn’t throw a single error -  which is exactly what made it tricky. Sharing it in case it saves someone a debugging session.The setupConnection to Snowflake: established. Foreign cata...
From RAG Demo to Production on Databricks: 7 Things Teams Should Validate FirstBy Naveen AyallaMany teams can build a RAG demo quickly.Upload documents, create embeddings, connect a model, ask a question, and show an answer.But production is differen...
Thanks for reading. I’m especially interested in hearing from people who have worked on real RAG or GenAI workflows.Which one has been the biggest challenge for your team?1. Choosing the right source data2. Access control and governance3. Improving r...
Body:Every day, data platforms generate thousands of audit events. But here's the problem: security teams are drowning in noise.Critical risks hide in plain sight. Manual investigations take hours. Compliance gaps surface too late. And there's no int...
Hi @WiliamRosa ,Good job! Really interesting use case. Thanks for sharing, and good luck in the contest!
Building an Incremental Customer Data Migration Workflow in DatabricksBy Naveen AyallaIntroductionIn many enterprise environments, customer data is spread across legacy systems that were originally designed for operational processing rather than mode...
Great write-up, Naveen. Very practical and clear.I really like how you focused not just on migration, but on building a reliable incremental workflow with proper duplicate handling and governance. That’s where real value comes from.Also, connecting D...
How to extract Tableau calculated fields, dimensions, and measures from a .twbx workbook and re-express them as a production-grade Databricks Metric View YAML — with the Sample Superstore dataset as a complete worked example, accelerated by AI coding...
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 72 | |
| 60 | |
| 43 |
