Answer firstA medallion architecture over SAP data is the same three layers everyone draws, with three jobs that are specific to the source. Bronze has to preserve deletes. Silver has to rebuild a business document that lives in five tables. Gold has...
Disclosure: I work at Onibex, the company that builds OneConnect, which is used as the worked example in this post. This is about the engineering approach, not a product pitch.Design of Databricks Pipelines for real-time data transformation to genera...
great explanation!
Part 5 - the final post in my enterprise data platform series - is up. This one is about what actually happened when we connected BI tools to the Views layer.We had two tools: Tableau for the finance team's analytical workbooks, and Databricks Dashbo...
For years, enterprise data architecture has followed a familiar pattern.An application writes customer orders, account updates, inventory changes, or transactions into an operational database.Then data engineering takes over.We capture changes throug...
Part 4 of my enterprise data platform series is up - this one cover why we added a fifth layer to the standard medallion architecture.We connected BI tools to the Gold layer and immediately hit four problems Gold alone couldn't solve:Schema breaks wh...
Part 3 of my series on building an enterprise data platform on Databricks is up - this one cover Gold layer design.The short version: Gold isn't just aggregated Silver. Silver maps to your source system. Gold maps to the business questions your consu...
Building an Incremental Customer Data Migration Workflow in DatabricksBy Naveen AyallaIntroductionIn many enterprise environments, customer data is spread across legacy systems that were originally designed for operational processing rather than mode...
Great write-up, Naveen. Very practical and clear.I really like how you focused not just on migration, but on building a reliable incremental workflow with proper duplicate handling and governance. That’s where real value comes from.Also, connecting D...
