Databricks

Welcome to Part 2 of our blog series on the Basics of Databricks Workflows! In Part 1 - Creating your pipeline, we explored the essential building blocks of creating a Databricks Workflow. We explored an example of how a retailer can orchestrate a p...

Introduction Modern data pipelines can be complex, especially when dealing with massive volumes of data from diverse sources. Managing the processing of this data is not too dissimilar to the responsibilities of a conductor in an orchestra, coordina...

Introduction In Part 1 of this blog series, we explored the various types of duplicates, considerations for remediation, and the impacts of unchecked duplicated records on strategic decision-making.  If you want to learn more about this, please read:...

Maintaining Slowly Changing Dimensions (SCD) is a common practice in data warehousing to manage and track changes in your records over time. It enables businesses to make more informed and strategic decisions based on historical patterns and trends. ...