- 4284 Views
- 1 replies
- 1 kudos
Handling the Chaos: Data Quality Strategies with PySpark Ingestion
Tips and Techniques for Ingesting Large JSON files with PySparkIntroductionSuppose you’ve ever struggled or grappled with consuming massive JSON files with PySpark. In that case, you are aware that insufficient data can always creep in and silently d...
- 4284 Views
- 1 replies
- 1 kudos
- 1 kudos
Maintaining data quality with PySpark is essential for reliable analytics, and combining validation checks with consistent monitoring can help catch issues before they affect downstream workflows. Corn dogs, quesadillas, wraps, and burgers all share ...
- 1 kudos
- 546 Views
- 0 replies
- 0 kudos
Showcase: Production Lakehouse Pipeline for Retail Pricing Analytics on Databricks (Plaza Vea, Peru)
Hi Databricks Community,I wanted to share a project I've been building over the past few months using Databricks on Azure , a production Lakehouse pipeline for retail pricing analytics.What it doesIngests ~21,000 products daily across 16 categories f...
- 546 Views
- 0 replies
- 0 kudos
- 512 Views
- 0 replies
- 0 kudos
Keep a PySpark Test Flood Out of Your Coding Agent's Context Window
Originally published at vmariiechko.com.This started with a broken Spark fixture on an ordinary day. Every test that needed Spark failed on the same setup error, and my coding agent had to read all of it before it could fix a thing. Reproduced and me...
- 512 Views
- 0 replies
- 0 kudos
- 1194 Views
- 0 replies
- 0 kudos
From Business Requirements to Lakeflow Pipelines: A Governed Metadata-Driven Delivery Pattern
IntroductionDifferent organizations use different names for these artifacts: business requirements, mapping specifications, source-to-target mappings, data contracts, transformation rules, or semantic definitions. The name matters less than the goal:...
- 1194 Views
- 0 replies
- 0 kudos
- 769 Views
- 0 replies
- 0 kudos
Gold Layer Design on Databricks — MERGE vs Overwrite, Partitioning, SCD Type 2 from SAP
Part 3 of my series on building an enterprise data platform on Databricks is up - this one cover Gold layer design.The short version: Gold isn't just aggregated Silver. Silver maps to your source system. Gold maps to the business questions your consu...
- 769 Views
- 0 replies
- 0 kudos
- 626 Views
- 0 replies
- 0 kudos
The Hardest Part of Our SAP Migration Wasn't the Data. It Was Timing
Part 2 of my series on building an enterprise data platform on Databricks — this one's about Silver.Part 1 covered why we ran two ingestion paths in parallel (GoldenGate CDC + JDBC batch) and kept them as separate bronze tables. If you missed it:http...
- 626 Views
- 0 replies
- 0 kudos
- 505 Views
- 0 replies
- 1 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Part 1 of a 5-part series on building an enterprise data platform on Databricks.When migrating a large retail conglomerate's SAP HANA platform to Databricks, we needed both historicalcompleteness and near-real-time freshness from day one.That require...
- 505 Views
- 0 replies
- 1 kudos
- 1217 Views
- 0 replies
- 1 kudos
SQL Scripting in Apache Spark™ 4.0
The Apache Spark™ 4.0 introduces a new feature for SQL developers and data engineers: SQL Scripting. As such, this feature enhances the power and extends the flexibility of Spark SQL, enabling users to write procedural code within SQL queries, with t...
- 1217 Views
- 0 replies
- 1 kudos