- 4300 Views
- 1 replies
- 1 kudos
Handling the Chaos: Data Quality Strategies with PySpark Ingestion
Tips and Techniques for Ingesting Large JSON files with PySparkIntroductionSuppose you’ve ever struggled or grappled with consuming massive JSON files with PySpark. In that case, you are aware that insufficient data can always creep in and silently d...
- 4300 Views
- 1 replies
- 1 kudos
- 1 kudos
Maintaining data quality with PySpark is essential for reliable analytics, and combining validation checks with consistent monitoring can help catch issues before they affect downstream workflows. Corn dogs, quesadillas, wraps, and burgers all share ...
- 1 kudos
- 562 Views
- 0 replies
- 0 kudos
Showcase: Production Lakehouse Pipeline for Retail Pricing Analytics on Databricks (Plaza Vea, Peru)
Hi Databricks Community,I wanted to share a project I've been building over the past few months using Databricks on Azure , a production Lakehouse pipeline for retail pricing analytics.What it doesIngests ~21,000 products daily across 16 categories f...
- 562 Views
- 0 replies
- 0 kudos
- 520 Views
- 0 replies
- 0 kudos
Keep a PySpark Test Flood Out of Your Coding Agent's Context Window
Originally published at vmariiechko.com.This started with a broken Spark fixture on an ordinary day. Every test that needed Spark failed on the same setup error, and my coding agent had to read all of it before it could fix a thing. Reproduced and me...
- 520 Views
- 0 replies
- 0 kudos
- 1204 Views
- 0 replies
- 0 kudos
From Business Requirements to Lakeflow Pipelines: A Governed Metadata-Driven Delivery Pattern
IntroductionDifferent organizations use different names for these artifacts: business requirements, mapping specifications, source-to-target mappings, data contracts, transformation rules, or semantic definitions. The name matters less than the goal:...
- 1204 Views
- 0 replies
- 0 kudos
- 776 Views
- 0 replies
- 0 kudos
Gold Layer Design on Databricks — MERGE vs Overwrite, Partitioning, SCD Type 2 from SAP
Part 3 of my series on building an enterprise data platform on Databricks is up - this one cover Gold layer design.The short version: Gold isn't just aggregated Silver. Silver maps to your source system. Gold maps to the business questions your consu...
- 776 Views
- 0 replies
- 0 kudos
- 632 Views
- 0 replies
- 0 kudos
The Hardest Part of Our SAP Migration Wasn't the Data. It Was Timing
Part 2 of my series on building an enterprise data platform on Databricks — this one's about Silver.Part 1 covered why we ran two ingestion paths in parallel (GoldenGate CDC + JDBC batch) and kept them as separate bronze tables. If you missed it:http...
- 632 Views
- 0 replies
- 0 kudos
- 512 Views
- 0 replies
- 1 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Part 1 of a 5-part series on building an enterprise data platform on Databricks.When migrating a large retail conglomerate's SAP HANA platform to Databricks, we needed both historicalcompleteness and near-real-time freshness from day one.That require...
- 512 Views
- 0 replies
- 1 kudos
- 1225 Views
- 0 replies
- 1 kudos
SQL Scripting in Apache Sparkâ„¢ 4.0
The Apache Sparkâ„¢ 4.0 introduces a new feature for SQL developers and data engineers: SQL Scripting. As such, this feature enhances the power and extends the flexibility of Spark SQL, enabling users to write procedural code within SQL queries, with t...
- 1225 Views
- 0 replies
- 1 kudos