Hi,
For me, data quality takes the most time — specifically the "silent drift" kind, where pipelines run green but produce subtly wrong data. Everything else is visible; that isn't.
What has worked best for me:
Data quality — catch it early
I push expectations to the bronze→silver boundary rather than the gold layer. Catching null-heavy or schema-drifting records right after ingest keeps the failure small and localized. Landing raw JSON in a VARIANT column and flattening explicitly in silver (instead of letting the reader infer types) has also saved me a lot of "why did this column suddenly become a string?" debugging.
Performance — measure before tuning.
We rely on predictive optimization to handle table maintenance automatically (OPTIMIZE/VACUUM/ANALYZE on Unity Catalog managed tables), liquid clustering to incrementally lay out data around the columns our queries actually filter on instead of guessing partition keys, and AQE to re-optimize shuffles, skew, and joins at runtime based on real statistics. The net effect: we spend far less time on maintenance and knob-turning, and more on making sure clustering keys and query patterns actually match. On serverless pipelines, autoscaling behavior still matters more than any manual Spark config — if it isn't scaling out, no knob fixes it.
Pipeline automation — SDP has done a lot of the heavy lifting.
We've been working with Lakeflow Spark Declarative Pipelines (SDP) and it has genuinely reduced the manual orchestration work — declaring what the data should look like instead of hand-writing the processing steps means fewer pipeline-specific quirks to maintain. The built-in expectations also make data quality checks continuous rather than a one-time gate, which pairs well with the bronze→silver approach above.
Monitoring — make the pipeline tell you it's unhealthy.
Threshold-based alerts on pipeline event logs (backlog depth, batch duration drift, input-vs-output row counts) have been more reliable than dashboards I have to remember to check. The row-count ratio is the underrated one: a pipeline can "succeed" while silently dropping records, and that ratio catches it.
On the forward-looking side, I'm really curious about Genie ZeroOps — if it can catch silent failures via Unity Catalog lineage and propose fixes without touching production until approved, that would directly target the "silent drift" problem I spend the most time on. Curious whether anyone in the community has had a chance to try it yet.