Why Your Delta Lake Tables Are Quietly Ballooning (And How to Fix It)If your data pipeline only appends a few gigabytes a day, but your cloud storage footprint is skyrocketing into hundreds of gigabytes, you aren’t alone. We recently watched one of o...
Most Databricks deployment pipelines on Azure still authenticate with a service principal client secret. There is a better way and it does not require managing a single credential. The standard pattern has a quiet problemIf you have set up Databricks...
Hey everyone!I've built and open-sourced ar-io-mlfow. This is a plugin that adds cryptographic provenance across the ML lifecycle (training runs, model registration, stage promotions, inference, and datasets).What it doesCreates signed Ed25519 crypto...
I also opened a discussion on Github, not sure which is the right place sorry if this isn't it - https://github.com/mlflow/mlflow/discussions/23355
In today’s data-driven world, the role of a data engineer is critical in designing and maintaining the infrastructure that allows for the efficient collection, storage, and analysis of large volumes of data. Databricks certifications holds significan...
Hello , can u hep me with study material for associate level exam, where do we get it? the free version if possible? any noteS?
Hi everyone — wanted to share a three-part series I recently published on Medium that examines architectural patterns from a real Databricks-based data consolidation project.The specific case is a logistics platform unifying two legacy systems into a...
This article wraps up a technical deep dive into building large-scale Lakehouse architectures, revisiting design decisions from a 2019 platform that processed billions of payment records.In the original platform, streaming pipelines ran on Spark Stre...
For a long time, one of the hardest questions in lakehouse architecture was:How do we let external engines access governed data without bypassing governance?Databricks is making this pattern much cleaner with Unity Catalog external access.The idea is...
I came across a blog post comparing Databricks and Google BigQuery for AI-ready data teams. The workload angle stood out.That feels like a useful way to frame the discussion here in the Databricks Community. A lot of platform questions come back to t...
Evaluating pure analytics capabilities is an outdated framework that treats the data warehouse as an isolated silo. Databricks is aggressively moving to handle the entire enterprise footprint including BI & Agentic universe. With the maturity of Data...
One of the more frustrating things when working with materialized views in Databricks was checking whether a view had refreshed incrementally. One way to verify it was by checking the event log, but that required running the pipeline and executing a ...
Hi. This is very helpful. Any idea whether incremental refresh ability is also true for non-algebraic functions like median etc. I was looking for a solution which will work for late arriving data and came across this. I also could not find any docum...
If your CI/CD pipelines suddenly started failing out of nowhere with this error:"error downloading Terraform: unable to verify checksums signature: openpgp: key expired"and you’re using Databricks CLI - you’re probably hitting the same issue I did.Th...
Une nouvelle primitive de permissions pour empêcher le croisement de données entre usages hébergés sur le même workspace Databricks.
lien Medium :https://medium.com/@kacn12872/azure-databricks-exclusive-groups-garantir-létanchéité-entre-cas-d-usage-sur-la-lakehouse-e340ce28f332
Hi everyone, I'm continuing to build a live cost estimator for Databricks to get immediate cost estimates every second instead of having to wait for the system tables to update. (see Live Cost Estimator - Databricks Community - 156374)I've finished t...
The Real Problem: Kafka Source Parallelism in SparkBefore discussing foreachBatch, multi-table writes, or any specific use case, it helps to understand the underlying issue. This is a problem with how Spark Structured Streaming consumes from Kafka, a...
How to handle MERGE with Schema Evolution in Delta LakeHi everyone,Schema evolution during MERGE is one of the trickiest parts of building robust Delta Lake pipelines. Databricks actually has a native SQL syntax for this — plus Python API options for...
Great post. Would also like to consider the following points:Guardrails: schema evolution is powerful — it can also accidentally add garbage columns if upstream sends unexpected fields.Recommendation: validate/allowlist schema changes in higher envir...
Most organizations don’t have a data problem anymore.They have a data access and usability problem.The dashboards exist. The warehouses are modernized. The lakehouse is running. Yet business teams still wait days for answers because analytics remains...
The Dashboard Era Is Ending. Conversation Is Replacing It.
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 69 | |
| 63 | |
| 44 |
