rss.livelink.posts-in-node

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

bifadden — Wed, 16 Sep 2026 23:56:09 GMT

Hello, i just completed a learning path but i didn't receive any voucher (when i connect to the webassessor i don't find any either)

Deletion Vectors Don’t Delete Data the Way You Think They Do: A Deep Dive into Delta Lake Maintenanc

Avinash_Narala — Wed, 16 Sep 2026 21:49:45 GMT

Hi everyone! 👋

In my previous post, I discussed how enabling Deletion Vectors helped reduce our Delta MERGE runtime from 22 minutes down to 6 minutes by eliminating write amplification.

However, deferring file rewrites introduces an important architectural consideration: Logical deletion is not physical deletion.

While Deletion Vectors mark rows as deleted logically, the underlying Parquet files still contain those rows physically. To maintain an efficient Lakehouse, it is critical to understand how different maintenance operations interact:

Deletion Vectors: Soft-delete rows logically to avoid immediate file rewrites during write operations.
REORG TABLE ... APPLY (PURGE): Physically rewrites files to apply deletion-vector changes and remove deleted rows.
VACUUM: Removes old, unreferenced files that fall outside the table retention window.

I wrote a complete breakdown covering why running PURGE immediately after every MERGE defeats the purpose of Deletion Vectors and how to structure your maintenance routines effectively.

You can read the full deep dive here:

🔗 Deletion Vectors Don’t Delete The Data The Way You Think They Do

How are you currently managing table maintenance and REORG PURGE schedules on your high-frequency MERGE workloads? Would love to hear how other teams handle this balance!

Lakehouse vs data warehouse vs data lake, drawn step by step: the one-line interview answer

techinlayers — Wed, 16 Sep 2026 20:30:44 GMT

A follow-up to my Databricks architecture post. The next question I kept getting was "what actually is a lakehouse, and why not just use a warehouse?", so I drew that one too.

The crisp answer: a data warehouse gives you ACID transactions, enforced schemas and fast BI, but storage is proprietary and it struggles with raw files and machine learning. A data lake is cheap, open object storage that takes any data type, but with no transactions it slowly turns into a swamp. So most teams ran both and kept two copies in sync. A lakehouse puts an open table format like Delta Lake on the lake, so you get transactions, schema enforcement, governance and fast queries directly on cheap open storage: one copy of the data serving BI and machine learning.

Full animated walkthrough (under 3 min): https://youtu.be/-c09yVpUtAs

The whole series in order, starting from episode 1 (a new episode every day): https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4

It is episode 2 of the same series, checked against the current docs. How do you explain the lakehouse in interviews? Would you add anything, like Iceberg or UniForm?

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

juanlozadab — Wed, 16 Sep 2026 17:46:27 GMT

This is great, looking foward to it!

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

sbussa0402 — Wed, 16 Sep 2026 17:21:37 GMT

Do we have any link to join the session or we need to go through self video links. I dont see any links as of now.

Announcement | Improving Lakebase Postgres compute cache

Tushar_Parekar — Wed, 16 Sep 2026 17:16:17 GMT

Lakebase Postgres separates compute from durable storage, making efficient compute-side caching essential for high-throughput, low-latency workloads. In this update, we share how larger shared buffers and huge pages help keep more hot data in DRAM, while reducing pressure on the storage layer.

Key highlights

A more efficient cache path: Traditional Postgres can double-buffer data in shared buffers and the operating system page cache. Lakebase uses a local file cache alongside Postgres shared buffers, while the durable storage layer remains authoritative.
Larger shared buffers for fixed-size computes: For fixed-size computes with 80 or more CUs, Lakebase now disables the local file cache and sizes shared buffers to 75% of available DRAM. This keeps more frequently accessed pages in the lowest-latency cache tier.
Huge pages reduce memory overhead: Large shared buffers can create significant page-table and translation overhead across Postgres processes. Dedicated 2 MB huge pages reduce page-table size and TLB pressure; benchmark testing showed up to roughly 40% lower tail read latency and up to 30% lower CPU utilization.
Production results show meaningful gains: Early rollout examples included about 2× higher throughput with 5× fewer storage reads on one endpoint, about 1.3× throughput on another, and a workload that used 5× less CPU while doubling throughput. Results depend on workload and access patterns.
Autoscaling is next: The current improvements target fixed-size computes because shared buffers are not yet dynamic. The next phase is bringing larger shared buffers to autoscaling computes, including dynamically resizing the buffers and the huge-page backing as compute changes size.

👉 Read the full post here

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

jabhiram7 — Wed, 16 Sep 2026 16:21:15 GMT

Hello, I was wondering where I can find course artefacts such as notebooks and queries or repo links ?

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

shivamsanwaria — Wed, 16 Sep 2026 15:16:47 GMT

does these courses provide learning lab also?

please provide me link.

CUSTOMER STORY | Powering Up: EnergyAustralia’s race to real-time

Tushar_Parekar — Wed, 16 Sep 2026 14:58:22 GMT

“Previously, technology was the bottleneck – business teams had to wait on lengthy development cycles and competing IT priorities before solutions reached production. With Databricks, business teams can build what we need ourselves, removing the dependency on specialist technology resources and allowing our business to move at the speed of the market.” - Chirag Shah, Trading Data & Systems Leader, EnergyAustralia

EnergyAustralia operates in one of the world’s fastest-moving wholesale energy markets, where electricity settles every five minutes and prices can shift dramatically with demand and renewable generation. Databricks is helping the company modernize its Trading data foundation so teams can bring together settlement data, battery telemetry and generation inputs, then build, test and deploy pricing and forecasting solutions more quickly.

Key highlights:

6x more settlement data after moving from 30-minute to five-minute settlements.
75% improvement in time to deliver a major initiative.
Daily to hourly pricing-curve refreshes, giving teams more current inputs for commercial decisions.
Business-led model development: More than three-quarters of the wholesale pricing team began contributing to full model development within three months.
Self-service applications and analytics: Teams build SQL, Python, user-facing applications and dashboards with Databricks Apps, Genie and Unity Catalog.
Greater cost transparency: EnergyAustralia can monitor platform usage down to the lab, workload and query, helping teams connect optimization with performance.

🔗 Check out the full story 👈

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

Elizeu_94 — Wed, 16 Sep 2026 14:57:54 GMT

Vamo Dale Brasil

Re: Community BrickTalk | One Platform, Any Source: Unifying Enterprise Data with Lakeflow Connect

Elizeu_94 — Wed, 16 Sep 2026 14:56:40 GMT

I'll be present

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

sujabannu136 — Wed, 16 Sep 2026 14:48:33 GMT

https://community.databricks.com/t5/learning-events/databricks-advanced-learning-festival-september-16-october-14/ec-p/166157#M10191

check this and click on the respective course module(blue highlighted text).

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

StephlK — Wed, 16 Sep 2026 13:48:27 GMT

I cannot find the link to join either.

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

jeffmat211 — Wed, 16 Sep 2026 13:46:04 GMT

Hi, For the Data Analyst Learning Plan there are 2 different ones, with Learning plan ID 'LP-KVO4WV' and 'LP-E045KV'. Which of these should we enroll in or do we enroll and complete both?

May I ask why are there 2 learning plans for the same course?

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

sbussa0402 — Wed, 16 Sep 2026 13:29:43 GMT

whereis link to join

Beyond the Dashboard: How Transferz Built a Truly Data-Driven Company Culture With AI/BI Genie

LPurcell — Wed, 16 Sep 2026 11:59:48 GMT

Discover how Transferz used AI/BI Genie to democratize data, achieving an 89,5% adoption rate amongst business users and 60% productivity increase amongst monthly active users through a 4-phase playbook.

Preventing Duplicate Records When Reprocessing Data in Databricks

Islam_hoti — Wed, 16 Sep 2026 11:56:40 GMT

A pipeline can finish successfully and still produce the wrong result after a retry. Imagine an orders load that writes its data, then fails during a later task. Repeating the load with an append can add the same orders again. Replacing existing rows without checking their version can introduce a different problem: an older delivery can overwrite a newer order status.
In this article, I walk through a small PySpark and Delta Lake example that handles repeated deliveries and older updates. The target stores the latest state of each order. Reprocessing the same input should leave that state unchanged. This property is called idempotency.
Start with the meaning of a duplicate
Consider these incoming records:
|order_id|source_version|status |
|--------|--------------|-------|
|101 |1 |Created|
|101 |2 |Shipped|
|101 |2 |Shipped|
|102 |1 |Created|
The two identical records for order 101 at version 2 are duplicates. Versions 1 and 2 represent different states of the same order. Because the target stores current state, the intended result is one row for order 101 with status Shipped and one row for order 102 with status Created.
For this example, the source must provide a non-null order ID and an increasing version per order. Every event contains the complete order state. A given order ID and version must always describe the same content, including across separate deliveries. Deletes, partial updates, and concurrent writers are outside this example.
Prepare the source before MERGE
Delta Lake MERGE supports inserting new rows and updating existing ones. However, duplicate source rows can still be inserted when their key is absent from the target. Multiple source rows that attempt to update the same target row can also make the operation ambiguous. Prepare one source row per target key before merging. Databricks MERGE documentation
The preparation below removes exact duplicates, rejects conflicting versions within the batch, and selects the highest version for each order. The window assigns row numbers in descending version order. Apache Spark row_number documentation
Processing repeated and older order events
Figure 1. Prepare one candidate per order, then compare it with the target. Equal or older versions leave the target unchanged.
Implement the pattern in PySpark
Run this example in a Databricks Python notebook with Delta Lake support. Select a sandbox catalog and schema where you can create tables. The setup creates a new demonstration table and deliberately stops if that table already exists. To repeat the entire example, choose another unused table name. To test a replay, rerun only the calls to merge_orders.
```python
from delta.tables import DeltaTable
from pyspark.sql import functions as F, Window
TARGET = "replay_safe_orders_demo"
SCHEMA = "order_id LONG, source_version LONG, status STRING"
if spark.catalog.tableExists(TARGET):
raise ValueError("Choose an unused demo table name before setup.")
spark.sql(f"CREATE TABLE {TARGET} ({SCHEMA}) USING DELTA")
def merge_orders(batch_df):
source = batch_df.select("order_id", "source_version", "status")
invalid = source.filter(
F.col("order_id").isNull()
| F.col("source_version").isNull()
| F.col("status").isNull()
)
if invalid.limit(1).count():
raise ValueError("Required source fields cannot be null.")
unique = source.dropDuplicates()
conflicts = (
unique.groupBy("order_id", "source_version")
.count()
.filter(F.col("count") > 1)
)
if conflicts.limit(1).count():
raise ValueError("One order version has conflicting payloads.")
window = Window.partitionBy("order_id").orderBy(
F.col("source_version").desc()
)
latest = (
unique.withColumn("rn", F.row_number().over(window))
.filter(F.col("rn") == 1)
.drop("rn")
)
(
DeltaTable.forName(spark, TARGET).alias("t")
.merge(latest.alias("s"), "t.order_id = s.order_id")
.whenMatchedUpdateAll(
condition="s.source_version > t.source_version"
)
.whenNotMatchedInsertAll()
.execute()
)
batch = spark.createDataFrame([
(101, 1, "Created"),
(101, 2, "Shipped"),
(101, 2, "Shipped"),
(102, 1, "Created"),
], SCHEMA)
merge_orders(batch)
merge_orders(batch) # Replay the same delivery.
older = spark.createDataFrame([(101, 1, "Created")], SCHEMA)
merge_orders(older) # An older version must not replace Shipped.
actual = [tuple(row) for row in spark.table(TARGET)
.select("order_id", "source_version", "status")
.orderBy("order_id").collect()]
assert actual == [(101, 2, "Shipped"), (102, 1, "Created")]
```
The expected final result contains two rows. The assertion checks both the records and their values, so it detects an incorrect status as well as an unexpected duplicate. Collecting the result is appropriate for this tiny example; use distributed comparisons for large datasets.
Why the version condition matters
The match uses only order_id. A new key is inserted. For an existing key, the update condition permits only a strictly higher source_version. An equal version is a replay, and a lower version is an older delivery. Both leave the stored row unchanged. Conditional updates are part of the documented MERGE semantics. Databricks MERGE documentation
This design depends on the source contract. If the producer changes a payload while reusing a version, the example cannot reliably resolve that conflict across deliveries. Validate that contract upstream, or maintain an event history that allows those conflicts to be detected.
Applying the idea to streaming
For a custom Structured Streaming pipeline, a similar merge can run inside foreachBatch. Databricks documents at-least-once write guarantees for that callback, so its write logic must tolerate repeated execution. A checkpoint alone does not make arbitrary callback side effects idempotent. Databricks foreachBatch documentation
A complete streaming implementation also needs a durable checkpoint, empty-batch handling, and appropriate consumption of stateful query output. For CDC requirements involving deletes, history, or out-of-order changes, evaluate Lakeflow AUTO CDC rather than extending this small demonstration into a general CDC engine. Databricks CDC guidance
Tests to add before production
|Test |Expected behavior |
|----------------------------------------------------|-----------------------------|
|Repeat the same delivery |No change to current state |
|Repeat a row within a batch |One candidate for that order |
|Send an older version later |Keep the newer target version|
|Send a genuinely newer version |Update the order |
|Send conflicting payloads for one version in a batch|Reject the batch |
|Send a null key or version |Reject the batch |
Keep the target unique by order ID and coordinate its writers. This example starts with an empty table and one writer; it does not repair historical target duplicates or enforce uniqueness across independent writers. Retain raw deliveries separately when you need an audit trail, and monitor rejected records instead of silently discarding them.
The practical lesson is to define both identity and ordering before writing the merge. A business key answers which record an event belongs to. A source version answers whether that event should replace the state already stored.
How do you handle repeated deliveries and older updates in your Databricks pipelines? Do you use custom MERGE logic or Lakeflow AUTO CDC?

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

germano — Wed, 16 Sep 2026 11:47:29 GMT

Same here, I'd like to know: if it's possible to take same pathway?, since last time, I didn't take the exam.

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

mannavas — Wed, 16 Sep 2026 11:34:13 GMT

How can I clear the previous status? Even after clicking 'Retake', the course is still not working.

Re: Databricks Advanced Learning Festival: September 16 - October 14 2026

Dee5 — Wed, 16 Sep 2026 11:05:31 GMT

Sorry buddy !! I read between lines and realized 1 account 1 discount coupon.