rss.livelink.threads-in-node

Announcement | Get Started with Genie One: Top AI Cowork Use Cases for Business Users

Tushar_Parekar — Wed, 02 Sep 2026 15:25:57 GMT

Genie One helps business teams turn everyday, repetitive work into repeatable workflows using governed data, connected tools, and natural language.

Key highlights

Automate business reviews: Use a standard format, pull in current data, add commentary, and schedule recurring reports.
Simplify meeting prep and follow-up: Create meeting briefs, summarize decisions and action items, and organize follow-up work.
Keep documents up to date: Draft and refresh policies, playbooks, FAQs, and other documents using approved templates and business data.
Monitor important signals: Track metrics such as revenue, ticket volume, job failures, or inventory and receive alerts when something changes.
Save workflows as skills: Turn useful instructions into reusable skills that teams can run again or schedule for regular tasks.

The best place to start is with one recurring task that takes time every week. Connect the right data sources, give Genie One clear instructions, review the first few results, and refine the workflow as needed.

👉 Read the full post here 👈

Havent received project expert badge

prathikkinim — Wed, 02 Sep 2026 14:46:17 GMT

Hi Team,

I wanted to let you know that I have completed the Databricks Data Engineer Professional certification recently.

I have received the Delivery Expert badge after completing the requirements few days back. However, it has now been more than 48 hours since I completed the exam, and I have not yet received the Project Expert badge.

Could you please check the status of the Project Expert badge and let me know if any additional action is required from my side?

Thank you for your help.

Best regards,
Prathik

After Databricks Summit 2026, I Feel Data Engineering Is Entering a New Phase

Brahmareddy — Wed, 02 Sep 2026 14:42:10 GMT

Dear Databricks Community, After coming back from Summit 2026, one thought stayed in my mind.

Data engineering is changing again. Earlier, most of our work was around pipelines, tables, jobs, transformations, reports, and dashboards. All of that is still important. But now, data is moving much closer to decisions, applications, AI, and real-time actions.

A few topics stood out to me strongly: LTAP, Lakebase, Genie, and Lakehouse RT.

LTAP made me think about how analytics and transaction processing are coming closer. For a long time, applications handled transactions and analytics systems handled reporting. Then we moved data between them using batch, streaming, CDC, or ETL pipelines.

But today, users want fresh insights. Applications need faster intelligence. AI systems need current context. So the gap between operational data and analytical data needs to become smaller.

Lakebase also feels like an important step. To me, it is not just another database. It shows how database architecture is evolving toward the lakehouse. If operational data can work more closely with trusted lakehouse data, it can reduce duplication, simplify architecture, and open new patterns for Data + AI applications.

Genie feels very practical too. Dashboards are useful, but they usually answer only the questions we already planned for. Business users always have follow-up questions. Why did this number change? Which team, region, or process is driving it? What happened this week?

Natural language analytics can make this easier. But one thing is clear: AI can only answer well when the data foundation is strong. Clean data, trusted metrics, governance, permissions, and context still matter a lot.

Lakehouse RT is another exciting direction. Earlier, real-time data was needed only for special use cases. Now, many business problems need faster updates. Fraud, inventory, customer experience, monitoring, recommendations, operations, and AI agents all need fresh data.

The real question is not only, “Can we process the data?”

It is, “Can we process it while it is still useful?”

My biggest takeaway from Summit 2026 is simple.

The future data platform is becoming more connected. Transactions, analytics, AI, applications, and real-time data are coming closer.

This means data engineers need to think beyond pipelines and reports. We need to think about the full solution.

What problem are we solving?
Who will use this data?
How fast do they need the answer?
Can AI make this workflow easier?
Can real-time data improve the decision?

For me, this is the exciting part of working with Databricks. A small idea can become a POC. A POC can become a dashboard. A dashboard can become a real-time insight. A real-time insight can become an action.

That is how I see the next phase of data engineering. It is not only about moving data faster. It is about helping people make better decisions sooner.

After Summit 2026, which area are you most excited to explore with Databricks: LTAP, Lakebase, Genie, Lakehouse RT, or another Data + AI use case?

Thoughts on Using Remix for Data-Focused Applications

ThiamLee — Wed, 02 Sep 2026 13:24:18 GMT

I’ve been exploring different approaches for building web applications that work with data-heavy workflows, and I recently came across Remix as an interesting option.

What I like about Remix is its focus on server-side data loading, forms, and handling application state without requiring everything to be managed on the client side. It seems like this approach could be useful for applications where users frequently interact with data.

I’m curious about how others approach this kind of setup.

For those who have worked with Remix or similar frameworks:

How has your experience been with data fetching and server-side logic?
Do you find the approach easier to maintain as an application grows?
Are there any challenges you’ve faced when connecting a Remix application with data platforms or APIs?

Would be interested to hear different experiences and opinions

Data Engineering 3.0 - From Trusted Data Products to Context Layers

balajij8 — Wed, 02 Sep 2026 13:13:51 GMT

Data Engineering 1.0 was largely about integrating, moving, consolidating massive data to support enterprise decisions. Enterprises built ETL pipelines, data warehouses, data lakes and orchestration platforms to bring data together reliably. Data Engineering 2.0 was about making the move to Cloud Lakehouse and making the data trustworthy and reusable. This created a much more mature enterprise data estate. But the arrival of generative and agentic AI changes the question again. An enterprise may now have thousands of tables, hundreds of data products, semantic models, business glossaries, APIs, documents and real-time data. An agent can potentially access much of this information. But access does not mean understanding. An agent investigating a business problem needs to know which information is relevant, how different entities are related, which definitions are authoritative, which rules apply, what happened previously, what is happening now, and what evidence supports a conclusion.

Enterprises may already have all of this information. The problem is that it is distributed across the data estate. This is the context gap - the distance between information that exists in the enterprise and the context an AI system needs to reliably perform a task.

The Agentic Shift

Traditional analytics was designed primarily to help users make vital decisions. A dashboard presents information. A business user interprets it. The user makes the decision. Machine learning automated parts of this process by using data to generate predictions or classifications. Agentic AI introduces a different operating model. An agent can retrieve information, reason over it, use tools, interact with systems and potentially take action. This means the data platform is no longer serving only a human consumer or an analytical model. It is increasingly serving an intelligent system that needs to understand the enterprise before it can act within it. This changes the role of data engineering.

The question is no longer only How do we make trusted data available? It becomes How do we make trusted enterprise information understandable and usable by an AI system for a specific decision? That is where I see the emergence of Data Engineering 3.0.

Data Engineering 3.0

Data Engineering 3.0 is the next evolution of the data engineering discipline. It is not a replacement for Data Engineering 2.0. It builds on it. The important shift is the unit of value. In Data Engineering 1.0, the question was whether data could be moved reliably at scale, processed and consumed in a robust manner. In Data Engineering 2.0, the question became whether data could be trusted and reused in accordance with governance and usage at scale via data products. In Data Engineering 3.0, the question becomes whether an intelligent system has the right context to perform a task reliably. We are moving from engineering data for consumption to increasingly engineering the information environment in which AI operates.

A Trusted Data Product is not the same as Context

Consider a bank that has already built well-governed data products. It may have a Customer data product containing customer information. It may have a Transaction data product containing transactions. It may have a Merchant data product. It may have a Device data product. It may have a Fraud data product containing historical fraud cases. Each of these can be high-quality, governed and independently reusable.

Now imagine an agent is asked Investigate this transaction and determine whether it should be escalated for fraud investigation. The agent does not simply need the Transaction data product. It may need to understand the customer associated with the transaction, the account involved, the merchant, the device used, recent transaction behaviour, previous fraud cases, current fraud indicators and the policies that determine when a transaction should be escalated. The data already exists. But the decision requires those pieces of information to be brought together with their meaning and relationships intact. That assembled information is the context for the decision. A Data Product provides trusted information. A Context Layer makes trusted information relevant to a particular task.

The Context Layer

An AI Context Layer is a decision-oriented capability that brings together the information an AI system needs to perform a specific task. It can include structured data, semantic definitions, relationships between entities, historical information, real-time signals, business policies and supporting evidence. It does not necessarily mean creating another physical copy of the enterprise data. In many cases, it can be a governed layer that composes information from existing data products and other enterprise systems when an agent needs it.

For the fraud investigation example, the context will include

Customer information and history
The transactions and accounts involved
Merchant characteristics
Device relationships
Recent transaction patterns
Previous fraud cases
Current fraud signals
Applicable fraud policies
Evidence supporting the recommendation

The important point is that the agent does not need the entire enterprise. It needs the right context for the decision it is making.

Context is more than Retrieval

The first instinct when discussing AI context is often to think about RAG. RAG is an important capability. It allows an AI system to retrieve relevant information before generating a response. But enterprise decision-making requires more than retrieving documents. An agent may need to understand that a particular transaction belongs to an account, that the account belongs to a customer, that the transaction was performed using a device associated with several accounts, and that the merchant has a particular risk classification. That is not simply a document retrieval problem. It is a problem involving entities, relationships, semantics, history, current state and rules.

Context may come from structured data, documents, knowledge graph, APIs or operational systems or real-time events. The Context Layer is therefore not a specific technology. It is an architectural capability for composing the information required by an intelligent system.

Where Ontology fits

An ontology can make business concepts and relationships explicit. For example, in a banking space, the enterprise may define relationships between Customer, Account, Transaction, Device and Merchant. That semantic structure helps an AI system understand that these are not independent datasets. It understands the business relationships between them. Ontology therefore provides part of the semantic foundation for a Context Layer. Ontology and context are not the same thing. Ontology answers questions such as

What are the important business concepts? , How are they related? , What does a concept mean? , The Context Layer answers a different question. What of this information and meaning does the agent need for this particular task? This distinction is important because it prevents the Context Layer from becoming another attempt to model the entire enterprise upfront.

From Data Quality to Decision Readiness

Data Engineering 2.0 made data quality a fundamental engineering discipline. We ask whether data is accurate, complete, fresh, consistent and governed. Data Engineering 3.0 introduces another dimension. Is the context sufficient for the decision? Consider the fraud example again. The Customer data, Transaction data, Merchant data and Fraud data may be accurate. Yet the agent could still make a poor decision if an important signal is missing from the context. This is why Data Quality and Context Quality are different problems. A data product can be high quality while the decision context built from several data products is incomplete. This suggests that data engineering will increasingly need to think about decision readiness.

Does the agent have the right information? Is it sufficiently fresh?, Are the relationships correct?, Are the business definitions consistent?, Are the relevant policies available? andCan the agent identify the evidence behind its conclusion? These questions sit on top of traditional data quality rather than replacing it.

Context Layer should be driven by the Decision

There is a natural temptation to approach this as another enterprise-wide architecture exercise. An organization may say We need to build an Enterprise Context Layer. The next step becomes an attempt to model every entity, relationship, rule, document and event across the organization. That is where the initiative can quickly become too large. The better starting point is the decision. Check Which agent task would benefit most from better enterprise context? Then define the minimum context required for that task. For a fraud investigation agent, it means starting with Customer, Account, Transaction, Merchant and Device along with the relationships, historical information and policies required for the investigation. The first objective is not to model the bank. It is to make one important decision more reliable. This is a very different way of approaching enterprise semantics.

Context should evolve with the Agent

The first version of a Context Layer will rarely be complete. And it does not need to be. Start with a defined task. Identify the information required. Build the context. Test the agent. Observe where it lacks information or misunderstands meaning. Then extend the context. This creates a practical feedback loop between the agent and the data platform. The agent becomes a source of evidence about where the enterprise’s existing information environment is insufficient for intelligent decision-making. That evidence can drive the next iteration of the data product or semantic layer. Over time, the Context Layer can expand from one decision to multiple decisions and from one domain to connected domains. The goal is not to build the biggest context layer. It is to build the most useful context for the decisions that matter.

What changes for Data Engineering?

Data Engineering 3.0 does not mean data engineers stop building pipelines. The fundamentals remain critical. Data still needs to be ingested, transformed, stored, governed, monitored and served. But the scope of the discipline expands. Data engineers will increasingly need to understand how enterprise information is represented for intelligent systems. That includes areas such as Semantic modeling, Ontology, Knowledge graphs, Entity resolution, Real-time data, Context retrieval, Agent tools, Data and AI governance, Context evaluation and Decision lineage.

The data engineer’s responsibility increasingly moves from Can we deliver the data? to Can an intelligent system understand and use the data correctly in the context of the decision?. That is a significant evolution in the role. The investments made during Data Engineering 2.0 become the foundation for Data Engineering 3.0. Trusted data products provide reliable information. Data quality provides confidence in that information. Governance provides controls. Lineage provides provenance. Semantic models provide business meaning. Real-time platforms provide current state. Ontologies and Knowledge Graphs can provide explicit relationships.

The next step is to bring these capabilities together around the decisions that AI agents need to perform. Context Layer is the new layer built via DE 3.0 on top of the Trusted Data Foundation enterprises have already created in DE 2.0.

Learn Databricks Lakehouse | From Fundamentals to Hands-On Labs

Tushar_Parekar — Wed, 02 Sep 2026 12:23:58 GMT

Databricks Training & Certifications

Learn Databricks Lakehouse 🏞️

One open platform for all your data, analytics, and AI.

⚡ One open platform 📊 Warehouse + Lake 🧠 Data + AI

Want one platform for all your data, analytics, and AI? The lakehouse combines the reliability and performance of a data warehouse with the openness and flexibility of a data lake - built on Delta Lake and governed by Unity Catalog on the Databricks Data Intelligence Platform.

The catalog offers five Lakehouse courses - from introductory concepts to hands-on Associate labs, most of them free. Follow the path below. 👇

1 Start here · Introductory

Introductory · 1H · Free

Lakehouse Architecture Fundamentals

A concept-level course based on The Data Lakehouse For Dummies. Go from “not sure what a lakehouse is” to holding a working conversation about lakehouse architecture - the limits of legacy architectures, the core technologies, and how it enables data intelligence and unified workloads on Databricks.

Free · Self-paced → Mandarin (中文) →

2 Get hands-on · Onboarding

Onboarding · 4H · Free

Databricks Get Started Days (Lakehouse Architecture + Data Warehousing)

A guided, instructor-led day across two tracks - lakehouse platform architecture (the well-architected lakehouse framework) and getting started with Databricks for data warehousing. Create a Free Edition account and try the follow-along demos hands-on.

Free · Instructor-led →

3 Go deeper: hands-on labs · Associate

Where the lakehouse meets Lakebase - serve and sync operational data in real time. Paid / subscription lab format.

Associate · 1H · Lab

Serving Lakehouse Data in Real Time with Lakebase

Serve ML features from Lakebase on the hot path. Measure why Delta cannot serve single-row reads, read features from a synced table with millisecond point lookups, and publish features to a managed online feature store.

Paid / Subscription · Lab →

Associate · 1H · Lab

Postgres to Lakehouse Change Data Feed (CDF)

Capture Lakebase changes back into Delta. Contrast Lakebase CDF with Delta CDF and synced tables, configure REPLICA IDENTITY FULL, interpret the five system columns in the destination Delta table, and choose among four downstream consumption patterns.

Paid / Subscription · Lab →

Browse all Lakehouse courses in the catalog →

Warehouse reliability. Data lake flexibility. One open platform. Learn the lakehouse the right way with official Databricks training.

create_auto_cdc_from_snapshot_flow Python session resolution fails if having multiple snapshot flows

david_aspegren — Wed, 02 Sep 2026 09:43:25 GMT

When a pipeline contains more than one create_auto_cdc_from_snapshot_flow flow (each driven by a custom Python next_snapshot_and_version function), flow resolution fails intermittently/consistently with:

RuntimeError: The original Spark session is being accessed instead of the per-flow cloned session during parallel analysis. This is commonly caused by spawning threads inside a flow function that access the Spark session.

I am having functions to figure out the next snapshot like this:
```
def next_x_snapshot_and_version(latest_version):
versions = spark.read.table(SOURCE_TABLE).select("file_modification_time").distinct()
```

Is this a bug?

Virtual Event | Closing the AI Context Gap: How to Teach AI How Your Business Actually Runs

Tushar_Parekar — Wed, 02 Sep 2026 09:23:56 GMT

Virtual Event

Time:
AMER Sept 23 / 9 AM PT
EMEA Sept 24 / 9 AM BST / 10 AM CEST
APJ Sept 24 / 9:30 AM IST / 12 PM SGT / 1 PM JST/KST / 2 PM AEST (Japanese and Korean captions available)

🔗 Register for the Workshop

Note: Marking RSVP on this Community event does not register you for the workshop. Please use the Register button above to complete your registration on the official event page.

AI doesn't have an intelligence problem, it has a context problem. Ground truth for business operations remains scattered across dashboards, documents, tickets, and chats. Join us for a virtual session on how to give AI a unified context layer to deliver trusted answers, operate autonomously, and integrate seamlessly into your team's workflow.

Key Takeaways:

Learn to enable a living context graph that continuously infers business terms, entities, and KPIs.
Ensure AI delivers trusted answers based on individual user access.
Turn insights into autonomous actions, agents, and apps without code.
Bring context-aware AI into Slack, Teams, mobile, and MCP-based experiences.

Agenda (60 Minutes):

Welcome (5 mins)
Solving the AI Context Problem (15 mins)
Under the Hood: How Genie Ontology Works (20 mins)
Genie Ontology in Action: Product Demo (20 mins)
Wrap Up (5 mins)

Databricks Data Ingestion

anushnagesh — Wed, 02 Sep 2026 09:09:04 GMT

Hi all,

I am currently exploring the data ingestion feature offered by databricks, specifically connecting to a SQL server.
I have gone through the UI and configured an ingestion pipeline that connects to one specific server and can read tables into databricks.

I am currently venturing into understanding if we can make an ingestion pipeline dynamic.

What I mean by this is, I have created a yaml file that can be run to setup an ingestion pipeline. It accepts variables from a metadata config table that can be used to populate the YAML with information such as, connection server name, pipeline name, source catalog, source schema, source table, destination catalog, destination schema and destination table name.

So the question I have is, can we have a generic pipeline whose configurations can be altered?
Use cases:
1. Once a pipeline is created can it be edited in the future to ingest different tables? As in using pipeline_01, I ingested table_1, table_2 present in Server_01. Can I later pass different configs to the YAML such that I use the same pipeline_01 to ingest table_03, table_04?

2. Can I use pipeline_01 to connect to a different to a connection server and ingest tables from this new server?

Currently I am not finding a way to achieve this functionality. This will cause an issue in the future when I have say 50 servers. It will be difficult to maintain 50 different pipelines and I don't think this is desirable as well.

Can I get some insights into Data Ingestion feature and if this a viable option for my use case?

Thank you,
Anush

Difference between Workspace and Unity Catalog experiments when using MLflow autologging?

kunduruanil — Wed, 02 Sep 2026 08:44:13 GMT

Hi everyone,

I am trying to understand the exact differences between using Workspace experiments versus Unity Catalog experiments, specifically in the context of MLflow autologging (mlflow.autolog()).

Does autologging behave differently depending on whether the experiment is registered in the Workspace or in Unity Catalog (using a 3-level namespace)? Are there any limitations, best practices, or specific configurations I should be aware of when using autologging with Unity Catalog compared to the traditional Workspace setup?

Any insights or documentation links would be greatly appreciated. Thanks!

How can i rename a column in a delta table?

gowri_databrick — Wed, 02 Sep 2026 07:06:30 GMT

I have a delta table and i want to rename one of its columns.

what is the recommended way to rename a column in databricks?

is there any difference between renaming a column using sql and using pyspark?

All 18 Lakeflow AUTO CDC configurations went green. Five failed my ship check

ivanvyd — Wed, 02 Sep 2026 01:54:46 GMT

Paper event slips trace 18 Lakeflow AUTO CDC configurations: 13 stay on the main sequence while five branch into production stop signs.

Change data capture (CDC) keeps a downstream table in step with row-level inserts, updates, and deletes instead of reloading the whole source. Lakeflow AUTO CDC handles the state-management work, including sequencing, deletes, and SCD history. It still needs a precise source contract: which clock wins, how ties break, what NULL means, and which changes deserve history.

I built this experiment to see what happens when source data is late, duplicated, contradictory, or noisy, and to separate a pipeline that finishes from a target I would trust. The suite pushes nine hostile CDC patterns across 13 isolated source tables and 18 AUTO CDC configurations: duplicates, late events, tied sequence values, conflicting clocks, sparse NULL updates, deletes, replays, sync-noise updates, and bitemporal corrections.

All 18 configurations completed. Five green configurations failed my ship check: three had complete ordering but violated the stated business rule, and two used incomplete ordering.

Start with scenario 4. Ordering the same rows by ingestion time kept ACTIVE; source event time produced SUSPENDED, the expected business state. Both configurations finished green.

I used pipeline status to confirm execution. I used target-state assertions to decide whether I would ship.

Independent experiment. I ran this test for my own engineering work. It is not official Databricks guidance. I’m happy to discuss your results, feedback, and the failure modes you think I missed.

Run the experiment · Inspect the result matrix

Results in one minute

10 handled: keep the sequence rule and test it against your source.
3 configuration-dependent: set the option that matches the business rule.
3 business-semantics stops: change the chosen clock, NULL meaning, or history policy.
2 ambiguous-order stops: add a source-side tie-breaker.

Execution stayed green across all five stop signs. Source semantics or ordering failed my ship check.

The five stop signs

Configuration Measured result Fix before production

3A Sequence collision	Two business states shared one sequence value. The contract names no winner.	Reject ties or add a stable source-side tie-breaker.
3B Tie-breaker ignored	The source supplied transaction_sequence, but the flow left it out of SEQUENCE BY.	Use a composite sequence.
4A Ingestion-time order	The target kept ACTIVE; source time required SUSPENDED.	Use the clock that defines business recency.
5A Default NULL handling	A sparse update replaced the existing email with NULL.	Define NULL semantics and use IGNORE NULL UPDATES when NULL means absent.
8A Track every column	Fifty sync-timestamp updates created 51 SCD2 rows.	Exclude operational metadata from history tracking.

The other 13 configurations matched the experiment’s business rule under a complete order and the required option.

How I tested it

The generator creates small, isolated source tables with one failure mode per scenario. One pipeline wraps each source in a streaming view and runs all 18 AUTO CDC configurations.

For late events and replays, I ran two updates. The full refresh established a baseline. The incremental update appended the withheld rows. The verifier required the two expected histories to change, the other 16 targets to remain equal, and every target to match its row-count and observed-state predicate. The final classification combined those measurements with the declared ordering and business-rule labels.

You can inspect the code in src/pipeline/pipeline.py and src/generators/dispatch.py. The result matrix and captured target rows carry the measured evidence. Official Databricks documentation used for the platform claims is indexed in docs/sources.md.

Four source decisions that change the answer

1. Pick the clock that means “newer”

The generator emits two events for the same key:

10:00 source time  ACTIVE
10:05 source time  SUSPENDED

Their ingestion timestamps reverse that order. Ordering by ingested_at keeps ACTIVE. Ordering by source_updated_at produces SUSPENDED, which matches the experiment’s business rule.

Two AUTO CDC targets show that ingestion time misses the business expectation while source time matches it.

Two AUTO CDC targets use the same rows and schema. The sequence clock changes the answer.

Arrival time helps you analyze transport; source time can define business recency. Pick the column from the business definition before you write the flow.

2. Give ties a real winner

The collision pattern gives two states the same source_sequence:

seq=10  status=ACTIVE
seq=10  status=SUSPENDED

I observed SUSPENDED, but the configured order cannot distinguish the rows, so I recorded the result as AMBIGUOUS_ORDER.

The source also provides transaction_sequence. A composite sequence such as STRUCT(source_updated_at, transaction_sequence) gives the flow a stable order. The composite configuration produced the expected SUSPENDED state, and the verifier classified it as HANDLED.

3. Decide what NULL means

The sparse-update pattern begins with email='x@example.com'. The next event updates the city and carries email=NULL.

The default configuration sets the target email to NULL. IGNORE NULL UPDATES keeps the existing email. Both behaviors can serve a valid source contract. Your producer must define whether NULL means “erase this value” or “this field was absent.”

4. Keep sync noise out of business history

The history-noise generator emits 50 updates. Among the retained target columns, only last_synced_at changes.

Tracking every included target column creates 51 SCD2 rows. Excluding last_synced_at creates one row because the business fields never change.

Tracking every included target column produces 51 SCD2 rows; excluding last_synced_at produces one.

Operational sync metadata accounts for all 50 extra versions.

Choose the columns that represent business history before you deploy SCD2. Whether operational changes belong there is a domain and audit decision; this experiment’s business rule counted only business-field changes.

Patterns AUTO CDC handled

Pattern Measured result

Identical duplicate and late replay	SCD1 kept one ACTIVE row. The replay left visible state unchanged.
Out-of-order update	SCD1 kept the newer state. SCD2 inserted the older event as a closed history row.
Delete followed by an older event	SCD1 kept the deletion. SCD2 inserted the late state before the delete boundary.
Full lifecycle replay	SCD1 and SCD2 matched their saved baselines after the replay.
Bitemporal correction	The target preserved business time and system time across five measured rows.

These are visible-state observations from appended replays. They do not establish transactional deduplication or test a checkpoint-only restart with no new source rows.

Bitemporal kept both clocks

Scenario 9 uses source_updated_at for business time and ingested_at for system time. The target stores __START_AT / __END_AT beside __SYSTEM_START_AT / __SYSTEM_END_AT.

Three events produced five rows because later ingestion times revised earlier valid-time intervals.

Five measured bitemporal rows show original and revised valid-time intervals across three system times.

Five rows preserve the original and revised valid-time intervals.

Databricks marks bitemporal storage as Beta. This run validates five tiny rows and does not test load. I would evaluate it when valid time differs from ingest time and consumers need as-of-system-time queries.

The full result map

All 18 measured configurations grouped by handled, configuration-dependent, business-semantics, and ambiguous-order outcomes.

All 18 measured configurations grouped by production outcome.

You can inspect the machine-readable matrix in results/normalized/summary_matrix.json and the captured rows in results/raw/target_state.json.

My production checklist

Name the business clock. Put the column that defines recency in SEQUENCE BY.
Search for ties per key. Add a source-side tie-breaker when two states can share one sequence value.
Write down NULL semantics. Use IGNORE NULL UPDATES only when NULL means absent.
Choose the state model. SCD1 keeps current state; SCD2 keeps ordered history.
Filter history noise. Exclude sync metadata unless it represents a business event.

Add a target-state predicate for each rule: update status checks execution, while the predicate checks the expected target state.

Reproduce the run

git clone https://github.com/ivanvyd/lakeflow-auto-cdc-torture-test.git
cd lakeflow-auto-cdc-torture-test
python -m pip install -e ".[dev]"
databricks auth login --host https://<workspace-url> --profile DEFAULT
make setup
make test
make results

You need Python 3.10 or newer, GNU Make, jq, the Databricks CLI, and access to a Databricks workspace. The reproduction guide covers the one-time setup and evidence capture.

Sixteen targets matched their baselines after update 2. The two SCD2 targets that received late history changed as expected. The verifier checked all 18 targets and produced 10 HANDLED, 3 CONFIGURATION_DEPENDENT, 3 BUSINESS_SEMANTICS, and 2 AMBIGUOUS_ORDER results.

The checked-in result set was captured on 2026-09-01 from clean commit 01d53b4; target_state.json records the pipeline and both update IDs.

Scope

This suite uses tiny controlled datasets. I did not test throughput, backpressure, or large joins. Each flow read one isolated source table; I did not test joins or interactions across streams. The delete-and-late-event case stayed inside the configured 48-hour tombstone-retention window. Bitemporal load behavior also sits outside this run.

The evidence comes from one workspace, one SQL warehouse, one customer key, serverless Advanced-edition compute on the CURRENT channel, and tiny deterministic inputs. Use separate tests for throughput, schema evolution, and multi-stream joins.

Bring your failure mode

Clone the repository and replace one generator with an event sequence from your source. Add the expected target state, run both phases, and compare the capture.

If the suite misses your case, open an issue with the source rows, SEQUENCE BY expression, storage type, expected target, and observed target. I’m happy to turn a clear failure report into another reproducible scenario.

Lakebase Postgre updating Delta Table.

rkhbo3003 — Wed, 02 Sep 2026 01:44:54 GMT

I am using Postgre for OLTP processing for POS application.Lag is reduced a lot, however when there is updation on Postgre table, I need to sync back to delta table. There is one way from delta table sync table (read only). Any design pattern sas to keep delta table and postgre table in sync.

From Semantic Similarity to Business Authority: Why Genie Ontology and OntoRank Matter

amitsharma1707 — Wed, 02 Sep 2026 00:59:50 GMT

Enterprise AI does not usually fail because the model lacks intelligence.

It fails because the model does not understand what the organization means.

Consider a simple question:

“What is our current exposure to active customer?”

Behind this question are several business decisions:

What qualifies as an “active” customer?

Should exposure include committed, outstanding, or available amounts?

Which customer identifier is authoritative?

Should rebooked amount be consolidated?

Which system is trusted: the servicing platform, CRM, MDM golden record, or a reporting mart?

What business date should be used?

A traditional text-to-SQL system may identify tables with similar column names and generate syntactically correct SQL. But syntactically correct SQL can still produce a completely incorrect business answer.

This is the context gap that Databricks Genie Ontology is designed to address.

What is Genie Ontology?
Genie Ontology is a unified, continuously improving context layer that gives Genie a business-aware map of the organization.

It combines:

Human-modeled context

Certified data products, Unity Catalog metric views, domains, business definitions, Pages, and governed assets.

Automatically inferred context

Knowledge extracted from tables, queries, dashboards, SQL patterns, Genie Agents, and platform usage.

Instead of treating enterprise knowledge as disconnected metadata, the ontology represents relationships among:

Business terms

Metrics

tables and columns

dashboards

queries

data products

people and teams

business rules

Genie Agents

This changes the question from:

“Which asset looks most similar to the user’s prompt?”

to:

“Which permitted source represents the most authoritative meaning for this question?”

Where OntoRank becomes important
Enterprises rarely have only one definition of a metric.

There may be multiple definitions of revenue, customer, active account, gross margin, or credit exposure—each created by different teams, at different times, for different purposes.

OntoRank is the PageRank-inspired authority-ranking concept associated with Genie Ontology.

Rather than ranking only by textual similarity, the context layer can consider signals such as:

Source authority and provenance

Asset certification

Frequency and breadth of usage

Relationships with other trusted assets

Freshness

Business relevance

User permissions

For example, imagine Genie discovers three definitions of “active customer”:

An old spreadsheet definition created three years ago

A frequently queried but uncertified reporting view

A certified Unity Catalog metric connected to the MDM golden customer and current amount balances

Keyword similarity alone might retrieve any of them.

An authority-aware approach should prioritize the certified, governed, fresh, and widely connected definition—while still enforcing the requesting user’s permissions.

Why this is bigger than better text-to-SQL
The real architectural shift is:

Metadata → Semantics → Context → Trusted action

A well-designed ontology can help an AI system understand:

Which source should be queried

Which metric definition should be applied

Which relationships and joins are valid

Which conflicting definition should take precedence

Which assets are deprecated

What the user is authorized to access

Why a particular source was used

This can make AI systems more accurate, explainable, reusable, and governance-aware.

But OntoRank does not eliminate data governance
Authority ranking is powerful, but popularity is not always correctness.

A widely used definition may still be outdated. A newly created certified data product may initially have little usage history. Poorly documented tables will continue to produce weak context.

Therefore, organizations should prepare the foundation:

Define important business terms

Create governed metric views

Certify authoritative data products

Deprecate obsolete assets

Maintain table and column descriptions

Capture lineage

Assign clear data ownership

Improve MDM and identity resolution

Test Genie answers against approved business scenarios

Genie Ontology can amplify a strong semantic and governance foundation—but it cannot magically repair an undefined business vocabulary.

My key takeaway
The next generation of enterprise AI will not be differentiated only by model size.

It will be differentiated by the quality of the context surrounding the model.

RAG helps AI find similar information.
Ontology helps AI understand relationships and meaning.
OntoRank helps AI decide what should be trusted.
Unity Catalog helps ensure that trust remains governed.

The most important question for data architects may soon change from:

“How do we expose our data to an AI agent?”

to:

“How do we make business meaning discoverable, authoritative, permission-aware, and machine-readable?”

I would love to hear from the Databricks Community:

How are you preparing your Unity Catalog metadata and metric views for Genie Ontology?

How should OntoRank balance popularity against formal certification?

Should users be able to inspect the ontology graph and understand why one definition outranked another?

What evaluation framework are you using to measure the business accuracy of Genie answers?

Operationalizing a Live Genie Space: Benchmarking, Governance & Continuous Improvement

Salman_Ahmed — Tue, 01 Sep 2026 20:25:33 GMT

In my previous article, I wrote an article about building a strong foundation for Databricks Genie Spaces through data modeling, metadata, and semantic design. This time, I want to share a few lessons from running a Genie Space against a real healthcare analytics use case.

The biggest surprise wasn't AI. It was how quickly Genie exposed problems that already existed in the data.

When we first deployed the Genie Space and ran it against our benchmark questions, the results were far from perfect. The out-of-the-box configuration answered only 6 out of 20 benchmark questions correctly. Even after multiple rounds of tuning, a repurposed legacy analytics dataset for BI workloads never achieved more than 75% benchmark accuracy.

What changed everything wasn't prompt engineering. It was dataset design.

Once we defined and built a live delta dataset with pre-defined metrics specifically for Genie, removed irrelevant columns, simplified business logic, standardized categorical values, and aligned the schema to the questions users were actually asking, benchmark accuracy eventually reached 100%.

While many teams may focus on configuring Genie and fine-tuning prompt. For our team, the biggest boost in accuracy came from preparing the underlying data for Genie.
Another lesson was the importance of benchmarking. Without benchmarks, every discussion becomes subjective.
- Quick Tip: Users may ask the same question in different ways. Databricks recommends using 2-4 different phrasings of the same question (but the same SQL code) to fully assess accuracy.

We also learned that not every configuration change improves accuracy. Below is a visual of our findings;

Table descriptions helped significantly.
General instructions provided some improvement.
Surprisingly, adding large numbers of synonyms wasn't always beneficial and sometimes introduced additional ambiguity.
The most effective configuration element by far was SQL examples.
- Quick Tip: Whenever a business rule involved non-trivial logic, such as event-level calculations, eligibility definitions, or classifications, SQL examples consistently outperformed additional instructions. Genie seemed to learn far more effectively from concrete examples than from lengthy explanations.
Another important observation was that Genie is not entirely deterministic. The same question can occasionally be presented differently depending on formatting choices, percentage calculations, null handling, or interpretation of date ranges. We found ourselves spending as much time standardizing outputs as improving answers.

This is where governance becomes essential

A successful Genie Space needs more than good metadata. It needs agreed definitions, controlled business logic, benchmark testing, and a clear process for evaluating changes. Otherwise, the results erodes trust.

The teams that will get the most value from Genie Spaces and turn a demo into a production-ready analytical product won't be the ones writing the most sophisticated prompts. They'll be the ones investing in benchmark-driven development, semantic consistency, and Genie Tailored purpose-built datasets.

Partitioning vs Liquid Clustering (per-table):

AshokB — Tue, 01 Sep 2026 20:16:23 GMT

Can PARTITION BY and CLUSTER BY (Liquid Clustering) be used simultaneously on the same table? If we use only PARTITION BY, is there a negative performance impact on materialized-view refreshes in Silver/Gold? Since materialized views read only incremental delta files, is Liquid Clustering redundant in this scenario, or does it still provide benefits for file compaction and read optimization?

Making Databricks Genie Spaces Actually Work: A Practical Framework for Client and Data Teams

Salman_Ahmed — Tue, 01 Sep 2026 18:42:28 GMT

Introduction

Over the past year, I've had quite a few conversations with teams exploring Databricks Genie Spaces. The pattern is usually the same. Someone sees a demo, watches a business user ask a question in plain English, and within seconds Genie returns a chart, a SQL query, and what appears to be a perfectly reasonable answer.

The reaction is almost always immediate.

"This could completely change how people use data and BI reports."

For years we've built dashboards, semantic models, reporting layers, and self-service analytics platforms with the goal of helping business users answer questions faster. Genie feels like the natural next step in that journey. Instead of learning SQL or navigating dozens of dashboards, users can simply ask a question and interact with data conversationally.

The technology itself is impressive. But after working on enterprise data platforms for many years, the challenge is rarely the AI model itself.

The challenge is trust.

Can users trust the answer? Can analysts reproduce it? Can data teams explain it? And perhaps most importantly, will different users receive consistent answers to the same question?

Those questions have far less to do with the language model and much more to do with the foundation underneath it. That's why whenever I'm asked how to improve Genie Spaces.

I rarely start by talking about prompts, I start by talking about data.

Why Most Genie Projects Fail Before Users Ask Their First Question

When teams evaluate Genie Spaces, their first instinct is often to improve prompts or add more instructions.

In my experience, that's usually the wrong starting point. Most quality issues originate from one of five areas:

Weak data modeling (Data Engineering)
Poor metadata quality (Data Scientist/Business Analyst/Data Analyst)
Undefined business metrics (Data Scientist)
Missing table relationships (Data Engineering)
Lack of benchmark testing (ML Engineer)

Databricks has been investing heavily in a governed semantic foundation through Unity Catalog Semantics, Genie Ontology, Live Tables, including metric views, domains, governed business definitions, and AI-aware context management. These capabilities help ensure that both humans and AI systems interpret data consistently.

Step-by-step plan

Step 1: Build the Data Foundation Before Building the Genie Space

The single most important success factor is the quality of the curated data layer.

Many data teams expose highly normalized source models and expect Genie to figure out the relationships. While technically possible, this often introduces ambiguity. Instead, design datasets specifically for consumption.

Recommended Design Approach

1. Denormalize Where Appropriate

Rather than expecting Genie to navigate a maze of joins every time a user asks a question, it's worth investing in curated business-ready delta tables. If answering a simple revenue question requires six or seven tables to be stitched together, the chances of selecting an incorrect relationship increase significantly. In most successful implementations I've seen, common dimensions are already joined, business entities are standardised, and duplicate relationship paths have been removed long before the data reaches Genie.

2. Pre-Calculate Common Business Logic

A common mistake is treating Genie as the place where business logic should be assembled. In reality, repetitive calculations and classifications belong in the data layer. Whether it's reporting periods, fiscal calendars, active customer definitions, or product lifecycle states, these concepts should already exist in a governed and reusable form. This allows Genie to focus on answering the question rather than reconstructing business logic every time.

3. Establish Canonical Metrics

One of the strongest capabilities available through Unity Catalog is the ability to define reusable metrics and semantic objects that provide consistent business logic across analytics workloads and AI consumers.

For example:

measures: - name: Total Revenue expr: SUM(purchase_amount) FILTER (WHERE status='approved') comment: Revenue from approved transactions display_name: Total Revenue synonyms: - revenue - sales - total sales - approved revenue

This ensures that every user asking about revenue receives answers based on the same calculation.

Step 2: Treat Genie Like Software and Create Benchmarks

I've been in sessions where a team asks Genie three questions, gets two correct answers, one questionable result, and immediately starts debating whether the prompt needs to be rewritten. The reality is that this kind of testing is far too subjective. Without a defined set of benchmark questions and expected outcomes, it's almost impossible to measure quality in a meaningful way. That's why it is best to establish a benchmark suite early, before wider adoption begins.

Create a Question Inventory

The best benchmark questions usually come directly from the people who use the data every day. Spend time with business stakeholders, analysts, and subject matter experts to understand the questions they regularly ask, whether that's tracking KPIs, understanding trends, explaining variances, or preparing executive reporting. Once you've collected those questions, document what a correct answer looks like. That includes not only the expected result, but also the level of aggregation, any business filters that should be applied, and how the answer should be presented. The goal isn't simply to test whether Genie returns an answer. It's to verify that the answer aligns with how the business expects the question to be interpreted.

Build a Regression Test Suite

A Genie Space is never really finished. The underlying data platform keeps evolving, new business requirements appear, and teams continuously refine their definitions and metrics. While those changes are important, they also introduce risk. I've found that the most successful teams maintain a set of benchmark questions that are executed regularly, especially after major updates. It provides a simple but effective way of confirming that answers users already trust continue to behave as expected, even as the platform grows and changes around them.

Step 3: Teach Genie How Your Business Thinks

Metadata is what helps bridge the gap in business thinking in natural flow and tables, columns, or schemas. The richer the business context around your data, the easier it becomes for Genie to understand what the user is really asking and translate that intent into a query that makes sense. In many cases, improving metadata delivers a bigger uplift in answer quality than yet another round of prompt tuning.

Table Descriptions Matter

Avoid generic descriptions like:

Customer transaction table

Instead use:

Contains finalized customer purchase records used for revenue reporting and financial performance analysis.

The second description provides significantly more business context.

Define Synonyms Explicitly

Business users rarely use technical column names.

For example:

Business Term Actual Field

Sales	Revenue
ARR	Annual Recurring Revenue
Customer Base	Active Customers
Gross Sales	Invoice Amount

Providing synonyms dramatically improves question interpretation.

Document Relationships

Another area that often gets overlooked is the way datasets relate to one another. In most enterprises, the same business entity appears across multiple tables, and there can be several possible paths between them. If those relationships aren't clearly defined, Genie may have to infer how the data is connected, which can lead to unexpected results.

Explicitly documenting relationships and validating the business meaning behind them significantly improves consistency. It's not enough to know that two tables can be joined; Genie also needs to understand how they should be joined and what business context that relationship represents.

Incorrect joins are a major source of AI-generated analytical errors.

Supply Example SQL

One of the most effective yet underutilized techniques is maintaining a library of gold-standard SQL.

Example:

SELECT fiscal_year, SUM(revenue) AS total_revenue FROM sales_gold GROUP BY fiscal_year ORDER BY fiscal_year;

These examples act as patterns that help Genie generate more reliable queries.

Use General Instructions Sparingly

Many teams attempt to solve every issue through lengthy instructions.

This typically creates maintenance problems.

A simpler decision framework is:

Problem Fix Location

Wrong table	Table metadata
Wrong column	Column description
Wrong value mapping	Example values
Wrong join	Relationship definition
Wrong calculation	Example SQL

Use narrative instructions only for business context.

Key Takeaways

Organizations often assume conversational analytics starts with AI.

In reality, it starts with data engineering.

Before focusing on prompts, invest in:

Curated Gold datasets
Metric definitions
Rich metadata
Relationship modeling
Benchmark testing

Genie Spaces are most successful when they are grounded in governed business semantics rather than isolated prompt instructions. Databricks' broader investment in Unity Catalog Semantics reflects this exact direction, creating trusted business context that can be reused across analytics and AI experiences.

2-Part Series

Part 1: Making Databricks Genie Spaces Actually Work: A Practical Framework for Client and Data Teams
Part 2: Operationalizing a Live Genie Spaces with Benchmarking, Governance, and Continuous Improvement

CUSTOMER STORY | Kraken governs utility data at scale with Unity Catalog

Tushar_Parekar — Tue, 01 Sep 2026 17:04:24 GMT

"Unity Catalog allows us to deliver data in the safest and most compliant way for our customers. Given the liability involved, that is not something we take lightly." - Javi Asensio, Head of Data and Analytics Engineering, Kraken

Kraken, the operating system for utilities, supports approximately 85 million contracted accounts across 13 countries. With Unity Catalog and Delta Sharing, Kraken is building a governed data foundation that helps utility providers access data securely, while giving internal teams better visibility across a complex, multi-account environment.

Key highlights:

95% of governance needs met natively: System tables and the Unity Catalog API provide visibility into access, queries, endpoints, and resource usage without relying on third-party tooling.
Data sharing in 1 day instead of 1.5 weeks: Delta Sharing helps Kraken make data available to clients faster, especially for customers already using Databricks.
Secure, fine-grained access: Account isolation, data masking, and role-based controls help protect sensitive personal, financial, and smart-meter data across regulated markets.
One governance layer at scale: Unity Catalog helps Kraken manage access, auditability, and compliance across dedicated Databricks environments for each client.

🔗 Check out the full story 👈

Webassesor last name change

Ds_1_0 — Tue, 01 Sep 2026 17:00:12 GMT

Hi, anyone knows how to contact databricks support? Or how long it takes to get any response from them?

I need to change my last name in webassesor, can't do this my self.

Already had to move my exam 4 times becasue support is not responsive. I've tried both... an email and the form..

Como aprender correctamente en el entorno de DataBricks y aprovecharlo al 100%

maycol25 — Tue, 01 Sep 2026 16:59:48 GMT

Hola a todos, soy ingeneriero en sistemas y deseo aprender a utilizar esta grandiosa herramienta para poder sacarle el mejor provecho posible.