New board topics in Databricks Community

Genie One Guardrails

SenthilKrishna — Tue, 20 Oct 2026 12:15:42 GMT

We are opening up the Genie AI to the business users. The Genie One seems to be a good fit, but the problem I noticed is the Guardrails and influence over the single orchestration. All the questions have been nicely interrupted by Genie one and works with agents, tables from all schemas, notebooks, etc. My issue here is that I want to prioritize the search to go on the agents first, Clarifying questions for underspecified prompts than assuming the metrics, categories and date ranges. I feel like that's possible by creating the persona via the skill under Genie One, and customize with guardrails. Is that a right approach?

Governed Tag Propagation (Beta) in Unity Catalog - How It Works and What to Consider

data_pulse — Tue, 20 Oct 2026 07:38:43 GMT

Databricks' Governed Tag Propagation (Beta) is an interesting addition to Unity Catalog, helping automatically carry governed tags across downstream tables and views using data lineage.

As data moves across medallion layers, this could significantly reduce the effort of manually maintaining consistent classifications across derived datasets.

The below flow illustrates how governed tags propagate through supported operations and some key lifecycle behaviours.

What I explored:

Tested the Beta with a few common Databricks operations:

CTAS, INSERT, MERGE and Views: Governed tags propagated successfully.
Column transformations: Tags followed renamed and transformed columns.
Delta CLONE: Table-level tags propagated, but column-level propagation wasn't supported.
Conflicting tag values: The configured default propagation value was applied.
Tag removal and reclassification: Removing an upstream tag didn't automatically remove the existing downstream tag during testing. However, explicit downstream removal worked, and subsequent reclassification triggered re-propagation.

Where could this help?

PII and sensitive-data classification: Maintaining consistent classifications through the lakehouse.
Governance automation: Reducing manual tagging across derived tables and views.
ABAC integration: Supporting tag-driven access policies, subject to propagation timing.
Data discovery and compliance: Improving visibility into classified downstream assets.

A few interesting questions remain:

While the initial results are promising, some areas would benefit from further exploration:

Tag removal: Should removing a source classification automatically revoke its downstream propagated tags, or is retaining them the safer behaviour?
Propagation latency: Since propagation is asynchronous, how should organisations handle the period before tag-dependent ABAC policies become effective?
Governance at scale: How should teams manage conflicting classifications, downstream overrides, and periodic tag reconciliation?

Overall, this looks like a useful step towards lineage-driven, automated data governance in Databricks. However, understanding the complete tag lifecycle is just as important as ensuring tags propagate successfully.

Would be interested to hear how others are approaching these scenarios or whether you've observed similar behaviour during the Beta.

Reference: https://docs.databricks.com/aws/en/admin/governed-tags/tag-propagation

Kinessana Reviews 2026: Is It Worth Buying?

ciplozine — Tue, 20 Oct 2026 06:12:12 GMT

Its wraparound design helps position the device around the intended area, while an extension strap helps secure it during use. A magnetic touch controller provides access to the available settings, allowing users to adjust the experience according to their comfort preferences.

Kinessana may appeal to people who experience occasional shoulder tension after work, exercise, travel, or other everyday activities. It is also designed for individuals who prefer relaxing at home rather than relying entirely on manual massage techniques.

Official Website –- https://kinessana.net

Kinessana

DevOps Fundamentals: 08 - Creating and Executing Unit Tests: Instructions doesn't match code

Daniel_B — Mon, 19 Oct 2026 21:04:29 GMT

Instructions:

To execute pytest within Databricks complete the following:

sys.dont_write_bytecode = True - By default, Python generates .pyc bytecode files in a pycache directory. Setting it to True prevents this, which is useful in read-only environments.

Code in cell directly underneath:

sys.dont_write_bytecode = False

Thanks. I hereby claim my $1000000 bug bounty. You're welcome.

Announcement | Geospatial queries are up to 38x faster with Lakehouse//RT

Tushar_Parekar — Mon, 19 Oct 2026 17:23:08 GMT

Spatial SQL and spatial joins are now supported on Lakehouse//RT, bringing low-latency, high-concurrency geospatial queries directly to governed lakehouse data. Teams can run spatial workloads without copying data into a separate geospatial serving layer.

Key Highlights

Up to 38x faster under high concurrency: In the reported SpatialBench comparison with 50 concurrent users, Lakehouse//RT delivered up to 38x faster performance than a traditional cloud data warehouse. Results vary by query and workload.
Run spatial SQL on governed data: Use native GEOMETRY and GEOGRAPHY data types, spatial joins, and the broader Databricks spatial function library on Delta and Iceberg data.
Reuse existing queries: Existing spatial SQL from Databricks SQL Serverless can run on Lakehouse//RT without query changes; create a Lakehouse//RT warehouse and point workloads to it.
Support real-time geospatial use cases: Analyze coverage areas, store trade areas, energy infrastructure, insurance risk, delivery zones, and location-based security signals at higher concurrency.
Keep the lakehouse architecture: Combine real-time performance with open formats, Unity Catalog governance, AI/BI maps, and the rest of the Databricks Data + AI Platform, without a separate serving stack.
Available in Beta: Spatial SQL and spatial joins on Lakehouse//RT are available now in Beta. Check regional and feature availability before deployment.

👉 Read the full story here

Certification Support Closed 4 Tickets Without Resolution — Refund & Festival Voucher Issue

Datastd — Mon, 19 Oct 2026 17:08:39 GMT

Hi everyone, especially the community managers.

I'm reaching out publicly because I've exhausted the regular support channels, and four tickets have been closed without resolving my issue. I would really appreciate the chance to speak directly with someone from the Databricks Certification team.

What happened

On October 8, 2026:

- I used a voucher won at the Databricks Festival to book the Data Engineer Professional exam through Webassessor.
- I accidentally scheduled the English exam for a slot starting just 2 minutes later and paid USD 100.
- I immediately attempted to cancel that booking.
- Just 14 minutes later, I booked the Portuguese exam, paid another USD 100, took the exam, and passed! 🎉

Unfortunately, the original English booking was never properly cancelled and eventually appeared as "Expired."

Contradictory support responses

- Oct 9 (ticket 01040221): Requested a USD 100 refund and restoration of my voucher.
- Oct 13: Databricks support explicitly acknowledged in writing: "We noticed that you have cancelled the exam and scheduled the exam again with the correct language." The ticket was then closed without resolution.
- Oct 15–19: Tickets 01045247, 01047289 and 01047318 were also closed without a solution.
- Oct 19: Support contradicted its earlier statement, claiming that the registration expired because I did not launch the exam, and denied both requests.

Why this deserves review

1. Support initially acknowledged my cancellation, then later denied it.
2. The booking system allowed me to schedule an exam starting in 2 minutes, despite a cancellation policy requiring 24 hours' notice. Under those rules, cancelling was impossible.
3. I contacted support the following day.
4. As a Brazilian consumer, I requested the refund within the 7-day withdrawal period guaranteed by Article 49 of Brazil's Consumer Protection Code.

What I'm requesting

- Refund of USD 100 for the unused English exam booking.
- Restoration of my Databricks Festival voucher.
- A direct contact from someone on the Certification team who can review the case, instead of another automated reply.

Current ticket: 01048926

I have already submitted complaints to PROCON and Reclame Aqui, but I would genuinely prefer to resolve this directly with Databricks.

I'm proud to have earned my Databricks certification, and I value this community. That's precisely why I hope this situation can be reviewed fairly rather than having another ticket closed without a meaningful resolution.

Could a community manager please help escalate this case and connect me with the Certification team? I'm happy to share all emails and receipts privately.

Thank you! 🙏

Announcement | Stripe Data Pipeline now syncs real-time Stripe data to Lakebase Postgres

Tushar_Parekar — Mon, 19 Oct 2026 14:44:44 GMT

Stripe Data Pipeline can now stream Stripe data directly into Databricks Lakebase Postgres, giving applications access to current payment, customer, subscription, refund, and payout data without custom API or webhook integrations.

Key Highlights

Stream Stripe data to Lakebase: Select the Stripe tables your application needs, and Stripe handles the backfill, ongoing delivery, and data audits through a no-code pipeline.
Power application logic with current data: Use SQL against Lakebase to support feature access, subscription changes, billing details, financial visibility, and personalized offers.
Reduce integration overhead: Avoid building and maintaining separate API polling, webhook handlers, retry logic, and reconciliation workflows to keep application data synchronized.
Use one Stripe pipeline for two destinations: Send Stripe data to Unity Catalog for governed analytics and AI, while syncing the operational data needed by applications to Lakebase Postgres.
Connect analytics and applications: Join Stripe data with other business data in Databricks for analytics, while Lakebase provides a Postgres interface for application workloads.
Get started in public preview: Real-time sync to Postgres is available in public preview and is free to get started. Availability depends on supported regions and configuration.

👉 Read the full story here

Table does not support batch scan.

harsh120 — Mon, 19 Oct 2026 13:15:27 GMT

After moving our incremental load jobs from DBR 16.4 (Spark 3.5) to DBR 18 (Spark 4.1.0), they fail with this error:

[UNSUPPORTED_FEATURE.TABLE_OPERATION] The feature is not supported:
Table `app_dev`.`vw_period` does not support batch scan. Please check the
current catalog and namespace to make sure the qualified table name is
expected, and also check the catalog implementation which is configured
by "spark.sql.catalog". SQLSTATE: 0A000

vw_period is a regular Unity Catalog view, and the table under it is a plain external Delta table, not a streaming table, materialized view or pipeline output. The same views, data and code ran without problems on 16.4.

Table and view Defination

-- external Delta table, about 20 rows in total
CREATE TABLE acme_prod.app_dev.calendar_entries (
  entry_id BIGINT, period_key BIGINT, period_label STRING,
  period_start DATE, period_end DATE, period_tz STRING
  -- other columns omitted
) USING DELTA;

CREATE OR REPLACE VIEW acme_prod.app_dev.vw_period AS
SELECT DISTINCT period_key, period_label, period_start, period_end, period_tz
FROM acme_prod.app_dev.calendar_entries;

-- external Delta table, partitioned by activity_type; the real one has about 170 columns
CREATE TABLE acme_prod.app_dev.activity_events (
  activity_id STRING, actor_key BIGINT, activity_type STRING,
  platform_label STRING, session_id STRING,
  occurred_at TIMESTAMP, is_automated BOOLEAN
  -- other columns omitted
) USING DELTA PARTITIONED BY (activity_type);

CREATE OR REPLACE VIEW acme_prod.app_dev.vw_activity_events AS
SELECT e.*, COALESCE(p.period_key, CAST(-1 AS BIGINT)) AS period_key
FROM acme_prod.app_dev.activity_events e
LEFT JOIN acme_prod.app_dev.vw_period p
  ON CAST(e.occurred_at AS DATE) BETWEEN p.period_start AND p.period_end
WHERE e.activity_type <> 'heartbeat';

What works and what fails

The first read works on DBR 18, and isEmpty() and rdd.isEmpty() both return normally on it:

src=spark.sql("""
  SELECT * FROM acme_prod.app_dev.vw_activity_events
  WHERE activity_type IN ('a', 'b')
    AND occurred_at >  TIMESTAMP '2026-10-13 09:02:00'
    AND occurred_at <= TIMESTAMP '2026-10-19 12:46:00'
    AND COALESCE(is_automated, false) = false
""")
src.createOrReplaceTempView("events_incr")

The next step is where it fails:

out = spark.sql("""
  SELECT CONCAT('X_', activity_id) AS event_key, actor_key,
         UPPER(activity_type) AS activity_type, CAST(occurred_at AS DATE) AS event_date,
         period_key,
         MAP('platform_label', platform_label, 'session_id', session_id) AS props
  FROM events_incr
""").filter("actor_key IS NOT NULL")

out.isEmpty()   # raises the error above

The plan for the first query is below. Under the outer view, vw_period is not expanded into its SELECT DISTINCT. It shows up as RelationV2 and is read by RemoteSparkConnectScan, while the outer view is expanded normally.

== Parsed Logical Plan ==
'Project [*]
+- 'Filter ((activity_type IN (...) AND (occurred_at > 2026-10-13 09:02:00.404))
            AND ((occurred_at <= 2026-10-19 12:46:43.923587)
                 AND (COALESCE(is_automated, false) = false)))
   +- 'UnresolvedRelation [app_dev, vw_activity_events], [], false

== Analyzed Logical Plan ==
Project [...]
+- Filter ((activity_type IN (...) AND (occurred_at > ...)) AND ((occurred_at <= ...) AND (coalesce(is_automated, false) = false)))
   +- SubqueryAlias acme_prod.app_dev.vw_activity_events
      +- View (acme_prod.app_dev.vw_activity_events, [...])
         +- Project [...]
            +- Filter NOT (activity_type = heartbeat)
               +- Join LeftOuter, between(cast(occurred_at as date), period_start, period_end)
                  :- SubqueryAlias acme_prod.app_dev.activity_events
                  :  +- Relation acme_prod.app_dev.activity_events[...] parquet
                  +- SubqueryAlias p
                     +- SubqueryAlias acme_prod.app_dev.vw_period
                        +- RelationV2[period_key, period_label, period_start, period_end, period_tz] acme_prod.app_dev.vw_period

== Optimized Logical Plan ==
Project [...]
+- Join LeftOuter, ((cast(occurred_at as date) >= period_start) AND (cast(occurred_at as date) <= period_end))
   :- Project [...]
   :  +- Filter (activity_type INSET (...) AND occurred_at > ... AND occurred_at <= ... AND NOT coalesce(is_automated, false))
   :     +- Relation acme_prod.app_dev.activity_events[...] parquet
   +- RelationV2[period_key, period_start, period_end] acme_prod.app_dev.vw_period

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- BroadcastNestedLoopJoin BuildRight, LeftOuter, ((cast(occurred_at as date) >= period_start) AND (cast(occurred_at as date) <= period_end)), true
   :- FileScan parquet acme_prod.app_dev.activity_events[...]
   :     Batched: false
   :     Location: PreparedDeltaFileIndex(1 paths)[s3://example-bucket/dev/tenant/master-data/activity_events]
   :     PartitionFilters: [isnotnull(activity_type), NOT (activity_type = heartbeat), activity_type INSET (...)]
   :     PushedFilters: [IsNotNull(occurred_at), GreaterThan(occurred_at, ...), LessThanOrEqual(occurred_at, ...)]
   +- Exchange SinglePartition, EXECUTOR_BROADCAST
      +- Project [period_key, period_start, period_end]
         +- RemoteSparkConnectScan acme_prod.app_dev.vw_period[period_key, period_start, period_end]
              PushedPredicates: [period_start IS NOT NULL, period_end IS NOT NULL]
              Runtime row count reported for this side: 5
              Size: 160 bytes

What I tried

Setting spark.sql.ansi.enabled to false made no difference.
Recreating both views, and using fully qualified names, made no difference. DESCRIBE EXTENDED on both views shows the correct catalog and namespace.
Calling localCheckpoint(eager=True) on the first DataFrame, before the second step, lets the whole job run on DBR 18. That is our workaround for now.
A second job joins another Delta table to vw_activity_events and adds a transform() over an array column in the same query. It fails the same way. Splitting the join from the transform() with a checkpoint avoids it. I haven’t isolated which expression triggers it.

Is this a known issue in DBR 18? Why is vw_period planned as RelationV2 and read through RemoteSparkConnectScan instead of being expanded, and what makes the second query’s scan fail when the first query’s succeeds? Is there a setting or recommended pattern to use until a fix is out?

CUSTOMER STORY | VisionHeight runs real-time queries ~3x faster, for 90% less

Tushar_Parekar — Mon, 19 Oct 2026 11:39:12 GMT

“Once we got Lakehouse//RT, it was very fast. No extracting data out, no maintaining infrastructure, and about a 90% cost reduction. It all just works in Databricks now.” - Nitsan Daniel, CTO and Co-founder, VisionHeight

VisionHeight is a pre-attack intelligence platform that helps cybersecurity teams detect threats before they strike by continuously mapping adversary infrastructure. Its customer-facing query experience previously relied on Apache Pinot, requiring a separate always-on cluster, data-copy pipelines and ongoing schema and infrastructure maintenance. By moving the workload to Lakehouse//RT, VisionHeight brought real-time serving onto its existing Databricks foundation.

Key highlights:

90%+ cost reduction: Serverless autoscaling replaced always-on Apache Pinot infrastructure.
Nearly 3× faster queries: VisionHeight’s heaviest query now returns in sub-second response times.
Production in three days: The team moved from evaluation to production in two to three days.
Zero data migration: Queries run directly against data already stored in the lakehouse, with no copying or new ingestion pipelines.
One platform instead of two: Data processing, machine learning, analytics, batch jobs and real-time serving now run together on Databricks.
Less operational overhead: VisionHeight no longer maintains a separate serving cluster or troubleshoots schema mismatches between platforms.

🔗 Check out the full story 👈

Your Genie Agent Passed. Who's Watching Production?

MouR — Mon, 19 Oct 2026 10:47:57 GMT

Five controls to put in place before rolling out Databricks Genie Agents.

The benchmarks passed. Business users are ready. But who checks failed questions, approves updates, watches query performance, and manages spending after go-live?

For a Databricks Genie Agent (formerly a Genie Space), I would put five operational controls in place before expanding access.

1. Release the agent from source control

Declarative Automation Bundles support Genie Agents through the genie_spaces resource. Keep the serialized definition in a version-controlled .geniespace.json file, with workspace-specific warehouse IDs and permissions configured per environment.

Use databricks bundle validate, databricks bundle plan, and databricks bundle deploy in CI/CD. This resource requires Databricks CLI 1.3.0 or later and the direct deployment engine. Cloning is useful for experiments; it is not a release process.

2. Make staging a release gate

Rerun benchmarks against staging data. Compare the generated SQL and returned results with verified answers. Test with restricted business-user permissions, including Unity Catalog row filters and column masks. Check the warehouse and data references before approving deployment.

A passing benchmark under an administrator’s access does not prove the same agent works safely for every user.

3. Watch the questions users actually ask

Use the Genie Agent Monitor tab for conversations, feedback, review requests, and the weekly digest. Analyze Space Usage can help surface repeated questions and gaps in agent context.

For operational signals, query system.access.audit for Genie activity, system.query.history for slow or failed SQL, and system.billing.usage for consumption. Alert on repeated errors, query delays, and unusual spend. Feedback does not automatically retrain Genie; someone must review it and retest changes.

4. Track two costs, not one

In Account Console → Usage → Budgets, create a Genie budget with resource type Unity Gateway and tag databricks-product: genie. Configure shared and per-user thresholds, group overrides, and alerts for billable usage. Blocking is approximate, not a guaranteed cap.

As of October 2026, interactive Genie One and Genie Agents usage is free through January 31, 2027, and budget controls do not apply to that promotional usage. Service-principal usage is billed. SQL warehouse compute is billed separately and is not covered by Genie LLM budgets.

5. Assign a production owner

Name the person or team responsible for feedback review, benchmark changes, access approvals, cost alerts, and rollback. Keep the previous agent configuration in Git. Consider Certified status after your internal approval process, and review production usage every week.

Benchmarks tell you how an agent performed on known questions. Release controls and monitoring show how it behaves when the questions change.

#Databricks #GenieAgents #DataEngineering #DataGovernance #DevOps

Lakebase Search: Vector, Keyword, and Hybrid Retrieval Inside Databricks Postgres

Isaac_18 — Mon, 19 Oct 2026 10:19:59 GMT

Lakebase Search brings vector search and BM25 keyword search directly into Lakebase, Databricks’ managed, serverless PostgreSQL offering.

It introduces two PostgreSQL extensions:

lakebase_vector — Approximate nearest-neighbor (ANN) vector search, compatible with pgvector.
lakebase_text — BM25 keyword search, compatible with PostgreSQL tsvector.

Together, these extensions enable semantic, keyword, and hybrid retrieval using SQL in the same database that stores your operational data.

In this post, we’ll explore how Lakebase Search works, when to use each retrieval mode, and how to build a working search implementation with SQL.

1. The Problem: Search Often Requires Multiple Systems

Modern applications commonly need three types of search:

Search mode	Question it answers	Example query
Keyword	Which records contain these exact terms?	SKU-12345, an error code
Semantic (vector)	Which records have a similar meaning?	“A fantasy book about a boy wizard”
Hybrid	Which records are relevant by both meaning and keywords?	“Harry Potter magic school adventure”

PostgreSQL has long provided reliable transactional capabilities, built-in full-text search, and vector search through extensions such as pgvector.

However, at scale, teams often operate three separate systems:

A transactional database for application data
A keyword search engine, such as Elasticsearch or OpenSearch
A vector database for semantic retrieval

This architecture introduces several challenges:

Data synchronization: Multiple copies of the same data must remain consistent, and delayed pipelines can produce stale search results.
Operational complexity: Teams must manage multiple query languages, APIs, indexes, and compute resources.
Multi-step retrieval: Applications may need to generate embeddings, query multiple search systems, merge results in application code, and retrieve complete records from PostgreSQL.
Additional infrastructure costs: Separate systems introduce duplicated storage and operational overhead.

Lakebase Search aims to simplify this architecture by bringing keyword and vector retrieval into the same PostgreSQL database, using SQL as the common interface.

2. What Does Lakebase Search Provide?

Lakebase Search introduces two extensions that support different retrieval strategies.

Extension	Purpose	Index type
lakebase_vector	ANN vector search with pgvector-compatible types, operators, and syntax	lakebase_ann
lakebase_text	BM25 keyword search using PostgreSQL tsvector	lakebase_bm25

Vector search

The vector search index uses IVF partitioning with RaBitQ quantization.

Key characteristics include:

Storage-backed indexes that persist across scale-to-zero events.
Support for large vector indexes.
Faster index-building potential compared with HNSW-based approaches, depending on workload and configuration.
Familiar pgvector-compatible vector types and operators.

Databricks positions the technology for very large indexes, including workloads with more than one billion vectors. Consult the official documentation for benchmark results and the conditions under which these performance claims apply.

Keyword search with BM25

BM25 is a ranking algorithm that considers term frequency, document length, and corpus-wide statistics.

PostgreSQL’s built-in ts_rank does not use corpus-wide inverse document frequency statistics in the same way. This can affect relevance ranking for large document collections.

Lakebase Search also uses Block-Max WAND to efficiently retrieve top-ranked results without scoring every matching document.

Hybrid search

Hybrid search combines vector similarity and keyword relevance in a single SQL statement.

There is no separate hybrid operator. Instead, you can retrieve candidates from both search methods and combine their rankings using an algorithm such as Reciprocal Rank Fusion (RRF).

We’ll implement this in Section 6.

3. Before You Start

Before enabling Lakebase Search, check the following prerequisites:

PostgreSQL version: PostgreSQL 16 or later.
Feature availability: Lakebase Search is a Beta feature.
Permissions: A workspace administrator must enable the feature through the Previews page.
Project configuration: Enable Lakebase Search under Settings → Lakebase Search in your project.

Important: Enabling Lakebase Search is irreversible according to the feature guidance described here and restarts all computes in the project, dropping active connections. Test it in a disposable project before enabling it in a production environment.

After enabling the feature, install the extensions:

CREATE EXTENSION IF NOT EXISTS lakebase_vector CASCADE;
CREATE EXTENSION IF NOT EXISTS lakebase_text;

The CASCADE option installs the required pgvector dependency for lakebase_vector.

4. Vector Search with lakebase_vector

Vector search retrieves records based on the similarity between numerical embeddings.

Each record stores an embedding in a VECTOR(n) column, where n must match the embedding model’s output dimension. The table column, index configuration, and query vector must all use compatible dimensions and distance metrics.

Choosing a distance metric

Metric	Operator class	Operator	Typical Use case
Cosine distance	vector_cosine_ops	<=>	Text embeddings and semantic search
L2 (Euclidean) distance	vector_l2_ops	<->	Workloads where absolute geometric distance matters
Negative inner product	vector_ip_ops	<#>	Similarity ranking, especially with normalized vectors

Choose the metric based on how your embedding model is intended to be used. The index operator class and query operator must match.

Step 1: Create a table

CREATE TABLE items (
    id        BIGSERIAL PRIMARY KEY,
    embedding VECTOR(3)
);

The three-dimensional vector is used here only to keep the example small. In a real application, use the dimension produced by your embedding model, such as 384 or another supported dimension.

Step 2: Insert sample embeddings

INSERT INTO items (embedding)
SELECT ARRAY[random(), random(), random()]::real[]
FROM generate_series(1, 1000);

This generates synthetic vectors for demonstration purposes. It does not generate meaningful semantic embeddings.

Step 3: Create an ANN index

CREATE INDEX ON items
USING lakebase_ann (embedding vector_l2_ops);

For larger datasets, building the index after bulk-loading data can be more efficient than creating it before the initial load.

Step 4: Query the nearest neighbors

SELECT *
FROM items
ORDER BY embedding <-> '[3,1,2]'
LIMIT 5;

This query returns the five nearest vectors using L2 distance.

Tuning and utility functions

Lakebase Search exposes options for managing index-building performance and query-time recall.

build_mode = 'fast': Can accelerate index builds at the expense of recall. The standard build mode prioritizes recall.
lakebase_ann.probes: Increasing the number of probes can improve recall, generally at the cost of query latency.
lakebase_ann.epsilon: Controls the re-ranking margin. The documented range is 0.0–4.0, with a default of 1.9.
lakebase_ann_prewarm(regclass): Loads an index into memory to help reduce cold-start latency.
lakebase_ann_index_info(regclass): Returns index metadata.

Verify the supported settings and their exact configuration syntax in the documentation for your Lakebase version.

5. Keyword Search with BM25

Vector search is useful for semantic similarity, but exact identifiers, product codes, error messages, and terminology often benefit from keyword retrieval.

Lakebase Search provides BM25 ranking through the lakebase_text extension.

Step 1: Create a table with a generated tsvector column

CREATE TABLE documents (
    id      SERIAL PRIMARY KEY,
    passage TEXT,
    vector  TSVECTOR GENERATED ALWAYS AS (
        to_tsvector('english', passage)
    ) STORED
);

The generated column automatically maintains the text-search representation when the underlying passage changes.

Step 2: Insert sample documents

INSERT INTO documents (passage)
VALUES
    ('Postgres is a powerful open-source relational database.'),
    ('Vector search finds semantically similar results.'),
    ('BM25 ranking improves full-text search relevance scores.');

Step 3: Create a BM25 index

CREATE INDEX documents_passage_bm25
ON documents
USING lakebase_bm25 (vector);

Build the index after loading the initial dataset so that the index can calculate corpus statistics from the populated collection.

Step 4: Execute a keyword query

SELECT
    id,
    passage,
    vector <@> to_bm25query(
        to_tsvector('english', 'database'),
        'documents_passage_bm25'
    ) AS score
FROM documents
ORDER BY score
LIMIT 5;

The <@> operator returns a negative BM25 score, so ascending order places the most relevant results first.

Operational consideration: BM25 ranking depends on corpus statistics calculated during index construction. Review the documented behavior for inserts and updates, and determine whether your workload requires index maintenance or rebuilding to keep ranking quality aligned with the evolving dataset.

6. Hybrid Search with Reciprocal Rank Fusion (RRF)

Vector and keyword retrieval produce scores on different scales. Combining their raw scores directly can therefore be misleading.

Reciprocal Rank Fusion avoids this issue by combining the rank positions of documents rather than their raw scores.

The RRF formula is:

RRF⁡(d)=1k+rv(d)+1k+rk(d)\operatorname{RRF}(d) = \frac{1}{k+r_v(d)} + \frac{1}{k+r_k(d)}

Where:

dd is a document.
rv(d)r_v(d) is its rank in the vector results.
rk(d)r_k(d) is its rank in the keyword results.
kk is a smoothing constant, commonly set to 60.

A document appearing near the top of both result lists receives a stronger combined score.

Step 1: Create a table for both retrieval methods

CREATE TABLE docs (
    id        SERIAL PRIMARY KEY,
    title     TEXT NOT NULL,
    body      TEXT NOT NULL,
    embedding VECTOR(3),
    body_tsv  TSVECTOR GENERATED ALWAYS AS (
        to_tsvector('english', body)
    ) STORED
);

Step 2: Insert sample documents

INSERT INTO docs (title, body, embedding)
VALUES
    (
        'Postgres overview',
        'Postgres is an open-source relational database.',
        '[0.1, 0.2, 0.3]'
    ),
    (
        'Vector search guide',
        'Vector search finds semantically similar results.',
        '[0.4, 0.5, 0.6]'
    ),
    (
        'Full-text search',
        'BM25 ranking improves keyword search relevance.',
        '[0.7, 0.8, 0.9]'
    );

These embeddings are illustrative and are not intended to represent actual semantic relationships.

Step 3: Create both indexes

CREATE INDEX ON docs
USING lakebase_ann (embedding vector_cosine_ops);

CREATE INDEX docs_body_bm25
ON docs
USING lakebase_bm25 (body_tsv);

Step 4: Retrieve and fuse the results

The following query retrieves candidates from both methods, assigns ranks, and combines them using RRF.

WITH vector_ranked AS (
    SELECT
        id,
        RANK() OVER (ORDER BY dist) AS rank
    FROM (
        SELECT
            id,
            embedding <=> '[0.1, 0.2, 0.3]' AS dist
        FROM docs
        ORDER BY dist
        LIMIT 40
    ) v
),
keyword_ranked AS (
    SELECT
        id,
        RANK() OVER (ORDER BY score) AS rank
    FROM (
        SELECT
            id,
            body_tsv <@> to_bm25query(
                to_tsvector('english', 'database'),
                'docs_body_bm25'
            ) AS score
        FROM docs
        ORDER BY score
        LIMIT 40
    ) k
)
SELECT
    d.id,
    d.title,
    COALESCE(1.0 / (60 + v.rank), 0)
      + COALESCE(1.0 / (60 + k.rank), 0) AS rrf_score
FROM docs d
LEFT JOIN vector_ranked v
    ON d.id = v.id
LEFT JOIN keyword_ranked k
    ON d.id = k.id
WHERE v.id IS NOT NULL
   OR k.id IS NOT NULL
ORDER BY rrf_score DESC, d.id
LIMIT 10;

How the query works

Vector retrieval: Finds the top 40 candidates by cosine distance.
Keyword retrieval: Finds the top 40 candidates by BM25 relevance.
Rank calculation: Assigns each candidate a position in its respective result list.
Rank fusion: Adds the reciprocal-rank contributions from both lists.
Final ranking: Returns the highest-scoring documents.

The LEFT JOIN and COALESCE expressions ensure that a document found by only one retrieval method still contributes to the final ranking.

The d.id tie-breaker makes result ordering deterministic for equal RRF scores, although stable pagination across changing data may still require an appropriate snapshot or cursor strategy.

Tuning tip: The candidate limit of 40 and the RRF constant of 60 are starting points, not universal defaults. Evaluate them against your dataset, relevance requirements, recall targets, and latency budget. Weighted fusion is another option when one retrieval method should contribute more strongly than the other.

7. Bringing Lakehouse Data into Lakebase with Synced Tables

You do not need to write every record directly into PostgreSQL.

Lakebase synced tables allow you to replicate a Unity Catalog Delta table into Lakebase, making lakehouse data available to operational applications and search workloads.

Available synchronization modes

Mode	Behavior	Change Data Feed required	Best suited for
Snapshot	One-time full copy	No	Initial setup and smaller datasets
Triggered	Updates on demand or on a schedule	Yes	Periodic refreshes
Continuous	Ongoing, near-real-time synchronization	Yes	Applications requiring frequent updates

For modes that require Change Data Feed (CDF), enable it on the source Delta table:

ALTER TABLE your_table
SET TBLPROPERTIES (
    delta.enableChangeDataFeed = true
);

Once the synced data is available in Lakebase:

Create the required vector and text-search columns.
Generate embeddings outside PostgreSQL, for example with Databricks Model Serving or ai_query, where supported by your workflow.
Store embeddings in a VECTOR(n) column.
Create the appropriate lakebase_ann and lakebase_bm25 indexes.
Validate synchronization behavior and search freshness.

An important design principle: Search freshness is constrained by the freshness of the underlying synchronized data and any downstream embedding or indexing work. Select the synchronization mode based on your application’s latency requirements.

8. When Should You Use Each Search Mode?

Use case	Recommended approach	Why
RAG over internal documents	Vector or hybrid	Retrieves semantically relevant passages, with hybrid search adding exact-term matching
Product and e-commerce search	Hybrid	Combines descriptions and semantic similarity with product names and SKUs
Similar historical support tickets	Vector	Finds issues with similar meaning even when wording differs
Source code, identifiers, and error codes	Keyword	Exact terminology and identifiers matter
Recommendations based on similarity	Vector	Finds items with similar embedding representations

These are starting points. The best approach depends on your data, query patterns, and relevance evaluation results.

9. Trade-Offs to Consider

Lakebase Search simplifies the search architecture, but it does not eliminate the need for careful system design.

Beta status and rollout risk

The feature is in Beta, and enabling it is described as irreversible. Test it in a separate project and confirm the current feature limitations before rolling it out.

Shared compute resources

Search queries and transactional operations share the database environment. Heavy indexing or retrieval workloads can affect application latency, so benchmark realistic workloads and size your compute accordingly.

Search-engine feature differences

Lakebase Search is not necessarily a drop-in replacement for every Elasticsearch or OpenSearch capability.

Evaluate requirements such as:

Faceted navigation and aggregations
Fuzzy matching and phrase search
Language-specific analyzers
Highlighting and snippets
Filtering and complex ranking logic

Embedding lifecycle management

Your application is responsible for producing and maintaining embeddings. The vector dimension must match the model output and schema, and switching embedding models generally requires regenerating embeddings and rebuilding or updating the relevant indexes.

Lakebase Search versus Databricks Vector Search

If your primary requirement is a managed vector index over Delta tables, evaluate Databricks Vector Search as well.

Lakebase Search is especially worth considering when retrieval needs to live alongside operational PostgreSQL data. The right choice depends on your data architecture, query requirements, latency targets, and operational constraints.

10. Quick Reference

Task	SQL
Install vector extension	CREATE EXTENSION IF NOT EXISTS lakebase_vector CASCADE;
Install text extension	CREATE EXTENSION IF NOT EXISTS lakebase_text;
Define vector column	embedding VECTOR(384)
Define text-search column	body_tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english', body)) STORED
Create ANN index	CREATE INDEX ON tbl USING lakebase_ann (embedding vector_cosine_ops);
Create BM25 index	CREATE INDEX idx ON tbl USING lakebase_bm25 (body_tsv);
Run vector search	SELECT * FROM tbl ORDER BY embedding <=> '[...]' LIMIT 10;
Run BM25 search	SELECT *, body_tsv <@> to_bm25query(to_tsvector('english', 'q'), 'idx') AS score FROM tbl ORDER BY score LIMIT 10;

Conclusion

Lakebase Search brings vector, keyword, and hybrid retrieval into the PostgreSQL database that already holds your operational data.

By combining lakebase_vector and lakebase_text, applications can use SQL to retrieve semantically similar content, rank keyword matches with BM25, and combine both result sets with Reciprocal Rank Fusion.

For workloads where search and transactions operate on the same data, this can reduce infrastructure complexity and eliminate some of the synchronization work associated with maintaining separate search systems.

However, Lakebase Search is still in Beta. Before adopting it for production workloads, validate its relevance, recall, latency, indexing behavior, and operational trade-offs against your own dataset.

Have you tried Lakebase Search? Share your implementation experience, performance results, and questions in the comments.

Author: Isaac Pandraju Feature status: Beta Last reviewed: October 2026

SCIM not returning account-level groups for Databricks App users

juan_maedo — Mon, 19 Oct 2026 09:40:00 GMT

We’re building an Azure Databricks App for a company use case, designed as a self-service tool for a group of non-technical users (around 100-200).

The main idea is that these users don’t have any permissions on the workspace. Access is granted only through an Entra ID group synced at the account level. This group is granted the CAN USE permission directly on the app through the Databricks Asset Bundle used for deployment, so users can log in successfully, and we are correctly capturing and recording their emails in the insights and logs. The Entra ID groups are not nested.

The problem comes with segregating sections or functionalities within the app. We can’t resolve the users’ security groups, because both the groups and the users exist only at the account level (the groups are added in the account groups section).

So far we have tried the workspace-level SCIM endpoint /api/2.0/preview/scim/v2/Me and the account-level SCIM endpoint /api/2.0/accounts/{account_id}/scim/v2/Users, both with the user’s OBO token (X-Forwarded-Access-Token) and with the app service principal token. We also tried making the app identity a workspace admin and granting it CAN MANAGE on the account group. None of these return the users’ group membership, so we can’t implement the segregation properly.

However, when a user has also been added to the workspace through other technical workspace groups, SCIM does return their group membership. Our hypothesis is that the workspace-level SCIM API only returns groups assigned to the workspace, so account-only groups are not visible to the app, even when the app identity has elevated permissions.

We are trying to avoid adding the users’ groups to the workspace, to keep things simple and maintain a scalable permissions model. Also, have a table of groups member its not maintainable for us their sync.

Is there a supported way for a Databricks App to resolve the account-level group membership of the current user without assigning those groups to the workspace?
Which api or permission of the App Identity could resolved this problem?
If any its possible, what is the recommended pattern for role-based access within a Databricks App for account-only users?

Databricks DiscountVoucher

TheVinhBui — Mon, 19 Oct 2026 09:19:36 GMT

When does Databricks send email Discount Voucher? Because I'm waiting for today and not get it?

Thank you.

Databricks Genie One - Managing past conversation initiated by Databricks service principal account

ken_npk — Sun, 18 Oct 2026 14:32:19 GMT

Hello everyone.

Recently, our team use Databricks service principal account, connecting to Genie One MCP server (via unity gateway) for data retrieval. However, we notice that there are currently no ways for us to manage the past conversations that had happened with service principal.

We not able to find SDKs or APIs to list all or perform deletion on past conversations initiated by service principal. As far as we know, there is no way for us to login as service principal and perform deletion via Genie One interface. We also noted that those functionalities only available for Genie Agent.

This limitation is problematic for us. In some cases, Genie One has misinterpreted our data during its reasoning process. Although we have corrected the relevant issues in the data semantic layer, Genie One continues to rely on past conversations containing incorrect interpretations when answering user questions. Since we are unable to delete these past conversations, it is difficult for us to prevent Genie One from reusing misleading context.

Are there any future plans on having a way for listing all or performing deletion on past conversations that were done by service principals? Or are there any alternative ways to manage the past conversations by service principals?

Appreciate any insights and suggestions from Databricks community. Thank you.

Why Your Genie Space Gives Wrong Answers: A Debugging Playbook

aayush_410 — Sun, 18 Oct 2026 08:26:03 GMT

Every team that builds a Databricks Genie space goes through the same moment. The demo went well, the first few questions came back clean, and then a stakeholder asks something slightly different and the number is wrong. Not obviously wrong, which would be easy. Plausibly wrong, which is worse.

The instinctive reaction is to open the instructions box and type another rule. Sometimes that works. Often it makes things quietly worse, because the real problem was somewhere else. This post is a debugging playbook for that moment. It gives you a repeatable way to work out why Genie produced a bad answer, which of your levers actually fixes it, and how to confirm the fix did not break something else.

The core idea is simple: treat a wrong answer the way you would treat a failing test. Reproduce it, isolate the cause, apply the smallest fix that addresses it, and check for regressions.

Start With the SQL, Not the Answer

Genie shows you the query it generated. That query is your stack trace. Before you change anything, read it carefully and compare it with the query you would have written yourself.
When you read the SQL, ask a fixed set of questions:

Did it pick the right table, or a table that merely looks right?
Are the joins correct, including the join keys and the join type?
Did it filter on the right columns, and are the filter values the ones the data actually contains?
Which date column did it use, and is that the one the business means?
Did it aggregate at the right grain, or did it double count by joining before aggregating?
Did it apply the business definition of the metric, or invent a plausible one?

Write down the first place where the generated query diverges from the correct one. That single divergence is the bug. Everything else in this post is about figuring out why Genie made that choice and how to change it.
It also helps to run the correct query yourself and compare results. If you cannot say what the right answer should be, you are not ready to debug the space yet. Find a domain expert or a trusted report first.

The Four Root Causes

Almost every wrong answer traces back to one of four root causes. Learning to tell them apart is the most valuable skill in running a Genie space, because each one has a different fix.

A metadata problem. The tables and columns are named or described in a way that misleads Genie.
A definition problem. Genie does not know how your business defines a term or calculates a metric.
A pattern problem. The logic is complex, and Genie keeps assembling it slightly wrong even though it understands the terms.
A scope problem. The question is outside what the space was designed to answer, or the data to answer it is not there.

The sections below take each one in turn, with symptoms, examples, and fixes.

Root Cause One: Metadata Problems

Genie reads your table names, column names, data types, and comments to decide what to query. If those are cryptic, duplicated, or misleading, Genie has to guess, and guesses are where wrong answers come from.

Symptoms.

Genie picks a column that sounds right but is not, such as order_date versus ship_date versus created_at.
Two tables hold similar data and Genie chooses the wrong one.
Genie filters on a status code and gets no rows because it guessed the wrong value.
Results change depending on how the question is phrased, in ways that do not make business sense.

Example. A user asks for revenue last quarter. The sales table has three amount columns: amt, amt_net, and amt_adj. None has a comment. Genie picks amt, which includes tax and refunds. The number is wrong, and nothing in the question could have told Genie which column was right.

Fixes.

Add comments to tables and columns in Unity Catalog. Write them for someone who has never seen your system. Say what the column means, what unit it uses, and when to prefer it over similar columns.
Rename or alias confusing columns in a curated view. A view called sales_curated with clean names is often better than exposing the raw table.
Remove columns Genie should never use. If a legacy column is deprecated, hide it from the space or the view rather than hoping Genie avoids it.
Document allowed values for status and category columns, for example status is one of open, closed, cancelled.
Make relationships explicit so join keys are clear.
How to confirm. Rerun the failing question and check that the generated SQL now uses the right column. Then rerun two or three similar questions to make sure the fix generalizes.
Metadata fixes are the cheapest and most durable. They also help every future question, not just the one you are debugging. If you take only one lesson from this post, make it this: fix the data documentation before you touch the instructions.

Feature Request: Cross-job dependencies with business-date awareness

indradbxlearn — Sun, 18 Oct 2026 08:19:09 GMT

Summary
Workflows can chain tasks inside one job, but there is no native way to say: “Run job X once per day, after jobs A, B, C, D and E have each completed successfully for the same business date, even though they are independent jobs with their own schedules.” We’d like native support for this.

The scenario

We have 5 dimension jobs: dimjob1 to dimjob5, each owned and scheduled independently.
We have one fact job, factjob1, which must run once per day, only after all 5 dimension jobs have succeeded for the current business date.
The dimension tables are also updated throughout the day by other jobs and manual processes. Those updates must not trigger or satisfy factjob1. Only the completion of dimjob1-5 counts.

Enterprise schedulers such as Tidal, Control-M and Autosys handle this with a simple job-level dependency.

Our current workaround

Each dimension job writes a completion record (job name, business date, status) to a custom control table at the end.
factjob1 has a gate task that checks all 5 records for today’s business date.
An If/else condition task acts on the gate’s result.
factjob1 is scheduled on a polling interval (e.g. every 15-30 minutes) because we can’t know when the dimension jobs will finish.
An “already succeeded today” guard prevents duplicate runs.
Custom alerting handles the case where dependencies aren’t met by the deadline.

Alternatives don’t fit either. A “Run Job” orchestrator forces us to merge the independent schedules and ownership into one pipeline, and table update triggers fire on any write, including the manual and ad hoc loads we need to ignore.

Why this is a problem

Six custom components to express one dependency rule.
Fragile: every dimension job must write its completion record correctly.
Rebuilt by every team in a slightly different way.
Polling wastes job runs and compute just to ask “are they done yet?”
Dependency status is invisible in the Workflows UI, so support teams can’t see why factjob1 hasn’t started.
Teams migrating from Tidal or similar tools have to redesign their dependencies, not just re-point them.

Proposed solution
Native job-level dependencies on a job’s trigger, for example (syntax is only a suggestion):

yaml

trigger:
  job_dependencies:
    jobs: [dimjob1, dimjob2, dimjob3, dimjob4, dimjob5]
    condition: ALL_SUCCEEDED
    window: DAILY              # business date / calendar window and timezone
    run_policy: ONCE_PER_WINDOW
    deadline: "08:00"          # alert if not satisfied by then

Key capabilities:

All-of-N job runs succeeded within a window (daily, or a custom business date).
Only job run outcomes count. Table writes from other processes are ignored.
Once-per-window execution, with no re-trigger on later reruns.
Support for independent schedules and owners, with no need to merge jobs.
Optional passing of the business date as a parameter to the downstream job.
SLA deadline alerting if dependencies aren’t met in time.
UI visibility of which upstream jobs are satisfied and which are pending.

Benefits

Six custom components shrink to a few lines of configuration.
No polling, no custom control tables, no convention for every job to follow.
Teams keep independent ownership and schedules.
Operations can see dependency status directly in the UI.
Migration from Tidal, Control-M and Autosys becomes much simpler.

Could you please log this as a feature request and let me know if it’s on the roadmap? Happy to share more detail or test an early version.

Per-agent guardrails: what do you alert on, tokens, retries, or cost per task?

DoTA — Sun, 18 Oct 2026 00:56:24 GMT

I'm trying to work out what a sensible guardrail looks like for each agent when several of them share a few serving endpoints.

A plain token budget feels too blunt. An agent that retries a failing tool four times looks like a heavy user, while a cheap agent stuck in a loop can burn far more than its neighbours before anyone notices.

The options I'm weighing:

1. A hard daily cap per agent on tokens, through AI Gateway rate limits.

2. An alert on tool invocations per request, to catch loops and retry storms early.

3. An alert on cost per completed task against that agent's own 7-day baseline.

For those running agents in production on Databricks: which of these do you actually use, and which one did you regret? And is anything in the system tables or inference tables good enough to build that baseline from, or do you end up logging your own per-task numbers?

Happy to hear partial answers too, even just what you stopped doing.

The Databricks Serverless Conversion Playbook

ShamenParis — Sat, 17 Oct 2026 18:07:16 GMT

The shift from classic compute to Serverless in Databricks is a game-changer, but the migration process requires a solid strategy. After diving deep into serverless conversions recently, I’ve put together a guided playbook.

Here are my top 5 takeaways for a smooth transition:

Audit & Standardize First Before moving anything, identify the standard patterns your company follows across notebooks, Python scripts, and SQL files. You need a clear baseline of your current architecture.
Bake Standards into a Databricks SKILL Take those identified standards and encode them into a custom Databricks SKILL. This turns your organizational knowledge into a reusable, automated asset for the migration.
Navigating the spark.conf.set Limitation Serverless doesn't support most spark.conf.set configurations (even custom ones). The Workaround: Pass configurations via widgets as an environment Python wheel stored in a Unity Catalog Volume, sharing it across teams. Alternatively, create a common environment notebook and inject it at the top of your jobs using %run.
Democratize the Conversion Once your SKILL is set up, share it across your data teams! They can use the SKILL and the standardized notebooks to independently check compatibility, convert their workloads, and test their jobs on Serverless compute.
Iterative Bulk Testing over "Big Bang" Don't try to convert everything at once. The best approach is domain-driven bulk testing. Take one repository or one specific data process, run the conversion, test the serverless execution thoroughly, and then scale the pattern.

I’ve compiled these steps and architectural diagrams into a full Playbook (attached below!).

Has your team started the move to Serverless yet?

#Databricks #DataEngineering #Serverless #ApacheSpark #DataArchitecture #DatabricksCommunity #CloudComputing

Palantir Foundry + Databricks: Move the Data or Just Read It?

MouR — Sat, 17 Oct 2026 17:50:04 GMT

Sometimes the cheapest migration is no migration at all.

An organization has already invested in Palantir Foundry, with established pipelines, validated data, Ontology models and operational applications. Another team wants to use the same data in Databricks.

Should they move everything, copy selected datasets, or simply read the data where it already lives?

Before migrating anything, consider what Databricks would add.

Why move something that already works?

If Foundry's Contour, Quiver, Workshop applications or existing BI integrations meet the business requirements, keeping those workloads in Foundry may be the most economical option.

If another team needs Foundry data alongside datasets already available in Databricks, federation may be sufficient.

Databricks can act as an additional analytics consumer without taking ownership of the source data, transformations or operational applications.

Federation vs. ingestion: Which costs less?

Databricks supports read-only federation of eligible Foundry-managed Apache Iceberg tables through Unity Catalog, currently in Public Preview.

This avoids building and maintaining a second ingestion pipeline. But federation isn't free. Databricks compute, cloud storage access and possible data transfer charges still apply.

Ingestion creates an independently managed copy, usually in Delta tables. It introduces storage, pipeline monitoring, reconciliation and ongoing maintenance costs.

Ingestion is worth considering when repeated transformations, large joins, ML workloads, independent writes or demanding reporting requirements justify those costs.

But what about latency?

Federation isn't automatically slower.

For supported Iceberg tables, Databricks reads the underlying data from cloud storage using its own compute. Performance depends on storage location, file layout, query complexity and concurrent demand.

For occasional analytics, federation may be entirely adequate. For frequently refreshed, heavily used dashboards, a curated local copy may provide more predictable performance.

The simplest test: Run representative queries both ways. Compare response time, data freshness, compute consumption and monthly operating costs.

If federation meets the performance target at an acceptable cost, why introduce another pipeline?

When does full migration make sense?

Full Foundry replacement is a separate business decision. It involves more than data movement: pipelines, Ontology relationships, actions, security rules and Workshop applications must also be considered.

Databricks Labs Lakebridge may assist with selected SQL assessment, conversion and validation tasks, but it does not provide a native one-click Foundry migration.

For platform consolidation, estimate the cost of rebuilding and operating the complete solution before committing to a move.

The takeaway

Keep validated workloads where they already deliver value. Use federation when Databricks only needs to consume the data. Ingest selectively when performance, processing or independent ownership justifies a copy.

Move the data because the workload needs it, not simply because another platform can store it.

How New Developers Learn Databricks Genie: A Practical Learning Path

aayush_410 — Sat, 17 Oct 2026 13:45:11 GMT

If you are new to the Databricks ecosystem, you have probably heard the name Genie in a demo, a keynote, or a hallway conversation. The pitch is simple and a little magical: ask a question about your data in plain English, and get back a table, a chart, and the SQL that produced it. For a new developer, that pitch raises as many questions as it answers. Is Genie a product for analysts or for engineers? Do I need to know machine learning? Where do I even start?
This post is a guided tour of how new developers actually learn Genie, and how you can do it faster. It is organized as a path rather than a feature list, because Genie is best learned in layers: first as a user, then as a builder, then as the person responsible for making it trustworthy, and finally as a developer who wires it into other systems. Each stage builds on the last, and skipping ahead is the most common reason people get frustrated.

What Genie Actually Is

Genie is the natural language interface of Databricks AI/BI. You create a Genie space, which is a curated workspace pointed at a set of Unity Catalog tables and views. Business users type questions, and Genie translates them into SQL, runs the SQL on a SQL warehouse, and shows the results. It also shows you the generated query, so nothing is hidden.
The important word there is curated. Genie is not a general chatbot that happens to know about your data. It is a system that behaves well in proportion to how much context you give it about your tables, your columns, your business terms, and your preferred ways of calculating things. A Genie space with five clean tables, good column descriptions, and a handful of example queries will outperform a space with fifty raw tables and no guidance, every time.
That single idea explains most of what new developers need to learn. The model is the easy part. The craft is in the setup.

Why Developers, Not Just Analysts, Should Learn It

Genie is marketed to business users, so developers sometimes assume it is not their problem. In practice, developers and data engineers end up owning it for three reasons.
First, Genie sits on top of Unity Catalog, which means the quality of your tables, naming, comments, and permissions directly shapes the quality of the answers. The people who build and govern those tables are usually engineers.

Second, Genie is increasingly something you build with, not just something you use. There is an API for running conversations programmatically, and Genie can be a component inside larger agent systems. If you are building internal tools, chat experiences, or automation, Genie becomes one of the pieces you call.
Third, when Genie gives a wrong answer, someone has to debug it. That person needs to read SQL, understand the data model, and know which knob to turn. That is a developer skill.
So the right framing is this: analysts consume Genie, but engineers make it good.

Stage One: Learn It as a User

The fastest way to start is to use a Genie space that someone else built. If your workspace has one, ask for access. If not, many Databricks workspaces include sample data, and the documentation walks through creating a space on top of it. Spend a few hours just asking questions.
While you do this, pay attention to four things.
Read the generated SQL every time. This is the single most valuable habit you can form. Genie shows its work, and the SQL tells you how it interpreted your question. When the answer looks right, check that the SQL matches your intent. When it looks wrong, the SQL usually shows you why: a wrong join, a misread column, a filter on the wrong date field.
Notice how phrasing changes results. Ask the same question three ways. Try asking for totals, then averages, then breakdowns by a category. Try vague questions and specific ones. You will quickly develop a feel for what Genie handles well and where it guesses.
Use follow-up questions. Genie keeps conversational context, so you can say things like break that down by region or now exclude returns. Learning to iterate this way is a different skill from writing one perfect query, and it is the way business users will actually work.
Collect the failures. Keep a running list of questions that gave wrong or confusing answers. This list becomes gold in Stage Four, when you start testing systematically.
By the end of this stage you should be able to explain, in two sentences, what Genie does and where it tends to go wrong. If you can do that, you are ready to build.
Stage Two: Build Your First Genie Space
Building a space is where learning accelerates, because you are forced to make the decisions that determine quality.
Start small. Pick one business domain, such as sales orders, support tickets, or inventory, and choose three to five tables that answer real questions in that domain. Resist the urge to add everything. A focused space is easier to reason about, easier to test, and cheaper to run.
Before you touch Genie itself, prepare the tables in Unity Catalog:
• Give tables and columns clear names. Cryptic abbreviations like cust_stat_cd force Genie to guess.
• Add comments to tables and columns. These descriptions are part of the context Genie reads, so write them for a reader who has never seen your system.
• Check data types and key relationships. If joins between tables are not obvious, document them or define the relationships.
• Make sure the people who will use the space have the right permissions. Genie respects Unity Catalog governance, so users only see data they are allowed to see.
Then create the space, attach your tables, and choose a SQL warehouse to run queries. Add a short description of what the space is for. Finally, add a few sample questions. These show up as starting points for users and also signal to Genie what kinds of questions are in scope.
Now test it. Ask the questions you collected in Stage One, plus the questions a real stakeholder would ask. Some will work. Some will not. That is the point. Every failure is a pointer to something you can improve, and the next stage is about how.
A useful exercise for new developers is to build the same small space twice: once with raw, undocumented tables, and once after cleaning up names and adding comments. The difference in answer quality is the most persuasive lesson in the whole subject, and no tutorial teaches it as well as seeing it happen on your own data.
Stage Three: Teach Genie Your Business
Once the basics work, you move from setup to teaching. Genie gives you several ways to add context beyond table metadata, and learning when to use each one is the core skill of a Genie developer.
General instructions. These are plain-language guidelines that apply to the whole space. They are the right place for rules like our fiscal year starts in February, revenue means net of refunds, or always exclude test accounts. Keep them short, specific, and unambiguous. Long, rambling instructions tend to hurt more than help.
Example SQL queries. When a question has a known correct answer, you can provide the question and the SQL that answers it. Genie uses these as patterns. This is especially powerful for questions that require non-obvious logic, such as a particular way of computing retention, or a multi-step calculation that should always be done the same way.
Reusable SQL expressions and functions. For metrics and definitions that appear repeatedly, such as active customer or gross margin, you can define them once and let Genie reuse them. This is how you stop every conversation from reinventing a key business definition slightly differently.
Trusted assets. Parameterized queries and functions that you have vetted can be marked as trusted. When a user's question matches one, the response can be flagged as coming from a verified source. For leadership-facing numbers, this distinction matters a great deal.
The skill to build here is diagnosis. When Genie gets something wrong, ask which kind of problem it is:
1. A metadata problem. The column names or comments are misleading. Fix the table documentation.
2. A definition problem. Genie does not know how your business defines a term. Add an instruction or a reusable expression.
3. A pattern problem. The logic is complex and Genie keeps getting it slightly wrong. Add an example query.
4. A scope problem. The question is outside what the space was designed to answer. Tell users, or expand the space deliberately.
New developers often reach for the same fix every time, usually piling more instructions into the general text box. Learning to match the fix to the failure is what separates a space that slowly improves from one that slowly becomes a mess.
Stage Four: Measure Quality Instead of Guessing
It is easy to tweak a Genie space, ask a few questions, and decide it feels better. It is much harder, and much more valuable, to know whether it is better.
This is where benchmarks come in. A benchmark is a set of questions paired with the SQL you consider the correct answer. You run the benchmark against your space, and Genie reports how its answers compare. Each time you change instructions, add an example, or edit table descriptions, you rerun the benchmark and see whether the score moved and which questions flipped.
A good way to build your first benchmark:
• Start with the failures you collected in Stage One and Stage Two.
• Add the ten to twenty questions your real users ask most often.
• Include a few deliberately tricky ones: ambiguous terms, edge cases, questions that need multiple joins.
• Write the reference SQL carefully, and have a domain expert review it. A benchmark with wrong answers is worse than no benchmark.
Also learn to use the feedback and monitoring tools. Users can mark answers as helpful or not, and space owners can review conversations to see what people are really asking. Real usage almost always surprises you. People ask questions you did not anticipate, use vocabulary you did not document, and expect the space to cover areas you never intended. Reviewing those conversations regularly is the cheapest source of improvement ideas you will ever have.
There is a mindset shift here worth naming. Traditional software has tests that pass or fail. A natural language interface has a pass rate that you push upward over time. Developers who accept that early, and who invest in evaluation before they invest in polish, tend to ship spaces that people actually trust.
Stage Five: Go Programmatic
Once you are comfortable building and evaluating spaces in the interface, the next step is treating Genie as a service you can call from code.
Databricks provides a Genie Conversation API that lets you start conversations, send messages, and retrieve results programmatically. That opens up a range of practical projects:
• A chat experience inside an internal web app or portal, so people can ask data questions without leaving the tool they already use.
• A Slack or Teams bot that answers common metric questions.
• Automated workflows that ask Genie a question on a schedule and route the answer somewhere useful.
• A component inside a larger agent that decides when a question needs structured data and hands it to Genie.
If you are learning the API, begin with the simplest possible script: authenticate, send one question to a space, wait for the response, and print the generated SQL and the result. Then add conversation follow-ups. Then add error handling for the cases where a query fails or returns nothing. Only after that should you think about wrapping it in an application.
A few practical lessons come up repeatedly. Authentication choices matter, because whether queries run as the end user or as a service identity determines what data comes back. Warehouse sizing and startup time affect how snappy the experience feels. And responses can be asynchronous, so your code needs to handle polling or waiting gracefully rather than assuming an instant answer.
Because Databricks is investing heavily in agent tooling, it is worth keeping an eye on how Genie fits into multi-agent setups, where one agent orchestrates several specialized ones and Genie handles the structured data questions. The details of those products change quickly, so treat the current documentation as your source of truth rather than any blog post, including this one.
Common Mistakes New Developers Make
After watching enough people learn Genie, the same mistakes show up again and again.
Adding too many tables. More tables mean more ambiguity. If two tables could plausibly answer the same question, Genie may pick the wrong one. Start narrow and expand only when you have a reason.
Skipping documentation. Developers who would never ship an undocumented API often ship an undocumented table to Genie. Column comments are not decoration. They are the interface.
Writing enormous instruction blocks. Instructions compete for attention. A dozen crisp rules beat three pages of prose. If an instruction is really a specific query pattern, make it an example SQL instead.
Trusting without verifying. Genie is good, not infallible. For anything that drives a decision, check the SQL, compare against a known number, or use a trusted asset. Teach your users the same habit.
Ignoring governance. Because Genie respects Unity Catalog permissions, a space is only as safe as the access controls behind it. Review who can see what before you share a space widely.
Treating launch as the finish line. A Genie space is a living thing. Data changes, the business changes, and users find new questions. Plan for ongoing maintenance from day one.
Where to Learn: A Realistic Resource Mix
No single resource will carry you through every stage, so mix a few.
The official Databricks documentation is the foundation. It is the most current source for how spaces are configured, what the instruction types are, and how the API behaves. Read the sections on setting up a space and on benchmarks closely.
Databricks Academy and the learning paths on the Databricks site offer structured courses on AI/BI and related topics, some of them free. They are useful for filling gaps in the surrounding platform, such as Unity Catalog and SQL warehouses, which Genie depends on.
The Databricks blog and community forums are where you find real-world patterns and war stories. Search for posts about improving Genie accuracy, and read the questions other developers ask. Seeing the problems other people hit is a shortcut to avoiding them.
Product release notes deserve a place on your reading list too. Genie evolves quickly, and features that were missing or in preview when you started may be generally available by the time you need them. When a tutorial seems out of date, the release notes will tell you why.
Finally, build something real. The most effective learning comes from pointing Genie at data you actually understand, such as a work dataset or a personal project, because you can instantly tell when an answer is wrong.
A 30-Day Plan for a New Developer
If you want a concrete schedule, here is one that works well.
Week one: use and observe. Get access to a Genie space or a sample workspace. Ask at least fifty questions. Read every generated query. Keep a log of failures and surprises. Skim the documentation overview so the vocabulary is familiar.
Week two: build. Choose one domain and three to five tables. Clean up names, add comments, and create your first space with a description and sample questions. Test it with the failures from week one and with questions from a real colleague.
Week three: teach and measure. Add general instructions, example queries, and reusable definitions based on the failures you see. Build a benchmark of at least fifteen questions and record a baseline score. Make changes one at a time and rerun the benchmark so you can tell which changes helped.
Week four: integrate and share. Write a small script against the Conversation API. Review conversations from your first real users. Decide what to improve next, and write down the lessons you have learned for the next person who joins the team.
By the end of those thirty days, you will have gone from curious to capable, with a working space, a measured quality baseline, and a handle on the programmatic side.
The Bigger Lesson
The most useful thing new developers learn from Genie is not about Genie at all. It is that natural language interfaces to data are a data quality and data modeling problem wearing a friendly disguise. The clearer your tables, definitions, and metrics are, the better any AI system will perform on top of them. The work you do to make Genie reliable, such as naming things well, documenting them, agreeing on definitions, and testing against known answers, also makes your analytics better for humans.
That is why learning Genie is worth the effort even if the product changes. The habits transfer: curate before you automate, show your work, measure instead of guessing, and treat quality as something you improve continuously.
Start small, read the SQL, and build one space you would be proud to hand to a colleague. The rest follows from there.