Get Started Discussions topics

Como funciona la facturación y pago en Databricks?

ResearcherPHD — Thu, 03 Sep 2026 02:59:36 GMT

Hola comunidad,

Tengo una duda sobre cómo funciona la facturación en cuentas Pay-As-You-Go y quería saber si a alguien más le ha pasado algo similar.

En mi panel de "Usage" veo un consumo acumulado de varios cientos de dólares durante el último año (principalmente Serverless SQL compute), tengo una tarjeta de crédito válida registrada como método de pago predeterminado, y en Settings > Subscription & billing no aparece ninguna alerta de pago fallido ni saldo pendiente. Sin embargo, nunca he visto un cargo de Databricks reflejado en mi tarjeta.

Mis preguntas:

1. ¿Es normal que el consumo se acumule sin generar un cargo mensual automático, o debería estar viendo cargos mes a mes?
2. ¿Existen créditos de bienvenida/trial que cubran el consumo por un tiempo antes de que se active el cobro real? Si es así, ¿dónde puedo ver ese saldo de créditos?
3. ¿Dónde puedo ver un historial real de facturas/pagos procesados (no solo de consumo) para confirmar si alguna vez se ha cobrado algo?

Adicionalmente, recientemente recibí un correo de "facturación" con apariencia sospechosa (remitente no oficial, datos de otra persona en el adjunto) pidiendo pagar a través de un portal de NetSuite, así que quiero entender bien qué es normal y qué no antes de asumir que debo algo.

Gracias de antemano a quien pueda orientarme.

Databricks Data Ingestion

anushnagesh — Wed, 02 Sep 2026 09:09:04 GMT

Hi all,

I am currently exploring the data ingestion feature offered by databricks, specifically connecting to a SQL server.
I have gone through the UI and configured an ingestion pipeline that connects to one specific server and can read tables into databricks.

I am currently venturing into understanding if we can make an ingestion pipeline dynamic.

What I mean by this is, I have created a yaml file that can be run to setup an ingestion pipeline. It accepts variables from a metadata config table that can be used to populate the YAML with information such as, connection server name, pipeline name, source catalog, source schema, source table, destination catalog, destination schema and destination table name.

So the question I have is, can we have a generic pipeline whose configurations can be altered?
Use cases:
1. Once a pipeline is created can it be edited in the future to ingest different tables? As in using pipeline_01, I ingested table_1, table_2 present in Server_01. Can I later pass different configs to the YAML such that I use the same pipeline_01 to ingest table_03, table_04?

2. Can I use pipeline_01 to connect to a different to a connection server and ingest tables from this new server?

Currently I am not finding a way to achieve this functionality. This will cause an issue in the future when I have say 50 servers. It will be difficult to maintain 50 different pipelines and I don't think this is desirable as well.

Can I get some insights into Data Ingestion feature and if this a viable option for my use case?

Thank you,
Anush

From Semantic Similarity to Business Authority: Why Genie Ontology and OntoRank Matter

amitsharma1707 — Wed, 02 Sep 2026 00:59:50 GMT

Enterprise AI does not usually fail because the model lacks intelligence.

It fails because the model does not understand what the organization means.

Consider a simple question:

“What is our current exposure to active customer?”

Behind this question are several business decisions:

What qualifies as an “active” customer?

Should exposure include committed, outstanding, or available amounts?

Which customer identifier is authoritative?

Should rebooked amount be consolidated?

Which system is trusted: the servicing platform, CRM, MDM golden record, or a reporting mart?

What business date should be used?

A traditional text-to-SQL system may identify tables with similar column names and generate syntactically correct SQL. But syntactically correct SQL can still produce a completely incorrect business answer.

This is the context gap that Databricks Genie Ontology is designed to address.

What is Genie Ontology?
Genie Ontology is a unified, continuously improving context layer that gives Genie a business-aware map of the organization.

It combines:

Human-modeled context

Certified data products, Unity Catalog metric views, domains, business definitions, Pages, and governed assets.

Automatically inferred context

Knowledge extracted from tables, queries, dashboards, SQL patterns, Genie Agents, and platform usage.

Instead of treating enterprise knowledge as disconnected metadata, the ontology represents relationships among:

Business terms

Metrics

tables and columns

dashboards

queries

data products

people and teams

business rules

Genie Agents

This changes the question from:

“Which asset looks most similar to the user’s prompt?”

to:

“Which permitted source represents the most authoritative meaning for this question?”

Where OntoRank becomes important
Enterprises rarely have only one definition of a metric.

There may be multiple definitions of revenue, customer, active account, gross margin, or credit exposure—each created by different teams, at different times, for different purposes.

OntoRank is the PageRank-inspired authority-ranking concept associated with Genie Ontology.

Rather than ranking only by textual similarity, the context layer can consider signals such as:

Source authority and provenance

Asset certification

Frequency and breadth of usage

Relationships with other trusted assets

Freshness

Business relevance

User permissions

For example, imagine Genie discovers three definitions of “active customer”:

An old spreadsheet definition created three years ago

A frequently queried but uncertified reporting view

A certified Unity Catalog metric connected to the MDM golden customer and current amount balances

Keyword similarity alone might retrieve any of them.

An authority-aware approach should prioritize the certified, governed, fresh, and widely connected definition—while still enforcing the requesting user’s permissions.

Why this is bigger than better text-to-SQL
The real architectural shift is:

Metadata → Semantics → Context → Trusted action

A well-designed ontology can help an AI system understand:

Which source should be queried

Which metric definition should be applied

Which relationships and joins are valid

Which conflicting definition should take precedence

Which assets are deprecated

What the user is authorized to access

Why a particular source was used

This can make AI systems more accurate, explainable, reusable, and governance-aware.

But OntoRank does not eliminate data governance
Authority ranking is powerful, but popularity is not always correctness.

A widely used definition may still be outdated. A newly created certified data product may initially have little usage history. Poorly documented tables will continue to produce weak context.

Therefore, organizations should prepare the foundation:

Define important business terms

Create governed metric views

Certify authoritative data products

Deprecate obsolete assets

Maintain table and column descriptions

Capture lineage

Assign clear data ownership

Improve MDM and identity resolution

Test Genie answers against approved business scenarios

Genie Ontology can amplify a strong semantic and governance foundation—but it cannot magically repair an undefined business vocabulary.

My key takeaway
The next generation of enterprise AI will not be differentiated only by model size.

It will be differentiated by the quality of the context surrounding the model.

RAG helps AI find similar information.
Ontology helps AI understand relationships and meaning.
OntoRank helps AI decide what should be trusted.
Unity Catalog helps ensure that trust remains governed.

The most important question for data architects may soon change from:

“How do we expose our data to an AI agent?”

to:

“How do we make business meaning discoverable, authoritative, permission-aware, and machine-readable?”

I would love to hear from the Databricks Community:

How are you preparing your Unity Catalog metadata and metric views for Genie Ontology?

How should OntoRank balance popularity against formal certification?

Should users be able to inspect the ontology graph and understand why one definition outranked another?

What evaluation framework are you using to measure the business accuracy of Genie answers?

Como aprender correctamente en el entorno de DataBricks y aprovecharlo al 100%

maycol25 — Tue, 01 Sep 2026 16:59:48 GMT

Hola a todos, soy ingeneriero en sistemas y deseo aprender a utilizar esta grandiosa herramienta para poder sacarle el mejor provecho posible.

LLM API Returning Inconsistent Responses When Using RAG

soumitradutta2 — Tue, 01 Sep 2026 07:00:00 GMT

Hi Everyone,

I am Soumitra Dutta,

I have an issue with LLM API and Retrieval-Augmented Generation (RAG). Sometimes the same query returns different answers even if the retrieved documents and the prompt are the same.

The retrieval results are useful, but sometimes the model does not include information from the context that was provided or generates information that does not appear in the source documents.

Have you had any problems with RAG? May it be structure, temperature, token limits or retrieval configuration?

If you have any tips for debugging and making the response be more consistent, it's appreciated.

Regards,
Soumitra Dutta

External location not accessble with job and general purpose cluster but works fine with serverless

Anmol_Chauhan — Mon, 31 Aug 2026 20:44:24 GMT

I m setting up a pipeline in databricks.

For data source I create an external location and add my pipeline run_as SP as read files and read metadata permission. When i trigger the pipeline with serverless cluster, i m able to read data from external location but when i use job cluster, my pipeline stuck at cluster initializing phase, cluster not started even after 1 hour. Then i tried my notebook with general purpose cluster, but processing stuck at reading external location cell of notebook.

I did not understand what i m missing. If there is an issue with access then how it works with Serverless? Please help if someone knows the solution.

Sample code i m using for reading:

df = spark.read.format("delta").load("abfss://container@storage.dfs.core.windows.net/path")

Path to become a Databricks Trainer

Mohit — Mon, 31 Aug 2026 14:35:36 GMT

Hello,

I am an independent consultant and a trainer in the data space, focused primarily on Databricks. I wanted to understand the path to becoming a Databricks trainer delivering ILT sessions on Databricks Academy or to Databricks partners.

It would be great if someone could share their experience and any prerequisites for applying to it.

Thank you!

Hospino : Hospital Management Software

thelogicplus — Fri, 21 Aug 2026 07:07:10 GMT

Have any one intergrated Hospino We’re exploring an integration between Hospino – Hospital Information System ( #Hospino.com and #Databricks, and I’d love to learn from the community.

DQX Forge - Extension for DQ in Databricks

arthurfr23 — Thu, 20 Aug 2026 13:44:32 GMT

Hey guys,

i'm just launched a VsCode extension for Data Quality proccess using DQX.

The idea is simplify the process using AI (you can do it manually to).

The extension construct data contracts, jobs and dashboards, all of that with few clicks and with less than 10 minutes.

If you can, please, test and send me a feedback, so i can improve that.

You can download directly in VsCode Extensions Marketplace

Cannot work with workspace

SuShuang — Tue, 18 Aug 2026 11:22:01 GMT

Hello everyone,

We’re currently experiencing issues across all our Databricks environments. We’re unable to open any notebooks and are receiving the error shown below.

Does anyone have any insights into the issue or know when it is expected to be resolved?

A network error occurred.

Child-Parent Partnership

Harshal54 — Tue, 18 Aug 2026 05:43:34 GMT

We have offices in both the US and India and are evaluating a parent-child structure in the Databricks Partner Program.

Most of our customers and business are US-based. Because of this, would there be any advantage to having the US entity as the parent versus the India entity as the parent from a partner program, co-sell, referral, or business development perspective?

Also, if we choose India as the parent and the US as the child, and the India parent achieves Silver status, would the Silver status and associated benefits extend across the entire hierarchy, or would the US entity still need to independently qualify?

We'd appreciate any insights from partners who have experience with a similar multi-country setup.

I want to help me with registration key

Kayunga — Mon, 17 Aug 2026 11:18:27 GMT

Is the Semantic Layer the Real Challenge With Databricks Genie?

kartikchoudhary — Mon, 17 Aug 2026 05:49:16 GMT

I've been thinking about this while looking at different Genie use cases.

It seems like the hardest part of implementing Genie isn't getting users to ask questions in natural language. It's making sure Genie understands what those questions actually mean in the context of the business.

For example, if someone asks:

“Why did our performance drop this week?”

Genie still needs to know:

Which metric are we talking about?
Which data is authoritative?
How are the tables related?
What does “performance” actually mean for that business?
Are there business rules that should affect the answer?

This becomes even more important for operational use cases. Questions around NPT, ROP, equipment performance, or cost can require data from multiple systems.

That makes me wonder if we're putting too much emphasis on the conversational interface and not enough on what sits underneath it.

Maybe the real Genie implementation project isn't building the Genie space. It's building the context that allows Genie to give a trustworthy answer.

Curious what others are seeing:

What has been the bigger challenge in your Genie projects — configuring Genie itself, or getting the underlying data, metrics, and business definitions ready for it?

Excel Add-in Regression - Sign-in opens external browser window instead of embedding in task pane, r

data_lover — Thu, 13 Aug 2026 17:46:56 GMT

Issue: The Databricks Excel Add-in launches authentication in an external browser window rather than within the Excel task pane. The Unity Catalog does display in the external browser window, but after sign-in, Excel remains stuck spinning/loading indefinitely.
Affected Platforms: Reproduced on both Windows and macOS using Microsoft 365 Excel.
Deployment Method: The add-in is side-loaded through a custom XML manifest rather than deployed tenant-wide from the Microsoft Marketplace.
Troubleshooting Completed:
- Verified and modified manifest trust and authentication settings.
- Cleared Excel Web Extensions Framework (WEF) caches and reloaded the manifest.
- Tested browser settings, pop-up permissions, cookies, and documented authentication workarounds.
- Reproduced on multiple independent Windows and Mac installations.
Business Impact: Users cannot refresh a recurring internal report through the add-in and must perform manual exports from the Databricks UI as a temporary workaround.
Suspected Cause: The issue may be tied to an August 2026 update to the Databricks Excel Add-in that introduced new features such as cascading filters and Genie One integration. The timing aligns with the appearance of the authentication problem. https://docs.databricks.com/aws/en/release-notes/product/2026/august
Request to Support: Investigate the regression and provide any known workarounds or fixes.

How Can I Analyze Gaming Website Traffic and Search Trends with Databricks?

tomodachilifero — Fri, 07 Aug 2026 07:18:20 GMT

I'm learning Databricks, and I'm interested in using it to analyze data from a gaming website focused on Tomodachi Life ROM .

I'd like to understand the best approach for building a simple analytics workflow.

Some questions I have are:

How can I import CSV or Google Analytics export data into Databricks?
Which tools or notebooks are best for analyzing traffic trends and user behavior?
Can I use SQL and PySpark together for this type of analysis?
What's the best way to create dashboards showing page views, traffic sources, and popular content?
Are there any beginner-friendly examples for website analytics projects?

Do you see a future where enterprise MDM platforms like Informatica MDM could be fully replaced ?

rkhand14_ltm — Thu, 06 Aug 2026 11:40:37 GMT

Do you see a future where enterprise MDM platforms like Informatica MDM could be fully replaced or reimagined on Databricks?

🚀 Databricks has evolved far beyond a data engineering platform. With its growing ecosystem of connectors, governance capabilities, AI features, and now Data Quality Monitoring, it continues to blur the lines between traditional platforms.

This got me thinking:

Do you see a future where enterprise MDM platforms like Informatica MDM could be fully replaced or reimagined on Databricks?

Can the Lakehouse architecture eventually handle:

✅ Master Data Management

✅ Golden Record Creation

✅ Match & Merge

✅ Survivorship Rules

✅ Hierarchies & Relationships

✅ Data Stewardship

Or do you believe specialized MDM platforms will always have a place in the enterprise landscape?

💬 Curious to hear from architects, data engineers, MDM specialists, and governance leaders.

What's your take?

#Databricks #MDM #DataGovernance #DataArchitecture #Lakehouse #DataEngineering #UnityCatalog #MasterDataManagement

Video Recordings of Previous Webinars

suryaprayaga — Wed, 05 Aug 2026 14:21:07 GMT

Hello.

There are a lot of webinars and events / meetings that happened in the past. Is there a way where we can get those recordings of events?

Surya

Best Practices and Questions for Configuring High QPS AI Search Endpoints in Databricks

pj-celebal-tech — Wed, 29 Jul 2026 06:38:30 GMT

Hello Databricks Community,

I am exploring the configuration and scaling of high QPS (Queries Per Second) endpoints for Databricks AI Search, as described in the official documentation. I have several questions and would appreciate insights from anyone who has experience with high-throughput search workloads:

What is the maximum QPS supported for AI Search endpoints, and how can I request higher QPS for my application?
How does Databricks provision infrastructure to meet the target QPS, and are there any guarantees for sustained throughput?
What factors influence the QPS limits for a given AI Search index (e.g., index size, data type, concurrency)?
How can I monitor and troubleshoot QPS-related issues, such as 429 (Too Many Requests) errors or latency degradation?
Are there recommended best practices for scaling AI Search endpoints to support real-time applications with high QPS requirements?
Can I dynamically adjust the target QPS after an endpoint is created, and what is the impact on performance and cost?
What are the differences between standard and high QPS endpoints, and how do I choose the right configuration for my workload?
Is there a way to estimate the required QPS for my application based on expected traffic and query complexity?
How does QPS scaling interact with index updates or sync operations—are there any limitations or considerations?
Are there any additional costs associated with requesting or maintaining high QPS endpoints?

If you have practical experience, tips, or documentation links, please share! I believe this discussion will help others planning to scale their AI Search workloads.

Thank you!

Has anyone prototyped Databricks Lakebase or deployed it in a production environment?

DPatil — Thu, 23 Jul 2026 15:25:39 GMT

Has anyone prototyped Databricks Lakebase or deployed it in a production environment? I'd be interested in hearing about your experience, including any lessons learned, performance observations, or challenges you encountered.

I’ve spent quite a bit of time reviewing this OLTP database and was able to go through several of its capabilities. I think it’s going to become very popular and much needed in the future.

Please your experience? I'm putting together architecture diagram and what I have explored later.

Labs

Wintermute — Thu, 23 Jul 2026 07:58:26 GMT

Hi, others have asked this before so apologies in advance but I can't seem to find any useful advice on how to access the labs. I have an active Databricks Academy Labs Subscription (stated as active when logged on) but I seem to only have access to free courses and no labs. I signed up as a customer originally, should I have signed up as a partner instead or is the problems something else.

Get Started Discussions topics

Como funciona la facturación y pago en Databricks?

Databricks Data Ingestion

From Semantic Similarity to Business Authority: Why Genie Ontology and OntoRank Matter

Como aprender correctamente en el entorno de DataBricks y aprovecharlo al 100%

LLM API Returning Inconsistent Responses When Using RAG

External location not accessble with job and general purpose cluster but works fine with serverless

Path to become a Databricks Trainer

Hospino : Hospital Management Software

DQX Forge - Extension for DQ in Databricks

Cannot work with workspace

Failed to load folder

Child-Parent Partnership

I want to help me with registration key

Is the Semantic Layer the Real Challenge With Databricks Genie?

Excel Add-in Regression - Sign-in opens external browser window instead of embedding in task pane, r

How Can I Analyze Gaming Website Traffic and Search Trends with Databricks?

Do you see a future where enterprise MDM platforms like Informatica MDM could be fully replaced ?

Video Recordings of Previous Webinars

Best Practices and Questions for Configuring High QPS AI Search Endpoints in Databricks

Has anyone prototyped Databricks Lakebase or deployed it in a production environment?

Labs