All Get Started Discussions posts

Re: Best Practices and Questions for Configuring High QPS AI Search Endpoints in Databricks

Lu_Wang_ENB_DBX — Fri, 31 Jul 2026 17:28:12 GMT

Answers to your questions:

Max QPS: Standard endpoints default to 20–200 QPS depending on index size, and with high QPS you can scale to 1,000+ QPS on standard endpoints.
How to request more: Set target_qps when creating or updating a standard endpoint via UI, SDK, or REST. Databricks then provisions extra capacity automatically.
Guarantees: It’s best-effort, not guaranteed; actual throughput depends on workload and traffic shape.
What affects QPS: index size, vector dimensionality, query complexity, filter usage, num_results, and whether you use managed embeddings / optimized query route.
Monitoring/troubleshooting: Check endpoint scaling_info.state, use the index URL with service principal OAuth, and look for SCALING_CHANGE_IN_PROGRESS, 429s, or rising latency.
Best practices: Use service principals, reuse the index object, keep num_results modest, load test, and split/replicate across endpoints if you need more total throughput.
Dynamic changes: Yes — you can update target_qps after creation. While scaling is in progress, capacity changes aren’t immediate; avoid changing it again until the state becomes applied.
Standard vs high QPS: High QPS is for standard endpoints only; storage-optimized endpoints don’t support it. Standard is the right choice when sustained query throughput matters.
Estimating needs: Use expected traffic + query shape, then load test. Databricks also provides a GenAI calculator for rough cost/capacity estimates.
Sync/update interaction: Changing target_qps does not require a sync to take effect, but the new capacity applies only after provisioning completes.
Cost: Higher target QPS increases endpoint cost, and you’re charged for the provisioned capacity even if traffic is lower.

Docs:

Re: Has anyone prototyped Databricks Lakebase or deployed it in a production environment?

DPatil — Wed, 29 Jul 2026 20:18:35 GMT

I was able to prototype this myself in the Databricks free edition. I successfully synced data from Lakebase to Delta Lake and vice versa with zero ETL code, including reverse ETL. In other words, Delta Lake can use a Lakebase summary table as a lookup to avoid expensive OLTP joins — which is a big win.

One important thing I learned: Lakebase databases live in Databricks‑managed storage, not in your own S3 bucket or AWS account the way classic Delta tables in the lakehouse do.

I'm sure some mission critical application that customer manage data they may not be comfortable with this approach.

Best Practices and Questions for Configuring High QPS AI Search Endpoints in Databricks

pj-celebal-tech — Wed, 29 Jul 2026 06:38:30 GMT

Hello Databricks Community,

I am exploring the configuration and scaling of high QPS (Queries Per Second) endpoints for Databricks AI Search, as described in the official documentation. I have several questions and would appreciate insights from anyone who has experience with high-throughput search workloads:

What is the maximum QPS supported for AI Search endpoints, and how can I request higher QPS for my application?
How does Databricks provision infrastructure to meet the target QPS, and are there any guarantees for sustained throughput?
What factors influence the QPS limits for a given AI Search index (e.g., index size, data type, concurrency)?
How can I monitor and troubleshoot QPS-related issues, such as 429 (Too Many Requests) errors or latency degradation?
Are there recommended best practices for scaling AI Search endpoints to support real-time applications with high QPS requirements?
Can I dynamically adjust the target QPS after an endpoint is created, and what is the impact on performance and cost?
What are the differences between standard and high QPS endpoints, and how do I choose the right configuration for my workload?
Is there a way to estimate the required QPS for my application based on expected traffic and query complexity?
How does QPS scaling interact with index updates or sync operations—are there any limitations or considerations?
Are there any additional costs associated with requesting or maintaining high QPS endpoints?

If you have practical experience, tips, or documentation links, please share! I believe this discussion will help others planning to scale their AI Search workloads.

Thank you!

Re: Has anyone prototyped Databricks Lakebase or deployed it in a production environment?

balajij8 — Thu, 23 Jul 2026 16:36:46 GMT

Lakebase is solving a key architectural requirement by finally allowing transactional workloads inside the Data Intelligence platform alongside the analytical data in Lakehouse.

Performance & Core

Speed & Concurrency - Because it operates as a Postgres engine under the hood, it handles high-concurrency row-level reads/writes effortlessly. You can use it to power real-time AI agents (acting as the agent's memory/state store) and the latency was consistently good.
Branching - The copy-on-write branching feature is good for CI/CD & also for compliance. You can use it for secure Time Travel and compliance trials. You can spin up isolated staging environments from production data in seconds without duplicating massive storage costs.

Architecture & Integration

Zero-Code APIs - If you are building frontend applications (like integrating with Databricks Apps), Lakebase’s Data API allows for incredibly fast, zero-code REST integrations. You can use it for pushing real-time patient vitals directly into the operational layer.
Governance - Wrapping Unity Catalog around both analytical lakehouse and the Lakebase OLTP layer is a key area for consideration. Choose controls based on need either at Unity Catalog level or directly at the Lakebase level (Postgres role controls)
OLTP Boundary - You can keep heavy BI and aggregations on Databricks SQL Warehouses, and strictly use Lakebase for fast, operational application serving.

Key Consideration

Scale-to-Zero - Lakebase's serverless scaling is great for lowering costs in dev environments. However, if you allow a production instance to scale entirely to zero, the first application request that triggers a wake-up will experience a cold-start latency spike. Make sure you configure your minimum compute to keep production warm.
Connection Pooling - Opening and closing direct connections for every single transaction will exhaust your limits and kill performance. Ensure the application architecture leverages connection pooling when interacting with Lakebase at a high volume.

You can refer the Lakebase architectures given below for your prototypes.

Security, Governance & Compliance

Integration & Bidirectional Data Sync

Applications

Re: Labs

Wintermute — Thu, 23 Jul 2026 15:49:29 GMT

I tried what you suggested and got it to work. Thank you!

Has anyone prototyped Databricks Lakebase or deployed it in a production environment?

DPatil — Thu, 23 Jul 2026 15:25:39 GMT

Has anyone prototyped Databricks Lakebase or deployed it in a production environment? I'd be interested in hearing about your experience, including any lessons learned, performance observations, or challenges you encountered.

I’ve spent quite a bit of time reviewing this OLTP database and was able to go through several of its capabilities. I think it’s going to become very popular and much needed in the future.

Please your experience? I'm putting together architecture diagram and what I have explored later.

Re: Labs

Brahmareddy — Thu, 23 Jul 2026 14:09:25 GMT

Hi @Wintermute, how are you doing today?

As per my understanding, I don’t think signing up as a customer instead of a partner is the issue.

First, make sure you are using the same email address that is linked to your Labs Subscription. Also, try opening the course directly and checking the course modules, because sometimes the labs are available inside the course and not shown separately.

You can also try logging out and back in, or opening the Academy in a private browser window.

Since your subscription shows as active but you still cannot access the labs, it may be an account access issue. I would suggest contacting Databricks Academy Support with a screenshot of your active subscription. They should be able to check and enable the correct access.

Hope this helps! Good day.

Re: Labs

zoltanctoth — Thu, 23 Jul 2026 13:34:05 GMT

@Wintermute I've found it. Copying from the message/notification that you probably see and need to open:

Thank you for purchasing Databricks Academy Labs on 7/23/26, 2:00 AM. To access Databricks Academy Labs, please navigate to "Catalogs in My Subscription", found in the User Menu (top left corner of Databricks Academy), and enroll in a listed course to get started. "

Re: Labs

AmitDECopilot — Thu, 23 Jul 2026 12:40:45 GMT

Hi,

I think this may be more of an account/subscription mapping issue than customer vs partner sign-up only.

Few things I would check:

Are you logged into the same Databricks Academy account where the Labs subscription is active?
Are you going through the Customer Academy learning portal, not a different academy login? Databricks docs mention using the Customer Academy account for Labs access.
Check the course catalog / subscription plan area for courses that actually include labs. Some free/self-paced courses may show course content but not lab access.
If the Labs subscription is active but no labs are visible, then I would raise a Databricks Academy support ticket with your account email and screenshot of the active subscription.

I don’t think I would create a new partner account unless Databricks support confirms that is required. It may create more confusion if the subscription is already tied to your current customer account.

Re: Labs

Wintermute — Thu, 23 Jul 2026 08:53:42 GMT

I tried logging on with different browsers. Edge, Brave and Firefox. I get the same result as you when using the filter. Also, the active subscription is visible. See screen shot.

Re: Labs

zoltanctoth — Thu, 23 Jul 2026 08:19:33 GMT

Just to add extra info, if I'm filtering for courses with labs, the list is empty

Re: Labs

zoltanctoth — Thu, 23 Jul 2026 08:15:01 GMT

I'm here with the same question, watching this

Labs

Wintermute — Thu, 23 Jul 2026 07:58:26 GMT

Hi, others have asked this before so apologies in advance but I can't seem to find any useful advice on how to access the labs. I have an active Databricks Academy Labs Subscription (stated as active when logged on) but I seem to only have access to free courses and no labs. I signed up as a customer originally, should I have signed up as a partner instead or is the problems something else.

Re: comprehensive guide or usecases for data engineering hands-on

Brahmareddy — Wed, 22 Jul 2026 13:51:00 GMT

Hi Manjunath

How are you doing today? I felt the same way starting out, so I know what you mean. The best structured resource is dbdemos.ai, maintained by the Databricks team. It installs full end to end demos into your workspace in one line (pip install dbdemos, then dbdemos.install), each with real style datasets, pipelines, and dashboards already wired up. Good ones to start with are Lakehouse Retail C360, Delta Live Tables and CDC, and MLOps end to end. One note, if you are on Free Edition, some may not fully install due to compute and outbound limits, so a trial or paid workspace works best.

Alongside that, build one real project on the sample data yourself. Take samples.tpch or nyctaxi, build a proper medallion pipeline (bronze to silver to gold), then put a dashboard or a Genie space on top. Doing one flow end to end teaches you more than ten scattered tutorials.

Hope this helps.

comprehensive guide or usecases for data engineering hands-on

Manjunath2684 — Wed, 22 Jul 2026 13:42:11 GMT

Hello members,

I have access to my databricks environment and I see it already includes sample data sets (under "samples" database).

I find the learning paths are very basic and not any close the real use cases. Are there any advanced hands-on use cases built around these sample datasets to try?

It would be great if we have some structured guide rather than fragmented topic. Appreciate any recommendation on this topic.

ISV Partnership

pragya17 — Wed, 22 Jul 2026 05:26:54 GMT

I have applied for ISV Partnership in databricks . How many days are taken to review the application or revert back from databricks Team.

Thanks,

Pragya Sharma

Re: New to the community!

suryaprayaga — Mon, 20 Jul 2026 17:10:16 GMT

Welcome to this wonderful group Chris!! While I am a Databricks & Snowflake practitioner myself, I have been constantly learning a lot of things in this group.

Looking forward to connecting with you.

New to the community!

cgambill — Mon, 20 Jul 2026 16:07:17 GMT

I’m Chris Gambill, founder of Gambill Data in Knoxville, Tennessee. I’ve spent 26 years working across data engineering, architecture, and leadership, primarily in telecom, manufacturing, and finance.

Gambill Data recently joined the Databricks Partner Program, and I’m a Databricks Certified Data Engineer Professional. While I’m new to this community, I’m not new to Databricks... I’ve been working with the platform for about six years.

I started my career building dts packages and Crystal Reports, so I’ve had a front-row seat to nearly three decades of change in the data profession. It has been exciting to watch, and occasionally exhausting to keep renaming everything (haha).

I’m looking forward to learning from the community, sharing what I’ve learned, and connecting with other Databricks practitioners. Don’t hesitate to reach out and say hello!

Re: Hello from a new member!

cgambill — Mon, 20 Jul 2026 13:37:41 GMT

Hi @arijitsam -Great to meet you! You’ve inspired me to write an introduction post as well. I usually focus on telecom, manufacturing, and financial services, but many of the same patterns hold across industries.

My preferred approach is to ingest into Bronze using Lakeflow Connect when an appropriate connector is available. Otherwise, I’ll typically use Auto Loader for file-based ingestion, keeping transformations at that stage to a minimum.

From there, I use LDP/DLT to move data from Bronze to Silver, applying expectations for data-quality rules and quarantining records that fail validation. Depending on the source and business requirements, Silver may be an append-only streaming table or an SCD Type 2 table maintained declaratively from CDC events or source snapshots.

I also favor a metadata-driven approach to pipeline development. Configuration, column mappings, and data-quality rules live in control tables, while lightweight Python modules and notebooks read that metadata to drive pipeline behavior and orchestration dynamically. This reduces duplicated code and makes onboarding or changing a feed largely a configuration exercise rather than another custom pipeline build.

The interesting design question with B2B feeds is usually how much schema variability exists between vendors. That can quickly determine whether schema evolution is an asset or a future firedrill...

Re: How to restore a workspace ( Cancelled) by mistake

Louis_Frolio — Mon, 20 Jul 2026 11:38:29 GMT

Hey @pk35 , I did some digging and here is what I found. However, before we get to recovery, the most important step is pinning down exactly what got cancelled, because there are three possibilities and they have very different outcomes.

You cancelled the whole Databricks plan (account console, Settings, Manage your plan, Cancel plan). Per the AWS docs this one is permanent. You cannot reactivate a cancelled plan, access to workspaces, notebooks, and data goes away, and any Customer Content in workspaces tied to the subscription is deleted within 30 days of cancellation. Your post is about two weeks old, so if this is your case, that 30-day clock is already partway through and time matters.
You deleted just the workspace (plan still active). There is no public self-service restore flow for a deleted workspace on AWS. Any recovery is a support matter and is not guaranteed.
You cancelled a Databricks subscription inside AWS Marketplace. This is the hopeful one. The AWS docs are explicit that you cannot cancel your Databricks plan through AWS Marketplace, and cancelling there only removes the Marketplace account as a payment method. If that is what happened, your plan and workspace may still be intact and you may just need to re-add a payment method.

Now the good news on your data. Because your storage is in your own S3 bucket under Unity Catalog, the underlying data is very likely still there. On AWS, when a workspace is deleted the default workspace catalog is retained, and its managed tables and volumes stay registered in the Unity Catalog metastore (the metastore lives at the account level, not inside the workspace). So your continued S3 and Unity Catalog access is a good sign that the data files survived, even if the workspace itself did not.

The harder part is notebooks and jobs. Those live in the Databricks control plane, not in your S3 bucket, so keeping S3 access does not bring them back on its own.

Here is what I would do next:

Confirm which of the three scenarios above applies. That decides everything else.
Open a Databricks Support case at https://help.databricks.com rather than email alone, ideally as the account admin or owner. If you do email, the correct address is help@databricks.com. Your note shows help@databrics.com, which is missing the k, so it may not have reached anyone. Cases created by email from a non-authorized contact can also get bounced.
Give support the specifics up front: account ID, workspace ID and URL, AWS region, approximate cancellation or deletion time, and whether this was a plan cancellation, a workspace deletion, or an AWS Marketplace change.
Move quickly. Restore is best-effort with no SLA, and if this was a plan cancellation the 30-day content-deletion window is already running.

One more thing that helps regardless of outcome. If any of your notebooks were Git-linked or deployed with Databricks Asset Bundles or Terraform, you can redeploy them into a fresh workspace without needing the old one restored. Paired with your intact S3 and Unity Catalog data, that gets you most of the way back.

Closing takeaway: figure out what was actually cancelled first, get a properly scoped support case in front of Databricks quickly, and take some comfort that your Unity Catalog data in S3 is the piece most likely to have survived.

References:

Manage subscription and billing (plan cancellation is permanent, the 30-day content deletion, and the AWS Marketplace payment-method detail): https://docs.databricks.com/aws/en/admin/account-settings/account
Delete a workspace on AWS (no self-service restore, plus the Unity Catalog catalog retention behavior): https://docs.databricks.com/aws/en/admin/workspace/delete-workspace
Support and case filing (including the help@databricks.com authorized-contact caveat): https://docs.databricks.com/aws/en/resources/support
Manage workspace objects (restoring from Trash, which is a different thing from recovering a deleted workspace): https://docs.databricks.com/aws/en/workspace/workspace-objects

Hope this helps, Louis