Best Practices and Questions for Configuring High QPS AI Search Endpoints in Databricks

[pj-celebal-tech]

Hello Databricks Community,

I am exploring the configuration and scaling of high QPS (Queries Per Second) endpoints for Databricks AI Search, as described in the official documentation. I have several questions and would appreciate insights from anyone who has experience with high-throughput search workloads:

1. What is the maximum QPS supported for AI Search endpoints, and how can I request higher QPS for my application?
2. How does Databricks provision infrastructure to meet the target QPS, and are there any guarantees for sustained throughput?
3. What factors influence the QPS limits for a given AI Search index (e.g., index size, data type, concurrency)?
4. How can I monitor and troubleshoot QPS-related issues, such as 429 (Too Many Requests) errors or latency degradation?
5. Are there recommended best practices for scaling AI Search endpoints to support real-time applications with high QPS requirements?
6. Can I dynamically adjust the target QPS after an endpoint is created, and what is the impact on performance and cost?
7. What are the differences between standard and high QPS endpoints, and how do I choose the right configuration for my workload?
8. Is there a way to estimate the required QPS for my application based on expected traffic and query complexity?
9. How does QPS scaling interact with index updates or sync operations—are there any limitations or considerations?
10. Are there any additional costs associated with requesting or maintaining high QPS endpoints?

If you have practical experience, tips, or documentation links, please share! I believe this discussion will help others planning to scale their AI Search workloads.

Thank you!

[Lu_Wang_ENB_DBX]

Answers to your questions:

• Max QPS: Standard endpoints default to 20–200 QPS depending on index size, and with high QPS you can scale to 1,000+ QPS on standard endpoints.
• How to request more: Set target_qps when creating or updating a standard endpoint via UI, SDK, or REST. Databricks then provisions extra capacity automatically.
• Guarantees: It’s best-effort, not guaranteed; actual throughput depends on workload and traffic shape.
• What affects QPS: index size, vector dimensionality, query complexity, filter usage, num_results, and whether you use managed embeddings / optimized query route.
• Monitoring/troubleshooting: Check endpoint scaling_info.state, use the index URL with service principal OAuth, and look for SCALING_CHANGE_IN_PROGRESS, 429s, or rising latency.
• Best practices: Use service principals, reuse the index object, keep num_results modest, load test, and split/replicate across endpoints if you need more total throughput.
• Dynamic changes: Yes — you can update target_qps after creation. While scaling is in progress, capacity changes aren’t immediate; avoid changing it again until the state becomes applied.
• Standard vs high QPS: High QPS is for standard endpoints only; storage-optimized endpoints don’t support it. Standard is the right choice when sustained query throughput matters.
• Estimating needs: Use expected traffic + query shape, then load test. Databricks also provides a GenAI calculator for rough cost/capacity estimates.
• Sync/update interaction: Changing target_qps does not require a sync to take effect, but the new capacity applies only after provisioning completes.
• Cost: Higher target QPS increases endpoint cost, and you’re charged for the provisioned capacity even if traffic is lower.

Docs:

• High QPS guide
• AI Search performance guide
• Cost management guide

[Brodybenson]

For high-QPS AI Search endpoints in Databricks, I’d start by separating the problem into latency, throughput, scaling, and cost. Make sure you know your target QPS, acceptable p95/p99 latency, query size, embedding/model latency, and whether traffic arrives in bursts or stays steady.

A few practical best practices:

• Load-test with realistic traffic rather than relying on a small number of concurrent requests.
• Monitor p50/p95/p99 latency, errors, throttling, CPU/memory utilization, and queueing.
• Tune endpoint capacity and autoscaling based on sustained QPS as well as traffic spikes.
• Keep retrieval efficient by limiting unnecessary result sizes and avoiding expensive filtering where possible.
• Cache repeated queries or embeddings when the workload allows it.
• Use retries carefully with exponential backoff so a temporary slowdown doesn’t turn into a retry storm.
• Separate ingestion/update workloads from serving traffic when possible so index updates don’t negatively affect query performance.
• Measure cost per request alongside latency; simply adding capacity may solve QPS problems but create an unnecessarily expensive architecture.

The main questions I’d ask before choosing a configuration are: What QPS are you targeting, what p95 latency do you need, how large is the index, how frequently does it change, and are requests mostly identical or highly variable?

If you’re also looking for a straightforward web-based option to explore, playeggycar.io is another site you can check out.