- 36125 Views
- 22 replies
- 47 kudos
Databricks Announces the Industry’s First Generative AI Engineer Learning Pathway and Certification
Today, we are announcing the industry's first Generative AI Engineer learning pathway and certification to help ensure that data and AI practitioners have the resources to be successful with generative AI. At Databricks, we recognize that generative ...
- 36125 Views
- 22 replies
- 47 kudos
- 47 kudos
Dear Certifications TeamI have completed full Generative AI Engineering Pathway, so I received module wise knowledge badge but I didn't received the overall certificate which mentioned in description which is Generative AI Engineer with one Star. Req...
- 47 kudos
- 1821 Views
- 1 replies
- 1 kudos
Resolved! Claude Code-Execution Call Format
Is it possible to use the code execution tool with instances of Claude hosted through Databricks? If I try to format the payload like in Anthropic's documentation, I get an error that the function isn't defined properly: tools=[{ "type": ...
- 1821 Views
- 1 replies
- 1 kudos
- 1 kudos
Greeting @jmartin1 , The Anthropic “code execution” tool isn’t supported through Databricks’ Foundation Model APIs. Databricks exposes an OpenAI-compatible tools interface, and the only supported tool type today is a function defined with a JSON sch...
- 1 kudos
- 9318 Views
- 3 replies
- 1 kudos
Behavior of Vector Index Sync with Delta Tables When Using OVERWRITE vs MERGE in Databricks
I'm working with vector search in Databricks using vector index sync with Delta tables, and I'm a bit unclear on how updates to the source table affect the vector index, specifically when using different write operations.If I overwrite the source Del...
- 9318 Views
- 3 replies
- 1 kudos
- 1 kudos
From community experience, vector index sync behavior depends heavily on how the Delta table is updated. With OVERWRITE, the table is effectively replaced, so the vector index typically treats this as a full refresh. Existing embeddings are dropped a...
- 1 kudos
- 1037 Views
- 3 replies
- 2 kudos
Resolved! How do I integrate third-party ML/AI libraries with Databricks GenAI workflows?
How can I use external AI libraries inside my Databricks GenAI projects?
- 1037 Views
- 3 replies
- 2 kudos
- 2 kudos
You can use third-party ML/AI libraries in Databricks GenAI by installing them on your cluster or notebook (%pip install library-name), importing them in your code, and then integrating their outputs into your GenAI workflows. For large models, use a...
- 2 kudos
- 2023 Views
- 5 replies
- 2 kudos
ai_parse_document + Genie with ai_query
Hi EveryoneI have used ai_parse_document to process multiple PDFs and store the parsed data in a table (one PDF per row). Later, I ran ai_query in natural language, which correctly scans all rows and returns answers from each PDF.However, when I use ...
- 2023 Views
- 5 replies
- 2 kudos
- 2 kudos
I think Genie is not optimized for this use case. Please run some experiments - chunk pdf to use multi-agent supervisor from agent bricks to combine Genie with Knowledge Base (although I haven't yet it yet)
- 2 kudos
- 1368 Views
- 1 replies
- 0 kudos
Resolved! What’s the recommended architecture for integrating Databricks + MLflow + GenAI for model training
If I want to use Databricks, MLflow, and GenAI together, what is the best way to organize and connect them so that I can train AI models and then use them in real apps?
- 1368 Views
- 1 replies
- 0 kudos
- 0 kudos
Broad question. I will recommend to follow Databricks offical documents and ML Training at their customer portal.Yet, I will try to answer it as belowUse the Databricks Lakehouse as the unified platform, with Unity Catalog (UC) providing centralized ...
- 0 kudos
- 6537 Views
- 6 replies
- 3 kudos
Resolved! ai_parse_document struggling to detect pdf
Hi helpful experts I'm writing my first PySpark Notebook that makes use of the new `ai_parse_document` function. I am basically following the code example from here: https://learn.microsoft.com/en-gb/azure/databricks/sql/language-manual/functions/ai...
- 6537 Views
- 6 replies
- 3 kudos
- 3 kudos
Hello @JN_Bristol,I discovered that ai_parse_document only works when the input is parsed as real Python bytes.The binaryFile format in Spark returns the content as an internal binary type (like a memoryview), and ai_parse_document can’t process that...
- 3 kudos
- 671 Views
- 1 replies
- 1 kudos
Databricks app services
Hi- I have built node.js based chatbot app that uses Azure OpenAI API to build connection and get query answers from it. I am using my organization deployed API on azure that requires cert.pem and cacert.pem certificates to authenticate. everything i...
- 671 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Daya3189 , Your local system likely has the organization's certificates installed in the Windows/Mac system trust store, which Node.js (or your browser) can read. On Databricks, the app runs in a secure, isolated Linux container. It has no knowle...
- 1 kudos
- 697 Views
- 1 replies
- 0 kudos
Slow Delta write when creating embeddings with mapPartitions
I’m trying to generate 35k+ embeddings in Databricks. What I’ve tried so far:Per-row UDF (very slow).Replaced UDF with rdd.mapPartitions to batch API calls, create one Azure client per partition, and call client.embed_documents(texts) in batches. Thi...
- 697 Views
- 1 replies
- 0 kudos
- 0 kudos
HiYou’ve optimised the embedding side really nicely already, batching in mapPartitions and creating one Azure client per partition is exactly what we recommend.For 35k rows, if embedding is fast but the Delta write/commit is slow, it’s almost always ...
- 0 kudos
- 2553 Views
- 3 replies
- 3 kudos
Resolved! Issue with ai_parse_document Not Extracting Text from Images in PDF
Hello Team,I hope you are doing well.I am a student currently exploring Databricks and learning how to work with the "ai parse document" function. While experimenting, I encountered a couple of issues related to text extraction from images inside PDF...
- 2553 Views
- 3 replies
- 3 kudos
- 3 kudos
Thank you for your reply!Yes, I have gone through your article — it explains very well how to extract text content from PDFs. However, I am facing a different issue.In my case, the PDF contains multiple images and paragraphs, but "ai_parse_document" ...
- 3 kudos
- 531 Views
- 1 replies
- 0 kudos
What is the difference between full fine-tuning, LoRA, and p-tuning on Databricks?
How are these three methods different?Full Fine-TuningLoRAP-Tuning
- 531 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Suheb Summary Table:Method What’s Tuned Speed/Cost Flexibility Use CaseFull Fine-TuningAll model weightsSlow/HighMaximumCustom tasks, large dataLoRASmall adapter layersFast/LowHighEfficient adaptationp-tuningPrompt embeddingsFastest/LowLimitedPro...
- 0 kudos
- 1065 Views
- 1 replies
- 0 kudos
Genie - Value dictionary
When I add the tables to the genie space, it automatically turns on the value dictionary for first 120 string fields, Is there a way to disable them by default and add only to the needed fields later?I am working in curating responses for around 15 t...
- 1065 Views
- 1 replies
- 0 kudos
- 0 kudos
"Value sampling is enabled by default for all Genie spaces." The only solution that I see is to build json and use as a serialized_space template https://docs.databricks.com/api/azure/workspace/genie/getspace
- 0 kudos
- 2515 Views
- 1 replies
- 1 kudos
Resolved! How do I build a robust multi-agent system (e.g. using Agent Bricks / Genie) on Databricks, while en
How can I set up several AI agents on Databricks that work together at the same time, and make sure they don’t mess up the data or break the system?
- 2515 Views
- 1 replies
- 1 kudos
- 1 kudos
@Suheb, Depends on your usecase. However, if it fits, I would recommend that you start with a multi-agent supervisor if you have the agents from the list below An existing Agent Bricks: Knowledge Assistant(/generative-ai/agent-bricks/knowledge-assist...
- 1 kudos
- 1077 Views
- 3 replies
- 4 kudos
Multi‑Agent Supervisor: url_citation (source links) not shown in Playground — why and how to enable?
I’m seeing a difference in citation behavior between a single Knowledge Assistant (KA) agent and the Multi‑Agent Supervisor setup.What I testedIn Agent Bricks → Knowledge Assistant, I created an agent that returns citations with links (e.g., url_cita...
- 1077 Views
- 3 replies
- 4 kudos
- 3816 Views
- 2 replies
- 2 kudos
How to Increase HTTP Request Timeout for Databricks App Beyond 120 Seconds?
I’ve built a Databricks App using Gradio that leverages predict_stream to get streaming responses from a multi-agent supervisor. The app coordinates reasoning across four knowledge agents, so the model uses a long chain-of-thought process before retu...
- 3816 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @snarayan ,I think you might be hitting timeout from model serving endpoint:Debug model serving timeouts - Azure Databricks | Microsoft LearnYou can try to increase timeout using environment variables using the Serving UI or programmatically using...
- 2 kudos
- 3002 Views
- 9 replies
- 2 kudos
Resolved! Custom MCP deployment
Hi Community!I have a question - could somebody please guide me how to deploy my custom MCP server to databricks?What I would like to achieve is the following:I have a unity catalog in databricks for which I would like to have MCPif the data in unity...
- 3002 Views
- 9 replies
- 2 kudos
- 2 kudos
@Hubert-Dudek ! Okay, I did not know about it! Unfortunately in my databricks workspace a can see Agent Bricks as Coming Soon
- 2 kudos
