- 126 Views
- 3 replies
- 0 kudos
Genie connection to copilot agent in copilot studio
Hello!I’m trying to add a tool — Azure Databricks Genie — in Microsoft Copilot Studio for my agent, but I’m running into some difficulties. Is it possible to establish this connection using a Pro cluster, or does it only work with a serverless cluste...
- 126 Views
- 3 replies
- 0 kudos
- 0 kudos
When you say they can't talk to him, what do you mean? Can they see him at all? or is it that Genie returns no results to them? Have you published the agent? https://learn.microsoft.com/en-us/microsoft-copilot-studio/publication-add-bot-to-microsoft-...
- 0 kudos
- 10909 Views
- 5 replies
- 4 kudos
How to use Parallel processing using Concurrent Jobs in Databricks ?
QuestionIt would be great if you could recommend how I go about solving the below problem. I haven't been able to find much help online. A. Background:A1. I have to text manipulation using python (like concatenation , convert to spacy doc , get verbs...
- 10909 Views
- 5 replies
- 4 kudos
- 4 kudos
I have to process data for n number of devices which is sending data in every 5 seconds.I have a similar scenario where I have to take last 3 hours of data and process it for all the devices for some key parameters. Now if I am doing it sequentially ...
- 4 kudos
- 99 Views
- 1 replies
- 0 kudos
ai_parse_document Not Extracting Text from Images in PDF
Hello Team,I hope you are doing well.I am a student currently exploring Databricks and learning how to work with the "ai parse document" function. While experimenting, I encountered a couple of issues related to text extraction from images inside PDF...
- 99 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @rajcoder! This post appears to duplicate the one you recently posted. A response has already been provided to your recent post. I recommend continuing the discussion in that thread to keep the conversation focused and organised.
- 0 kudos
- 94 Views
- 0 replies
- 0 kudos
Building a Claude Code-Compatible Proxy for Databricks (with MCP, Git Tools, and Prompt Caching)
Many Databricks engineers have asked whether it's possible to use Claude Code CLI directly against Databricks-hosted Claude models instead of Anthropic's cloud API. This enables repo-aware AI workflows—navigation, diffs, testing, MCP tools—right insi...
- 94 Views
- 0 replies
- 0 kudos
- 6107 Views
- 4 replies
- 2 kudos
Accessing Databricks Volumes from a Serving Endpoint Using a Custom Model Class in Unity Catalog
Hi everyone,I’m looking for accessing Unity Catalog (UC) Volumes from a Databricks Serving Endpoint. Here’s my current setup:I have a custom AI model class for inference, which I logged into Unity Catalog using mlflow.pyfunc.log_model.I’ve created a ...
- 6107 Views
- 4 replies
- 2 kudos
- 2 kudos
Serverless Model Serving does not mount the UC Volumes FUSE path (/Volumes), so references to “/Volumes/…” inside a custom pyfunc’s model code will fail at container build or runtime. The correct pattern is to package any required files (like your ...
- 2 kudos
- 163 Views
- 2 replies
- 1 kudos
Can serverless environments not use SynapseML's LightGBM?
When I use LightGBM, I get the following error on the line below: 'str' object has no attribute 'getParam'.Is this because serverless cannot run the JAR files that SynapseML depends on?File /local_disk0/.ephemeral_nfs/envs/pythonEnv-b0d5f8ce-8426-443...
- 163 Views
- 2 replies
- 1 kudos
- 1 kudos
Sorry, I just found out while checking the official documentation that starting from November, dependency JAR files are supported.I’ll give it a try and see how well it works.If anyone has any insights, I would greatly appreciate your guidance.
- 1 kudos
- 295 Views
- 2 replies
- 3 kudos
Resolved! Databricks Model Serving Endpoint Fails: “_USER not found for feature table”
Hi Databricks Community,I’m trying to deploy a model serving endpoint that uses Databricks Feature Store (Unity Catalog, online tables).My offline and online feature tables are created and visible in Databricks.The model is logged with FeatureEnginee...
- 295 Views
- 2 replies
- 3 kudos
- 3 kudos
Thanks for the reply It is very useful and comprehensive.I managed to find another solution to the problem so I wanted to share some additional details on this topic:I was using 15.4 LTS ML Runtime, this could have caused the problem - I did not swit...
- 3 kudos
- 3924 Views
- 5 replies
- 1 kudos
Resolved! Lacking support for column-level select grants or attribute-based access control
In the Unity Catalog launch and its accompanying blog post, one of the primary selling points was a set of granular access control features that would at least partially eliminate the need to create a multitude of separate table views and the attenda...
- 3924 Views
- 5 replies
- 1 kudos
- 1 kudos
@at-khatri Databricks ABAC is in Public Preview nowhttps://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/
- 1 kudos
- 307 Views
- 1 replies
- 4 kudos
Hackathon Project: Recipe Recommendation Engine with Traditional ML + Genie on Databricks Free Edit
Hi everyone, For the Databricks Free Edition Hackathon, I wanted to show that traditional ML still has a big role today, and how it can work hand-in-hand with Databricks’ newer AI tooling. As a concrete use case, I created a recipe recommendation eng...
- 307 Views
- 1 replies
- 4 kudos
- 4 kudos
This is amazing @hasnat_unifeye. Well done and good luck for the hackathon.
- 4 kudos
- 213 Views
- 1 replies
- 0 kudos
AutoML Deprecation?
Hi All,It looks like AutoML is set to be deprecated with the next major version (although the note isn't specific on if that's 18). I haven't seen any announcement or alert about this impending change. Did I just miss it? I know we have teams using t...
- 213 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @kevin11 ,I guess it's their standard way of library deprecation policy. In their docs they mentioned that when a library is planned for removal, Databricks takes following steps to notify customers:So they've added those note to AutoMl docs:And y...
- 0 kudos
- 327 Views
- 1 replies
- 0 kudos
Resolved! How to store & update a FAISS Index in Databricks
I’m currently using FAISS in a Databricks notebook to perform semantic search in text data. My current workflow looks like this:encode ~10k text entries using an embedding model.build a FAISS index in memory.run similarity searches using index.search...
- 327 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ashfire , Here’s a practical path to scale your FAISS workflow on Databricks, along with patterns to persist indexes, incrementally add embeddings, and keep metadata aligned. Best practice to persist/load FAISS indexes on Databricks Use faiss...
- 0 kudos
- 3606 Views
- 2 replies
- 0 kudos
DAB - Add/remove task depending on workspace.
I use DAB for deploying Jobs, I want to add a specific Task in dev only but not in staging or prod. Is there any way to achieve this using DAB ?
- 3606 Views
- 2 replies
- 0 kudos
- 0 kudos
You can define specific resources by target in DAB as shown here. This is valid for jobs and/or tasks:For instance, in my case:I think, best option (but not available as far as I know) would be to be able to define "include" sections by target, inste...
- 0 kudos
- 1175 Views
- 2 replies
- 0 kudos
Distributed Training quits if any worker node fails
Hi,I'm training a Pytorch model in a distributed environment using the Pytorch's DistributedDataParallel (DDP) library. I have spin up 10 worker nodes.The issue which I'm facing is that during the training, if any worker node fails and exits, the ent...
- 1175 Views
- 2 replies
- 0 kudos
- 0 kudos
Distributed training with PyTorch’s DistributedDataParallel (DDP) is not inherently fault-tolerant—if any node fails, the whole job crashes, and, as you noted, checkpointing cannot auto-recover the process without hypervisor or application-level orch...
- 0 kudos
- 3960 Views
- 1 replies
- 0 kudos
FeatureEngineeringClient and Unity Catalog
When testing this code ( fe.score_batch( df=dataset.drop("Target").limit(10), model_uri=f"models:/{model_name}/{mv.version}", ) .select("prediction") .limit(10) .display() ) I get the error: “MlflowException: The...
- 3960 Views
- 1 replies
- 0 kudos
- 0 kudos
Your issues are tied to authentication and network/configuration differences between Unity Catalog and Workspace models in Databricks, specifically when using the FeatureEngineeringClient. Key Issues FeatureEngineeringClient + Unity Catalog: You get...
- 0 kudos
- 3785 Views
- 1 replies
- 0 kudos
Why is spark mllib is not encouraged on the platform?/Why is ML dependent on .toPandas() on dbricks?
I'm new to Spark,Databricks and am surprised about how the Databricks tutorials for ML are using pandas DF > Spark DF. Of the tutorials I've seen, most data processing is done in a distributed manner but then its just cast to a pandas dataframe. From...
- 3785 Views
- 1 replies
- 0 kudos
- 0 kudos
You are noticing a common pattern in Databricks ML tutorials: data is often processed with Spark for scalability, but training and modeling are frequently done on pandas DataFrames using single-node libraries like scikit-learn. This workflow can be c...
- 0 kudos
