- 556 Views
- 9 replies
- 1 kudos
Genie connection to copilot agent in copilot studio
Hello!I’m trying to add a tool — Azure Databricks Genie — in Microsoft Copilot Studio for my agent, but I’m running into some difficulties. Is it possible to establish this connection using a Pro cluster, or does it only work with a serverless cluste...
- 556 Views
- 9 replies
- 1 kudos
- 1 kudos
I'm afraid I don't have much further suggestions. I'd suggest you raise a ticket with Microsoft on this.
- 1 kudos
- 60 Views
- 1 replies
- 1 kudos
What are the recommended practices for handling skewed datasets in Databricks?
What should you do when your dataset is uneven—some values appear too many times and others appear very few times—while working in Databricks?
- 60 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Suheb ,Refer to really good guide prepared by Databricks team. When you have a skewed dataset the primary things you can do are following:1. Filter skewed values2. Apply Skew hints3. AQE skew optimization4. SaltingMuch detailed description of abo...
- 1 kudos
- 141 Views
- 3 replies
- 0 kudos
Migrated model to Unity catalog not seeing referenced serving endpoint
There was a model which was migrated from workspace model registry to unity catalog. At the time of initial creation of that model, dependencies to other databricks serving endpoints were configured using "DatabricksServiceEndpoint" config in mlflow....
- 141 Views
- 3 replies
- 0 kudos
- 0 kudos
Workspace model registry worked with workspace-scoped serving endpoints. UC models and UC serving endpoints use metastore-wide semantics and different lookup rules. The saved path inside the model metadata still points to workspace-level endpoints th...
- 0 kudos
- 84 Views
- 1 replies
- 0 kudos
UC Model Deployment across data bricks instances
Hello, We have multiple data bricks instances each represents an environment dev,qa,rel,prod etc.. We developed a model in the dev workspace and registered in the UC model registry using mlflow. Now, we are trying to find a best way to deploy this r...
- 84 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use UC's centralized model registry and MLflow’s copy APIs. If all target workspaces attach to the same Unity Catalog metastore, reference and promote models via their 3‑level UC names; use MLflow’s copy_model_version to “copy” the exact arti...
- 0 kudos
- 10997 Views
- 5 replies
- 4 kudos
How to use Parallel processing using Concurrent Jobs in Databricks ?
QuestionIt would be great if you could recommend how I go about solving the below problem. I haven't been able to find much help online. A. Background:A1. I have to text manipulation using python (like concatenation , convert to spacy doc , get verbs...
- 10997 Views
- 5 replies
- 4 kudos
- 4 kudos
I have to process data for n number of devices which is sending data in every 5 seconds.I have a similar scenario where I have to take last 3 hours of data and process it for all the devices for some key parameters. Now if I am doing it sequentially ...
- 4 kudos
- 170 Views
- 1 replies
- 0 kudos
ai_parse_document Not Extracting Text from Images in PDF
Hello Team,I hope you are doing well.I am a student currently exploring Databricks and learning how to work with the "ai parse document" function. While experimenting, I encountered a couple of issues related to text extraction from images inside PDF...
- 170 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @rajcoder! This post appears to duplicate the one you recently posted. A response has already been provided to your recent post. I recommend continuing the discussion in that thread to keep the conversation focused and organised.
- 0 kudos
- 171 Views
- 0 replies
- 0 kudos
Building a Claude Code-Compatible Proxy for Databricks (with MCP, Git Tools, and Prompt Caching)
Many Databricks engineers have asked whether it's possible to use Claude Code CLI directly against Databricks-hosted Claude models instead of Anthropic's cloud API. This enables repo-aware AI workflows—navigation, diffs, testing, MCP tools—right insi...
- 171 Views
- 0 replies
- 0 kudos
- 6251 Views
- 4 replies
- 2 kudos
Accessing Databricks Volumes from a Serving Endpoint Using a Custom Model Class in Unity Catalog
Hi everyone,I’m looking for accessing Unity Catalog (UC) Volumes from a Databricks Serving Endpoint. Here’s my current setup:I have a custom AI model class for inference, which I logged into Unity Catalog using mlflow.pyfunc.log_model.I’ve created a ...
- 6251 Views
- 4 replies
- 2 kudos
- 2 kudos
Serverless Model Serving does not mount the UC Volumes FUSE path (/Volumes), so references to “/Volumes/…” inside a custom pyfunc’s model code will fail at container build or runtime. The correct pattern is to package any required files (like your ...
- 2 kudos
- 201 Views
- 2 replies
- 1 kudos
Resolved! Can serverless environments not use SynapseML's LightGBM?
When I use LightGBM, I get the following error on the line below: 'str' object has no attribute 'getParam'.Is this because serverless cannot run the JAR files that SynapseML depends on?File /local_disk0/.ephemeral_nfs/envs/pythonEnv-b0d5f8ce-8426-443...
- 201 Views
- 2 replies
- 1 kudos
- 1 kudos
Sorry, I just found out while checking the official documentation that starting from November, dependency JAR files are supported.I’ll give it a try and see how well it works.If anyone has any insights, I would greatly appreciate your guidance.
- 1 kudos
- 348 Views
- 2 replies
- 3 kudos
Resolved! Databricks Model Serving Endpoint Fails: “_USER not found for feature table”
Hi Databricks Community,I’m trying to deploy a model serving endpoint that uses Databricks Feature Store (Unity Catalog, online tables).My offline and online feature tables are created and visible in Databricks.The model is logged with FeatureEnginee...
- 348 Views
- 2 replies
- 3 kudos
- 3 kudos
Thanks for the reply It is very useful and comprehensive.I managed to find another solution to the problem so I wanted to share some additional details on this topic:I was using 15.4 LTS ML Runtime, this could have caused the problem - I did not swit...
- 3 kudos
- 3967 Views
- 5 replies
- 1 kudos
Resolved! Lacking support for column-level select grants or attribute-based access control
In the Unity Catalog launch and its accompanying blog post, one of the primary selling points was a set of granular access control features that would at least partially eliminate the need to create a multitude of separate table views and the attenda...
- 3967 Views
- 5 replies
- 1 kudos
- 1 kudos
@at-khatri Databricks ABAC is in Public Preview nowhttps://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/
- 1 kudos
- 349 Views
- 1 replies
- 4 kudos
Hackathon Project: Recipe Recommendation Engine with Traditional ML + Genie on Databricks Free Edit
Hi everyone, For the Databricks Free Edition Hackathon, I wanted to show that traditional ML still has a big role today, and how it can work hand-in-hand with Databricks’ newer AI tooling. As a concrete use case, I created a recipe recommendation eng...
- 349 Views
- 1 replies
- 4 kudos
- 4 kudos
This is amazing @hasnat_unifeye. Well done and good luck for the hackathon.
- 4 kudos
- 239 Views
- 1 replies
- 0 kudos
AutoML Deprecation?
Hi All,It looks like AutoML is set to be deprecated with the next major version (although the note isn't specific on if that's 18). I haven't seen any announcement or alert about this impending change. Did I just miss it? I know we have teams using t...
- 239 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @kevin11 ,I guess it's their standard way of library deprecation policy. In their docs they mentioned that when a library is planned for removal, Databricks takes following steps to notify customers:So they've added those note to AutoMl docs:And y...
- 0 kudos
- 399 Views
- 1 replies
- 0 kudos
Resolved! How to store & update a FAISS Index in Databricks
I’m currently using FAISS in a Databricks notebook to perform semantic search in text data. My current workflow looks like this:encode ~10k text entries using an embedding model.build a FAISS index in memory.run similarity searches using index.search...
- 399 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ashfire , Here’s a practical path to scale your FAISS workflow on Databricks, along with patterns to persist indexes, incrementally add embeddings, and keep metadata aligned. Best practice to persist/load FAISS indexes on Databricks Use faiss...
- 0 kudos
- 3679 Views
- 2 replies
- 0 kudos
DAB - Add/remove task depending on workspace.
I use DAB for deploying Jobs, I want to add a specific Task in dev only but not in staging or prod. Is there any way to achieve this using DAB ?
- 3679 Views
- 2 replies
- 0 kudos
- 0 kudos
You can define specific resources by target in DAB as shown here. This is valid for jobs and/or tasks:For instance, in my case:I think, best option (but not available as far as I know) would be to be able to define "include" sections by target, inste...
- 0 kudos
