- 2713 Views
- 3 replies
- 1 kudos
How to use Databricks secrets on MLFlow conda dependencies?
Hi!Do you know if it's correct to use the plain user and token for installing a custom dependency (an internal python package) in a mlflow registered model? (it's the only way I get it working because if not it can't install the dependency) It works,...
- 2713 Views
- 3 replies
- 1 kudos
- 1 kudos
Did someone solve this? I'm currently forced to download some libraries using an artifactory pypi mirror. However, I wouldn't want to have my secrets pasted in the conda.yaml file as plain text.
- 1 kudos
- 15686 Views
- 7 replies
- 3 kudos
Resolved! How to PREVENT mlflow's autologging from logging ALL runs?
I am logging runs from jupyter notebook. the cells which has `mlflow.sklearn.autlog()` behaves as expected. but, the cells which has .fit() method being called on sklearn's estimators are also being logged as runs without explicitly mentioning `mlflo...
- 15686 Views
- 7 replies
- 3 kudos
- 3 kudos
Good question—mlflow autologging can easily capture more runs than expected if not configured properly. Managing it carefully improves experiment tracking. Similar control and optimization are important in bussid mod workflows, where users fine-tune ...
- 3 kudos
- 285 Views
- 2 replies
- 0 kudos
Memory error in LightGBM training data processing
I am developing a LightGBM model on Databricks, and I am using the Native API because it offers the widest range of options and allows me to try various approaches.The training data is loaded from a table in the Catalog as a Spark DataFrame. However,...
- 285 Views
- 2 replies
- 0 kudos
- 905 Views
- 2 replies
- 1 kudos
Resolved! Which types of model serving endpoints have health metrics available?
I am retrieving a list of model serving endpoints for my workspace via this API: https://docs.databricks.com/api/workspace/servingendpoints/listAnd then going to retrieve health metrics for each one with: https://[DATABRICKS_HOST]/api/2.0/serving-end...
- 905 Views
- 2 replies
- 1 kudos
- 1 kudos
Your observation is correct—this behavior is expected.Endpoints with entity_type = FOUNDATION_MODEL_API do not expose health metrics via the /metrics endpoint, which is why you’re getting 404 responses. These endpoints are fully managed, multi-tenant...
- 1 kudos
- 320 Views
- 1 replies
- 0 kudos
Resolved! AWS GovCloud Feature Availability Question
Hi! I'm trying to determine if Mosaic Vector Search (or is it simply called Vector Search) is available on AWS GovCloud?This shows it is not: https://docs.databricks.com/aws/en/resources/feature-region-supportAnd it's not mentioned here: https://docs...
- 320 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @MattBuck ,It's not available on AWS GovCloud.1) The first link you attached is the authoritative source for feature availability by region. If you can't find it there it means the feature is not available in specific region2) And I think this lim...
- 0 kudos
- 326 Views
- 1 replies
- 0 kudos
Resolved! Recommended Python UDFs for On-Demand Feature Computation in Databricks
The Databricks documentation page on on-demand feature computation (https://docs.databricks.com/aws/en/machine-learning/feature-store/on-demand-features#what-are-on-demand-features) mentions using Python UDFs for computing on-demand features. What ty...
- 326 Views
- 1 replies
- 0 kudos
- 0 kudos
HI @nb92 , Only Scalar Python UDFs are allowed for on-demand feature computation. This page provides the recommended approach. Best regards,
- 0 kudos
- 1149 Views
- 7 replies
- 3 kudos
MLFlow Detailed Trace view doesn't work in some workspaces
I've created a Databricks Model Serving Endpoint which serves an MLFlow Pyfunc model. The model uses langchain and I'm using mlflow.langchain.autolog().At my company we have some production(-like) workspaces where users cannot e.g. run Notebooks and ...
- 1149 Views
- 7 replies
- 3 kudos
- 3 kudos
Funnily enough, the problem also disappeard on my end this morning Previously, I saw a networking issue in my logs, but that also went away. Let's hope it stays that way!
- 3 kudos
- 1201 Views
- 1 replies
- 0 kudos
Resolved! Using Qwen with vLLM
There are many conflict and dependency issues when trying to install VLLM and use the Qwen models (on serverless), even the v2 families.I tried following this guide https://docs.databricks.com/aws/en/machine-learning/sgc-examples/tutorials/sgc-raydat...
- 1201 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pfzoz -- the "Model architectures failed to be inspected" error you are hitting is a well-known compatibility issue between vLLM, the transformers library, and the Qwen2/2.5-VL model family. The root cause is that vLLM's model registry subprocess...
- 0 kudos
- 901 Views
- 3 replies
- 0 kudos
Resolved! TrainingArguments fails
Hello,I am working on an ML project for text classification and I have a problem.The following piece of code stalls completely. It prints 'start' but never 'end'.from transformers import TrainingArguments print("start") args = TrainingArguments(outpu...
- 901 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @lingareddy_Alva ,Thank you for your reply. I have since been given a cluster with the ML Runtime and the code now works. So I consider the problem solved.
- 0 kudos
- 2341 Views
- 2 replies
- 0 kudos
Identity Resolution
Looking for best solutions for identity resolution. I already have deterministic matching. Exploring probabilistic solutions. Any advice for me?
- 2341 Views
- 2 replies
- 0 kudos
- 0 kudos
Check open source Zingg which runs natively within Databricks https://github.com/zinggAI/zingg
- 0 kudos
- 298 Views
- 1 replies
- 0 kudos
Job compute fails due to BQ permissions
Hello,My databricks workspace is associated to GCP project analytics.But me and my team mostly work on GCP project data-science, which contains the only BQ dataset that we have write access to.I'm trying to automate a pipeline to run on job compute a...
- 298 Views
- 1 replies
- 0 kudos
- 0 kudos
What identity is the job running as? Do you have any settings on the all-purpose cluster that you are not setting on the job-cluster? Maybe you need to provide roles/bigquery.jobUser on project analytics to the job compute service account?
- 0 kudos
- 1177 Views
- 3 replies
- 1 kudos
Resolved! Unable to Access Azure Blob Storage from Databricks Community Edition Notebook
Hi everyone,I’m currently using the Databricks Community Edition and trying to access data stored in Azure Blob Storage from my .ipynb notebook. The storage account is part of my student free Azure subscription.However, I’m not able to establish a co...
- 1177 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi, I think you are referring to Databricks Free edition, in which case this doesn't support the connection to external storage such as Azure Blob storage. Thanks,Emma
- 1 kudos
- 817 Views
- 1 replies
- 1 kudos
Resolved! Issue Running Job on Serverless GPU
I have a job that runs a notebook, the notebook uses serverless GPU (A10) and it keeps failing with a "Run failed with error message Cluster 'xxxxxxxxxxx' was terminated. Reason: UNKNOWN (SUCCESS)". The base environment is 'Standard v4' and I have tr...
- 817 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @rtglorenabasul, Thanks for sharing the details. The behaviour you’re seeing is consistent with an issue in how the job is bringing up Serverless GPU compute, rather than with the notebook code itself. Having done some checks, that error usually m...
- 1 kudos
- 1629 Views
- 4 replies
- 3 kudos
Resolved! Generic Spark Connect ML error. The fitted or loaded model size is too big.
When I train models in the serverless environment V4 (Premium Plan), the system occasionally returns the error message listed below, especially after running the model training code multiple times. We have tried creating new serverless sessions, whic...
- 1629 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @jayshan, I'm sorry for the delayed response to your question. And, thanks for the extra details and for sharing your workaround. This behaviour is tied to how Spark Connect ML works in serverless mode, rather than a traditional JVM/GC leak. On se...
- 3 kudos
- 1660 Views
- 5 replies
- 3 kudos
Resolved! Vector search index initialization very slow
Hello,I am creating a vector search index and selected Compute embeddings for a delta table with 19M records. Delta table has only two columns: ID (selected as index) and Name (selected for embedding). Embedding model is databricks-gte-large-en.Ind...
- 1660 Views
- 5 replies
- 3 kudos
- 3 kudos
Why the deltaSync doesn't compute the embedding in parralel instead of sequential.That a major gap in the architecture no ?
- 3 kudos
