- 1388 Views
- 1 replies
- 0 kudos
Lakehouse monitoring generates broken queries
Hi everyone,I’m setting up Databricks Lakehouse Monitoring to track my model’s performance using an inference-regression monitor. I’ve completed all the required configuration and successfully launched my first monitoring run.The quality tables are g...
- 1388 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @the_p_l ,I want to confirm that I understand your situation correctly. You mentioned that you are not adding any custom code to the deployed Lakehouse Monitoring setup, and you believe the issue is related to the inline comments generated during ...
- 0 kudos
- 2085 Views
- 2 replies
- 0 kudos
Issue Importing transformers Library on Databricks
I'm experiencing an issue when trying to import the "transformers" library in a Databricks notebook. The import statement causes the notebook to hang indefinitely without any error messages. The library works perfectly on my local machine using Anaco...
- 2085 Views
- 2 replies
- 0 kudos
- 0 kudos
@Deniz_Bilgin yeah some packages are not compatible with runtime. Use a Stable Version Try installing a known working version:%pip install transformers==4.41.2Dependency Issues:Conflicts with preinstalled libraries like urllib3 (e.g., version ...
- 0 kudos
- 7245 Views
- 4 replies
- 3 kudos
Resolved! Data Drift & Model Comparison in Production MLOps: Handling Scale Changes with AutoML
BackgroundI'm implementing a production MLOps pipeline for part classification using Databricks AutoML. My pipeline automatically retrains models when new data arrives and compares performance with existing production models.The ChallengeI've encount...
- 7245 Views
- 4 replies
- 3 kudos
- 3 kudos
Here are my thoughts to the questions you pose. However, it is important that you dig into the documentation to fully understand the capabilites of Lakehouse Monitoring. I will also be helpful if you deploy it to understand the mechanics of how it wo...
- 3 kudos
- 2300 Views
- 3 replies
- 1 kudos
Endpoint deployment is very slow
HI team I am testing some changes on UAT / DEV environment and noticed that the model endpoint are very slow to deploy. Since the environment is just testing and not serving any production traffic, I was wondering if there was a way to expedite this ...
- 2300 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @WiliamRosa Thanks for your response on this. I have been using the setting you described aboved, with the exception of `scale_to_zero`. PFA screenshot of the endpoint settings. My deployment is a simple Pytorch Deep Learning model wrapped in a `s...
- 1 kudos
- 3656 Views
- 7 replies
- 1 kudos
PERMISSION_DENIED: Endpoint databricks-claude-3-7-sonnet is not allowed
I am working on a use case in my personal Databricks account - AgenticDataPipeline. I am getting below error when using Anthropic Claude 3.7 Sonnet AI model15:52:56.267 | ERROR | [Pipeline] Task failed: ACCOUNT DATA NORMALIZATION: ops_bronze.account...
- 3656 Views
- 7 replies
- 1 kudos
- 1 kudos
I don’t have a working solution as of now. Trying other options but the models are not working.
- 1 kudos
- 6863 Views
- 1 replies
- 1 kudos
Resolved! VLLM dependency Issues with DBR 17.0
I am trying to install vllm and its appropriate dependencies on DataBricks Cluster DBR 17.0.I have the vllm wheel and its dependent wheels within the databricks catalog in a Volumes folder.I am running the following command below to install all the w...
- 6863 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @nish7 here are some helpful suggestions, I did some digging and confirmed that the issue you’re encountering stems from conflicting dependencies. Specifically, there’s a hard version clash around the numba library: => vLLM 0.10.1.1 requires exac...
- 1 kudos
- 10110 Views
- 3 replies
- 2 kudos
Resolved! Avoid Using MLFlow to log runs
I am running some code to smooth columns of data and fill in missing values. The algorithms being used are automatically logged to an experiment using MLFlow which makes a 15 second job take 9 minutes and causes out of memory errors in another instan...
- 10110 Views
- 3 replies
- 2 kudos
- 2 kudos
very helpful. WIthout this command, optuna tries to register the info of each experiment, which make it so slow.
- 2 kudos
- 4666 Views
- 2 replies
- 4 kudos
Resolved! Custom docker container for GPU compute using python 3.12
Hello!I have a GPU compute using my own custom docker container. I am trying to upgrade from python 3.10 to 3.12 since 3.10 EOL is next year, but I cannot find any official documentation around this for Databricks runtime.My current working solution ...
- 4666 Views
- 2 replies
- 4 kudos
- 4 kudos
Yeah, I have been down the rabbit hole of trying to force 3.12 into GPU containers to no avail. With 3.10 end of life next year, I hope that Databricks is working on an official solution.
- 4 kudos
- 2453 Views
- 3 replies
- 3 kudos
Resolved! How to enable Public Preview features in Databricks Free Edition?
I’m using Databricks Free Edition but don’t see an option to enable Public Preview features. Is this supported in Free Edition, or only in paid tiers?I was trying to use the ai_query feature, but since it’s in Public Preview, I couldn’t find a way to...
- 2453 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @atharvaghodekar! Currently, the Free Edition doesn’t provide the ability to enable Preview features.
- 3 kudos
- 3123 Views
- 4 replies
- 4 kudos
Resolved! Distributed Optuna and MLflow
Hello All, I just tried running the following notebook (https://docs.databricks.com/aws/en/notebooks/source/machine-learning/optuna-mlflow.html) on the Databricks Free Edition platform , through Microsoft Account Authentication. It takes 15 minutes ...
- 3123 Views
- 4 replies
- 4 kudos
- 4 kudos
Great. Thank you. That worked. I still need more compute and networking resources to make it justifiable, but this confirms that it works !!!
- 4 kudos
- 5447 Views
- 3 replies
- 2 kudos
Problem with ipywidgets and plotly on Databricks
Hi everyone, I am encountering a problem when using ipywidgets with plotly on Databricks. I am trying to pass interactive arguments to a function and then plot with plotly. When I do the followingdef f(m, b) : plt.figure(2) x = np.linspace(-10,...
- 5447 Views
- 3 replies
- 2 kudos
- 2 kudos
I'm having this same exact issue. @moseb any chance you were able to figure it out?
- 2 kudos
- 2482 Views
- 6 replies
- 12 kudos
ML course affected by phasing out of community edition
Hi there - I'm a professor at CMU teaching a course on scalable ML, and the last few semesters we have run assignments using the DataBricks Community Edition, which is no longer open for new accounts. The free accounts aren't a great substitute for ...
- 2482 Views
- 6 replies
- 12 kudos
- 12 kudos
@wcohen have you looked into Databricks support for universities: https://www.databricks.com/university There's also this section which could be good to point out to your students:All the best,BS
- 12 kudos
- 2228 Views
- 4 replies
- 11 kudos
Resolved! [Databricks User Research] AI Assistance for Data Science Work
Hi all! I'm running a user research series on the topic of AI Assistance for data science work (e.g. exploratory data analysis, feature engineering, model eval, etc.). If you are a Databricks Notebooks user, I would love to pick your brain on your pr...
- 2228 Views
- 4 replies
- 11 kudos
- 11 kudos
@DataBrickatorPerfect timing! I'm a heavy Databricks Notebooks user working on performance optimization and ML pipelines.My daily workflows involve:EDA on 100M+ record Iceberg tablesReal-time feature engineering with Spark StreamingCustom indexing st...
- 11 kudos
- 1626 Views
- 2 replies
- 5 kudos
Resolved! Does Databricks AutoML support multi-target/multi-output classification?
Hi Databricks community,I'm working on a classification problem where I need to predict 50 different target columns simultaneously using the same input features (X), commonly known as multi-target or multi-output classification.My question: Does Data...
- 1626 Views
- 2 replies
- 5 kudos
- 5 kudos
Long story short after lots of research: No.. Sadly this is not possible. Yet the two mentioned options could be considered as workaround
- 5 kudos
- 3515 Views
- 3 replies
- 3 kudos
Resolved! Error in automl.regress
Hi,I'm running example notebook from https://docs.databricks.com/aws/en/machine-learning/automl/regression-train-api on a node with ML cluster 17.0 (includes Apache Spark 4.0.0, Scala 2.13) and getting error at from databricks import automlsummary = ...
- 3515 Views
- 3 replies
- 3 kudos
- 3 kudos
Ilir, greetings!Thank you for a prompt response. Unfortunately, none of the suggested solutions works. I checked with Genie:"The error occurs because databricks-automl is not available for Databricks Runtime 17.0.x. Databricks AutoML is not supported...
- 3 kudos
