- 1901 Views
- 3 replies
- 0 kudos
Problem loading a pyfunc model in job run
Hi, I'm currently working on a automated job to predict forecasts using a notebook than work just fine when I run it manually, but keep failling when schedueled, here is my code: import mlflow # Load model as a PyFuncModel. loaded_model = mlflow.pyf...
- 1901 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey AmineM! If your MLflow model loads fine in a Databricks notebook but fails in a scheduled job on serverless compute with an error like: TypeError: code() argument 13 must be str, not int the root cause is almost always a mismatch between the ...
- 0 kudos
- 574 Views
- 4 replies
- 2 kudos
Resolved! What is the most efficient way of running sentence-transformers on a Spark DataFrame column?
We're trying to run the bundled sentence-transformers library from SBert in a notebook running Databricks ML 16.4 on an AWS g4dn.2xlarge [T4] instance.However, we're experiencing out of memory crashes and are wondering what the optimal to run sentenc...
- 574 Views
- 4 replies
- 2 kudos
- 2 kudos
If you didn't get this to work with Pandas API on Spark, you might also try importing and instantiating the SentenceTransformer model inside the pandas UDF for proper distributed execution. Each executor runs code independently, and when Spark execut...
- 2 kudos
- 49 Views
- 0 replies
- 0 kudos
GenAI experiment tracing does not render markdown images
When traces include base64 encoded images in Markdown, they do not render properly. This makes the analysis of traces including images difficult.Just for context, the same trace in other tracing tools like LangSmith renders as expected. An example of...
- 49 Views
- 0 replies
- 0 kudos
- 98 Views
- 1 replies
- 0 kudos
Inference Tables Empty
Hello,I have been using Databricks Free Platform for a while. Everything seems to work well. However, I've been trying to generate the payload from the deployed endpoint and I got always an empty inference table.When I check the configuration, I got ...
- 98 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @salesbrj ,Most probably this will be related to limitation in Free Edition. In limitations section I can see following entry:No custom models on GPU or batch inferencehttps://docs.databricks.com/aws/en/getting-started/free-edition-limitations
- 0 kudos
- 1160 Views
- 3 replies
- 1 kudos
Distributed SparkXGBRanker training: failed barrier ResultStage
I'm following a variation of the tutorial [here](https://assets.docs.databricks.com/_extras/notebooks/source/xgboost-pyspark-new.html) to train an `SparkXGBRanker` in distributed mode. However, the line:pipeline_model = pipeline.fit(data) Is throwing...
- 1160 Views
- 3 replies
- 1 kudos
- 1 kudos
You have already mentioned you did turn off autoscaling, please try the num_workers too Step 1: Disable Dynamic Resource Allocation: Use spark.dynamicAllocation.enabled = false Step 2: Configure num_workers to Match Fixed Resources After disabling dy...
- 1 kudos
- 6087 Views
- 7 replies
- 3 kudos
Unable to Use VectorAssembler in PySpark 3.5.0 Due to Whitelisting
Hi,I am currently using PySpark version 3.5.0 on my Databricks cluster. Despite setting the required configuration using the command: spark.conf.set("spark.databricks.ml.whitelist", "true"), I am still encountering an issue while trying to use the Ve...
- 6087 Views
- 7 replies
- 3 kudos
- 3 kudos
I also had this error trying to use ML on free edition. Is ML features working for free edition.
- 3 kudos
- 184 Views
- 2 replies
- 4 kudos
Resolved! Can't use pyspark bucketizer
As title suggests, I am struggling to use pyspark bucketizer as I repeatedly get the following error:File <command-8301298062763331>, line 4 2 from pyspark.ml.feature import Bucketizer 3 spark = SparkSession.builder.appName("test").getOrC...
- 184 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @wise_centipede ,In your Serverless compute select Environment Version: 4 and it will work With version below 4 I've got the same error as you:And when I've upgrade serverless environment ot version 4 it works as expected
- 4 kudos
- 695 Views
- 1 replies
- 0 kudos
Lakehouse monitoring generates broken queries
Hi everyone,I’m setting up Databricks Lakehouse Monitoring to track my model’s performance using an inference-regression monitor. I’ve completed all the required configuration and successfully launched my first monitoring run.The quality tables are g...
- 695 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @the_p_l ,I want to confirm that I understand your situation correctly. You mentioned that you are not adding any custom code to the deployed Lakehouse Monitoring setup, and you believe the issue is related to the inline comments generated during ...
- 0 kudos
- 1142 Views
- 2 replies
- 0 kudos
Issue Importing transformers Library on Databricks
I'm experiencing an issue when trying to import the "transformers" library in a Databricks notebook. The import statement causes the notebook to hang indefinitely without any error messages. The library works perfectly on my local machine using Anaco...
- 1142 Views
- 2 replies
- 0 kudos
- 0 kudos
@Deniz_Bilgin yeah some packages are not compatible with runtime. Use a Stable Version Try installing a known working version:%pip install transformers==4.41.2Dependency Issues:Conflicts with preinstalled libraries like urllib3 (e.g., version ...
- 0 kudos
- 135 Views
- 1 replies
- 0 kudos
Help Finding Course Notebook Machine Learning Practitioner Learning Plan
Hi,This is Vandna. I’m currently taking the Machine Learning Practitioner Learning Plan course, but I’m unable to locate the corresponding notebook. Could you please share the link to the notebook or guide me on where and how I can access it?Thank yo...
- 135 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Vandna-Shobran! The Machine Learning Practitioner Learning Plan modules are free self-paced and do not include hands-on labs. To access the labs, you would need to either: Enroll in the ILT (Instructor-Led Training) courses - This will grant y...
- 0 kudos
- 761 Views
- 4 replies
- 3 kudos
Resolved! Data Drift & Model Comparison in Production MLOps: Handling Scale Changes with AutoML
BackgroundI'm implementing a production MLOps pipeline for part classification using Databricks AutoML. My pipeline automatically retrains models when new data arrives and compares performance with existing production models.The ChallengeI've encount...
- 761 Views
- 4 replies
- 3 kudos
- 3 kudos
Here are my thoughts to the questions you pose. However, it is important that you dig into the documentation to fully understand the capabilites of Lakehouse Monitoring. I will also be helpful if you deploy it to understand the mechanics of how it wo...
- 3 kudos
- 405 Views
- 3 replies
- 2 kudos
Unable to register Scikit-learn or XGBoost model to unity catalog
Hello, I'm following the tutorial provided here https://docs.databricks.com/aws/en/notebooks/source/mlflow/mlflow-classic-ml-e2e-mlflow-3.html for ML model management process using ML FLow, in a unity-catalog enabled workspace, however I'm facing an ...
- 405 Views
- 3 replies
- 2 kudos
- 2 kudos
Maybe add missing: mlflow.set_tracking_uri("databricks")mlflow.set_registry_uri("databricks")
- 2 kudos
- 418 Views
- 3 replies
- 1 kudos
Endpoint deployment is very slow
HI team I am testing some changes on UAT / DEV environment and noticed that the model endpoint are very slow to deploy. Since the environment is just testing and not serving any production traffic, I was wondering if there was a way to expedite this ...
- 418 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @WiliamRosa Thanks for your response on this. I have been using the setting you described aboved, with the exception of `scale_to_zero`. PFA screenshot of the endpoint settings. My deployment is a simple Pytorch Deep Learning model wrapped in a `s...
- 1 kudos
- 988 Views
- 7 replies
- 1 kudos
PERMISSION_DENIED: Endpoint databricks-claude-3-7-sonnet is not allowed
I am working on a use case in my personal Databricks account - AgenticDataPipeline. I am getting below error when using Anthropic Claude 3.7 Sonnet AI model15:52:56.267 | ERROR | [Pipeline] Task failed: ACCOUNT DATA NORMALIZATION: ops_bronze.account...
- 988 Views
- 7 replies
- 1 kudos
- 1 kudos
I don’t have a working solution as of now. Trying other options but the models are not working.
- 1 kudos
- 778 Views
- 1 replies
- 1 kudos
Resolved! VLLM dependency Issues with DBR 17.0
I am trying to install vllm and its appropriate dependencies on DataBricks Cluster DBR 17.0.I have the vllm wheel and its dependent wheels within the databricks catalog in a Volumes folder.I am running the following command below to install all the w...
- 778 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @nish7 here are some helpful suggestions, I did some digging and confirmed that the issue you’re encountering stems from conflicting dependencies. Specifically, there’s a hard version clash around the numba library: => vLLM 0.10.1.1 requires exac...
- 1 kudos
