Hi Community,I've playing around with AutoML and started with a simple forecast for Databricks samples.I used a copy of table samples.tpch.orders.To my supprise only integer types were available as Predicat Target. The field I was interested in forec...
I am trying to configure the Gemini Vertex API in Databricks. In simple Python code, everything works fine, which indicates that I have correctly set up the API and credentials. Error message: {"error_code":"INVALID_PARAMETER_VALUE","message":"INVALI...
The edit_mode for Databricks jobs cannot be overridden using the bundle. Based on the jobs REST API docs, there is a functionality to set this parameter but in the bundle docs, it's not available. How can I use this in the bundle to override the para...
The `edit_mode` property cannot be set by design. It is set to `UI_LOCKED` on bundle deployment to let viewers of the job in the UI know that any changes they make to the job instance are going to be clobbered the next time someone runs a bundle depl...
Hey.I'm building a DevOps/MLOps pipeline to train/register simple scikit learn model.I created a simple Databricks Workflow to execute training and register task on specific .git branch. (Workflow is setup with Databricks Repo on specifc branch, with...
I had same issue while trying to call notebook from workflow. I was able to do what you did. But it needs new experiment name for each run, so I had to do this:# Set the experimentexperiment_name = f"/Workspace/MLOps/{env}/experiment/{experiment}_{ti...
Hey Guys, I'm trying to make automated process to run ML training sessions using mlflow and databricks jobs.While developing the model on my local machine using IDE, When finished I have a template notebook that get as parameters the mlflow project p...
Hi,Were you able to figure out this one? I have same issue trying to call training notebook from workflow. Each run needs a new experiment name which I can do but then it creates a new experiment ID/name for each workflow run. Where as when you run f...
Hello Community,I am seeking assistance with an issue related to serving a custom Hugging Face model (M2M100). I successfully registered the model in Unity Catalog using the MLflow Python libraries without any problems. However, when attempting to se...
see https://community.databricks.com/t5/machine-learning/serving-endpoint-container-image-creation-fails/td-p/83458
how can i pass parameter from Azure data factory rest web API to delta live Databricks?I get this error: "Py4JJavaError: An error occurred while calling o382.getArgument.: com.databricks.dbutils_v1.InputWidgetNotDefined: No input widget named *** def...
@girl_chan The error you are encountering usually occurs when a widget referenced in a Databricks notebook is not defined. In your case, the issue is likely related to how you pass parameters from Azure Data Factory (ADF) to Delta Live Tables (DLT) i...
Hi Databricks support,I'm training an ML model using mlflow on DBR 13.3 LTS ML, Spark 3.4.1 using databricks.automl_runtime 0.2.17 and databricks.automl 1.20.3, with shap 0.45.1. My training data has two float-type columns with three or fewer unique ...
@NandiniN Were you able to use my reprex above to investigate this issue at all? Thank you.
Is it possible to rollback changes made to a cluster? The problem I'm trying to solve is to recover from an accidental change made by a user on a cluster that affects interactive and job runs. Cluster policies help, but the policy still provides the ...
@User16826990884 Along with what @sajith_appukutt mentioned, we can achive this viaVersion Control for Cluster Configurations: Store cluster configurations in JSON files in GitHub or another version control system.In case of accidental changes, you c...
The automated notebook pipeline in an AutoML experiment applies StandardScaler to all numerical features in the training dataset as part of the PreProcessor. See below.But I want a more nuanced and varied treatment of my numeric values (e.g. I have l...
I'm looking to scale xgboost to large datasets which won't fit in memory on a single large EC2 instance (billions to tens of billions of rows scale). I also require many of the bells & whistles of regular in-memory xgboost slash lightgbm including:Th...
Hi EveryoneI tried to follow the same steps in Topic from Text on similar data as example. However, when I tri to fit the model with data I get this error.IllegalArgumentException: requirement failed: Column features must be of type equal to one of t...
Hi @amirA ,The LDA model expects the features column to be of type Vector from the pyspark.ml.linalg module, specifically either a SparseVector or DenseVector, whereas you have provided Row type.You need to convert your Row object to SparseVector.Che...
Hi, surprisingly we have found 2 new folders with some short logfiles in our Databricks workspace:ls -lFr sparse_logs/ tensorboard/tensorboard/:-rwxrwxrwx 1 root root 88 Sep 2 11:26 events.out.tfevents.1725275744.0830-063833-n68nsxoq-10-139-64-10.20...
Hi everyone,I’m looking for accessing Unity Catalog (UC) Volumes from a Databricks Serving Endpoint. Here’s my current setup:I have a custom AI model class for inference, which I logged into Unity Catalog using mlflow.pyfunc.log_model.I’ve created a ...
Additionally, I log the model as shown below, with MicrosoftResnet50Model being my custom inference class with load_context and predict methods:with mlflow.start_run():model_info = mlflow.pyfunc.log_model(REGISTERED_MODEL_NAME,python_model=MicrosoftR...
What is the best practice for applying MLFlow to clustering algorithms? What are the kinds of metrics customers track?
Good question! I'll divide my suggestions into 2 parts:(1) In terms of MLflow Tracking, clustering is pretty similar to other ML workflows, so not much changes.(2) In terms of specific parameters, metrics, etc. to track, clustering is very different...
