cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

wcohen
by New Contributor
  • 214 Views
  • 6 replies
  • 11 kudos

ML course affected by phasing out of community edition

Hi there - I'm a professor at CMU teaching a course on scalable ML, and the last few semesters we have run assignments using the DataBricks Community Edition, which is no longer open for new accounts.  The free accounts aren't a great substitute for ...

  • 214 Views
  • 6 replies
  • 11 kudos
Latest Reply
BS_THE_ANALYST
Honored Contributor III
  • 11 kudos

@wcohen have you looked into Databricks support for universities: https://www.databricks.com/university There's also this section which could be good to point out to your students:All the best,BS

  • 11 kudos
5 More Replies
DataBrickator
by Databricks Employee
  • 194 Views
  • 4 replies
  • 11 kudos

Resolved! [Databricks User Research] AI Assistance for Data Science Work

Hi all! I'm running a user research series on the topic of AI Assistance for data science work (e.g. exploratory data analysis, feature engineering, model eval, etc.). If you are a Databricks Notebooks user, I would love to pick your brain on your pr...

  • 194 Views
  • 4 replies
  • 11 kudos
Latest Reply
ck7007
New Contributor II
  • 11 kudos

@DataBrickatorPerfect timing! I'm a heavy Databricks Notebooks user working on performance optimization and ML pipelines.My daily workflows involve:EDA on 100M+ record Iceberg tablesReal-time feature engineering with Spark StreamingCustom indexing st...

  • 11 kudos
3 More Replies
spearitchmeta
by New Contributor II
  • 191 Views
  • 2 replies
  • 4 kudos

Resolved! Does Databricks AutoML support multi-target/multi-output classification?

Hi Databricks community,I'm working on a classification problem where I need to predict 50 different target columns simultaneously using the same input features (X), commonly known as multi-target or multi-output classification.My question: Does Data...

  • 191 Views
  • 2 replies
  • 4 kudos
Latest Reply
spearitchmeta
New Contributor II
  • 4 kudos

Long story short after lots of research: No.. Sadly this is not possible. Yet  the two mentioned options could be considered as workaround

  • 4 kudos
1 More Replies
staskh
by New Contributor III
  • 300 Views
  • 3 replies
  • 3 kudos

Resolved! Error in automl.regress

Hi,I'm running example notebook from https://docs.databricks.com/aws/en/machine-learning/automl/regression-train-api on a node with ML cluster 17.0 (includes Apache Spark 4.0.0, Scala 2.13) and getting error at from databricks import automlsummary = ...

staskh_0-1756215619847.png
  • 300 Views
  • 3 replies
  • 3 kudos
Latest Reply
staskh
New Contributor III
  • 3 kudos

Ilir, greetings!Thank you for a prompt response. Unfortunately, none of the suggested solutions works. I checked with Genie:"The error occurs because databricks-automl is not available for Databricks Runtime 17.0.x. Databricks AutoML is not supported...

  • 3 kudos
2 More Replies
spearitchmeta
by New Contributor II
  • 206 Views
  • 2 replies
  • 1 kudos

Data Drift & Model Comparison in Production MLOps: Handling Scale Changes with AutoML

BackgroundI'm implementing a production MLOps pipeline for part classification using Databricks AutoML. My pipeline automatically retrains models when new data arrives and compares performance with existing production models.The ChallengeI've encount...

  • 206 Views
  • 2 replies
  • 1 kudos
Latest Reply
BigRoux
Databricks Employee
  • 1 kudos

Have you explored Lakehouse Monitoring?  It provides a comprehensive solution for drift detection.  You can read more here: https://docs.databricks.com/aws/en/lakehouse-monitoring/   Hope this helps, Louis.

  • 1 kudos
1 More Replies
naveen_marthala
by Contributor
  • 11691 Views
  • 3 replies
  • 3 kudos

Resolved! How to PREVENT mlflow's autologging from logging ALL runs?

I am logging runs from jupyter notebook. the cells which has `mlflow.sklearn.autlog()` behaves as expected. but, the cells which has .fit() method being called on sklearn's estimators are also being logged as runs without explicitly mentioning `mlflo...

  • 11691 Views
  • 3 replies
  • 3 kudos
Latest Reply
Joe_Breath1
New Contributor III
  • 3 kudos

It looks like MLflow auto-logging is kicking in by default whenever you call .fit(), which is why you’re seeing runs even without explicitly using mlflow.sklearn.autolog(). To fix this, you can disable the global autologging and only trigger it when ...

  • 3 kudos
2 More Replies
dangkhai
by New Contributor II
  • 305 Views
  • 3 replies
  • 0 kudos

Having a different performance while use GPU and CPU

I'm building a model that is mostly sklearn libraries, but I'm also using TF-IDF and RandomForest. In theory, they only need a CPU to work properly, but in fact, when I use a physical computer with about 32 GB of RAM, it runs very fast. There are som...

  • 305 Views
  • 3 replies
  • 0 kudos
Latest Reply
BR_DatabricksAI
Contributor III
  • 0 kudos

Hello @dangkhai : Have the below link and see parallelism enabled during transformation. ParallelTextProcessing/parallelizing_text_processing.ipynb at master · rafaelvalero/ParallelTextProcessing

  • 0 kudos
2 More Replies
Yuki
by Contributor
  • 2321 Views
  • 1 replies
  • 0 kudos

When does everyone utilize the model register?

Hi, I'm Yuki,I'm considering when I should use register_model.In my case, I'm running the training batch once a week and if the model is good, I want to update the champion.I have created the code to register the model if the score is the best.# star...

  • 2321 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kumaran
Databricks Employee
  • 0 kudos

Hi @Yuki, Thank you for contacting the Databricks community. If you run register_model with the same run twice, you’ll create multiple versions pointing to the same source. To avoid that, you can check if the run is already registered before creati...

  • 0 kudos
Nishat
by New Contributor
  • 2336 Views
  • 1 replies
  • 0 kudos

Custom Multi-agent deployment error

Hi. I am deploying a custom multi-agent system comprising of a genie agent and a RAG solution. While deploying, I am getting the following error:I am using 16.1 ML (Node: Standard_D4ads_v5 16GB,4 core) cluster and I am using the following code for lo...

Nishat_0-1746428192246.png Nishat_1-1746436645946.png Nishat_0-1746437076157.png Nishat_0-1746436851190.png
  • 2336 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kumaran
Databricks Employee
  • 0 kudos

Hi @Nishat, Thank you for contacting the Databricks community. Which LLM model are you currently utilising?

  • 0 kudos
skosaraju
by New Contributor III
  • 2279 Views
  • 1 replies
  • 0 kudos

Infer_signature for a dictionary datasets during mlflow registration

Hello community,Can you please guide me here. I am trying to build custom Ensemble model where I will be passing a dictionary of datasets to the fit() and predict() with the keys being the model_names and value being the respective datasets for each ...

  • 2279 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kumaran
Databricks Employee
  • 0 kudos

Hi @skosaraju, Thank you for contacting Databricks support! Because infer_signature() cannot handle a dict of DataFrames directly, you will need to convert this structure into a dictionary of row dicts, or manually build a ModelSignature. Option A: F...

  • 0 kudos
ecram
by New Contributor
  • 1144 Views
  • 1 replies
  • 0 kudos

How to Reduce Log Latency for AI Gateway-Enabled Inference Tables in Model Serving?

Hi everyone,I've recently deployed a custom model using Databricks Model Serving with AI Gateway-enabled inference tables. The model is built with:Python 3.11.11LightGBM 4.5.0MLflow 2.13.1I’ve noticed that the inference logs can take up to 1 hour to ...

  • 1144 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kumaran
Databricks Employee
  • 0 kudos

Hi @ecram, Thank you for contacting Databricks community. As per the doc below, you'll see the latency for 1 hour for log delivery in the inference table. https://docs.databricks.com/aws/en/ai-gateway/inference-tables#:~:text=You%20can%20expect%20log...

  • 0 kudos
Aravinda
by New Contributor III
  • 370 Views
  • 5 replies
  • 2 kudos

Resolved! Databricks Machine Learning Practitioner Plan - DBC section unavailability

Hi Everyone,I am not able to locate any DBC folders for each course present in the machine learning practitioner plan.Earlier, we used to have DBC sections where we can access the course and lab materials.Do we have any solution to this??? Or can som...

  • 370 Views
  • 5 replies
  • 2 kudos
Latest Reply
Aravinda
New Contributor III
  • 2 kudos

Thanks @szymon_dybczak !!

  • 2 kudos
4 More Replies
guilhermeneves
by New Contributor II
  • 426 Views
  • 2 replies
  • 3 kudos

Resolved! Unexpected ProtoBuf Version Changes on DBR 15.4 LTS Causing databricks-feature-client Failures

Hi all,I noticed an issue starting from Friday, Aug 15 through Monday, Aug 18. Our clusters running DBR 15.4 LTS experienced unexpected upgrades of the protobuf library:Friday: protobuf was at version 4.24.1 (as documented here: DBR 15.4 LTS release ...

  • 426 Views
  • 2 replies
  • 3 kudos
Latest Reply
Advika
Databricks Employee
  • 3 kudos

Hello @guilhermeneves! Could you confirm if pip was used to install any Python packages in your workload? The version shifts you’re seeing are likely related to unpinned pip installs: when package versions aren’t pinned, dependency changes from third...

  • 3 kudos
1 More Replies
senura96
by New Contributor
  • 204 Views
  • 1 replies
  • 0 kudos

mlflow.store.artifact.cloud_artifact_repo. issue in Databricks APPs backend

INFO mlflow.store.artifact.cloud_artifact_repo: Failed to complete request, possibly due to credential expiration. Refreshing credentials and trying again... (Error: API request to https://dbc-d4dd6237-edea.cloud.databricks.com/api/2.0/fs/create-down...

  • 204 Views
  • 1 replies
  • 0 kudos
Latest Reply
WiliamRosa
New Contributor II
  • 0 kudos

Why MLflow and Databricks library versions?Did you try increasing the MLflow artifact upload/download timeout, for example:import osos.environ["MLFLOW_ARTIFACT_UPLOAD_DOWNLOAD_TIMEOUT"] = "120"

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels