cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

kng88
by New Contributor II
  • 2647 Views
  • 6 replies
  • 7 kudos

How to save model produce by distributed training?

I am trying to save model after distributed training via the following codeimport sys   from spark_tensorflow_distributor import MirroredStrategyRunner   import mlflow.keras   mlflow.keras.autolog()   mlflow.log_param("learning_rate", 0.001)   import...

  • 2647 Views
  • 6 replies
  • 7 kudos
Latest Reply
Xiaowei
New Contributor III
  • 7 kudos

I think I finally worked this out.Here is the extra code to save out the model only once and from the 1st node:context = pyspark.BarrierTaskContext.get() if context.partitionId() == 0: mlflow.keras.log_model(model, "mymodel")

  • 7 kudos
5 More Replies
fa
by New Contributor III
  • 1282 Views
  • 2 replies
  • 5 kudos

How can I view the storage space taken by a registered model using MLFlow?

The information viewed about the registered models on the Models tab is very minimal. Just showing the tags we pass in and version information. How can I get more details about the model such as the size on disk?

  • 1282 Views
  • 2 replies
  • 5 kudos
Latest Reply
Octavian1
Contributor
  • 5 kudos

Hi,I have used the MLFlow client, but I am not sure where to find the size of the model image.The response to client.search_registered_models() I am getting is the following:<RegisteredModel: aliases={}, creation_timestamp=17061..., description='', l...

  • 5 kudos
1 More Replies
Evan_MCK
by Contributor
  • 4398 Views
  • 4 replies
  • 1 kudos

Resolved! Shap Values for predictions from registered model

I have saved a model in the model registry using MLFlow. How can I find the shap values for this model once I have generated predictions in batch mode? Shap tree explainer does not support the mlflow pyfunc model type. When I use mlflow.shap.log_exp...

  • 4398 Views
  • 4 replies
  • 1 kudos
Latest Reply
peacocktv
New Contributor II
  • 1 kudos

Thank you so much for this kind of valuable post its amazing post it may helpful for each visitors. For more information go through my websites here:peacocktv.com/tv | peacocktv.com/tv/xbox

  • 1 kudos
3 More Replies
Robbie
by New Contributor III
  • 5122 Views
  • 5 replies
  • 7 kudos

Why are my MLflow results not showing up in the Experiment UI view?

The issue:None of my MLflow experiment results show up in the Experiment UI. Context:I encountered this issue recently, despite having successfully used the MLFlow UI for the past few weeks.Note: I can still access the experiment runs in a notebook, ...

MLflow Experiment UI view (no results showing up) Accessing via `mlflow.get_runs(...)`
  • 5122 Views
  • 5 replies
  • 7 kudos
Latest Reply
RichardKous
New Contributor II
  • 7 kudos

Hello ! Well, fret not, my friend! I've stumbled upon my own little paradise, visit now , and let me tell you, it boasts a video collection that's nothing short of extraordinary. The performers on this platform? Seasoned pros with an unmatched level ...

  • 7 kudos
4 More Replies
smedegaard
by New Contributor III
  • 2446 Views
  • 3 replies
  • 5 kudos

Resolved! Difference between MLFlow recipes and projects?

MLFlow projects are described asAn MLflow Project is a format for packaging data science code in a reusable and reproducible way, based primarily on conventions. In addition, the Projects component includes an API and command-line tools for running p...

  • 2446 Views
  • 3 replies
  • 5 kudos
Latest Reply
smedegaard
New Contributor III
  • 5 kudos

Thanks for the answer @Priyadarshini G​ . Although a project has a pre-defined folder structure and standard files, it also "... includes an API and command-line tools for running projects, making it possible to chain together projects into workflows...

  • 5 kudos
2 More Replies
ptawil
by New Contributor III
  • 1979 Views
  • 2 replies
  • 4 kudos

Runtime error using MLFlow and Spark on databricks

Here is some model I created:class SomeModel(mlflow.pyfunc.PythonModel): def predict(self, context, input): # do fancy ML stuff # log results pandas_df = pd.DataFrame(...insert predictions here...) spark_df = spark...

  • 1979 Views
  • 2 replies
  • 4 kudos
Latest Reply
Nikhil3107
New Contributor III
  • 4 kudos

Any updates on this? I am running into the same issue@Patrick Tawil​ were you able to solve this problem? If so, do you mind sharing?

  • 4 kudos
1 More Replies
zhh210
by New Contributor III
  • 2604 Views
  • 5 replies
  • 2 kudos

Resolved! mlflow down in workspace?

Mlflow started failing all of a sudden for no reason when logged in databricks community edition:Any idea why this is happening or is there a way to restart the mlflow server?

image
  • 2604 Views
  • 5 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Zheng Han​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...

  • 2 kudos
4 More Replies
CBRS
by New Contributor II
  • 869 Views
  • 2 replies
  • 1 kudos

Mlflow :loading script failed !!

I am using mlflow to track experimentation with databricks but todaty i tried to access my experimetations in dtabricks and i face the error .

image
  • 869 Views
  • 2 replies
  • 1 kudos
Latest Reply
CBRS
New Contributor II
  • 1 kudos

I didn't manage to solve the error . I guess it is related to databricks community cloud because I tested with another account and it all the same.

  • 1 kudos
1 More Replies
Saeid_H
by Contributor
  • 2022 Views
  • 2 replies
  • 0 kudos

Logging spark pipeline model using mlflow spark , leads to PythonSecurityException

Hello,I am currently using a simple pyspark pipeline to transform my training data, fit model and log the model using mlflow.spark. But I get this following error (with mlflow.sklearn it works perfectly fine but due to size of my data I need to use p...

  • 2022 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Saeid Hedayati​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 0 kudos
1 More Replies
its-kumar
by New Contributor III
  • 3235 Views
  • 2 replies
  • 0 kudos

MLFlow Remote model registry connection is not working in Databricks

Dear community,I am having multiple Databricks workspaces in my azure subscription, and I have one central workspace. I want to use the central workspace for model registry and experiments tracking from the multiple other workspaces.So, If I am train...

  • 3235 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Kumar Shanu​ :The error you are seeing (API request to endpoint /api/2.0/mlflow/runs/create failed with error code 404 != 200) suggests that the API endpoint you are trying to access is not found. This could be due to several reasons, such as incorr...

  • 0 kudos
1 More Replies
DK
by New Contributor II
  • 1012 Views
  • 1 replies
  • 1 kudos

Unable to call logged ML model from a different notebook when using Spark ML

Hi, I am a R user and I am experimenting to build an ml model with R and with spark flavoured algorithms in Databricks. However, I am struggling to call a model that is logged as part of the experiment from a different notebook when I use spark flavo...

  • 1012 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Dip Kundu​ :It seems like the error you are facing is related to sparklyr, which is used to interact with Apache Spark from R, and not directly related to mlflow. The error message suggests that an object could not be found, but it's not clear which...

  • 1 kudos
thomasm
by New Contributor II
  • 1923 Views
  • 3 replies
  • 1 kudos

Resolved! Online Feature Store MLflow serving problem

When I try to serve a model stored with FeatureStoreClient().log_model using the feature-store-online-example-cosmosdb tutorial Notebook, I get errors suggesting that the primary key schema is not configured properly. However, if I look in the Featur...

  • 1923 Views
  • 3 replies
  • 1 kudos
Latest Reply
NandiniN
Valued Contributor III
  • 1 kudos

Hello @Thomas Michielsen​ , this error seems to occur when you may have created the table yourself. You must use publish_table() to create the table in the online store. Do not manually create a database or container inside Cosmos DB. publish_table()...

  • 1 kudos
2 More Replies
Orianh
by Valued Contributor II
  • 1524 Views
  • 1 replies
  • 2 kudos

MLflow log pytorch distributed training

Hey Guys,I have few question that i hope you can help me with.I start to train pytorch model in distributed training using petastorm + Horovod like databricks suggest in docs.Q 1:I can see that each worker is train the model, but when epochs are done...

  • 1524 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@orian hindi​ :Regarding your questions:Q1: The error message you are seeing is likely related to a segmentation fault, which can occur due to various reasons such as memory access violations or stack overflows. It could be caused by several factors,...

  • 2 kudos
Idan
by New Contributor II
  • 1288 Views
  • 2 replies
  • 1 kudos

Using code_path in mlflow.pyfunc models on Databricks

We are using Databricks over AWS infra, registering models on mlflow. We write our in-project imports as from src.(module location) import (objects).Following examples online, I expected that when I use mlflow.pyfunc.log_model(...code_path=['PROJECT_...

  • 1288 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Idan Reshef​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 1 kudos
1 More Replies
DebK
by New Contributor III
  • 2801 Views
  • 6 replies
  • 6 kudos

Resolved! MLFlow is throwing error for the shape of input

I am running the code for prediction which will take the model from mlflow deployment. Code I have copied from the example given by mlflow experiment tab.import mlflow logged_model = 'runs:/id/model'   # Load model as a PyFuncModel. loaded_model = ml...

  • 2801 Views
  • 6 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Koushik Deb​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 6 kudos
5 More Replies
Labels