cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Data_Cowboy
by New Contributor III
  • 1195 Views
  • 3 replies
  • 0 kudos

Resolved! Problems with xgboost.spark model loading from MLflow.

When loading an xgboost model from mlflow following the provided instructions in Databricks hosted MLflow the input sizes I am showing on the job are over 1 TB. Is anyone else using an xgboost.spark model and noticing the same behavior? Below are som...

image.png image
  • 1195 Views
  • 3 replies
  • 0 kudos
Latest Reply
dbx-user7354
New Contributor III
  • 0 kudos

Thank you very much @Data_Cowboy !!! I had the same issue. I even had 14 TiB  Databricks should really fix this

  • 0 kudos
2 More Replies
kng88
by New Contributor II
  • 2468 Views
  • 6 replies
  • 7 kudos

How to save model produce by distributed training?

I am trying to save model after distributed training via the following codeimport sys   from spark_tensorflow_distributor import MirroredStrategyRunner   import mlflow.keras   mlflow.keras.autolog()   mlflow.log_param("learning_rate", 0.001)   import...

  • 2468 Views
  • 6 replies
  • 7 kudos
Latest Reply
Xiaowei
New Contributor III
  • 7 kudos

I think I finally worked this out.Here is the extra code to save out the model only once and from the 1st node:context = pyspark.BarrierTaskContext.get() if context.partitionId() == 0: mlflow.keras.log_model(model, "mymodel")

  • 7 kudos
5 More Replies
145093
by New Contributor II
  • 3120 Views
  • 2 replies
  • 2 kudos

MLFlow model loading taking long time and "model serving" failing during init

I am trying to load a simple Minmaxscaler model that was logged as a run through spark's ML Pipeline api for reuse. On average it takes 40+ seconds just to load the model with the following example: This is fine and the model transforms my data corre...

simple model load sometimes the model takes almost 3 min just to load
  • 3120 Views
  • 2 replies
  • 2 kudos
Latest Reply
DanSimpson
New Contributor II
  • 2 kudos

Hello,Any solutions found for this issue?I'm serving up a large number of models at a time, but since we converted to PySpark (due to our data demands), the mlflow.spark.load_model() is taking hours.Part of the reason to switch to spark was to help w...

  • 2 kudos
1 More Replies
jonathan-dufaul
by Valued Contributor
  • 2020 Views
  • 5 replies
  • 5 kudos

Does FeatureStoreClient().score_batch support multidimentional predictions?

I have a pyfunc model that I can use to get predictions. It takes time series data with context information at each date, and produces a string of predictions. For example:The data is set up like below (temp/pressure/output are different than my inpu...

  • 2020 Views
  • 5 replies
  • 5 kudos
Latest Reply
EmilAndersson
New Contributor II
  • 5 kudos

I have the same question. I've decided to look for alternative Feature Stores as this makes it very difficult to use for time series forecasting.

  • 5 kudos
4 More Replies
Jaeseon
by New Contributor II
  • 1493 Views
  • 3 replies
  • 3 kudos

Resolved! Distributed training on building object detection model on PyTorch and PySpark.

I'm currently immersed in a project where I'm leveraging PyTorch to develop an object detection model using satellite imagery. My immediate objective is to perform distributed training on this model using PySpark. While I have found several tutorials...

  • 1493 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Jaeseon Song​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
2 More Replies
thomasm
by New Contributor II
  • 1862 Views
  • 3 replies
  • 1 kudos

Resolved! Online Feature Store MLflow serving problem

When I try to serve a model stored with FeatureStoreClient().log_model using the feature-store-online-example-cosmosdb tutorial Notebook, I get errors suggesting that the primary key schema is not configured properly. However, if I look in the Featur...

  • 1862 Views
  • 3 replies
  • 1 kudos
Latest Reply
NandiniN
Valued Contributor III
  • 1 kudos

Hello @Thomas Michielsen​ , this error seems to occur when you may have created the table yourself. You must use publish_table() to create the table in the online store. Do not manually create a database or container inside Cosmos DB. publish_table()...

  • 1 kudos
2 More Replies
invalidargument
by New Contributor II
  • 523 Views
  • 1 replies
  • 0 kudos

Model storage requirements management

Hi.We have around 30 models in model storage that we use for batch scoring. These are created at different times by different person and on different cluster run times.Now we have run into problems that we can't de-serialize the models and use for in...

  • 523 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Jonas Lindberg​ :To address the issues you are facing with model serialization and versioning, I would recommend the following approach:Use MLflow to manage the lifecycle of your models, including versioning, deployment, and monitoring. MLflow is an...

  • 0 kudos
pol7451
by New Contributor
  • 548 Views
  • 2 replies
  • 0 kudos

Automating model history with multiple downstream elements

Hey, We got two models A and BModel A is fed from raw data that is firstly Clean / enriched and forecasted The results from model A are what are fed into model Bthe processes for cleaning, enriching, forecasting, model A and model B are all under ver...

  • 548 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @polly halton​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 0 kudos
1 More Replies
Saeid_H
by Contributor
  • 4179 Views
  • 6 replies
  • 5 kudos

Register mlflow custom model, which has pickle files

Dear community,I want to basically store 2 pickle files during the training and model registry with my keras model. So that when I access the model from another workspace (using mlflow.set_registery_uri()) , these models can be accessed as well. The ...

  • 4179 Views
  • 6 replies
  • 5 kudos
Latest Reply
arzex
New Contributor II
  • 5 kudos

آموزش تولید محتوا

  • 5 kudos
5 More Replies
Orianh
by Valued Contributor II
  • 751 Views
  • 2 replies
  • 0 kudos

TF SummaryWriter flush() don't send any buffered data to storage.

Hey guys, I'm training a TF model in databricks, and logging to tensorboard using SummaryWriter. At the end of each epoch SummaryWriter.flush() is called which should send any buffered data into storage. But i can't see the tensorboard files while th...

  • 751 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @orian hindi​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...

  • 0 kudos
1 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 540 Views
  • 1 replies
  • 7 kudos

Have you heard about databricks latest open-source language model called Dolly? It’s a ChatGPT like model that uses the tatsu-lab/alpaca dataset with ...

Have you heard about databricks latest open-source language model called Dolly? It’s a ChatGPT like model that uses the tatsu-lab/alpaca dataset with examples of questions and answers. To train Dolly, you can combine this dataset (simple solution on ...

Screenshot 2023-03-26 215509
  • 540 Views
  • 1 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Thanks for posting this! I am so excited about the possibilities that this can do for us. It's an exciting development in the natural language processing field, and it has the potential to be a valuable tool for businesses looking to implement chatb...

  • 7 kudos
Tilo
by New Contributor
  • 1945 Views
  • 3 replies
  • 3 kudos

Resolved! MLFlow: How to load results from model and continue training

I'd like to continue / finetune training of an existing keras/tensorflow model. We use MLFlow to store the model. How can I load the wieght from an existing model to the model and continue "fit" preferable with a different learning rate.Just loading ...

  • 1945 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Tilo Wünsche​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 3 kudos
2 More Replies
zachclem
by New Contributor III
  • 2302 Views
  • 2 replies
  • 1 kudos

Resolved! Logging model to MLflow using Feature Store API. Getting TypeError: join() argument must be str, bytes, or os.PathLike object, not 'dict'

I'm using databricks. Trying to log a model to MLflow using the Feature Store log_model function. but I have this error: TypeError: join() argument must be str, bytes, or os.PathLike object, not 'dict' I'am using the Databricks runtime ml (10.4 LTS M...

  • 2302 Views
  • 2 replies
  • 1 kudos
Latest Reply
zachclem
New Contributor III
  • 1 kudos

I updated by Databricks Run Time from 10.4 to 12.1 and this solved the issue.

  • 1 kudos
1 More Replies
notsure
by New Contributor
  • 771 Views
  • 1 replies
  • 1 kudos

Model serving with Serverless Real-Time Inference - How could I call the endpoint with json file consisted of raw text that need to be transformed and get the prediction?

Hi!I want to call the generated endpoint with a json file consisted of texts directly, could this endpoint take the raw texts, transform the texts into vectors and then output the prediction?Is there a way to support so?Thanks in advance!!!

  • 771 Views
  • 1 replies
  • 1 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi, the updated document is : https://docs.databricks.com/machine-learning/model-inference/serverless/serverless-real-time-inference.html, (mentioned in the document stated above: This documentation has been retired and might not be updated. The prod...

  • 1 kudos
Charley
by New Contributor II
  • 3994 Views
  • 1 replies
  • 1 kudos

error status 400 calling serving model endpoint invocation using personal access token on Azure Databricks

Hi all, I've deployed a model, moved it to production and served it (mlflow), but when testing it in the python notebook I get a 400 error. code/details below:import osimport requestsimport jsonimport pandas as pdimport numpy as np# Create two record...

  • 3994 Views
  • 1 replies
  • 1 kudos
Latest Reply
nakany
New Contributor II
  • 1 kudos

data_json in the score_model function should be defined as followsds_dict = {"dataframe_split": dataset.to_dict(orient='split')} if isinstance(dataset, pd.DataFrame) else create_tf_serving_json(dataset)

  • 1 kudos
Labels