cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

kng88
by New Contributor II
  • 2468 Views
  • 6 replies
  • 7 kudos

How to save model produce by distributed training?

I am trying to save model after distributed training via the following codeimport sys   from spark_tensorflow_distributor import MirroredStrategyRunner   import mlflow.keras   mlflow.keras.autolog()   mlflow.log_param("learning_rate", 0.001)   import...

  • 2468 Views
  • 6 replies
  • 7 kudos
Latest Reply
Xiaowei
New Contributor III
  • 7 kudos

I think I finally worked this out.Here is the extra code to save out the model only once and from the 1st node:context = pyspark.BarrierTaskContext.get() if context.partitionId() == 0: mlflow.keras.log_model(model, "mymodel")

  • 7 kudos
5 More Replies
Jaeseon
by New Contributor II
  • 1493 Views
  • 3 replies
  • 3 kudos

Resolved! Distributed training on building object detection model on PyTorch and PySpark.

I'm currently immersed in a project where I'm leveraging PyTorch to develop an object detection model using satellite imagery. My immediate objective is to perform distributed training on this model using PySpark. While I have found several tutorials...

  • 1493 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Jaeseon Song​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
2 More Replies
User16461610613
by New Contributor II
  • 1245 Views
  • 1 replies
  • 2 kudos

Free Databricks Training on AWS, Azure, or Google Cloud Good news! You can now access free, in-depth Databricks training on AWS, Azure or Google Cloud...

Free Databricks Training on AWS, Azure, or Google CloudGood news! You can now access free, in-depth Databricks training on AWS, Azure or Google Cloud.  Our on-demand training series walks through how to:Streamline data ingest and management to build ...

image
  • 1245 Views
  • 1 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Moderator
  • 2 kudos

Thank you for sharing this!!

  • 2 kudos
gaponte
by New Contributor III
  • 2624 Views
  • 5 replies
  • 5 kudos

Unable to install SynapseML on clusters

I would like to run a distributed training using LightGBM but I cannot install SynapseML. I have tried doing so on a few different clusters (note: our clusters are running on AWS, not sure if that matters. Also, I am running the Databricks ML Runtime...

  • 2624 Views
  • 5 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Greg Aponte​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 5 kudos
4 More Replies
matebreeze
by New Contributor
  • 864 Views
  • 0 replies
  • 0 kudos

MLflow model serving: KeyError: 'python_function'

Hello, I am training a logistic regression on text with the help of an tf-idf vectorizer.This is done with MLflow and sklearn in databricks.The model itself is trained successfully in databricks and it is possible to accomplish predictions within the...

  • 864 Views
  • 0 replies
  • 0 kudos
MoJaMa
by Valued Contributor II
  • 652 Views
  • 1 replies
  • 0 kudos
  • 652 Views
  • 1 replies
  • 0 kudos
Latest Reply
MoJaMa
Valued Contributor II
  • 0 kudos

Yes!You will have to pip install mlflowin your environment as a first step. For more details, see: https://docs.databricks.com/applications/mlflow/access-hosted-tracking-server.html

  • 0 kudos
Labels