cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Jaeseon
by New Contributor II
  • 1548 Views
  • 3 replies
  • 3 kudos

Resolved! Distributed training on building object detection model on PyTorch and PySpark.

I'm currently immersed in a project where I'm leveraging PyTorch to develop an object detection model using satellite imagery. My immediate objective is to perform distributed training on this model using PySpark. While I have found several tutorials...

  • 1548 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Jaeseon Song​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
2 More Replies
HT
by New Contributor II
  • 4593 Views
  • 5 replies
  • 2 kudos

Trying out Dolly - how to load pytorch_model.bin so it's not downloading it every time the cluster is restarted

Hi, I am new to LLM and am curious to try it out. I did the following code to test from the databricks website:import torch from transformers import pipeline instruct_pipeline = pipeline(model="databricks/dolly-v2-12b", torch_dtype=torch.bfloat16, tr...

  • 4593 Views
  • 5 replies
  • 2 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 2 kudos

Just set the HF cache dir to a persistent path on /dbfs:import os os.environ['TRANSFORMERS_CACHE'] = "/dbfs/..."

  • 2 kudos
4 More Replies
Marco98
by New Contributor II
  • 1114 Views
  • 1 replies
  • 2 kudos

Pytorch DDP on Databricks

Hello!I am currently trying to use Pytorch Lightning inside Databricks and I am currently using a cluster with 2 gpus. Whenever I try to train my Transformer model with 1 gpu in DP strategy everything works fine, but when I try to use both the 2 gpus...

  • 1114 Views
  • 1 replies
  • 2 kudos
Latest Reply
Chakra03
New Contributor II
  • 2 kudos

Hi @Marco Capusso​ , I am facing the similar issue could you find some fix. It would be great if you share some details around it.

  • 2 kudos
Labels