cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Jaeseon
by • New Contributor II
  • 2850 Views
  • 3 replies
  • 3 kudos

Resolved! Distributed training on building object detection model on PyTorch and PySpark.

I'm currently immersed in a project where I'm leveraging PyTorch to develop an object detection model using satellite imagery. My immediate objective is to perform distributed training on this model using PySpark. While I have found several tutorials...

Machine Learning
Reply
  • 2850 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Jaeseon Song​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
2 More Replies
HT
by • New Contributor II
  • 10279 Views
  • 5 replies
  • 2 kudos

Trying out Dolly - how to load pytorch_model.bin so it's not downloading it every time the cluster is restarted

Hi, I am new to LLM and am curious to try it out. I did the following code to test from the databricks website:import torch from transformers import pipeline instruct_pipeline = pipeline(model="databricks/dolly-v2-12b", torch_dtype=torch.bfloat16, tr...

Machine Learning
Reply
  • 10279 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
sean_owen
Databricks Employee
  • 2 kudos

Just set the HF cache dir to a persistent path on /dbfs:import os os.environ['TRANSFORMERS_CACHE'] = "/dbfs/..."

  • 2 kudos
4 More Replies
Marco98
by • New Contributor II
  • 1769 Views
  • 1 replies
  • 2 kudos

Pytorch DDP on Databricks

Hello!I am currently trying to use Pytorch Lightning inside Databricks and I am currently using a cluster with 2 gpus. Whenever I try to train my Transformer model with 1 gpu in DP strategy everything works fine, but when I try to use both the 2 gpus...

Machine Learning
Reply
  • 1769 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Chakra03
New Contributor II
  • 2 kudos

Hi @Marco Capusso​ , I am facing the similar issue could you find some fix. It would be great if you share some details around it.

  • 2 kudos
Labels