cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

anastassia_kor1
by New Contributor
  • 5734 Views
  • 2 replies
  • 1 kudos

Error "Distributed package doesn't have nccl built in" with Transformers Library.

I am trying to run a simple training script using HF's transformers library and am running into the error `Distributed package doesn't have nccl built in` error.Runtime: DBR 13.0 ML - SPark 3.4.0 - Scala 2.12Driver: i3.xlarge - 4 coresNote: This is a...

  • 5734 Views
  • 2 replies
  • 1 kudos
Latest Reply
patputnam-db
Databricks Employee
  • 1 kudos

Hi @anastassia_kor1,For CPU-only training, TrainingArguments has a no_cuda flag that should be set.For transformers==4.26.1 (MLR 13.0) and transformers==4.28.1 (MLR 13.1), there's an additional xpu_backend argument that needs to be set as well. Try u...

  • 1 kudos
1 More Replies
nolanreilly
by New Contributor
  • 802 Views
  • 0 replies
  • 0 kudos

Impossible to read a custom pipeline? (Scala)

I have created a custom transformer to be used in a ml pipeline. I was able to write the pipeline to storage by extending the transformer class with DefaultParamsWritable. Reading the pipeline back in however, does not seem possible in Scala. I have...

  • 802 Views
  • 0 replies
  • 0 kudos
Labels