cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

sanjay
by Valued Contributor II
  • 34757 Views
  • 2 replies
  • 1 kudos

Resolved! torch.cuda.OutOfMemoryError: CUDA out of memory

Hi,I am using pynote/whisper large model and trying to process data using spark UDF and getting following error.torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 172.00 MiB (GPU 0; 14.76 GiB total capacity; 6.07 GiB already allocated...

  • 34757 Views
  • 2 replies
  • 1 kudos
Latest Reply
JMTech18
New Contributor II
  • 1 kudos

Try to run these codesimport torchtorch.cuda.empty_cache()And make sure to find the optimize batch size otherwise the error can occur again

  • 1 kudos
1 More Replies
elgeo
by Valued Contributor II
  • 3067 Views
  • 3 replies
  • 2 kudos

Table name as a parameter in SQL UDF

Hello experts,We would like to create a UDF function with input parameter a table_name. Please check the below simple example:CREATE OR REPLACE FUNCTION F_NAME(v_table_name STRING,                   v_w...

  • 3067 Views
  • 3 replies
  • 2 kudos
Latest Reply
alm
New Contributor III
  • 2 kudos

Did you find a solutions? I'm having the same problem

  • 2 kudos
2 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 1227 Views
  • 1 replies
  • 5 kudos

Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outs...

Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outside of Databricks. While serving the model via REST API is ideal for external use cases, it is recom...

1677238735536
  • 1227 Views
  • 1 replies
  • 5 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 5 kudos

Thank you for sharing this!!!

  • 5 kudos
anvil
by New Contributor II
  • 2630 Views
  • 3 replies
  • 4 kudos

Are UDFs necessary for applying models from ML libraries at scale ?

Hello,I recently finished the "scalable machine learning with apache spark" course and saw that SKLearn models could be applied faster in a distributed manner when used in pandas UDFs or with mapInPandas() method. Spark MLlib models don't need this k...

  • 2630 Views
  • 3 replies
  • 4 kudos
Latest Reply
Manoj12421
Valued Contributor II
  • 4 kudos

MlLib is in the maintenance model and udf is not used by creating model in most cases

  • 4 kudos
2 More Replies
Vicky1215
by New Contributor II
  • 6103 Views
  • 6 replies
  • 2 kudos

Solution for - "PythonException: 'ModuleNotFoundError: No module named 'spacy'

I am actually trying to extract the adjective and noun phrases from the text column in spark data frame for which I've written the udf and applying on cleaned text column. However, I am getting this error.from pyspark.sql.functions import udffrom pys...

  • 6103 Views
  • 6 replies
  • 2 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 2 kudos

only init script will work here

  • 2 kudos
5 More Replies
Dan_Z
by Databricks Employee
  • 664 Views
  • 0 replies
  • 0 kudos

spark.apache.org

mapInPandas is one of the most powerful Spark functions. It uses an arrow-like in-memory data structure to split up Spark Data Frames into chunks and feeding them to a function that takes a Pandas DF as input and output. Check it out here:https://spa...

  • 664 Views
  • 0 replies
  • 0 kudos
Labels