cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

elgeo
by Valued Contributor II
  • 1344 Views
  • 3 replies
  • 2 kudos

Table name as a parameter in SQL UDF

Hello experts,We would like to create a UDF function with input parameter a table_name. Please check the below simple example:CREATE OR REPLACE FUNCTION F_NAME(v_table_name STRING,                   v_w...

  • 1344 Views
  • 3 replies
  • 2 kudos
Latest Reply
alm
New Contributor III
  • 2 kudos

Did you find a solutions? I'm having the same problem

  • 2 kudos
2 More Replies
sanjay
by Valued Contributor II
  • 20572 Views
  • 1 replies
  • 1 kudos

Resolved! torch.cuda.OutOfMemoryError: CUDA out of memory

Hi,I am using pynote/whisper large model and trying to process data using spark UDF and getting following error.torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 172.00 MiB (GPU 0; 14.76 GiB total capacity; 6.07 GiB already allocated...

  • 20572 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Sanjay Jain​ : The error message suggests that there is not enough available memory on the GPU to allocate for the PyTorch model. This error can occur if the model is too large to fit into the available memory on the GPU, or if the GPU memory is bei...

  • 1 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 604 Views
  • 1 replies
  • 5 kudos

Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outs...

Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outside of Databricks. While serving the model via REST API is ideal for external use cases, it is recom...

1677238735536
  • 604 Views
  • 1 replies
  • 5 kudos
Latest Reply
jose_gonzalez
Moderator
  • 5 kudos

Thank you for sharing this!!!

  • 5 kudos
anvil
by New Contributor II
  • 1609 Views
  • 3 replies
  • 4 kudos

Are UDFs necessary for applying models from ML libraries at scale ?

Hello,I recently finished the "scalable machine learning with apache spark" course and saw that SKLearn models could be applied faster in a distributed manner when used in pandas UDFs or with mapInPandas() method. Spark MLlib models don't need this k...

  • 1609 Views
  • 3 replies
  • 4 kudos
Latest Reply
Manoj12421
Valued Contributor II
  • 4 kudos

MlLib is in the maintenance model and udf is not used by creating model in most cases

  • 4 kudos
2 More Replies
Vicky1215
by New Contributor II
  • 3855 Views
  • 7 replies
  • 2 kudos

Solution for - "PythonException: 'ModuleNotFoundError: No module named 'spacy'

I am actually trying to extract the adjective and noun phrases from the text column in spark data frame for which I've written the udf and applying on cleaned text column. However, I am getting this error.from pyspark.sql.functions import udffrom pys...

  • 3855 Views
  • 7 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Aditya Singh​(Customer)​ , We haven’t heard from you on the last response from @Aviral Bhardwaj​ ​ and @sherbin w​​, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please do share that with the ...

  • 2 kudos
6 More Replies
Dan_Z
by Honored Contributor
  • 339 Views
  • 0 replies
  • 0 kudos

spark.apache.org

mapInPandas is one of the most powerful Spark functions. It uses an arrow-like in-memory data structure to split up Spark Data Frames into chunks and feeding them to a function that takes a Pandas DF as input and output. Check it out here:https://spa...

  • 339 Views
  • 0 replies
  • 0 kudos
Labels