- 2569 Views
- 3 replies
- 2 kudos
Table name as a parameter in SQL UDF
Hello experts,We would like to create a UDF function with input parameter a table_name. Please check the below simple example:CREATE OR REPLACE FUNCTION F_NAME(v_table_name STRING, v_w...
- 2569 Views
- 3 replies
- 2 kudos
- 30807 Views
- 1 replies
- 1 kudos
Resolved! torch.cuda.OutOfMemoryError: CUDA out of memory
Hi,I am using pynote/whisper large model and trying to process data using spark UDF and getting following error.torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 172.00 MiB (GPU 0; 14.76 GiB total capacity; 6.07 GiB already allocated...
- 30807 Views
- 1 replies
- 1 kudos
- 1 kudos
@Sanjay Jain : The error message suggests that there is not enough available memory on the GPU to allocate for the PyTorch model. This error can occur if the model is too large to fit into the available memory on the GPU, or if the GPU memory is bei...
- 1 kudos
- 1087 Views
- 1 replies
- 5 kudos
Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outs...
Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outside of Databricks. While serving the model via REST API is ideal for external use cases, it is recom...
- 1087 Views
- 1 replies
- 5 kudos
- 5 kudos
Thank you for sharing this!!!
- 5 kudos
- 2265 Views
- 3 replies
- 4 kudos
Are UDFs necessary for applying models from ML libraries at scale ?
Hello,I recently finished the "scalable machine learning with apache spark" course and saw that SKLearn models could be applied faster in a distributed manner when used in pandas UDFs or with mapInPandas() method. Spark MLlib models don't need this k...
- 2265 Views
- 3 replies
- 4 kudos
- 4 kudos
MlLib is in the maintenance model and udf is not used by creating model in most cases
- 4 kudos
- 5249 Views
- 7 replies
- 2 kudos
Solution for - "PythonException: 'ModuleNotFoundError: No module named 'spacy'
I am actually trying to extract the adjective and noun phrases from the text column in spark data frame for which I've written the udf and applying on cleaned text column. However, I am getting this error.from pyspark.sql.functions import udffrom pys...
- 5249 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi @Aditya Singh(Customer) , We haven’t heard from you on the last response from @Aviral Bhardwaj and @sherbin w, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please do share that with the ...
- 2 kudos
- 512 Views
- 0 replies
- 0 kudos
spark.apache.org
mapInPandas is one of the most powerful Spark functions. It uses an arrow-like in-memory data structure to split up Spark Data Frames into chunks and feeding them to a function that takes a Pandas DF as input and output. Check it out here:https://spa...
- 512 Views
- 0 replies
- 0 kudos