- 33921 Views
- 2 replies
- 1 kudos
Resolved! torch.cuda.OutOfMemoryError: CUDA out of memory
Hi,I am using pynote/whisper large model and trying to process data using spark UDF and getting following error.torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 172.00 MiB (GPU 0; 14.76 GiB total capacity; 6.07 GiB already allocated...
- 33921 Views
- 2 replies
- 1 kudos
- 1 kudos
Try to run these codesimport torchtorch.cuda.empty_cache()And make sure to find the optimize batch size otherwise the error can occur again
- 1 kudos
- 2938 Views
- 3 replies
- 2 kudos
Table name as a parameter in SQL UDF
Hello experts,We would like to create a UDF function with input parameter a table_name. Please check the below simple example:CREATE OR REPLACE FUNCTION F_NAME(v_table_name STRING, v_w...
- 2938 Views
- 3 replies
- 2 kudos
- 1194 Views
- 1 replies
- 5 kudos
Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outs...
Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outside of Databricks. While serving the model via REST API is ideal for external use cases, it is recom...
- 1194 Views
- 1 replies
- 5 kudos
- 5 kudos
Thank you for sharing this!!!
- 5 kudos
- 2540 Views
- 3 replies
- 4 kudos
Are UDFs necessary for applying models from ML libraries at scale ?
Hello,I recently finished the "scalable machine learning with apache spark" course and saw that SKLearn models could be applied faster in a distributed manner when used in pandas UDFs or with mapInPandas() method. Spark MLlib models don't need this k...
- 2540 Views
- 3 replies
- 4 kudos
- 4 kudos
MlLib is in the maintenance model and udf is not used by creating model in most cases
- 4 kudos
- 5924 Views
- 6 replies
- 2 kudos
Solution for - "PythonException: 'ModuleNotFoundError: No module named 'spacy'
I am actually trying to extract the adjective and noun phrases from the text column in spark data frame for which I've written the udf and applying on cleaned text column. However, I am getting this error.from pyspark.sql.functions import udffrom pys...
- 5924 Views
- 6 replies
- 2 kudos
- 644 Views
- 0 replies
- 0 kudos
spark.apache.org
mapInPandas is one of the most powerful Spark functions. It uses an arrow-like in-memory data structure to split up Spark Data Frames into chunks and feeding them to a function that takes a Pandas DF as input and output. Check it out here:https://spa...
- 644 Views
- 0 replies
- 0 kudos