I am trying to run ray on databricks for chunking and embedding tasks. The cluster I’m using is:g4dn.xlarge1-4 workers with 4-16 cores1 GPU and 16GB memoryI have set spark.task.resource.gpu.amount to 0.5 currently.This is how I have setup my ray clus...
I'm using a shared access cluster and am getting this error while trying to upload to Qdrant. This is the error. Anyway I can make it worked on shared access mode? It works on the personal cluster.[UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The...
I'm using a shared access cluster and am getting this error while trying to upload to Qdrant. #embeddings_df = embeddings_df.limit(5)
options = {
"qdrant_url": QDRANT_GRPC_URL,
"api_key": QDRANT_API_KEY,
"collection_name": QDRANT_COLLEC...
The "dense_vector" column does not output on show(). Instead I get the error below. Any idea why it doesn't work on the shared access mode? Any alternatives? from fastembed import TextEmbedding, SparseTextEmbedding
from pyspark.sql.pandas.functions ...
@jacovangelder I think the resources are sufficient since it works on the personal cluster which has lesser resources. I tried to run the code you sent on my shared access mode cluster and it still didn't work. Maybe I need to make some changes to th...
Got this error along with the one above, even though the model cached in the UC Volume.from fastembed import TextEmbedding, SparseTextEmbedding
from pyspark.sql.pandas.functions import pandas_udf, PandasUDFType
from pyspark.sql.types import StructTyp...
