topic DB vector search tutorial with GTE large - BPE tokenizer correct? in Machine Learning

DB vector search tutorial with GTE large - BPE tokenizer correct?

Kronos — Sat, 21 Jun 2025 22:34:57 GMT

Hi,

I was trying to implement a vector search use case based on the databricks example notebook with GTE large:

https://docs.databricks.com/aws/en/notebooks/source/generative-ai/vector-search-foundation-embedding-model-gte-example.html

For chunking, the notebook uses BPE tokenization cl100kbase which is the same as in the equivalent example notebook which uses an openAI model.

Is this correct? I couldn't find any info on the used tokenization encoding in the original paper of GTE large as well as in the web. Does GTE large really also use BPE with exactly the same encoding as for the newer open AI models, or is this an error in the tutorial notebook? And one should rather use AutoTokenizer from huggingface?

Thanks!

Re: DB vector search tutorial with GTE large - BPE tokenizer correct?

MariuszK — Wed, 25 Jun 2025 12:14:53 GMT

I created an vector index using databricks-gte-large-en and it was working fine.

Re: DB vector search tutorial with GTE large - BPE tokenizer correct?

Kronos — Thu, 26 Jun 2025 12:13:44 GMT

I just found the definition and it is indeed word piece tokenization.

So I think the tutorial is wrong.

https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5/blob/main/tokenizer.json