Dev News Daily ENDE

Sentence Transformers adds pooling-aware training: 97.8% of quality with 4x fewer vectors

Multi-vector retrieval models such as ColBERT store one vector per document token, which makes them precise and expensive. A post by Tom Aarsen on the Hugging Face blog, published 11 November, walks through a way to shrink that storage with less loss: train the encoder knowing its vectors will later be merged.

The merging step is HierarchicalTokenPooling in Sentence Transformers. It groups similar token vectors with Ward clustering, replaces each group by its normalised mean and leaves query vectors untouched. The budget can be set three ways: a hard cap per document (max_tokens), a fraction of the original count (pool_factor), or a distance threshold that lets the count vary with the content. By default the first token is protected from merging.

The cost of storage is spelled out in the post. At 4 bits per residual coordinate, 200 vectors of 128 dimensions take 12.8 KB per document, or 12.8 GB for a million documents; pooling each document to 50 vectors brings that to 3.2 GB.

The experiment compares two models trained from ModernBERT-base on the same 120,000 MS MARCO triplets, one epoch, same seed. One uses an ordinary contrastive loss; the other wraps it with an extra objective that scores both unpooled and factor-4 pooled documents. On NanoBEIR the ordinary model scores 51.84 nDCG@10 unpooled and 49.03 at pool_factor=4, keeping 94.6% of its quality with about a quarter of the vectors. The pooling-aware model scores 52.39 unpooled and 51.23 at factor 4, keeping 97.8%. The gap widens as compression increases, from 1.64 points at factor 2 to 2.20 at factor 4. The author notes that only the loss differed and no hyperparameter search was run.

There is also an optional speed-up: CUDA kernels for MaxSim scoring and Ward clustering, the latter described as 30 to 150 times faster than SciPy, depending on settings.

Sentence Transformers adds pooling-aware training: 97.8% of quality with 4x fewer vectors
Sentence Transformers adds pooling-aware training: 97.8% of quality with 4x fewer vectors — Dev News Daily

Why it matters

For teams running late-interaction search, the number that decides cost is vectors per document. The result suggests that compression is cheaper in quality when the model is trained for it, and the pool factor can still be changed at query time without retraining.

Primary source
Hugging Face Blog
https://huggingface.co/blog/token-pooling