Vector Search Engine

Yahoo! JAPAN Research released the approximate nearest neighbor (ANN) search engine NGT as open-source software (OSS) in September 2016. NGT has achieved world-leading performance, and is adopted by a number of Yahoo! JAPAN services through Vald as well. Yahoo! JAPAN Research added Quantized Graph (QG) in January 2021 and Quantized…

ElasticSearch is a popular search engine used by developers to implement search functionality in their applications. With each new release, ElasticSearch continues to add powerful features to its arsenal, making it easier for developers to build more efficient and accurate search systems. One such feature that has recently gained a…

How to chat with any documents, PDFs, and books using OpenAI ChatGPT API and Text Embedding — Nowadays, it seems that all applications have either “Chat” or “GPT” in their names. …

While working on this blog post I had a privilege of interacting with all neural search frameworks key developers / leadership: Malte Piestch (Haystack), Amin Ahmad (Vectara), Florian Hönicke (Jina AI), David Mezzetti (txtai), Daniel Vassilev (Relevance AI), George Sivulka and Swetha Revanur (Hebbia), Aarne Talman and myself (Muves). The…

This blog is the second post in a series on building a billion-scale vector search using Vespa. The series covers the cost and serving performance tradeoffs related to approximate nearest neighbor search at a large scale. In the first post, we introduced some challenges related to building large-scale vector search…

Hey everyone, I wanted to just quickly document this in case someone else finds themself working on a similar problem. Step 1 — Set a date property in Weaviate { "dataType": ["date"], "name": "timeEntered", "description": "The time this object entered Weaviate." } Step 2 — (If uploading in Python, timestamp…

Doug Turnbull asks an interesting question on Linkedin; Will new vector databases dislodge traditional search engines? The short answer is no, but it depends on what you classify as a traditional search engine. Features like phrase search, exact search, BM25 ranking, dynamic summaries, and result facets are features we take…

This blog post describes HNSW-IF, a cost-efficient solution for high-accuracy vector search over billion scale vector datasets. — The first blog post on billion-scale vector search covered methods for compressing real-valued vectors to binary representations and using hamming distance for efficient coarse level search. …

The new version of our engine — v0.8.0, went live. Let’s go through the new features it has: On-disk payload storage allows storing more with less RAM usage. Distributed deployment support is available. And we continue improving it, so stay tuned for new updates. The payload can be indexed in the process without rebuilding the segment. Advanced filtering support now includes filtering by similarity score.