Milvus is an open-source vector database with state-of-the-art features for similarity search and AI applications. One of the essential features of Milvus is the support for various Index types; Indexes help to optimize data querying and retrieval.
Index types
Currently, Milvus supports eight types of Indexes:
- FLAT: FLAT is best suited for scenarios that seek perfectly accurate and exact search results (100% recall rate) on a small, million-scale dataset. FLAT does not compress vectors and is the only index guaranteeing exact search results. Results from FLAT can be a point of comparison for results produced by other indexes with less than 100% recall.
- IVF_FLAT: IVF_FLAT is a quantization-based index best suited for scenarios that seek an ideal balance between accuracy and query speed (low-dimensional data with less than 100,000 entries). IVF_FLAT is a system that divides vector data into clusters and returns similarity search results based on comparisons between the target input and the vectors in the most similar cluster(s). IVS_FLAT drastically reduces query time when considering the number of clusters queried. Please note that IVF_FLAT does not perform any compression, so the index files it produces are roughly the same size as the original, raw non-indexed vector data.
- IV_SQ8: IVF_SQ8 is a quantization-based index and is best suited for scenarios that seek a significant reduction in disk, CPU, and GPU memory consumption in situations where these resources are limited. IV_SQ8 can achieve this by converting each FLOAT (4 bytes) to UINT8 (1 byte) by performing scalar quantization. The IV_SQ8 index is suited for scenarios of low-dimensional data with less than 100,000 entries.
- HNSW Index: Hierarchical Navigable Small World (HNSW) is a graph-based index best suited for scenarios with a high demand for search efficiency. This Index creates a tree-like structure to index the vectors. Therefore, it is useful when the dataset is enormous (large dataset with many dimensions) and requires a fast search. It builds a multi-layer navigation structure for an image according to certain rules. In this structure, the upper layers are more sparse, and the distances between nodes are farther; the lower layers are denser, and the distances between nodes are closer. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then enters the next layer to begin another search. After multiple iterations, it can quickly approach the target position.
- IVF_PQ: IVF_PQ is a quantization-based index best suited for scenarios that seek high query speed of high-dimensional data even at the cost of accuracy. It uniformly decomposes the high-dimensional vector space into Cartesian products of low-dimensional vector spaces, then quantizes the low-dimensional vector spaces. Product quantization calculates distances between the target vector and the clustering center of each low-dimensional space, reducing time and space complexity. The IVF_PQ index file is smaller (and less accurate) than IVF_SQ8.
- ANNOY: ANNOY is a tree-based index best suited for scenarios that seek a high recall rate. It is handy for datasets (less than 1 million entries) with high dimensions (and when searching for similar vectors).
- BIN_FLAT: This Index is suitable for low-dimensional data with less than 100,000 entries. This index is the same as FLAT, except BIN_FLAT is used for binary embeddings.
- BIN_IVF_FLAT: This Index is suitable for high-dimensional data datasets with less than 10 million entries when searching for similar vectors. This index is the same as FLAT, except BIN_IVF_FLAT is used for binary embeddings.
In Conclusion
Milvus supports various Index types that help to optimize data querying and retrieval for different use cases. Choosing the right Index type can significantly improve the search performance of your application.
If you are still determining which Index type to use for your specific use case, trying out different Indexes and comparing their performance is recommended. Milvus can provide lightning-fast search performance for your AI applications with the right Index type and configuration.
Next Steps
- Download Milvus and get started
- Read more about Milvus Index types
- Check out the Milvus benchmarks in this paper
