Tuning indexing process in Manticore search
Manticore is a fork of Sphinx Search, great solution for full text search. Indexing process can take a lot of time, especially when we have to deal with hundreds of millions of documents. Let’s find out which performance tuning options we have to make indexing process faster.
We’re going to edit config file which is usually available at /etc/manticoresearch/manticore.conf
.
mem_limit
Most important indexer option is mem_limit
. It limits amount of RAM available for indexer and is defaulted to 128M
. Which is fairly small value for modern loads.
Let’s test default indexing performance based on the following plain index config (using Clickhouse as source):
source test_src {
type = tsvpipe
tsvpipe_command = clickhouse-client -q "SELECT crc64(text)-1, text FROM test FORMAT TSV"
tsvpipe_field = text
}index test_idx {
type = plain
source = test_src
path = /var/lib/search/test
}
Indexer will do 118
reads and 3382
writes during the process:
root@desktop:/# sudo -u manticore indexer --all --rotate
...
indexing index 'test_idx'...
collected 1000000 docs, 861.6 MB
creating lookup: 1000.0 Kdocs, 100.0% done
sorted 127.2 Mhits, 100.0% done
total 1000000 docs, 861681456 bytes
total 65.635 sec, 13128252 bytes/sec, 15235.62 docs/sec
total 118 reads, 0.090 sec, 3416.2 kb/call avg, 0.7 msec/call avg
total 3382 writes, 1.039 sec, 439.5 kb/call avg, 0.3 msec/call avg
rotating indices: successfully sent SIGHUP to searchd (pid=303562).
If we increase buffer to 2G
:
indexer {
mem_limit = 2G
}
We can see the difference in indexing process:
total 60.313 sec, 14284143 bytes/sec, 16580.07 docs/sec
total 4 reads, 0.135 sec, 91577.7 kb/call avg, 33.9 msec/call avg
total 2529 writes, 0.995 sec, 573.1 kb/call avg, 0.3 msec/call avg
That’s because larger buffer allowed indexer to make bigger read batches and fetch all source data in 4
instead of 118
reads. We also see indexing time improved, but that was tested on NVME disk and the difference is quite small. We would feel more difference in cases of HDD and larger indexing volumes.
write_buffer
This buffer is used for writing temp index files. Larger values will lead to less disk writes and we want exactly this when indexing large amounts of data. Since default value is only 1Mb
we would want to see much bigger value here:
indexer {
...
write_buffer = 1G
}
Note, that indexer
can use up to 4
writing buffers, which in our case will lead to maxium RAM consumption of 4 x 1Gb = 4Gb
. So pick values 4 times smaller than you actually can dedicate for indexer writing buffer. And don’t forget that mem_limit
amount of RAM will be used in addition to the writing buffers.
Summary
When indexing large amounts of data tune Manticore indexing options because defaults are irrelevant for modern hardware. These two options will make the biggest effect:
indexer {
mem_limit = 2G # will consume max 2G
write_buffer = 1G # will consume max 4*1 = 4G
}
This sample configuration will consume max 6G
of RAM and will improve indexing performance compared to default settings.