send/receive messages in Elixir and rate control

TL;DR — Erlang VM is very efficient as it crunches small amounts of messages with no effort at all. Letting message queue to build up decreases performance 2–5 times. Avoid writing your own message rate control code — look into GenStage or Flow . Also see power tip below for extra memory management tips.

Subscribe to this publication, so that you’ll get instant updates when I post new benchmarks. If you interested in specific benchmarks — write a comment and I’ll try to write another article on that.

Crawling to stop

I write an application, which in some extreme cases has to produce big data structure, iterate over it and send to another process over 500k elements for serializing and network delivery. And on big requests this application was really getting very slow, so I decided to investigate and benchmark.

Basically there are 2 processes:

• data producer, which iterates map and it is quite fast
• data consumer, which does serialization and it is slow

While doing benchmarks I noticed that consumer message queue was quickly building up. You can check it in observer application or by checking

Process.info(pid)[:message_queue_len]

Consumer process was processing messages slowly and falling much behind. This was leading to queue build-up, extra memory consumption and it was bringing consumer to crawl.

To show this behaviour I decided to write some tests.

Flood message test exhibits exactly this behaviour and sync message test stops every 1000 messages and waits acknowledgment from consumer that it processed all messages so far.

Seems that removing messages from queue is quite slow, even if it is just first message in queue and not a selective receive.

With message count increase flooding algorithm slows down more and more.

And here come detailed benchmark results:

Flood message test

processing time (usec)

processing time per message

Sync message test

max backlog scale is on the right (count of messages in queue)

Conclusion

If you expect to pass big number of messages in Erlang/Elixir between processes — use rate control to avoid overflowing consumer capacity.

As synchronization code can be complex and error prone you may want to avoid writing it and use amazing GenStage and Flow libraries.

Power tip — limit maximum memory usage

I also learned that after requests from community Erlang developers added possibility to specify maximum memory size for process. Message queue also counts towards that size.

You can see your current memory consumption with

Process.info(pid)
...
total_heap_size: 1363, 
garbage_collection: [
  max_heap_size: %{error_logger: true, kill: true, size: 0} 
]
...

• total_heap_size shows current memory usage
• max_heap_size shows current limits for memory usage. If process exceeds that limit and kill: true then it will exit with reason :killed .

Messages are not stored in heap by default and thus not subject for memory limits, but you can change it with Process.flag :message_queue_data, :on. Read documentation carefully as it warns about potential pitfalls 👹

To set memory limit you should call Process.flag :max_head_size, size_in_words from the process itself. You cannot setup this limit from another process. Read more in documentation, especially about kill and error_logging options.

Raw logs for flooding test

 mix test test/consumer_producer_speed/flood_message_test.exs

Working on 5000 messages
Producer finished in 5137 usec
Consumer finished in 5171 usec
Total time is 5216 usec
Working on 50000 messages
Producer finished in 33571 usec
Consumer finished in 58147 usec
Total time is 58181 usec
Working on 100000 messages
Producer finished in 84952 usec
Consumer finished in 140963 usec
Total time is 141085 usec
Working on 200000 messages
Consumer message queue 84789
Producer finished in 137880 usec
Consumer message queue 46180
Consumer finished in 286446 usec
Total time is 286504 usec
Working on 300000 messages
Consumer message queue 58503
Consumer message queue 173046
Producer finished in 231076 usec
Consumer message queue 127002
Consumer message queue 64410
Consumer finished in 512896 usec
Total time is 512942 usec
Working on 400000 messages
Consumer message queue 73390
Consumer message queue 215159
Producer finished in 269078 usec
Consumer message queue 218277
Consumer message queue 178304
Consumer message queue 132055
Consumer message queue 74577
Consumer finished in 797096 usec
Total time is 797137 usec
Working on 500000 messages
Consumer message queue 54332
Consumer message queue 147997
Consumer message queue 313679
Producer finished in 351862 usec
Consumer message queue 285836
Consumer message queue 240478
Consumer message queue 194166
Consumer message queue 136101
Consumer message queue 67867
Consumer finished in 985400 usec
Total time is 985446 usec
Working on 750000 messages
Consumer message queue 95366
Consumer message queue 253735
Consumer message queue 421951
Producer finished in 453883 usec
Consumer message queue 553606
Consumer message queue 521791
Consumer message queue 486570
Consumer message queue 451242
Consumer message queue 407082
Consumer message queue 362922
Consumer message queue 318740
Consumer message queue 264233
Consumer message queue 209726
Consumer message queue 146133
Consumer message queue 66341
Consumer finished in 1881851 usec
Total time is 1882313 usec
Working on 1000000 messages
Consumer message queue 76092
Consumer message queue 188889
Consumer message queue 294198
Consumer message queue 442198
Consumer message queue 659314
Producer finished in 705898 usec
Consumer message queue 745786
Consumer message queue 710983
Consumer message queue 682350
Consumer message queue 646238
Consumer message queue 610126
Consumer message queue 574014
Consumer message queue 537902
Consumer message queue 495807
Consumer message queue 458330
Consumer message queue 420262
Consumer message queue 375082
Consumer message queue 319121
Consumer message queue 262581
Consumer message queue 198146
Consumer message queue 126074
Consumer message queue 46052
Consumer finished in 2840617 usec
Total time is 2841275 usec

Raw output for sync test

mix test test/consumer_producer_speed/sync_message_test.exs

Working on 5000 messages
Producer finished in 4470 usec
Consumer finished in 4494 usec
Total time is 4529 usec
Working on 50000 messages
Producer finished in 48726 usec
Consumer finished in 48855 usec
Total time is 48872 usec
Working on 100000 messages
Producer finished in 96522 usec
Consumer finished in 96981 usec
Total time is 97017 usec
Working on 200000 messages
Consumer message queue 19
Producer finished in 193077 usec
Consumer finished in 193412 usec
Total time is 193443 usec
Working on 300000 messages
Consumer message queue 49
Consumer message queue 0
Consumer message queue 77
Producer finished in 303052 usec
Consumer finished in 304001 usec
Total time is 304023 usec
Working on 400000 messages
Consumer message queue 14
Consumer message queue 176
Consumer message queue 368
Consumer message queue 193
Producer finished in 412590 usec
Consumer finished in 412616 usec
Total time is 412652 usec
Working on 500000 messages
Consumer message queue 487
Consumer message queue 599
Consumer message queue 8
Consumer message queue 45
Producer finished in 486721 usec
Consumer finished in 487573 usec
Total time is 487618 usec
Working on 750000 messages
Consumer message queue 170
Consumer message queue 45
Consumer message queue 6
Consumer message queue 45
Consumer message queue 338
Consumer message queue 549
Consumer message queue 45
Producer finished in 756512 usec
Consumer finished in 756535 usec
Total time is 756590 usec
Working on 1000000 messages
Consumer message queue 273
Consumer message queue 610
Consumer message queue 423
Consumer message queue 130
Consumer message queue 437
Consumer message queue 459
Consumer message queue 450
Consumer message queue 22
Consumer message queue 26
Consumer message queue 22
Producer finished in 1010887 usec
Consumer finished in 1011262 usec
Total time is 1011279 usec
Working on 1500000 messages
Consumer message queue 170
Consumer message queue 606
Consumer message queue 6
Consumer message queue 628
Consumer message queue 494
Consumer message queue 137
Consumer message queue 144
Consumer message queue 403
Consumer message queue 45
Consumer message queue 21
Consumer message queue 407
Consumer message queue 134
Consumer message queue 105
Consumer message queue 174
Consumer message queue 174
Producer finished in 1523932 usec
Consumer finished in 1524405 usec
Total time is 1524423 usec
Working on 2000000 messages
Consumer message queue 20
Consumer message queue 7
Consumer message queue 45
Consumer message queue 116
Consumer message queue 2
Consumer message queue 178
Consumer message queue 4
Consumer message queue 45
Consumer message queue 388
Consumer message queue 4
Consumer message queue 60
Consumer message queue 184
Consumer message queue 406
Consumer message queue 392
Consumer message queue 617
Consumer message queue 734
Consumer message queue 592
Consumer message queue 45
Consumer message queue 377
Consumer message queue 312
Producer finished in 2044247 usec
Consumer finished in 2044263 usec
Total time is 2044304 usec