https://github.com/joshnuss/blip
Blip is an attempt to write a super small StatsD server using Elixir. It is compliant with Etsy’s specification.
Should be no shock that Elixir is scary good at network utilities (thanks BEAM!).
While hacking on this, I picked up some some tips & tricks along the way, so I’m passing the savings along to y’all. Here’s how all the magic works…
UDP Server
Metric data begins its journey as a UDP packet. Blip can handle many packets in parallel (one Task per packet).
The UDP frontend is implemented using Erlang’s gen_udp :
TIP: To listen on multiple ports, you can simply start multiple UDPServer’s (one per port).
Recording
After the metric arrives, it needs to be recorded. That happens in 3 steps.
- Parse the data, and determine its validity
- Buffer it, so we don’t tax the backend
- Persist the data on some fixed schedule
Parsing
Several types of messages are supported:
- Counters: keep a running total of how many times something occurred. Format:
bucket-name:value|c - Gauges: a measure that changes linearly.
Format:bucket-name:value|g - Sets: Similar to a counter, but each value replaces the last value(with counters each value increments the value)
Format:bucket-name:value|s - Timings: record how long something took in milliseconds
Format:bucket-name:value|ms
My first approach was to use String.split/2 to extract the data, but at the Empex Conference, Wojtek Mach presented how he used Nimble Parsec to parse date time. Unlike String.split/2, Nimble Parsec uses a declarative approach that is way more readable. So the code was refactored to use it and it made it much easier to follow. (also it’s only 14 LOC #yay)
Buffering
Once the parsing part is done, the value is temporarily stored in a bucket. The bucket flushes its data on a periodic timer. We do this to reduce the write pressure on the backend data store. If we didn’t buffer we would flood the database — dunno about you, but I don’t like floods much.
Of course, the bucket is just another simple GenServer:
Backend
For now, only a console based backend is implemented, but I hope to add more in the future (Postgres?).
Command Line Interface
Blip is compiled into an escriptbinary. You can run mix escript.build to compile it, and then ./blip <portA...portN> to start ‘er up.
Here’s how to send it test messages using netcat:
./blip 2052 &
echo "requests:99|c" | nc -u -w0 127.0.0.1 2052That’s all folks.
What I’ve learned is: it’s incredibly simple to build industrial grade tools with Elixir. You get the power of the BEAM and the simplicity of Ruby’s syntax. Can’t beat that/it’s a double win!
You can find Blip on GitHub, give it a try: https://github.com/joshnuss/blip
