Outbound SSL Performance in Node.js
When browsing the internet, we all know that employing encryption via SSL is extremely important. At PayPal, security is our top priority. We use end-to-end encryption, not only for our public website, but for all our internal service calls as well. SSL encrypton will, however, affect Node.js performance at scale. We’ve spent time tuning our outbound connections to try and get the most out of them. This is a list of some of those SSL configuration adjustments that we’ve found can dramatically improve outbound SSL performance.
SSL Ciphers
Out of the box, Node.js SSL uses a very strong set of cipher algorithms. In particular, Diffie Hellman and Elliptical Curve algorithms are hugely expensive and basically cripple Node.js performance when you begin making a lot of outbound SSL calls at default settings. To get an idea about just how slow this can be, here is a CPU sample taken from a service call:
918834.0ms 100.0% 0.0 node (91770)
911376.0ms 99.1% 0.0 start
911376.0ms 99.1% 0.0 node::Start
911363.0ms 99.1% 48.0 uv_run
909839.0ms 99.0% 438.0 uv__io_poll
876570.0ms 95.4% 849.0 uv__stream_io
873590.0ms 95.0% 32.0 node::StreamWrap::OnReadCommon
873373.0ms 95.0% 7.0 node::MakeCallback
873265.0ms 95.0% 15.0 node::MakeDomainCallback
873125.0ms 95.0% 61.0 v8::Function::Call
873049.0ms 95.0% 13364.0 _ZN2v88internalL6InvokeEbNS0
832660.0ms 90.6% 431.0 _ZN2v88internalL21Builtin
821687.0ms 89.4% 39.0 node::crypto::Connection::ClearOut
813884.0ms 88.5% 37.0 ssl23_connect
813562.0ms 88.5% 54.0 ssl3_connect
802651.0ms 87.3% 35.0 ssl3_send_client_key_exchange
417323.0ms 45.4% 7.0 EC_KEY_generate_key
383185.0ms 41.7% 12.0 ecdh_compute_key
1545.0ms 0.1% 4.0 tls1_generate_master_secret
123.0ms 0.0% 4.0 ssl3_do_write
...Let’s focus on key generation:
802651.0ms 87.3% 35.0 ssl3_send_client_key_exchange
417323.0ms 45.4% 7.0 EC_KEY_generate_key
383185.0ms 41.7% 12.0 ecdh_compute_key87% of the time for the call is spent in keygen!
These ciphers can be changed to be less compute intensive. This is done in the https (or agent) options. For example:
var agent = new https.Agent({
"key": key,
"cert": cert,
"ciphers": "AES256-GCM-SHA384"
});The key here is exclusion of the expensive Diffie-Hellman algorithms (i.e. DH, EDH, ECDH). With something like that, we can see a dramatic change in the sample:
...
57945.0ms 32.5% 16.0 ssl3_send_client_key_exchange
28958.0ms 16.2% 9.0 generate_key
26827.0ms 15.0% 2.0 compute_key
...You can learn more about the cipher strings from openSSL documentation.
SSL Session Resume
If your server supports SSL session resume, then you can pass sessions in the (undocumented as yet) https (or agent) option session. You can also wrap your agent’s createConnection function:
var createConnection = agent.createConnection;agent.createConnection = function (options) {
options.session = session;
return createConnection.call(agent, options);
};
Session resume will decrease the cost of your connections by performing an abbreviated handshake on connection.Keep AliveEnabling keepalive in agent will mitigate SSL handshakes. A keepalive agent, such as agentkeepalive, can 'fix' Node's keepalive troubles but is unnecessary in Node 0.12.Another thing to keep in mind is agent maxSockets, where high numbers can result in a negative performance impact. Scale your maxSockets based on the volume of outbound connections you are making.Slab Sizetls.SLAB_BUFFER_SIZE determines the allocation size of the slab buffers used by tls clients (and servers). The size defaults to 10 megabytes.This allocation will grow your rss and increase garbage collection time. This means performance hits at high volume. Adjusting this to a lower number can improve memory and garbage collection performance. In 0.12, however, slab allocation has been improved and these adjustments are no longer necessary.Recent SSL Changes in 0.12Testing out Fedor's SSL enhancements.Test DescriptionRunning an http server that acts as a proxy to an SSL server, all running on localhost.v0.10.22Running 10s test @ http://127.0.0.1:3000/
20 threads and 20 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 69.38ms 30.43ms 268.56ms 95.24%
Req/Sec 14.95 4.16 20.00 58.65%
3055 requests in 10.01s, 337.12KB read
Requests/sec: 305.28
Transfer/sec: 33.69KBv0.11.10-pre (build from master)Running 10s test @ http://127.0.0.1:3000/
20 threads and 20 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 75.87ms 7.10ms 102.87ms 71.55%
Req/Sec 12.77 2.43 19.00 64.17%
2620 requests in 10.01s, 276.33KB read
Requests/sec: 261.86
Transfer/sec: 27.62KBThere isn't a lot of difference here, but that's due to the default ciphers, so let's adjust agent options for ciphers. For example:var agent = new https.Agent({
"key": key,
"cert": cert,
"ciphers": "AES256-GCM-SHA384"
});v0.10.22Running 10s test @ http://localhost:3000/
20 threads and 20 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 59.85ms 6.77ms 95.71ms 77.29%
Req/Sec 16.39 2.36 22.00 61.97%
3339 requests in 10.00s, 368.46KB read
Requests/sec: 333.79
Transfer/sec: 36.83KBv0.11.10-pre (build from master)Running 10s test @ http://localhost:3000/
20 threads and 20 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 38.99ms 5.96ms 71.87ms 86.22%
Req/Sec 25.43 5.70 35.00 63.36%
5160 requests in 10.00s, 569.41KB read
Requests/sec: 515.80
Transfer/sec: 56.92KBAs we can see, there is a night and day difference with Fedor's changes: almost 2x the performance between 0.10 and 0.12!
Wrap Up
One might ask "why not just turn off SSL, then it's fast!", and this may be an option for some. Actually, this is typically the answer I get when I ask others how they overcame SSL performance issues. But if anything enterprise SSL requirements will increase rather than decrease; and although a lot has been done to improve SSL in Node.js, performance tuning is still needed. Hopefully some of the above tips will help in tuning for your SSL use case.
