Post Mortem: Gemini Exchange Downtime on November 29, 2017
As many are aware, Bitcoin attained new peaks of value on November 28th and into November 29th. On the morning of November 29th, our Web interface experienced an unprecedented increase in traffic which impacted the performance and availability of our Web interface and to a lesser extent our API servers. At no time was this increase in traffic an attack on any of our systems and our customers’ funds remain secure. This was solely a result of an unprecedented surge of requests on our platform.
Throughout the events that lasted from 10:11AM EST through 07:11PM EST, we made repeated attempts to tune the system to handle the extremely high traffic on the Web server. Most of these attempts were met with new, higher spikes in activity. The API remained unimpacted during the majority of the time that our Web interface was experiencing the excessive traffic, and our FIX and matching engines continued to fully operate throughout.
The environment stabilized at 7:11PM EST and, after monitoring the mitigation for a period of time, we communicated to our customers at 8:45PM EST on our status page that both our Web and API interfaces were in full functioning order.
As many of you may know, in August we migrated our primary trading platform and network PoP (Point of Presence) to our own hardware in the Equinix NY5 data center in Secaucus, New Jersey. The downtime on November 29th was due to an application tuning issue, and our servers in the data center had plenty of room to scale. Still, in preparation for events such as the one above, we have forecasted and prepared for increasing capacity and were already installing our new hardware when this occurred. We plan to bring this new capacity online in the near future to continue scaling our infrastructure to better service our customers and community. The security and availability of our Web, API, and FIX interfaces is our first concern and we strive to have them available for all our customers at all times.
Onward and Upward!