BlockCypher Saves the Day
TLDR; BlockCypher helped us keep caffeinated. We highly recommend their API.
October 17th, we had an outage. Customers would load the app, send bitcoin- and never receive a voucher to spend at Starbucks.
Support tickets started flooding in. Users that already had a balance were fine- they could spend as normal in stores. But those who wanted to reload their balance were left in the cold. The hot wallet was filling up, but we weren’t issuing new cards.
The Situation
In some ways, it was a good day for an outage. Usage is typically lighter on the weekend, so fewer customers were affected.
In other ways, it was the worst possible time for an outage. Because most users were still able to use the app, we didn’t recognize there was an issue until late Saturday. And our CTO, Chris, had just flown out for a wedding on the other side of the country.
I tweeted that the team was on it… but that night, the team was just me. Regardless, I resolved to make sure our customers weren’t disappointed buying their Sunday morning coffee. I started my own pot and got to work.
Diagnosis
Some bitcoin services use payment processors like Coinbase or BitPay. For us, those options never gave us enough control over the customer experience. So last August, we moved off Coinbase- and in March, we removed all remaining dependencies on the Coinbase API.
Still, instead of relying on a payment processor, we relied on a blockchain notification service. These services will notify a site of events on the blockchain, including incoming and outgoing transactions on a wallet.
As far as I could tell, ours was working fine. We were still getting notifications of payments to deposit addresses. When I looked a little deeper, though, I realized that these notifications were getting delayed. Instead of getting a notification in minutes, we got them up to 5 hours after they hit the blockchain- not really acceptable if you’re waiting to pay for your coffee! And because payment requests expire after 10 minutes to limit volatility, those payments were being refunded instead of being credited to users’ accounts.
I reached out to the team behind our notification system, but as you might expect on a Saturday night, I didn’t get a response.
A Solution
I decided that night to replace our notification system. If I succeeded, our users would still enjoy their Sunday morning coffee.
I chose to use BlockCypher.
Internally we’d talked about integrating with them for months. But the discussion always ended- we already had a blockchain notification service, so why did we need another? We’d heard they were the best, and we love the team, but no reason to make more work for ourselves.
They were the obvious choice, but it was now midnight on a Saturday. Could I finish and test an integration in the next 3 hours, in time for the morning rush on the east coast?
Apparently, yes. Their API is the best kind- well-documented, and only as complicated as necessary.
Welcome to BlockCypher
I finished the integration, and put it into production for new accounts. I disabled refunds for the time being- better safe than sorry. At 5 AM the dust had settled, and after monitoring the system for a few hours, I took a nap.
I woke up a few hours later to another challenge. I’d put out the fire for new users, but now it was time to migrate old accounts to the new system.
This was the real sticking point. I needed notifications for thousands of previously issued deposit addresses. Otherwise, new users would get a great experience- and loyal customers might still suffer delays.
If you’re familiar with the state of blockchain APIs, you’ll know why I was worried. Our experience with players in the space so far had been disappointing- many focus more on new features over stability.
Normally in these situations, I’d call someone at the company to make sure they could handle what I was about to do. But I was short on time, so I went for it, and crossed my fingers that their infrastructure could handle the spike- and it did.
Service fully restored, I went to sleep, and woke up the next morning to this gem from BlockCypher CEO, Catheryne Nicholson.
Hi Matt,
I see you have 14.8k webhooks in our database already.
Welcome to BlockCypher.
Best,
Catheryne Nicholson
Thanks for not actually saying “I told you so” Catheryne!
Our Experience So Far
Since the integration, I haven’t had to think much about BlockCypher. And that’s high praise for an infrastructure company- they made what should be easy, easy.
In the month since we deployed, we haven’t had any reported notification delays. And we’ve only had 1 error related to notifications- a bug on our end.
If your business depends on the blockchain, we highly recommend you look into the BlockCypher API. Tell them Fold sent ya!
Thoughts or questions? We’re discussing this post on ZapChain To learn more about BlockCypher, follow them on Twitter or discuss on ZapChain.
Originally published at foldapp.com on November 24, 2015.
