User Data Collection Update —
Over the last two weeks we’ve been combating an on-going spam and DoS attempt with our service Temporal exploiting the fact that we don’t do any mandatory verification of user identities. When we started Temporal one of our foundational goals was to respect user privacy as much as possible, and to avoid collecting user information unless absolutely necessary. This meant that when supplying your email to sign up was an optional feature. While technically speaking you are required to input an email as part of the call to our API, this email wasn’t required to be verified.
Up until now this hasn’t been problematic, however within the last two weeks a person or person(s) have begun conducting spam and DoS attempts to Temporal and our IPFS nodes. With go-ipfs, the biggest performance sink is the pinning system, which when pinning a large number of objects, performance gets exponentially worse. Generally speaking this isn’t a problem as long as we dont have people abusing the lack of user verification, and free tiers we provide. The kind of usage we were getting was very uncharacteristic of our normal usage. Namely uploads of extremely small sizes (12 -> 112 bytes) and tens of thousands of them. With the size of the uploads being made, and the amount of data allowed by our free tier, the attacker would be able to upload approx 600k files of this size before reaching their free tier limit. As detailed in a blog post comparing TemporalX nodes performance vs Go-IPFS performance, a pinset of roughly 5 million pins is required to drop node performance by 50% when it comes to ingesting new data. So it would take an attacker roughly 8–9 accounts to upload 5 million pins of the 112 byte size.
Due to this strange usage we sent the email on file a brief email inquiring about their usage, and to see if perhaps they had an automated script gone haywire. There was no response to our email after 24 hours, however the uploads were still increasing in frequency. As such we sent a final emailing noting we are disabling their account, and that they should email us if they believe this is an incorrect move on our part. Less than 24 hours later the attacker was back with the exact same activity, and uploading the exact same type of files, in the exact size range. Account was disabled, and again within 24 hours they were back.
The act of banning these accounts is not sustainable, and will never be efficient as by the time we identify the account conducting this kind of activity, several hours would already pass, and thousands more uploads would be into the system. This lead to us needing a more proactive solution to combat this kind of spam. While talking with some people, it seemed like the only solution to this would be user identity verification likely via SMS. This aspect of user verification however is a direct conflict to our beliefs of not wanting to collect user information. So we decided to see if other solutions would work.
The first update we made was introducing a new default account tier called “Unverified”, and requiring email validation before getting access to the free tier. Additionally we rolled our per-ip rate limiting on our APIs, an upgrade to our previous “X requests/second for all IPs”. Unfortunately however, this simply slowed down the attacker, and they continued on with their plans, albeit at a slower pace. We left this update sit for a few days to see how it would do, and it was clearly not sufficient. The attacker was coming from a series of publicly identifiable IP address blocks, so we decide to put a blanket ban on all IPs associated with those blocks temporarily, whichis not sufficient since you simply need to use a VPN and you’ll defeat this protection measure.
As such it was clear that we needed to implement some form of user verification, as this would make it more difficult for the attacker go about their business. While this goes against our beliefs of not wanting to collect user information, unless we made some change this attack would not stop, at all. So we decided to roll out a change where unless you verified your email, you would be unable to authenticate, and subsequently be unable to make any kinds of uploads. Additionally we rolled out a reCAPTCHA integration onto our playground as a secondary preventative measure.
While discussing the reCAPTCHA integration, one person asked us if we had no thoughts on the privacy implications of this, as cloudflare as recently abandoned reCAPTCHA in favour of hCAPTCHA. While this is a valid point, one must understand that we’re a small team. Our backend API is written in Golang, and hCAPTCHA has no golang libraries, so we would be forced to write one from scratch to use their API. This would cause us to have to spend more time in the midst of an on-going attack, and in the midst of all the other projects we have in motion. This just wasn’t a practical decision because we needed defensive measures now, not a week or two from now. If reCAPTCHA really is a problem for people, then we would welcome a PR into the open-source Temporal codebase that enables hCAPTCHA, but for reasons previously mentioned it is not viable for us to do this integration ourselves.
The end result of all this is an unfortunate change to our user data collection, making email verification a mandatory activity to be able to use Temporal. We personally dont like that we have to do this, but there is no other option. If we don’t do this, then the attackers will be able to bog down our systems, and prevent other users from having speedy access to our infrastructure and to IPFS. This doesn’t mean we’ll spam you marketing emails, updates, etc… and we will still use emails for exactly what we do now, which is limited to payment processing notifications, and notifications about errors occurring while processing your request. We hope this is enough to prevent this kind of attack against our service, but if it isn’t we will likely need to roll out SMS verification. We’ve contemplated some form of blockchain based identity verification, but the unfortunate reality of that is that its a UX nightmare. The bulk of the mainstream world, and our users are not blockchain experts, they’re regular people, regular businesses. Requiring blockchain based verification would be a UX disaster for these people, so we had to settle with email verification, and hopefully not have to do SMS verification.
As for who this attacker or attacker is, we’ve collected a healthy bit of information through their attacks, and have some suspicions as to their real identity, but public speculation is not fair to the suspect d parties, and is a waste of time and energy. We truly regret having to roll our required email verification, but the old adage is true, one bad apple spoils the bunch.
Email verification requirement has already been rolled out. All previous accounts will have their emails automatically verified, however it is likely the attacker has some other accounts that we have yet to identify, so there is a chance we will have to ban some accounts post roll out of this new requirement. If your account is accidentally banned please contact us.
Join Temporal’s online community on Twitter, Telegram, or visit our company Website, Medium or Github for more information.
