ArbiterSports Was Hacked. Don’t Use Them Ever Again.
Tech incompetence of this scale can’t just be patched. It’s cultural. Systemic. They don’t deserve a second chance.
In a letter dated Aug 24th, 2020, ArbiterSports (parent company of ArbiterPay and RefPay) announced that they had suffered a security breach that compromised users’ sensitive personal information.
Here’s the letter they sent me:
This breach was an unforgivable failure
- ArbiterSports’ security measures failed to adhere to the most basic security best practices established 30+ years ago (detailed analysis and explanation below).
- There are prior complaints of similarly terrible security flaws dating back to at least 2015.
- They compromised our “account username and password, name, address, date of birth, email address, and Social Security number.” This is more than sufficient for hackers to pull off identity theft attacks on us.
- They don’t specify the scope of the hack but 8,158 users were exposed just in Iowa alone. How many tens of thousands more from the much bigger states?
- They were unaware that they had been hacked for “a few weeks”.
- They paid a deletion ransom even though there is no such thing as proof of deletion nor is it possible to prove that the data wasn’t already copied and distributed elsewhere.
Next steps
- If the deletion ransom was paid with bitcoin or some other cryptocurrency, ArbiterSports must share the blockchain transaction id for analysis (rationale below).
- ArbiterSports must publicly disclose the details of this hack beyond the bare minimums required by law.
- Governing bodies, schools, and club sports should immediately terminate any business relationships they have with ArbiterSports, ArbiterPay, or other subsidiaries.
Any organization that can make a series of mistakes this bad cannot be trusted with our sensitive personal information going forward.
Detailed discussion
While I am not a digital security expert, I am a professional programmer with 20 years of experience building tech startups. I also earned a Computer Science degree with honors from Princeton University.
Security Best Practices: Password hashing
Google “how do I store user passwords”, click on any result. After the page loads, search for the word “hash”. You now know more than ArbiterSports.
In their letter ArbiterSports says, “The passwords and Social Security numbers were encrypted in the file” but the attackers were “able to decrypt the data”.
Passwords should never be stored as decryptable data.
Best practices mandate that passwords be “hashed” — transformed into seemingly nonsense letters and numbers. The key feature of a hash is that it’s a one-way function; it is not possible to take the resulting gobbledy-gook data and reverse the calculations to reveal the original password.
If hackers were to gain possession of these hashes, they would only have garble. They could not know the users’ actual passwords. It’s not that it would be difficult to get the passwords out. As far as modern math is concerned, it just isn’t possible.
But because the attackers were able to decrypt the password data, that tells us that ArbiterSports could not have been using hashing to protect user passwords.
Hashing was developed in the late 1970s and has been in broad mainstream use since at least the early 1990s.
Hashing is one of the earliest foundations of cryptography and one of the most basic must-have protections for passwords in any user database.
How could the entire tech staff at ArbiterSports not know about it?!
(to be fair there is an alternate possibility where they might have hashed the passwords but failed to apply the “salt” best practice which would have left them vulnerable to a trivial dictionary table attack; so not really any better)
Security Best Practices: Email is not secure
“I am hoping someone from Arbiter reads this since they don’t seem to know what they are doing over there. Password encryption is the least of the basic technical issues I have seen.” — RefCT, officiating.com forum
While searching for info on this hack, I came across an old web forum complaint here. In January 2015 this user reset his password and ArbiterPay emailed his new password back to him. That’s like writing top secret info on the back of a postcard and dropping it in the mail. Any mail carrier can see the secrets! The ArbiterSports website was confirmed to have the same behavior. Not surprisingly, hashing was even discussed then.
And they were still emailing passwords at least through April 2016. That poster also knows more about hashing than ArbiterSports.
At least now in 2020 I can confirm that ArbiterSports and ArbiterPay are no longer emailing passwords.
Security Best Practices: Strong Encryption
Unlike passwords, it does make sense for Social Security numbers to be stored in the database with two-way encryption (the data can be encrypted AND decrypted). ArbiterSports would need to be able to decrypt SSNs in order to display them for user accounts, write them into tax forms, etc.
In this realm AES-256 is the industry standard. Google “standard encryption algorithm”. Not hard to find.
We know that the encrypted SSNs were decrypted by the hackers. But there’s no way they hacked AES-256. It’s used by the NSA. All the world’s computers would take much longer than the age of the universe to crack it.
But AES-256 does have one vulnerability: it relies on an encryption key that must be kept secret.
So either:
- ArbiterSports was not using strong, standard encryption and left themselves vulnerable to brute-force attacks or
- They were using AES-256 but they didn’t adequately secure their encryption key.
Whichever is true the conclusion is the same: incompetent security practices.
Proof of Deletion is not a thing
ArbiterSports said that they “reached an agreement and obtained confirmation that the unauthorized party deleted the files.” This strains all credulity. There are almost no conceivable ways that a hacker can provide convincing proof that they’ve deleted the stolen files.
First of all the hackers had possession of the stolen data since “some point in the prior few weeks.” In that time the hackers would have gone to work at cracking whatever encryption was in place (the aforementioned passwords and SSNs).
ArbiterSports knows their cracking was successful because the hackers would have sent them a sample of the unencrypted data as proof.
So the hackers were sitting on now-unencrypted data, potentially for weeks, while they attempted other infiltrations to paralyze ArbiterSports.
And we’re to believe that the hackers offered proof that they deleted all of that unencrypted, stolen data? If you copy a file on your computer, there’s no way for me to know what you’ve done.
And if you delete a file — or don’t — I have no way to know either way.
Who would believe such a claim?!
If they believe it, they’re fools.
If they don’t believe it, they’re lying to us to make us think they’ve adequately cleaned up the mess. But they haven’t. Leaked data is a Pandora’s box; once released it can never be taken back.
Our sensitive personal info is forever vulnerable out in the wild now.
The deletion ransom should be audited and tracked
These sorts of ransom attacks are generally paid out in bitcoin or some other cryptocurrency. Despite popular misconceptions most blockchain data is not hidden nor is it necessarily even anonymous. A significant amount of “chain analysis” can be done to “follow the money” as hackers begin moving it.
I’ve done chain analysis before on the Ethereum blockchain. Everyone running a full bitcoin node has a copy of every bitcoin transaction that’s ever taken place (no joke; blockchains don’t work anything like people might think they do).
If ArbiterSports paid out their deletion ransom in a cryptocurrency, they should provide the transaction id (aka “tx”) and/or the cryptocurrency address it was sent to. The crypto community publicly tracks known scammer or ransom addresses and keeps a close eye when any of those funds are moved.
Hackers generally try to move their crypto to exchanges; that is where they can convert it to a more easily spendable form. If the hackers aren’t careful when they create their accounts and transfer in their funds, they can leave traces that can reveal clues to their identity.
Keeping the ransom transaction a secret prevents this crypto community defense mechanism from doing its job. ArbiterSports says that they “notified law enforcement and are supporting their investigation” but in general law enforcement lacks the expertise necessary for this kind of chain analysis and tracking. They might not even be trying.
Find the ArbiterSports security alert. I’ll wait.
Aside from the physical letter ArbiterSports mailed to my address, I cannot find any online information about this security breach on their website. The only search result is from the Iowa attorney general’s office which publicly posted the notice which, by Iowa law, ArbiterSports was required to disclose and detail how Iowans were affected.
No word whatsoever on their @ArbiterSports twitter account. Their most recent tweet at the time of this writing (Aug 29th) is July 15th, ironically the day they learned they were hacked.
There is no description in the letter I received of how many user accounts were compromised overall. In Iowa alone they notified the attorney general that 8,158 Iowa users’ info was leaked.
The letter I received simply says the hacked data “contained information about our users”. The letter explains that the stolen data was “a backup copy of a database made for business continuity reasons”. In other words: this is a backup that they would use if an emergency crashed their main system. That likely means that it had ALL the data necessary to run their system.
If that’s the case, then ALL of their users have been compromised. Not just me. Not just 8,158 people in Iowa. Every. User.
As of 2008 they had 260,000 registered officials. How many more have they signed up in the last 12 years?
ArbiterSports’ lack of transparency about this hack is appalling. There is no news alert on their website. Presumably hundreds of thousands of people are now at risk of identity fraud, destruction of their credit scores, and more. But ArbiterSports wants to keep this as quiet as possible and only make the bare minimum disclosures that are required by law.
Future users and organizations who might sign up for ArbiterSports / ArbiterPay / RefPay deserve to know the risks they’re facing due to the company’s technical incompetence.
The first Google hit should be the company website.
The second Google hit should be about this security breach.
Actions you should take
Delete your bank info from their site(s).
Contact their customer support and ask them to delete your account.
I suppose you should take them up on their offer for a year of free credit monitoring. Not sure if it really does any good, though.
Share this with other officials and demand that your sport’s governing body, your schools, and your sports clubs stop using ArbiterSports services.
Be sure to include ArbiterSports’ CEO in your emails (kyle.ford@arbitersports.com) to ensure that he’s aware of these discussions. Once we have his attention I look forward to giving him a chance to correct any errors I may have made in this writeup.
Closing
I have judged plenty of head-to-head gymnastics meets, large invitationals, State Sectionals, and even had the honor of judging our State Finals twice. But I will not accept payment through ArbiterSports / ArbiterPay / RefPay ever again.
I hope other officials in my sport and across every sport will heed my warnings and adopt the same stance.
Sincerely,
Keith Mukai
If ArbiterSports would like to respond or clarify anything, I’m happy to discuss and publish any reasonable responses. DM me here or on twitter @KeithMukai. Note that all correspondence on this subject will be made public.
