How a good deed became a shakedown
As CTO and Co-Founder of The Block, a news and research company focused on the cryptocurrency industry, my job is to create and maintain a publishing platform for our talented team of crypto-native journalists and researchers. Our engineers and designers operate behind the scenes to keep The Block humming, our readers and paying members happy, and to ensure that our websites, email newsletters, and membership services are globally available, 24-hours a day.
After a year on the job, I’m proud of what a small team of two engineers (👋@joeles and @web_mech) and one designer (👍@jamescabrera) have built: we’ve extended Wordpress to support multi-tier memberships, crypto payments, paywall previews, and members-only Slack Teams, we’ve launched over 30 publicly visible features since January. In addition to our public work on theblockcrypto.com, we have deployed a significant amount of infrastructure to support our business: continuous integration and delivery pipelines, automated backups, log rotation and aggregation, and a sizable number of backend API integrations. All told, there are nearly a hundred distinct scripts and integrations that silently, tirelessly work in the background to ensure that our engineers remain productive and that our data is backed up and safe.
One critical piece of infrastructure that The Block relies on is a Java application called Jenkins. Jenkins is an “automation server” which means it can be scripted to perform the same task over and over again without any human interaction, which saves our engineering team hundreds of hours per month. We use Jenkins for a variety of tasks: it backs up and replicates our databases and gigabytes of images, it creates and initializes demo environments (dev instances) so that we develop multiple features in parallel, it performs dozens of automated tests against new feature code before deploying it to production. Jenkins is the centerpiece of our backend and without it, The Block’s engineering team would be much, much less productive.
In order for Jenkins to “automate all the things,” it requires administrative privileges to almost all of our web-based services and applications including AWS, Github, Slack, and Pantheon, which we use for Wordpress hosting. Rather than storing a username and password to access these services, Jenkins uses a unique security token for each service. It is critical that these tokens are kept in a secure place, with one of our security tokens and a basic understanding of how to communicate with our services via their APIs, a bad actor could gain administrative access to our production environments or code repositories.
A Security Report and Response
I explain all of this because it was recently reported to me that The Block had failed to secure several security tokens in Jenkins. The first issue was exposing a small number of read-only Jenkins jobs to the outside world. This was originally done to support services which needed to communicate with Jenkins but which would do so from a dynamic (versus static) IP address. The second issue was our gradual adoption of AWS Key Management Service (KMS). Most of our security tokens are encrypted using KMS, except the tokens for the services we first integrated with, specifically, Github and Pantheon. Finally, a single character (-ex instead of -e) in a single Jenkins script activated verbose logging on one test job which was visible to the public:
None of these mistakes in isolation is a serious security issue; taken together though, they enabled the public exposure of a read-only Github token and an administrative token for Terminus, which is the command-line interface for Pantheon.
After receiving the security report, we immediately confirmed the details and communicated our intent to address the issues. The engineering team removed all public access to Jenkins, invalidated all exposed security tokens, and implemented changes to our Jenkins jobs to ensure no jobs could run in verbose mode. We’ve started an investigation into our Jenkins access logs to look for unusual activity and we have opened a ticket with the security team at Pantheon. Our investigation has turned up no evidence that the exposed security tokens were misused. We have taken the proactive step of issuing a password reset for all of our users and notifying them of the potential data exposure. We are auditing our entire infrastructure and information technology security policies to ensure that an issue like this doesn’t happen again.
The Shakedown
Normally, this is the part of the story where I graciously thank the researcher which reported the security issue under the accepted practices of “responsible disclosure.” We responded to the researcher within minutes of receiving the security report, we immediately patched the issue, we started an audit to better understand if the exposed information had been misused, and I contacted the researcher with what I believed was an appropriate bug bounty.
The researcher disagreed with my assessment and insisted that they “have gotten north of $10K in a very similar situation” for other reports like this one. My response was as a seed-stage, venture-backed startup, we could not afford to pay out large cash rewards like banks or publicly traded companies and that we made no guarantees of large bug bounties as don’t yet have a public bug bounty program. The researcher then suggested that if we couldn’t afford a higher cash payment for their services, we should instead provide free advertising for their startup on The Block as a reward, and threatened to publicize our error if we didn’t meet his demands. There are many reasons we are unable to provide free advertising, but primarily, we didn’t believe their startup or services were appropriate for our audience. Finally, the researcher stated that they would accept “no less than $1,500” as a reward and to “Don’t expect me to settle for a penny less either, as I have a business to run.”
The conversation broke down after we pointed out that their tactics seemed awfully like extortion to us and we refused to increase the bounty. Perhaps realizing they’d not get what they wanted and might have crossed both a legal and moral line the researcher responded as follows:
Keep the money.
This email exchange, as well as a number of other relevant factors, have made it painfully evident that you are not competent enough to properly secure your infrastructure, and by extension, your customer’s information. I strongly fear that this will continue to be the case in the future.
From the start you have attempted to downplay the severity of the disclosure, and despite my well-intentioned report and respectful manner in which I interacted with you, the way you have treated me in the process has been hostile (going as far as accusing me of blackmail) and has left a terribly sour taste in my mouth.
I am therefore morally obligated to disclose this encounter to the wider public, but will be giving you two days to notify relevant stakeholders before doing so. Legally speaking you were obligated to inform them within 2 days of finding out, the deadline of which has since passed.
Do not contact me any further, and good luck.
So, there you have it. According to this researcher, the primary measure of organizational engineering competency is your ability to pay large rewards for speculative security work. The report that this researcher will be releasing tomorrow would most likely have never been made public if we had just paid them tens of thousands of dollars. At this point, we have involved our legal team to assess the threats this researcher has made towards our organization and whether any criminal or civil violations occurred.
I hope that by disclosing what happened we can continue to maintain the trust of our readers and members, which is why we notified our members promptly even though there was no evidence that their registration information had been accessed.
When I joined The Block, it was with the goal of bringing transparency to the world of cryptocurrency journalism. I am humbled by the fact that hundreds of thousands of readers rely on The Block to provide the best reporting and research in the industry and I will continue to work as hard as possible to exceed your expectations.
Best, Jake McGraw
Co-Founder & CTO, The Block
