Hello, World!
My name is Mike Malone and I’m CTO at Betable, a venture backed technology startup based out of San Francisco. In the past I’ve been a fairly active writer of things, speaker at conferences, and contributor to open source and open standards projects. That stopped pretty abruptly about four years ago when I joined Betable, a tiny company with the ambitious goal of reinventing and safely opening up the $500 billion gambling industry to developers who can’t afford their own licenses.
When I joined, Betable was just an idea. We immediately started documenting a basic system architecture (on a white board in the founder’s garage, of course). Six months later we launched a payments flow and our first games. Fast forward about three years… last quarter we shuffled over $150 million through our accounting system. Lots of interesting stuff happened between then and now. To focus on that stuff, I took some time off from sharing my thoughts online. Now that the dust is settling a bit I’m back, and I’m excited to start documenting and sharing that journey here.
As an online gambling operator Betable is a special kind of e-commerce business that, legally, is treated something like a multinational bank that also sells liquor. Our regulatory requirements are as onerous as many publicly traded companies. There are a lot of things we need to get right the first time (e.g., security, payments, random number generation). As a platform, we’re built for scale and developer ease of use. We’ve also got all the normal non-functionals that you’d expect at a startup (move fast, keep it simple, be agile, avoid complacency, yada yada yada). The point is we’re an interesting hybrid of an organization. We need mature processes and technologies to manage risk. At the same time, we’ve got to stay agile, move fast, and attract startup talent to be successful. Balancing these requirements is not easy.
For the most part, we’re not managing this stuff the “normal” way you’d see in an enterprise. Rather, we’ve been pushing the boundaries of agile processes and modern technologies. The accounting system mentioned above uses Apache Cassandra. Our servers? 100% cloud-based and under config management. Overall architectural style? Chatty RESTish microservices speaking HTTPS and/or communicating over an event bus. QA and Deployments? Automated and on demand with a code-review and CI pipeline, and a whole lot of tests. Languages? We know a bunch of them. Pretty normal stuff for a startup (that does things right). Not so normal for a bank/liquor store. So, heads up to anyone who says this won’t work for their organization because insert stupid regulatory requirement here — you’re wrong. The only reason this stuff won’t work is organizational apathy.
That’s not to say there aren’t gotchas… there are lots. For example, it’s not easy getting regulatory waivers so you don’t have to implement password reset questions (because they’re stupid and insecure), or explaining how deploying a system via git + configuration management is a safe alternative to signed gold-masters like JARs or ISOs. We’ve also had our fair share of screw ups. Have you ever run into this Go language quirk? Combined with a bit of test failure fatigue, that little quirk ended up costing us more than $50,000. Stories like this make us all better. On this blog we fully intend to share the bad along with the good.
Oh yea, we also have lots of normal distributed systems, microservice, operational, and general computing problems too. Want to know how to manage more than forty microservices with thirteen engineers and a two person ops team? We do that. Need a high availability durable state machine? We built our own distributed database for that. Need simple security measures to keep all of your microservices safe? Got ’em. Want to keep track of tons of micro-transactions? We’ve got some ideas. Need to support ad-hoc queries, but your online database is a NoSQL distributed key-value store? We’ve got that problem too (spoiler alert: don’t use Hadoop). Anyways, we’ve got some stories to tell!
In the interest of intrigue I’ll save further discussion for future posts. I’m very excited to finally be sharing more about Betable engineering publicly. Follow us here and on twitter to stay up to date, and leave a comment if there’s anything you’re particularly excited to hear more about.
