⚡️Can't find the right DB? Just build one!
CeleraOne builds server-side rendered paywalls — powered by a custom DB + a custom programming language!
CeleraOne build server-side rendered paywalls for digital publishers. As a part of the Axel Springer Group, it’s also a member of the Global Axel Springer Developer Community.
Even as a software developer it's not everyday that you talk to someone who genuinely builds their own low-level technologies — especially not high-performance tech like a realtime streaming in-memory database that serves millions of requests every day.
That's why when I first heard that CeleraOne had built a custom in-memory database, I got curious. As it turns out, it's even crazier than I thought — they even built their own programming language, to allow their customers to define programs that act as elaborate database requests which can customize the offers website visitors are being presented with.
But I'm getting ahead of myself.
Let's start at the beginning.
"Fighting for every bit"
Christoph’s job can be summarised in one sentence: He’s fighting “for every bit” as he puts it, being the Lead Developer for the custom database technology at CeleraOne.
CeleraOne offers server-side rendered paywalls to news-websites which decide in realtime, whether a website visitor is a subscriber, and if not which offer is most likely going to make them a subscriber. Server-side rendering to avoid Ad-blockers from circumventing it, realtime because increased page-load times also increase the bounce-rates, and custom offers because newspapers desperately need every paying subscriber that they can get.
The system is shipped in custom-fitted, physical machines that have hundreds of Gigabytes of RAM inside, running their proprietary in-memory streaming database. This puts Christoph and his team in an unusual situation for a web technology team: If they run out of RAM, that's it. They have no auto-scaling to save them.
So they fight for every bit. "Database as a library" as Christoph calls it — whenever new datatypes are needed to model rulesets around user behavior, the database is recompiled. A customer needs to define rules around certain website domains? Then those domains are baked into the database through a lookup table whose index has just as many bits as it needs. Only storing 3 URLs in that table? Then the index is going to have 4 bit addresses.
COPL — the friendly, custom programming language
Christoph has a doctorate in compiler construction and worked at the TU Berlin until 2017, so it's not too surprising that a custom programming language is hustling away inside CeleraOne's database (even though to be fair the custom language was already there when he joined).
He admits that the decision to develop a custom programming language for their custom database at first doesn't make much sense when you think about it from a business perspective. But that's only because it's a little hard to explain why a custom language is needed in this case.
With the database running in-memory to deliver realtime performance, any program that would want to interact with that data needs to run in the same process. That in turn means that any faulty program can not only corrupt data, but also crash the complete system with a memory error.
With the danger of a complete system crash, a custom programming language suddenly does make a whole lot of sense. It allows Christoph and his team to provide customers with just enough power to construct elusive rulesets (e.g. granting unsubscribed users 5 free articles per month, offering special soccer subscriptions for people who seem to like sports etc.) while being able to maintain the safety of the system — which is critical for newspapers to get paid.
The custom programming language is called COPL (CeleraOne Programming Language) and is a statically typed, functional programming language which compiles to machine code (for obvious performance reasons) and — since it’s a custom language running in the same process as the DB — can directly manipulate in-memory values of the CeleraOne database.
It's hard to tell at first glance that COPL is a custom programming language though. The language itself is very well readable, and is offered up to users in a clean looking online editor that features syntax highlighting and lets users test rules, before activating them for live traffic.
What about Rust though?
As the most loved programming language of 2020, that is both type-safe and compiles to machine code (the two requirements for the team) I had to ask though: Why not use Rust, instead of COPL —as their embedded programming language?
The answer dumbfounded me a bit.
If they were to use Rust, it would essentially create the reverse problem of what programming languages usually struggle with. If normally the typical complaint about a language might be that its standard library is too constrained ("It would be so cool if language X had this feature I love from language Y"), CeleraOne's COPL faces the opposite problem: If Christoph and the team can't control the standard library, they can't ensure the safety of the overall system.
Christoph explains it like this: They used to support calling C functions from COPL, but after a while realised that it's too risky and needed to be rolled back. With any other programming language (e.g. Rust) it would have been pretty difficult to remove parts of the language features. Since the team is dealing with their own language though, they simply had to change 30 lines of code and recompile. Pretty elegant.
Overall I went out of this interview, being incredibly impressed. With my background as a JavaScript developer, I'm from the opposite side of the spectrum it seems. I don't even remember what byte-length the different variable types have in JavaScript, because generally I don't have to.
While Christoph and his team are fighting for every single bit, in my world I was more concerned with, for example, uploading gigantic MP4 video files reliably via a web uploader, making sure that even on a shaky wifi connection uploads of the individual file parts were retried in case there was a network problem. I wasn't "fighting for bits" — I was rather throwing bits at the problem, until it was solved.
And yet, Christoph's story is just such a cool reminder that at the end of the day, being a developer means you can build anything.
Literally anything — even a custom realtime in-memory streaming database for millions of requests with its own programming language, named COPL.
—
Connect with Christoph on Github
💼 Curious about working at CeleraOne?
Check out our open positions
— — — — — — — —
Hi, I’m Jonas and I’m currently building up a Global Axel Springer Developer Community. Subscribe to our Weekly Tech Newsletter to get a fresh story like this every Monday and to receive details on how you can join our global developer community ✌🏻
Want me to help tell your story? Reach out to me 👇🏻
🚀 Subscribe to the Weekly Tech Newsletter
