Wire is hiring again: a DevOps + FP position [Berlin]

NB: This post is for people who like to have more details before applying for a job. If you just want to send your resume in, please do it here.


Hi! I’m Artyom Kazak, a backend developer at Wire (until about two weeks ago). We’re looking for an experienced DevOps engineer with functional programming background.

NB: We have been using Haskell in production since 2014, and that consistently worked out great. You don’t need to know Haskell though — we’ll be happy to teach you!

Our product is a messenger for teams and businesses. Kinda like Slack. Or Skype. But not quite. I’m not an elevator pitch kind of person, but if I had to give an elevator pitch, a) I would die from anxiety and b) then I would say something like this:

We intend to eat both Skype and Slack’s lunches, while being entirely open-source¹, using end-to-end encryption for everything², and spending extra effort on advancing state of the art in messaging protocols³ and consumer cryptography⁴ whenever we can.
¹ Backend and client apps are released under (A)GPL-3 and published on GitHub: https://github.com/wireapp/.
² Links to audit reports, the protocol description, the whitepaper, etc are gathered in this post: https://wire.com/en/blog/security-privacy-transparency-journey/.
³ Right now we are collaborating with Cisco, Google, Facebook and others on designing a new protocol for large-scale (50k people) E2EE group conversations with optional federation: https://wire.com/en/blog/mls-future-of-messaging/.
⁴ Since it is very hard to say what shiny new toys the NSA folks will have access to in ten years’ time, we are already investigating whether we can roll out quantum-resistant encryption: https://wire.com/en/blog/post-quantum-resistance-wire.

We have our Haskell bases covered pretty nicely, so now we are looking for people who are more on the ops side. In other words, you don’t have to know Haskell, but you must be willing to learn it.

We are a Swiss company, but development happens in our very nicely located Berlin office, which is where you will be working as well. If you are not in Berlin, we will provide visa support and help with relocation. Also, while it is not a remote position, nobody will mind if you travel a lot or if you like working from home — as long as you don’t do it literally all the time.

I mentioned that we are focusing on business users, but if you’re curious to try try out Wire, we have a consumer version — completely free and without ads. You can also poke at our backend code on GitHub at wireapp/wire-server — about 60000 lines of Haskell, licensed under brutal and unforgiving AGPL-3. And of course, our client apps are fully open-source as well.

Left to right: desktop (JavaScript+Electron), Android (Scala), iOS (Swift+Objective-C). Disclaimer: we can not promise that using Wire will suddenly make you as photogenic as the people in the screenshot above.

As for the salary — I can’t give exact figures, but the salary is “competitive”, which is HR speak for “higher than average”. Apart from relocation and visa support, we also provide a laptop and — for the first month — a free apartment in the city center, right near the office.

Finally, I should mention that English is the only official language at Wire. We have beginner German courses, but if you don’t feel like learning a new language, you can easily get by in Berlin without knowing any German. (I’ve spent more than a year here, completely avoided learning German, and I’m doing just fine.)

Okay, hopefully that was enough of an introduction. Let’s talk about the actual job!

Job description (tl;dr 80% operations, 20% Haskell)

Our backend is hosted on AWS and currently managed with Ansible; our biggest ongoing project is migrating all stateless pieces of the infrastructure to Kubernetes. We want to make it possible for anyone to self-host our backend — including tricky cases like “bare metal, closed network, no AWS allowed”.

You will have some involvement with the old infrastructure, but mainly you will be building the new one. Some of our customers already run our backend on Kubernetes (have a look at https://github.com/wireapp/wire-server-deploy/) — we will need your help to take this to the next level!

The Wire backend itself is written in Haskell. We have several services that handle registrations and logins, push messages via websockets, send out emails, upload pictures to S3, protect users’ privacy by proxying requests to Spotify/YouTube/Giphy, and more. You can expect to spend 20% of your time working with Haskell — investigating and fixing bugs, hunting down memory leaks, and contributing to our custom tools.

Responsibilities

The list of things that we have to deal with is pretty standard — cloud services, orchestration, automation, databases, search, continuous deployment, monitoring. Here are the details:

  • All our infrastructure runs on AWS — in particular, we use EC2, S3, ELB, SES, SQS, and DynamoDB.
  • Even though we’re not planning to migrate from AWS any time soon, we don’t want our customers to depend on AWS in order to host our backend. Making it completely AWS-independent would require doing some research and potentially writing quite a lot of Haskell code — to either support non-AWS dependencies, or work around quirks of almost-but-not-quite-AWS-compatible ones.
  • Our customers may have some specific setups and/or requirements. While we are usually able to suggest a “battle tested” setup (including base OS, firewall rules, etc.), sometimes we have to do requirement gathering. You should be comfortable with that, and also with gently hinting to our customers what may be a good/bad idea about their requirements.
  • Most of the data is stored in Cassandra and Elasticsearch clusters. Cassandra sometimes misbehaves or gets slow, either because Cassandra is just like that, or because somebody wrote a suboptimal query that leaves lots of tombstones. (That person may or may not have been me.) Regardless, somebody has to help poor Cassandra when she gets into trouble.
  • Ansible is currently used — and will likely continue to be used — for managing database clusters (Cassandra, Redis, Elasticsearch, MinIO), setting up CI, renewing SSL certificates, etc. For a sample Ansible role, see wireapp/ansible-cassandra.
  • As mentioned earlier on, our stateless services will eventually run on Kubernetes. This part of the job involves writing Helm charts (templated Kubernetes configuration files), setting up custom environments for other developers to play in, and doing research into topics like “What is the best way to back up etcd?” or “How to do load-balancing if the cluster is not hosted on a cloud provider and there is no button ‘launch me an ELB’?”. See wireapp/wire-server-deploy for a taste of how our Helm charts look like.
  • We have a self-hosted installation of Concourse for continuous integration and deployment. A sample issue here would be, for instance, “aw dammit, why has Concourse stopped doing continuous integration and deployment”. (Spoiler alert: the answer is usually “it ran out of space”. But sometimes it’s not!)
  • Monitoring is done with Prometheus and Grafana, and for logging we have Kibana. Luckily, those are pretty resilient, so you will spend more time using them than fixing them. (Hopefully.)
  • Our internal tools (deployment helpers, database migrators, and so on) are written in Haskell and sometimes they need to be extended or debugged. For writing new tools or one-off scripts we pick whatever is best for the task — doesn’t have to be Haskell.

Finally, once you have learned the ins and outs of our infrastructure, we will add you to on-call rotation. While on-call incidents happen quite rarely, it still means that there will be periods of time when you will have to carry your phone and laptop around.

Requirements

You must have experience with automation tools (e.g. Ansible, Chef, Puppet) and with orchestration tools (e.g. Kubernetes, Mesos, Docker Swarm). Note that those are just examples — the leap from any orchestration tool to Kubernetes is much smaller than the leap from zero to Kubernetes. In fact, we do not require experience with any of the specific technologies we are using — the important thing is that you must have enough overall experience to not feel intimidated by them.

Haskell knowledge is also not a strict requirement, but you must have experience with at least one functional language. Being good at operations is more important for this position than being good at Haskell specifically.

Lastly, you should be mostly alright with the infrastructure we already have. We’re not against changes, but if you are a person who would push heavily for a migration from Kubernetes to NixOps, that would not be ideal.

Everything else is optional.

How to apply

Fill out this form. You will need to attach a resume and a cover letter. If you have any recommendations (even if just a tweet that says something vaguely nice about you), throw them in as well.

Two tips for the cover letter:

  • While having relevant experience is nice, there’s never going to be a 100% overlap between what you know and what we do. The ability to learn is pretty much as important as your experience (if not more important). So, if you have any examples that show that you can learn non-trivial things, including them will help.
  • We have no managers in the backend team, i.e. self-sufficiency is a very useful trait for working here. If you have successful side-projects or if you were a project/tech lead at your previous job, that’s a bonus.

Working at Wire

The backend team is largely involved in medium-term projects that don’t require tight coordination with other teams.

This has quite a few benefits — say, very flexible working hours. If you hate getting up before 11am, you won’t ever have to. If you have a date planned for today’s afternoon, nobody is going to bat an eye. If you feel like working from home today, just notify the rest of the team and off you go (er, stay). Or maybe you want to go to Cyprus and work from there for a couple of weeks — that’s completely okay too. (Psst! Cyprus is great. You should totally go there.)

The relative lack of deadlines also means that you can always take some time to try your hand at anything you want to get better at. For instance, once you have worked here for a while, you can serve as a tech lead for a company-wide sprint and coordinate feature development — it is demanding, but fun. Or you can attend meetings and help us divine what our customers want from us. Or give internal talks. (Or write job ads! Welcome to the club.)

A fair warning: we don’t really have a system in place for extracting work from people if they don’t have internal motivation and a feeling of responsibility. Some people work best when they have more freedom, and some — when they have more structure. Either is fine! But if you’re in the latter category, Wire likely won’t work out for you.

Interview process

If you CV is interesting, we will call you — to chat about your experience, reasons for looking for a new job, expectations from Wire, and so on. If you have any questions after reading this post, we’ll answer them as well.

[You don’t have to brush up on binary search at this stage.]

Then somebody from our team will do a technical interview, where you will have to face deviously cunning questions along the lines of “Can you tell a bit more about $(some thing you mentioned on your resume)?” and “How would you approach $(such-and-such problem)?”.

[Binary search is still not needed.]

If it seems that we like each other, we’ll invite you to visit our Berlin office, all expenses paid, for an onsite interview. Which will once again consist mostly of talking. Admittedly, a whiteboard might be involved — if you want to illustrate something (or just show off your calligraphy skills).

[As you could have guessed already, at no point will binary search come up. Or any similar silliness.]

If all is good, you sign the contract and you are hired! Otherwise you will hear from us as soon as possible — and you are welcome to try again the next time we have an open position.

The Wire office

We are located in one of the Hackesche Höfe buildings in the center of Berlin. That’s us right here:

Rosenthaler Str. 40, 10178 Berlin

We have yoga, massage, free drinks, breakfasts every morning, lunches on Tuesdays, and two coffee machines: a simple one and a fancy one. The backend team has its own cozy room in the office, but you can work anywhere else you like — on a couch on the rooftop, or in a distraction-free meeting room, or perhaps in the nearest Starbucks.

Oh boy, I’m going to get so much flak for recommending Starbucks. I hope that nobody from my team actually reads up to this point.

A terrace where you can sit, enjoy the sun and look at the tourists below. Starbucks: not pictured.

Living in Berlin

(guest section from Tiago)

The first thing newcomers notice when they arrive in Berlin is that public transportation just works. Pretty much 24/7, timetables are accurate, and getting from A to B is generally easy. (But hey, don’t forget to punch your ticket before boarding the trains… if you buy it on a tram though, it’s already punched for you when you buy it onboard — sounds complicated, but you will get used to it and will soon be helping out the many visitors!).

The city is pretty large and packs about 3.7 million inhabitants; unlike most European cities though, there’s no “typical” city center with an old town where everyone gathers and where most restaurants/shops/hangout places are located: instead, you will quickly learn the names of the boroughs inside the “ring” (train line that goes around Zone A) and sound like a local! Each one has its one identity and style (different cafés, types of food, you name it).

Is Berlin multi-culti? You bet! The number of foreigners keeps rising — Disclaimer: I have actually not done factual research on this so I am basing it on experience and the fact that I see more and more meetups, standup comedy shows in many different languages. Also, if you ever need to visit a doctor, you are likely to find one that speaks your mother tongue!

I have heard that Berlin is the Silicon Valley of Europe… is it really? Well, I have never been to SF so I can’t really compare! What I can say is that the amount of meetups on all different topics (ranging from Haskell to Blockchain), networking events and other interest groups is simply impressive: you can pretty much go to one every evening if that is your thing.

Is music your thing? Well, welcome then! Home to Berghain (have a look at the latest film: Berlin Bouncer) and other epic locations, Berlin has a place for everyone. It’s no coincidence that Soundcloud’s founders decided to set up base here and turned an otherwise quiet café named Sankt Oberholz into a vibrant cowork place where 9 out of 10 people are trying to create the next big thing in tech! Do you want to hang out in a pool inside a ship in the middle of the river? Well, just look up “Badeschiff”!

You can expect to spend 30–40% of your salary on taxes and health insurance — which is still less than in Sweden (take that, Sweden). Also, Berlin is significantly cheaper than a lot of other European capitals, and healthcare and social security are very good. If you want to go into further details, https://www.brutto-netto-rechner.info/ is your friend (German only!).

If you have a family, Germany is probably one of the best places to live in. We have many people at Wire with kids in kindergarten and school, so if you have children, you’ll have someone to ask for assistance or advice. And, of course, we will help with visas and relocation for your family.

♫ This is the end

If you have any questions, you can always write at jobs@wire.com. If you’re in Berlin, feel free to pop by the office and get a tour — just write to backend@wire.com first to notify us.

If you want to try out Wire, the download links for all platforms are here. If you don’t know anyone else who uses Wire, you can add me — my nick is @lightgreen.

If you know any companies that want their employees to communicate securely, and if they have not looked at Wire previously, please point them to sales@wire.com. We would be happy to give a demo if they need one.

Cheers!