Dance like no one is watching. Code like everyone is.
As you may know (and expect) Unsplash is built on the shoulders of giants by leveraging open source and proven projects. There’s a bird’s eye view by Luke here.
I’m not here to tell the same ol’ story about how open source is the best and using it is the way to go.
Since I’m the product of an open source community I take it very seriously, not only by doing open source and talks myself but also pushing the cultural part of it.
Lately I’ve been working on tools that try to solve specific problems we’ve encountered but with a twist: treating it like open source, coding like everyone is watching.
Penumbra
The 24 hour bookstore
The name of this library comes from a book that I really enjoyed some years ago called Mr. Penumbra’s 24 hour bookstore that tells the story of a web designer working in a library, Ruby and the importance of books.
It was the perfect name for our internal Elastic Search library ^_^.
Penumbra was born with a specific purpose: being the nuts and bolts of our new search autocomplete.
However, there were some specifics to be resolved since it was supposed to be stupidly fast and accurate with the recommendations.
I think that the best tools are the ones that solve a specific problem without adding layers of complexity while being honest and clear about what’s going on.
Any fancy stuff happening behind the scenes? Nah, just a `proc`.
Since we were building something for us — with the idea of making it open source — I started with this paradigm:
People want a tool to make the job easier, not to be completely oblivious to what’s happening so Penumbra is a tool. It does not force you to do anything, it will just try to make things easier.
For example: we have a lot of photos so reindexing takes time. That’s a problem. As a result, Penumbra’s sole focus is to make that fast and safe so that you spend less time being worried, and instead spend more time on what actually matters: building a better algorithm.
MRQ
Mr Q — Map/Reduce Queue
The name comes from the beloved MrQ from the James Bond series. That sweet man in charge of all the gizmos and gadgets… and also happens to describe what it actually does: a map/reduce queue for Ruby.
We have a lot of sources of information here at Unsplash, different systems that extract and process all we can get from a photo.
But one thing is important: most of these systems are not dependent on each other, they just extract data so they can be run in parallel.
For that we use concurrent-ruby and push the tool to a feature complete state-meaning it will do one thing and one thing only: map then reduce.
Every map runs in a thread fetching information and the reduce will give you everything once all the sources are ready.
Present Proofing
I like to feel like I’m able to predict the future. That I’ll be able to know where the codebase should be a week, a month.
I like to feel that I do, that I can reach the code utopia I feel we deserve. That I’m no different from Wayne Gretzky saying this:
I skate to where the puck is going to be, not where it has been.
The truth is that I don’t. I have no clue where we are going to be.
Don’t get me wrong, I really know our codebase and I have an idea of where it’s headed-but code is only a tiny percentage of our product.
Businesses will change, performance will fluctuate, team mates might be different, an AI might overtake the world.
However, the thing I’m certain of is the now. So instead of future-proofing for a future that most likely will not be as I imagine it, I prefer to present-proof the code.
And that’s by solving the actual problem that we’re having in the most simple, elegant and real way.
If something is painful for the team, if it takes too much time, if it’s not reliable or if we don’t have the control we need, we must change it.
Ruby will change, JavaScript will definitely change, we will change so instead of imagining the best and fanciest palace that will be nice to look at, but completely useless, let’s just build the stairs we need; one step at a time.
