furr.io dev blog #1 — to infinity, and beyond!

Most of my friends know I lead a secret double life as a furry, however, the most exposure I get to the fandom right now is through a local offshoot of a furry-oriented Facebook page that I started and currently co-admin.

So, naturally, as a professional web developer who possesses a metric fuck-ton of — ahem — shall we say, images begotten from the aforementioned community, I also wanted to make an attempt to categorize and/or tag them. Further, as a Facebook admin, I have to say, administrating a Facebook page is extremely inconvenient; there’s a lot of opportunities for people to harass and exploit the Facebook Page system.

Last year, I originally attempted to create an alternative with a gallery with an infinite-scroll on the Meteor platform. Unfortunately, I ran into limitations with this approach, and with 1.3, Meteor has changed since then (you’ve changed, man). I’ve also since discovered the benefits of writing one’s own stack. Namely, it still has a build system that doesn’t give you as much control as Webpack does, and its performance is suspect. Further, I’m not fond of the idea of deploying with MongoDB and the Meteor application server. Some Meteor devs might object to these reasons,

Now, I’ve been a fan of Meteor for a long time now, and I’d still recommend it for developers who don’t have the same capabilities and skillset as I do (I’ve been developing web applications for nearly ten years, and some of my oldest complex JavaScript dates to 2008). Meteor will take you very, very, very far. But that said, you must love Meteor. You must have all your faith in it. And recently, I’ve lost my faith in that stack, mostly because I want freedom!

For example, instead of relying upon Meteor’s famous/infamous MongoDB LiveQuery mechanisms to manage communicating document state, I’d prefer to use a more efficient messaging queue and do durability bookkeeping afterwards. These kind of things generally aren’t sought-after by a Meteor developer. Further, the ecosystem has only just now been exploring React/Redux, but I’ve been experimenting with more functional approaches using a React/Redux stack since late last year. I just feel like I’ve reached the limits of how far I’d like to go with Meteor, and I’ve outgrown it.

One thing I’ve also found is that boilerplates will only take you so far. As a solo developer, you must be intimately familiar with every single part of your application stack. My advice: find a good React/Redux guide, start with a tooling like Webpack and Babel, and feel free to jump around and mix and match code samples if you feel confident enough to move past the venerable todo list and kanban board recipes.

So, as you might have deduced, for the frontend, I’m using Webpack, React, and Redux. Not a trivial choice of frontend stack, to be honest; there’s a lot of support out there because it’s gotten really popular, but it takes a lot of tooling, configuration, documentation and tutorial-reading, and a decent understanding of some basic functional programming tenets. Choosing this stack is not a decision to take lightly, but if you do choose it, I can guarantee that your skills as a JavaScript developer will advance a great deal more than choosing other stacks.

As for backend, I’m planning to use trusty old Express, as Koa never really caught on, generators are… arcane, and it supports streaming, which will be extremely necessary for the image processing component. As for websocket frameworks, the days of Socket.io have come and gone, and there are a great deal more approaches these days, and I plan to explore them deliberately. And, unlike in Meteor applications, not everything has to be a websocket.

I’ll be using RethinkDB, but without the Thinky ORM, which, in my mind, adds unnecessary cruft. I’d prefer to use the rethinkdbdash driver it’s based off of. Not everything has to be highly abstracted. I know, I know. JavaScript developers don’t often say that…

The reason I prefer RethinkDB is because it supports live cursor updates much like Meteor’s LiveQuery functionality, but on the database level, instead of the application level. That, I think, is essential for the approach I’m taking. Further, I wanted a messaging queue. RabbitMQ will do nicely for that, and both are hosted on Compose, which is something I’ve used before for both and I’ve been satisfied with the attention to security and performance and configurability with both on that platform.

So far, I’m liking what I’m reading about Jimp, and I’ll likely give that a go, especially for its native streaming support.

For auth, I’m not looking forward to storing PII, including passwords, and going down a JWT rabbithole. I see that as a security liability. I’d rather just use something like Auth0. (That’s with a zero)

As far as deployment goes, Google Compute Engine seems to be the strongest contender for a scalable IaaS in my book. Sorry, I’ve used EC2, Modulus, Digital Ocean VPS, Azure, IBM BlueMix, and Heroku over the years, and GCE seems to be the most compelling so far. (Have I missed any? Anyone remember DotCloud?)

This will not be an inexpensive stack to deploy. With Compose RethinkDB and RabbitMQ, GCE, S3, and Auth0, I’m thinking it would take about $100 a month just to get started. That said, I plan it to be a production-ready stack, and not a prototype or ‘toy’ stack. That was a feeling I always got when I wrote Meteor applications. The ones I wrote always felt like prototypes or toys.

So, with all that introductory material out of the way, now for an update on how development is proceeding so far.

Also, I don’t have a huge amount of hobbies I’m really into other than writing JavaScript web applications, building on the startup I co-founded, and maybe the occasional game or TV show. For me, writing code is a helluva lot of fun. So, I spend my weekends doing just that. So, I decided to build (or re-build) furr.io this weekend.

Some background on this image tagging and categorization system I mentioned earlier… This won’t be just any image tagging system. It’s not going to be a simple gallery that anyone can contribute to. No, instead, I wanted to take a few cues from the Slack messaging platform, but with an imageboard take. I don’t want to use Slack itself because their uploading functionality is inconvenient and ill-suited for this application.

So, imagine a workflow where you upload a bunch of images on your hard drive, they get processed (thumbnails, perceptual hashing for de-duplication, and color detection for fun), stored on S3 and kept track of in the database. Now that they’re on your private image pool, you can then scroll through, search by artist, category, or tag name (which, with de-duplication, this information is handily shared).

Then, and this is the cool part, you share them in private channels that also act as a sort of tagging mechanism. The images get tagged automatically by being in that channel. They’d achieve a tag acceptance threshold once they’ve been favorited in that channel (and thus, added to your private image pool), or uploaded under that tag several times. Cool, right?

Only problem is, the sheer volume of images I wanted to throw at it. I, like many of my fellow furries tend to collect a lot of images. I’m talking in the tens of thousands… Per folder.

My machine could handle rendering about a thousand react-dropzone preview URIs without crashing, but once it got any higher, I’d see crashy-crashy-sadfaces and big red “INSUFFICIENT_RESOURCES” errors in my console. By the way, react-dropzone is a very well-executed component.

So, I naturally had to set up a lazy-loading infinite scroll solution. The first one I tried was react-infinite. It seemed more than capable, but it took some hammering to get it to work with several images across. I’d suppose it’d be a limitation of sorts, since it only works with rows of items, not rows and columns of items, like how I wanted. I had to implement a reducer that grouped images across the window width, so it actually scrolls by rows. This is the code I wound up with for that:

It’s rather novel because it has that tricky reducer. It’s basically seeing what the width of the page is, dividing it by the width of the images to see how many images it can fit across, and then grouping them into individual divs for each row.

And yes, I used a traditional for loop. Yes, you can still do that with React in 2016. And yes, it made sense the way I did it, because I didn’t want to have to slice a new array each time I wanted to access a portion of the array of images passed to it. It just seemed arbitrary and inefficient.

The result is this:

It actually works very well, and seemingly is ready for arbitrary numbers of images. Now, I wouldn’t say it’d work for hundreds of thousands or millions at once, but I’m happy with it and it seems to go pretty far.

Next steps are to set up a backend where you can pre-tag your private image pool and actually upload them.

Also, I skipped over the part where I implemented a Redux reducer in “Ducks” format. I’ll revisit that when I add uploading functionality. Also, I used Rebass for a basic component library, but I’ve found I’m adding a lot of custom styles on a per-component basis in order to hammer that into place. The grid actually looked a lot nicer, in terms of gutters and spacing before when using Rebass Card and CardImage components, but those used Flexbox, which didn’t seem to be really well-supported by the opinions of react-infinite.

I haven’t open-sourced the repo just yet, and I’m not sure if I will, just because it’s a private application repo, and I haven’t cleaned up / refactored my components nicely, and I still need to finish up some tests, as of the time of writing of this article.

One thing I’d like to know is, what do people think of my approach? Are people interested, and also, would you pay for access to such a system? If so, how much? Otherwise, I’d just like to hear what people think so far.