DevOps

Partitioning in PostgreSQL

Rotating your API request logs table.

Here at Semantics3, among other things, we build APIs to enable easy access to a gargantuan Ecommerce product database. Right from the beginning, we have seen the need to store an audit log for…

GPU-enabled EC2 instance for deep learning

As a team which builds data-powered software for ecommerce-focussed companies, we are always on the lookout for incorporating new development practices into our workflow.

Measuring application performance for great good

In Numbers We Trust

Layered middleware for Internet services

At Semantics3, we are heavily invested in micro-services. We have a suite of light-weight HTTP services deployed that support not just our user-facing products but also different parts of our Data pipeline. Suffice it to say that HTTP is the lingua franca of…

Getting beyond those pesky iptables rules

Most of the development here at Semantics3 occurs on a shared development machine hosted at AWS. While this helps us control our dev environment with high precision, it causes some difficulties when trying to test features on a local computer. For example…

Talk@NUS Hackers: Building a Distributed Web Crawler

I gave a general technical talk on building a distributed web crawler at last Friday’s NUS Hackers meetup. Here are my presentation slides, which I used during the talk.

How We Built Our 60-Node (Almost) Distributed Web Crawler

Web crawling is one of those tasks that is so easy in theory (well you visit some pages, figure out the outgoing links, figure out which haven’t been visited, queue them up, pop the queue, visit the page, repeat), but really hard…

Talk@BigData.sg: Systems Architecture at Semantics3

I spoke about our back-end infrastructure at the May meetup of BigData.sg. Here are my presentation slides, which I used during the talk.