Hipster batch technique on Microservices

Hipster batch is one of powerful technique for gaining your microservices highly availability.

Requirements — Basic understanding of AWS S3

Start from Selling burgers!

Let imagine with a simple scenario, I am opening a burger shop. My shop sells 3 kinds of burger ‘Donkey Burger’ ‘Beef Burger’ and ‘Fish Burger’. Unfortunately, the price of donkey meat, beef, fish, vegetables and burger bun changes every night. So, I have to update the prices on the menu everyday to respect the cost of the ingredients. Otherwise, I will lose profit or customers. Sounds crazy! but it is business!

Best Fish Burger, Donkey Burger and Beef Burger

Building Menu service

With a simple design, Menu service has to talk to Donkey service, Beef service, Fish Service, Vegetable Service and Bun Service for asking the price of ingredients. Then, calculate the burger prices according to those costs. As a result of that, we can design our real-time system like this:

Menu Service in very beginning

This Menu Service may expose several endpoints.

1. /menus for list burgers with prices
2. /menus/id for details of burger respect to id

However, when I re-consider the system. I feel that every time consumers make a request to Menu service. It requires at least 3 calls from Menu service to external services. (Donkey, Beef, Fish, Vegetable,Bun) Since, Those external services could have different SLA which could effect to Menu Service when one of them is down or slowly response. Moreover, we’ve known that the menu prices are changed only once a day. It’s unnecessary to calculate the prices at real time. That’s why ‘Hipster batch’ come into a play in this scene.

What is Hipster batch?

Hipster batch is a old-style scheduled batch processing to create and publish feed files. When combined with cloud technology like Amazon’s S3 file storage and hypermedia linking, this can create a highly available, yet simple and testable solution. https://www.thoughtworks.com/radar/techniques/hipster-batch

Implement Hipster batch to Menu Service

Firstly, We will build Menu Batch service instead of Menu service.. just a few simple steps

1. Menu Batch service is triggered (use AWS Auto Scaling)
2. It starts querying ingredient prices from external services and processing burger prices
3. The processed prices are stored in S3 bucket in JSON format
4. Terminate Menu Batch service when finished

Every night Menu Batch service is periodically run then storing processed data in S3. After that, API consumers could directly request to S3 instead of consuming Menu Service.

S3 structure & format

According to auditability of the burger prices. I keep revisions of the prices as separated folders. So that, every single day, I store 2 kind of resources. 1. menu resource (ex: 20160831_131722.json) and 2. menu item resource (ex: c50ecf12-c964–4df7–8e8f-f41743d5aa3e.json) Then, update menu/index.json to point to the latest menu resource.

S3 Bucket structure

JSON format, I personally like HAL format as Hypermedia representation. http://stateless.co/hal_specification.html

So that, menu/index.json looks like:

menu/index.json

In menu/index.json gives me discoverability to navigate to the latest version of the menu. For this example: s3://burger-shop/menu/2016–08–31/20160831_131722.json

The reason of the latest menu file structure as ‘{date}_{timestamp}.json’ (ex: `20160831_131722.json`) because there could be very much situations which require re-running batch multiple times during the day. So that, I can keep revisions and make sure that the existing data will not been overridden.

Then, Look at our latest menu at `menu/2016–08–31/20160831_131722.json`

Finally, I can see the burgers (menu_items) price here as an embedded resource of menu. The menu item also provides link to navigate into detail of the resource itself.

The new structure

Then, Look at overall structure again. Menu Service is removed. Instead, Menu Batch Service and S3 come.

Menu Batch service & S3 bucket

How cool?

1. Gain higher available service. Since, S3 is designed for 99.99% availability
2. S3 is very cheap. Don’t even compare with live service instances.

3. S3 is security by default. I can grant read permission to read only service. And grant write permission to write service. or so…

4. Service consumers don’t need to understand what rules S3 JSON files are stored (ex: stored menu as menu/date_timestamp.json ). Instead, just look into `index.json` then start navigating from there.

In Real World / Further implements

1. Hipster Batch can fetch data from other sources apart from API services. such as `Donkey Service can be raw CSV in Box` or `Beef Service can be MySQL` or even another S3 bucket
2. Structure & Format of S3 can be different from this article. example: I can choose JSON API over HAL format
3. Manual batch trigger can be done via AWS API Gateway + AWS Lambda to enable Auto Scaling. Rather than login into AWS Console / AWS CLI

More + Reference + Inspiration

1. https://www.youtube.com/watch?v=BqLd8PtcciE from Beth Skurrie

Thanks for reading! Any Questions? or Feedback?

ปล. พูดคุยเป็นภาษาไทยได้นะครับ