Fault Tolerance in Asynchronous, Choreographed, Distributed System

Choreography vs Orchestration

Service Orchestration — curtesy of user Andrei https://stackoverflow.com/a/29808740
Service Choreography — curtesy of user Andrei https://stackoverflow.com/a/29808740

The Problem

Asynchronous Processing

API call with queue and queueMetadata
  1. We have a micro-service A invoking the API
  2. API returns result to a queue
  3. Micro-service B picks up the result for further processing
  4. Micro-service B sees an error in the message but cannot do anything but fail, it does not have any knowledge of micro-service A, or the API in between
  • Introduce tight coupling between B and A and B could invoke A again to retry the operation
  • Similarly, we can introduce coupling between B and the API. B can retry calling the API but it would have to know what are the parameters to invoke the API

RetryToken

  1. Micro-service A invokes API
  2. API returns result to a queue
  3. Micro-service B picks up the result for further processing
  4. Micro-service B checks for errors in the message
  5. B validates retryToken and if valid, issues HTTP request to retryToken value, effectively repeating step 1

--

--

--

In his 12+ years in the industry, Ondrej has worked on variety of products from search engines to mobile apps.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Let world know you by using GitHub to host your website.

Sinatra! not Frank?

An Overview of Ethereum and Solidity

Solidity and Ethereum logos

“WePiggy’s Billionaires” WPC Mining Competition on Optimism Begins!

How to modernize identity governance and user access reviews with Workday and Okta?

Hack Your Brain to Improve Coding Skills

How long does it take to learn programming?

Resilient Executive: Which Key Digital Technology Enablers Should You Worry About for Future…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ondrej Stastny

Ondrej Stastny

In his 12+ years in the industry, Ondrej has worked on variety of products from search engines to mobile apps.

More from Medium

Scanning Microservices for Dependencies

AMQP (RabbitMQ) vs Kafka for asynchronous communication

Static + Dynamic Code Analysis with SonarQube

Saga design pattern and transaction in Microservices architectures