Orchestration Focused Design

August 11, 2014

Recently I’ve started worshipping the Crossfit cult. A lot of people in Crossfit take to social media to share their involvement with the sport. Coming from the programming world it was easy to start following Crossfit people on Instragram and Facebook but I didn’t want to invest the time of:

• Finding all the right people to follow; and
• Having to check 2–3 feeds for Crossfit news.

For a weekend project I made thechipper.io. This project grabs all of the top Crossfit feeds and puts them in one place.

This post is about coupling not Crossfit. Starting with the architecture of the project.

This is a standard architecture for an application of this type but the rules for the design aren’t:

1. All data objects are structs without any extra methods.
2. All database models are dumb transaction processors, nothing more.
3. Logic to be held in services.
4. Services are to be stateless.
5. State is to be held in main loops and external interfaces.

The patterns here are very familiar after writing code in Elixir or other functional languages.

The experiment here is if you code with these goals in mind and extract any object oriented-ness out to an adapter setting (for external interfaces only) do you end up writing code that is easier to reason about and maintain?

Serving posts is the simplest slice of code to breakdown:

class Server < Sinatra::Base
  get "/posts.json" do
    page = (params[:page] || 1).to_i
    PostService.all_posts(page).to_json
  end
end

module PostService
  class << self
    def all_posts(page)
      total = Post.order(Sequel.desc(:created_at)).count
      posts = Post.order(Sequel.desc(:created_at)).paginate(page, 40).all
      PostRequest.new total: total, posts: posts, :page => page
    end
  end
end

class Source < Sequel::Model; end
class Post < Sequel::Model; end

class PostRequest < Hashie::Dash
  property :total
  property :posts
  property :page
end

• All services are modules without any state.
• All models are dumb structs with transaction processors attached to them or just fancy hashes.

Fetching posts from Instagram or Facebook is more complicated but uses the same design rules.

# fetch_instagram.rb
loop do
  sources = SourceService.for_instagram
  sources.each do |source|
    source = InstragramService.ensure_system_id_is_set source

    InstragramService.posts_for(source.system_id).each do |instagram_attrs|
      PostService.create InstagramToPostTransformer.transform(instagram_attrs)
    end

    sleep 5
  end
end

module InstragramService
  class << self
    def user_id_for(username)
      ...
    end

    def posts_for(system_id)
      ...
    end

    def ensure_system_id_is_set(source)
      ...
    end
  end
end

module SourceService
  class << self
    def for_instagram
      ...
    end

    def for_facebook
      ...
    end

    def update_system_id(source, system_id)
      ...
    end
  end
end

module PostService
  class << self
    def create(post_attrs)
      ...
    end
  end
end

module InstagramToPostTransformer
  class << self
    def transform(instagram_attrs)
      ...
    end
  end
end

I have removed the logic from the code to keep it brief.

fetch_instagram.rb illustrates a different approach to coupling. The typical approach is to have one way coupling where each layer passes data down the stack. In this approach the main loop (fetch_instagram.rb) is the orchestration layer and it has the logic to push and pull data between all of the other layers in the stack.

This becomes a major system design rule, that only orchestration layers can depend on other tiers. There aren’t any other dependencies allowed.

Comparing this approach to that I used in a recent large application:

The design of the interface would change to:

The goal of this approach is:

• Controllers/routes tests are automatically full stack tests and can be consistent across the system.
• Testing functions in every other tier of the system is isolated and requires very little setup.
• Programmers can forget about what state an object is in and focus on transforming data.

The downside of this is that controllers grow in size but the upside is that complexity is brought to the surface rather than having iceberg style services. It also allows for some tiers to be introduced, like transformers for converting between data structures and validators that just check data integrity.

Summing up, the rules for designing systems in this manner are:

1. All data objects are structs without any extra methods.
2. All database models are dumb transaction processors, nothing more.
3. Logic to be held in services.
4. Services are to be stateless (i.e. use modules).
5. State is to be held in main loops and external interfaces.
6. Only orchestration layers (routers, controllers, main loops) can depend on other tiers.

Having used this on a small system with great success, I’m confident it will help with large system design greatly.

I love flame wars so tweet me your thoughts: @cjwoodward and use the hashtag #orchestrationfocuseddesign.