Authorization is act of granting access/privileges to a resource. Authentication precedes authorization step to assert the identity of a particular user. Here we have used JWT tokens to provide a decentralized authorization for micro services.

JWT in short is self-contained authorization token issued by server post successful authentication of user credentials. The token is in encoded form with 3 portions. First portion is an encoded header, second one is encoded body, and the last one is encoded signature. Body can contain the claims for the token issued by the server during authentication step. While the header and body are simply encoded but not encrypted, can still be asserted for integrity using the signature portion on the server before serving the request.

The primary idea of decentralized authorization is to have authorization to be granular and lives within the service that owns the data/resource. In a way it is also called Native authorization pattern where the authorization code and service are written in same programming language.

Taking a step back, Identifying the right pattern for authorization can become confusing and challenging when there are many ways of implementation.

In our case, it’s an online shopping portal with only one group of users who are customers who can place/cancel orders and track delivery status. All we wanted to evaluate primarily as part of authorization was to check whether the user has access to the requested resource. Apart from token validation checks.

Here I would like to mention different authorization patterns we have evaluated. I would be illustrating the same using a sample set of microservices as shown below. We have started with extremely naive thinking and slowly moved to a maintainable solution with few comprisable tradeoffs.

The Setup

In the above diagram, we have 3 services.

Service A: Takes external requests and talks to downstream service.

Service B: Takes both external and internal requests and talks to downstream service C.

Service C: Serves the requests coming from different sources from the database.

Also, there is an external ingress which accepts requests from Client and an internal ingress which facilitates communication between services.

Points and Requirements for evaluation:

• Standard JWT token for authentication. It has claim data of the “userId” with a pre defined expiry time.
• Have to check if the user has access to requested resource.
• Should allow some endpoints to be used for both internal and external purposes without compromising security.
• E.g: Let’s say we need to fetch checkout for a client request. When it is called from outside, should have user Id along with checkoutId in the request. Whereas, when an internal service (like a retry/consolidation service) would like to utilize the same endpoint, it should be able to fetch using checkoutId itself without the need to send user Id. As these internal requests need not be initiated from the user itself.
• Ability to restrict certain endpoints for internal consumption only.
• Less cost of maintaining authorization policies.
• Assume there is a dedicated service which does authentication.

Authorization approaches

Authorization on API gateway intercepting requests, responses & headers (Proxy/Gateway pattern)

• Place some sort of API gateway in front of all services. All external traffic is routed through API gateway.
• This will allow us to possibly centralize authorization to the Api gateway itself and all downstream services need not have concern of authorization.

Disadvantages include:

• Approach is very much centralized with too much of authorization logic specific to different types of endpoints. This can suffer maintainability in the long run.
• Also reading and parsing entire request/response/headers will have performance cost.
• In case of intercepting response and doing authorisation will have adverse consequences and can fail miserably for mutation requests
• Needed additional endpoints on each service to fetch user Id for a resource type by the API gateway.

Authorization to be performed on Service A (Variation of Native Authorization pattern)

• Fetching user Id accessible for a given requested resource from Service C’s DB
• Authorization is done in Service A where it doesn’t have direct access to the data, nor the service is owner of the data.

Disadvantages:

• Have to create additional endpoints on service C for fetching userId for various resource types.
• Also doesn’t feel natural to do authorization at a point where the service is clearly not owner of the resource.

Authorization on DB Query layer on Service C (Variation of Native Authorization pattern)

• When a DB query is executed, we make sure user Id will always be part of where clause of the query.

This gives fine grained control on the resources being accessed, but have disadvantages like

• Purely depends on developer awareness to not forget to add userId in to the where clause.
• Authorization responsibility is leaking to query layer, and it is not very obvious.
• Also doesn’t provide flexibility to differentiate requests from internal and external sources without sending that information all the way from handler to Query layer. This will eventually pave way to create endpoints with internal and external scopes.
• Certain service may have actual data needed, but do not have userId mapping within same database to perform a join and use the where clause.

Authorization middleware at service C (Variation of Native Authorization pattern)

• Service owning data can perform pre-authorization using middleware for each request handler.
• Ability to turn off authorization checks for internal endpoints (more details on this below) and the logic stays in middleware and is not leaked to rest of the service layers.
• Authorization is decentralized and lives in the corresponding service and thus enables more maintainability for the developers.

Disadvantages

• Duplicated logic in multiple services unless explicitly pay attention to de-duplicate of logic across services. (E.g. If something is repeated, can be moved to an authorization policies library)
• No single place to know all the authorization policies in the entire system.

Out of all the above approaches, we feel authorization middleware at service level is more approachable and maintainable given its advantages of being decentralized. In our case, given the number of user groups and accesses required for each group, we don’t anticipate the policy implementations exploding unmanageably. Also, this approach with the use of some additional marker headers can help to move this authorization from the first service (doesn’t necessarily own data) which receives the external request to the service that owns the data.

How did we implement?

We leveraged Istio service mesh and added an envoy filter which does the following.

• Adds X-App-User-Id header on verifying the JWT token expiry and validity.
• Adds X-App-Ingress-Source header with value as external if request originate from external ingress.
• Also adds X-App-Authz-Mandatory header if request is originating from external ingress.
• Also added tests to make sure above filters are not altered without knowing as it can potentially harm security.

What is the intent of these headers?

X-App-User-Id: Determines the User Id of the request which is parsed from JWT token.

X-App-Ingress-Source: Determines the source of request, value external determines that it’s a request from customer device.

X-App-Authz-Mandatory: Determines if authorization check needs to be performed on the receiving service.

Additionally, we created couple of shared middleware implementations to make sure these headers are honored while request moves from service to service. For the purpose of this article, we are going to have code snippets in Golang.

Parsing headers from incoming web request:

Below example is a middleware that parses x-app-authz-mandatory and x-app-user-id headers from incoming web request and add its it to the context object. As a standard go pattern, context object is sent as parameter to every function within code.

// middleware.go
package middlewares

import (
 "github.com/gofiber/fiber/v2"
)

// PreAuthHeader middleware to add pre auth headers to context
func PreAuthHeader(c *fiber.Ctx) error {
 c.Locals("preAuthHeaders", getPreAuthHeader(c.Get))
 return c.Next()
}

func getPreAuthHeader(headerGetter headerGetter) map[string]string {
 return map[string]string{
  "x-app-authz-mandatory": headerGetter("x-app-authz-mandatory"),
  "x-app-customer-id": headerGetter("x-app-customer-id"),
 }
}

Add headers to outgoing web request:

Below example is a middleware that reads x-app-authz-mandatory and x-app-user-id from context and adds as additional request headers to outgoing web request from any service.

// RoundTripper implementation to be used by HTTP client
package roundtrippers

import (
 "net/http"
)

// PreAuth RoundTripper to add pre auth headers
type PreAuth struct {
 next http.RoundTripper
}

// RoundTrip implementation of round tripper with pre auth headers
func (trt *PreAuth) RoundTrip(req *http.Request) (resp *http.Response, err error) {
 if headers, ok := req.Context().Value("preAuthHeaders").(map[string]string); ok {
  for k, v := range headers {
   req.Header.Add(k, v)
  }
 }
 return trt.next.RoundTrip(req)
}

// Adding roundtripper while creating http client instance
func NewClientWithRoundTrippers(client http.Client) http.Client {
 transport := http.DefaultTransport
 if client.Transport != nil {
  transport = client.Transport
 }

 preAuthRT := roundtrippers.NewPreAuth(transport)

 return http.Client{
  CheckRedirect: client.CheckRedirect,
  Timeout:       client.Timeout,
  Transport:     preAuthRT,
 }
}

Above middleware's ensure both X-App-User-Id and X-App-Authz-Mandatory headers are propagated to all downstream services.

Below are the scenarios explaining how headers will be propagated:

1. Request originated from customer devices (E.g: Request to Service B directly)

E.g. Create order request received from client application.

When request is received from the customer device, it will pass through External ingress and will land to respective service. Envoy filter does the following:

• Authorize and add X-App-User-Id by reading the JWT claims.
• Adds X-App-Authz-Mandatory header to mark that the request needs mandatory authorization as it is originated from customer device.
• Adds X-App-Ingress-Source header with value external to mark the request is routed through external ingress.

2. Request originated from internal service (E.g: Service B)

E.g: Get order request by order Id from internal service.

When request is originated from internal service like Service B, authorization is skipped as both X-App-Authz-Mandatory and X-App-Ingress-Source headers won't be available. However, its optional to add X-App-User-Id header as it doesn't dictate any mandatory authorization but acts as additional information to upstream service.

3. Request originated from customer device and served via an internal aggregator/proxy (E.g: Request to Service C routed via Service A)

E.g: Get invoice request when made from customer device. Invoice service can talk to order service to get details of the order.

When request is received from the customer device, it will pass through External ingress and Service A and will land to Service C. Envoy filter does the following:

• Authorizes and adds X-App-User-Id by reading the JWT claims.
• Adds X-App-Authz-Mandatory header to mark that the request needs mandatory authorization as it is originated from customer.
• Adds X-App-Ingress-Source header with value external to mark the request is routed through external ingress

In this case,X-App-Authz-Mandatory header will be sent to upstream services by using the PreAuthRoundTripper middleware at Service A. This header will allow Service C to perform authorisation instead of Service A as Service C being the owner of the data.

• However, X-App-Ingress-Source header is not propagated through.

Another point to note here is, Scenarios 2 and 3 above solves multi purposing endpoints for external and internal access.

Let’s take a case, “Get Address for an order” is called from customer device. It must return unauthorized (401) error when customer is not associated with order itself. Whereas when the same endpoint is used in the internal context to fetch address for the given order id shouldn’t bother about the userId.

Restrict some endpoints from external access:

Whatever best authorization strategy you can have, it may not be thought through and implemented on day 1. Since we also have some existing liability of some endpoints not suitable for authorization.

E.g. A proxy endpoint which is supposed to be accessible by internal services but do not have sufficient information to do authorization at the middleware.

X-App-Ingress-Source allows us to restrict external access to certain endpoints. X-App-Ingress-Source is not propagated to downstream services. This is marker header to know that the request is directly received from an external ingress. RestrictExternalAccess middleware if added to any handler, checks if the header is present and the value = externaland returns unauthorized response. This helps when you have endpoints that needs to be restricted to be called from public domain while the rest of them continue to be available. However, this header is expected to be used only as a stop gap solution.

// Implementation of the RestrictExternalAccess middleware
package middlewares

import (
 "github.com/gofiber/fiber/v2"
 "errors"
)

// RestrictExternalAccess rejects direct requests from external clients which have
// "X-App-Ingress-Source: external" header in the request
func RestrictExternalAccess(c *fiber.Ctx) error {
 if c.Get("X-App-Ingress-Source") == "external" {
  return errors.Unauthorized("This endpoint is not directly accessible from external ingress")
 }
 return c.Next()
}

// Example of how the RestrictExternalAccess middleware can be used


// ....
// ....
// Restricting external access to particular endpoint
app.Get("/profiles/:id", middlewares.RestrictExternalAccess, h.GetProfiles)
// Rest of the code about starting web server, registering handlers etc ommitted for brevity

Conclusion:

With Istio providing header injection, middleware providing support to implement authorization policies at each service level. We will have a decentralized authorization in place which allows high maintainability and also allowing to defer authorization to the service owning the data.

If you have reached till here, it means you may have few questions like — How are we going to handle if we have multiple groups of users? I think, the same Authorization middleware’s would have to be enriched to work based on claims which can be sent as separate header by authentication service once the token validation is successful.

Advantages:

• Decentralized, the logic resides in the service that owns the data.
• Authorization is moved to middleware instead of retaining it as part of the service code.
• Helps to have custom logic on how to retrieve userId from each type of request especially if the userId is not standardized across the requests.
• Maintainability is high.
• Identifying the authorization logic related to particular resource access is available with the service itself.
• Each authorization function for an endpoint is focused and granular and allows modification easily.
• Highly extensible when new endpoints require new authorization.

Disadvantages:

• Authorization policy is tied to the implementation of the service; thus re-usability can be low.
• Cannot find all the policies in one place for the purpose of re-use and auditability.
• Istio playing major role in adding request headers. Slight misconfiguration can cause the authorization to be skipped entirely. Hence having tests for envoy filter configuration is very much necessary.
• Benefits of centralized authorization like caching can be low.

References:

• An example of delegating authentication and adding additional request header is available here — https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/lua_filter#script-examples
• Envoy filter configuration samples — https://github.com/istio/istio/wiki/EnvoyFilter-Samples

Golang channels are used to establish communication between go routines. The same channels are also used to inform Done state of a go routine.

Here is how we typically do it in Golang if you have to spin off a bunch of go routines and stop/conclude them using a dedicated Done channel

package main

import (
 "sync"
 "time"
)

func main() {
   // Initialise a done channel
   doneChan := make(chan bool)
   var wg sync.WaitGroup
  
   // Creates 10 goroutines
   for i := 0; i < 10; i++ {
      wg.Add(1)
      go waitOnDone(i, &wg, doneChan)
   }
  
   // Wait for 10 seconds and close the doneChan
   time.Sleep(time.Second * 10)
   close(doneChan)
  
   // main program to wait until all go routines are closed
   wg.Wait()
}

func waitOnDone(idx int, wg *sync.WaitGroup, doneChan chan bool) {
   defer wg.Done()
   for {
      // Do some computation here
      select {
        // If its able to read, it means channel is closed
        case <-doneChan:
           println("received done on: ", idx)
           return
      }
   }
}

Here are the couple of things about Golang that made this possible:

1. when a channel is closed it can be read at least once by all the waiting go routines on the channel.

2. A select statement can read from multiple channels at once.

Usually a separate channel can be used to signal end of a go routine. This channel is commonly referred as Done channel, although its not a specific type but the name itself.

The Rust way

Rust also has channels as part of the standard library. But I chose to use the channels from crossbeam crate.

There are two distinct ways to read data from a Rust channel. One is to use recv() method and other one is try_recv() method. The first one is blocking call while the second one is non-blocking and returns immediately with an empty error value when there is nothing sent on the channel.

Would Rust allow you to read a single done channel from multiple threads when tx is dropped/closed? The answer is Yes.

Unlike Golang, Rust do not have in built support to read from multiple channels at once. This can be achieved in the following way.

Sequentially calling try_recv() methods for different channels ignoring the empty error on each call would effectively simulate what select does in Golang.

extern crate crossbeam;

use std::thread;
use std::time;

fn main() {

    // done channel
    let (done_tx, done_rx) = crossbeam::channel::unbounded::<()>();
    let mut handles = vec![];

    // some other compute channel which you would use for business logic
    let (compute_tx, compute_rx) = crossbeam::channel::unbounded::<u8>();

    // Spin off 10 worker threads
    for i in 0..10 {
        let done_rx_clone = done_rx.clone();
        let compute_rx_clone = compute_rx.clone();

        let h = thread::spawn(move || {
            loop {

                // Try checking on channel for disconnection
                match done_rx_clone.try_recv() {
                    Err(e) => {
                        match e {
                            crossbeam::channel::TryRecvError::Empty => {},
                            crossbeam::channel::TryRecvError::Disconnected => {
                                println!("looks like done on thread {}", i);
                                break;
                            },
                        }
                    },
                    _ => {}
                }

                // Try checking on compute channel if there is any data to process
                match compute_rx_clone.try_recv() {
                    Ok(v) => {
                        println!("received value on thread {}: {}", i, v)
                    },
                    Err(e) => {
                        match e {
                            crossbeam::channel::TryRecvError::Empty => {},
                            crossbeam::channel::TryRecvError::Disconnected => {
                                println!("err on compute channel {}: {}", i, e);
                                break;
                            },
                        }
                    },
                }

            }
        });

        handles.push(h);
    }


    // Drop done_tx after certain time to emulate done channel
    let _wh = thread::spawn(move || {
        thread::sleep(time::Duration::from_secs(10));
        println!("sending signal");
        drop(done_tx);
    });

    // Send some data on compute_tx
    let _ch = thread::spawn(move || {
        loop {
            thread::sleep(time::Duration::from_secs(2));
            compute_tx.send(u8::MAX).unwrap();
        }
    });

    // Wait for all handles to finish
    for h in handles.into_iter() {
        h.join().expect("error joining thread");
    }

    println!("all threads handled");
}

Beware of tight loops

Would non blocking read from multiple channels without a delay result in an effect of on infinite while loop? Let us find out.

When I ran the above program on my Mac, CPU shot up to ~1000% ⏫ consistently. Here is a snippet from top command

This CPU spike seems to happen as we are doing nothing when the channel is empty error resulting in an infinite tight loop.

// Try checking on compute channel if there is any data to process
match compute_rx_clone.try_recv() {
    Ok(v) => {
        println!("received value on thread {}: {}", i, v)
    },
    Err(e) => {
        match e {
            crossbeam::channel::TryRecvError::Empty => {},
            crossbeam::channel::TryRecvError::Disconnected => {
                println!("err on compute channel {}: {}", i, e);
                break;
            },
        }
    },
}

While I don’t have a perfect answer for this. A quirk helped to resolve this. By adding a sleep on receiving an empty error on the compute channel brought down the CPU utilisation to almost ~ 0.1% ⏬

Here is the updated snippet where the sleep is added.

// Try checking on compute channel if there is any data to process
match compute_rx_clone.try_recv() {
    Ok(v) => {
        println!("received value on thread {}: {}", i, v)
    },
    Err(e) => {
        match e {
            crossbeam::channel::TryRecvError::Empty => {
                // sleep for 1 second on empty channel
                thread::sleep(time::Duration::from_secs(1))
            },
            crossbeam::channel::TryRecvError::Disconnected => {
                println!("err on compute channel {}: {}", i, e);
                break;
            },
        }
    },
}

You should have noticed that adding sleep to one of the try_recv() reduced the CPU significantly. You wouldn’t have to do it for the other channel as the tight loop is already broken.

Thoughts to ponder

Is there a better way to do this?

I am not sure, this has worked for me so far.

Can you avoid sleep on compute channel?

You can bring down the sleep to a reasonable value based on your need. As long it breaks the tight loop, you are good.

Can we use the same compute channel for the purpose checking for Done without a need for dedicated channel?

Of course you could do that too as long you as you have ability to drop all references to compute_tx, otherwise stick to a dedicated done channel as described and keep it consistent across app.

Have I explored select! from crossbeam?

Not yet.

👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇

🚀Join FAUN Developer Community & Get Similar Stories in your Inbox Each Week

Golang like Done channel in Rustlang was originally published in FAUN.dev() 🐾 on Medium, where people are continuing the conversation by highlighting and responding to this story.

So many of you have gotten accustomed to work from home by now. This pandemic has thrown a lot of challenges at our face when most of us are not ready for. While Work From Home is kind of boon, living with it for a long time comes with its own unique challenges. One of the simplest challenge I have faced in the initial days was to make my family understand that I am in a meeting where Mic/Camera/both are activated. It was becoming difficult sometimes to respond back to the family members for very trivial things while I am listening to a crucial conversation on zoom or trying to answer an important question on an ongoing video chat. Of course they understand that I am in work and it is difficult for me to respond. But these things can slowly be the trends at your home.

Gestures are great way to convey the same. But they don’t always go smooth on the receiving side especially when you do them for visitors.

As a technologist, we should be able to tackle this better.

• How about sending text messages to family members when you are on/off meeting(May be overwhelming)
• Share your calendar with family members? (How to deal with unscheduled ones?)
• Mobile app to show your availability(Too much of work?)
• How about placing visual indicators at our desk? (Can be a Good starter)

I have picked up the last idea out of all the above ones which seems doable with relatively little effort. Back to hardware mode.

Outcome:

Two simple LED light indicators of distinct colours, one for Microphone and another for Camera. Indicators turn on automatically based on your activity on your mac(picked up Mac as a starter) be it on Zoom, Meet, Discord or Facetime etc.

What do I need?

1. Majority of people use either one or two meeting applications on a daily basis. Check for developer documentation and find out whether they expose any apis to read Mic and Camera usage status.
2. Unfortunately, I will not able to do this at least for zoom meetings when I last checked in their developer docs. Thats too disappointing to find out.
3. An app to detect Mic or Camera usage on your computer.
4. Doing this is not so easy especially to work with OS APIs. As I use Mac and I am no expert in OSX development. I am looking for some open source alternatives and found out a tool called “Oversight” from a security researcher Patrick Wardle. More about the tool here
5. Again unfortunate that I don’t get any apis to query device status. But hold on, the tool writes the device status to a log file which I can scrape. Hoo yeah.
6. Two 10mm LED bulbs of different colours and jumper wire connectors which I can get from an online store.
7. I need a controller which has wifi support. I chose NodeMCU V2 for this.
8. I have a Raspberry Pi with on OLED display running in my living room all the time. I wanted to make this a central hub for any device communications within my home rather than my work laptop.
9. Idea was that my work laptop just updates device status to raspberry pi and NodeMCU queries it every few seconds and control the LED state accordingly. Later I also wanted to leverage this for my windows laptop, Mobile and tablet as well.

Here is a short demo video

On my Raspberry PI

To make things easy, I have assigned static IP to my Raspberry pi4 device based on its Mac address in my home router configuration.

Wrote a simple Golang HTTP webserver with endpoints to query as well as update device status.

package main

import(
	"net/http"
	"fmt"
	"nodemculistener/devicestate"
)

var devState *devicestate.Indicator

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/mic", handleMicState)
	mux.HandleFunc("/camera", handleCameraState)
	mux.HandleFunc("/*", handleUnknown)

	setup()
	if err := http.ListenAndServe(":3000", mux); err != nil {
		fmt.Println("listen error: ", err.Error())
	}
}

func setup() {
	devState = devicestate.NewIndicator()
}

func handleMicState(rw http.ResponseWriter, r *http.Request) {
	if r.Method == http.MethodGet {
		fmt.Println("mic status", devState.IsMicEnabled())
		rw.Write([]byte(fmt.Sprintf("%v", devState.IsMicEnabled())))
	} else if r.Method == http.MethodPut {
		qParams := r.URL.Query()
		devState.SetMic(qParams.Get("state") == "on")
		rw.WriteHeader(http.StatusOK)
	} else {
		rw.WriteHeader(http.StatusBadRequest)
	}
}

func handleCameraState(rw http.ResponseWriter, r *http.Request) {
	if r.Method == http.MethodGet {
		fmt.Println("camera status", devState.IsMicEnabled())
		rw.Write([]byte(fmt.Sprintf("%v", devState.IsCameraEnabled())))
	} else if r.Method == http.MethodPut {
		qParams := r.URL.Query()
		devState.SetCamera(qParams.Get("state") == "on")
		rw.WriteHeader(http.StatusOK)
	} else {
		rw.WriteHeader(http.StatusBadRequest)
	}
}

func handleUnknown(rw http.ResponseWriter, r *http.Request) {
	fmt.Println("unknown req", r)
}

On My Mac

Installed OverSight which starts writing log to a standard location. /System/Volumes/Data/Users//Library/Application Support/Objective-See/OverSight/OverSight.log

There is a catch here:

Oversight has an additional functionality which sends system tray notifications when device is activated or deactivated with information of the process like PID and name etc. The catch lies here, if you click on Allow on the notification (let’s say for zoom process) badge, The process will be whitelisted by oversight and you will not further receive notifications and neither will write to log file for Zoom again.

To solve this, I have turned off notifications from the Mac system settings for Oversight application and also cleared all the whitelisted entries from Oversight application.

Script to scrape OverSight logs and updates device status tp Web server running on Raspberry pi

#!/bin/bash

echo "Base URL configured: $BASE_URL"

function updateMicStatus() {
	if [ $1 -eq 0 ]; then
		curl -X PUT "$BASE_URL/mic?state=on"
		echo "mic status updated to on"
	else
		curl -X PUT "$BASE_URL/mic?state=off"
		echo "mic status updated to off"
	fi
}

function updateCameraStatus() {
	if [ $1 -eq 0 ]; then
		curl -X PUT "$BASE_URL/camera?state=on"
		echo "camera status updated to on"
	else
		curl -X PUT "$BASE_URL/camera?state=off"
		echo "camera status updated to off"
	fi
}

cat "/System/Volumes/Data/Users/krishnak/Library/Application Support/Objective-See/OverSight/OverSight.log" | grep "Audio Device" | tail -n 1 | grep -q -w "active"
AUDIO_INIT_STATUS=$?
cat "/System/Volumes/Data/Users/krishnak/Library/Application Support/Objective-See/OverSight/OverSight.log" | grep "Video Device" | tail -n 1 | grep -q -w "active"
CAMERA_INIT_STATUS=$?

updateMicStatus $AUDIO_INIT_STATUS
updateCameraStatus $CAMERA_INIT_STATUS

tail -1f "/System/Volumes/Data/Users/krishnak/Library/Application Support/Objective-See/OverSight/OverSight.log" | awk '/Audio Device/ && /[^[:alpha:]]active/ { system("curl -X PUT $BASE_URL/mic?state=on") } /Audio Device/ && /inactive/ { system("curl -X PUT $BASE_URL/mic?state=off") }' &
tail -1f "/System/Volumes/Data/Users/krishnak/Library/Application Support/Objective-See/OverSight/OverSight.log" | awk '/Video Device/ && /[^[:alpha:]]active/ { system("curl -X PUT $BASE_URL/camera?state=on") } /Video Device/ && /inactive/ { system("curl -X PUT $BASE_URL/camera?state=off") }'

Run the script from Mac automator as an application:

BASE_URL="http://192.168.11.111:3111" <Path To Folder>/monitor.sh

Creating a automator application on Mac:

Followed the process from here https://michal.karzynski.pl/blog/2013/01/13/how-turn-shell-commands-mac-os-x-services/

On My NodeMCU

NodeMCU program to listen to device state changes:

#include <Arduino.h>
#include <WiFiManager.h>
#include "RestClient.h"
#include <ESP8266HTTPClient.h>
#include "Config.h" //Keep your URL and SSID credentials in this

#define MICLED D5
#define CAMERALED D6

WiFiManager wifiManager;
String hostname;

int readMicStatus();
int readCameraStatus();
String getServerHostname();

void setup() {
  pinMode(MICLED, OUTPUT);
  pinMode(CAMERALED, OUTPUT);

  WiFi.mode(WIFI_STA); // explicitly set mode, esp defaults to STA+AP
  Serial.begin(9600);
  //reset settings - wipe credentials for testing
  //wm.resetSettings();

  // Automatically connect using saved credentials,
  // if connection fails, it starts an access point with the specified name ( "AutoConnectAP"),
  // if empty will auto generate SSID, if password is blank it will be anonymous AP (wm.autoConnect())
  // then goes into a blocking loop awaiting configuration and will return success result
	// wifiManager.setConfigPortalBlocking(false);

  if(!wifiManager.autoConnect(HOTSPOT_SSID, HOTSPOT_PWD)) {
        Serial.println("Failed to connect");
        delay(5000);
        ESP.restart();
  } else {
        //if you get here you have connected to the WiFi    
        Serial.println("connected...yeey :)");
  }
}

void loop() {
	// wifiManager.process();
  if (readMicStatus() == 1) {
    digitalWrite(MICLED, HIGH);
  } else {
    digitalWrite(MICLED, LOW);
  }

  if (readCameraStatus() == 1) {
    digitalWrite(CAMERALED, HIGH);
  } else {
    digitalWrite(CAMERALED, LOW);
  }

  delay(2000);
}

int readMicStatus() {
  int ret = -1;
  HTTPClient http;
  http.begin(MIC_ENDPOINT);
  int httpCode = http.GET();

  if (httpCode > 0) {
    String payload = http.getString();
    Serial.print("mic status: ");
    Serial.println(payload);
    if (payload == "true") {
      ret = 1;
    } else {
      ret = 0;
    }
  }
  http.end();
  return ret;
}

int readCameraStatus() {
  int ret = -1;
  HTTPClient http;
 
  http.begin(CAMERA_ENDPOINT);
  int httpCode = http.GET();

  if (httpCode > 0) {
    String payload = http.getString();
    Serial.print("camera status: ");
    Serial.println(payload);
    if (payload == "true") {
      ret = 1;
    } else {
      ret = 0;
    }
  }
  http.end();
  return ret;
}

LED wiring:

WiFiManager library:

When NodeMCU is not able to connect to your home router/hotspot for various reasons like SSID/password change. Instead of writing new SSID and password configuration by physically connecting and flashing NodeMCU with your computer, it turns on a Wifi Hotspot with pre defined SSID and allows you to connect via mobile/laptop and allows your add/correct router details. This library solves the most common problem of maintenance and I liked it.

Config.h:

Keep your URL and SSID credentials in a separate header file rather than in the code directly and don’t forget to ignore it from git tracking. I placed these details in include/config.h file and included it in the main CPP file. This is one of the preferred and simplest approaches for maintaining secrets with CPP.

**PlatformIO.org IDE for IOT development:**

Arduino is a great IDE for programming NodeMCU boards. But it definitely lacks some things which a regular C++ IDE would have like

• Auto code completion
• Static code analysis
• Library manager
• Debugger
• Test framework

I personally found PlatformIO IDE solves these problems as well as supporting other existing features like inbuilt board management, serial plotter etc too.

More Ideas/Coming Soon:

• Trim Oversight code of additional functionality and do better integration
• Make an API call using Alexa skills/IFTTT to change colour of your smart bulb instead of LEDs
• Instead of using (Mac + NodeMCU + Raspberry Pi), this setup can be simplified by directly using either of wifi enabled NodeMCU or Raspberry Pi with your Mac as they both support GPIO pins for connecting LEDs.
• Send push notifications to your family members phones using services like Twilio/Pushbullet etc.

Note about VSCode extensions:

While I generally develop on mac but the actual code is sitting on my Raspberry pi SSD. I use a cool “Remote SSH” VSCode extension to access files over SSH connection on VSCode editor as if they were on your local machine.

Moreover, VSCode is intelligent enough to let me choose my PlatformIDE extension to be installed on the Raspberry pi over SSH thus allowing Platform IDE to detect and work with connected devices on Raspberry pi natively. How cool is that.

Reference Links:

Patrick Wardle — OverSight: Exposing Spies on macos

Disclaimer: Installing a third party tool like OverSight comes up with their own set of questions about security, privacy etc. This tool is open source and go see for yourself before you want to use it.

Originally published at github.com

Minio is an S3 compliant data storage service. It can be hosted on premises and even supports distribution across multiple nodes. To meet certain data protection regulations, data is required to be encrypted the moment it is written to disk. Minio supports two types of encryption schemes

• SSE-S3 (Server side encryption) — Encryption key is managed on server side typically using a KMS
• SSE-C (Client side encryption) — Encryption key is managed by clients and provided as request headers to Minio

Goal of this blog is to guide you through setting up Minio with server side encryption. For server side encryption a KMS(key management system) is required. We have chosen Hashicorp vault as KMS here. Minio also supports a Key Encryption Service(KES) which is a stateless cryptographic operations service for Minio with the keys provided from KMS.

KES supports only one form authentication currently which is mTLS. Minio authenticates itself to KES with client TLS certificate for every connection. At a time, KES also supports different policies to be applied for different clients(Eg: Minio). KES acts like a lightweight gateway to use KMS thus abstracting out KMS footprint to clients.

This blogs covers following in step wise manner:

1. Concepts Overview — Some of my exploration notes
2. Generate a root CA certificate pair
3. Hashicorp vault setup
4. Minio KES setup
5. Minio Standalone Setup
6. Minio Distributed Mode Setup
7. Minio Benchmarking using S3 benchmark
8. Additional information

Points 2 through 5 provides information on Minio Standalone setup. Where as Point 6 is required if we want to turn the Minio setup into distributed mode.

Point 7 discusses about benchmarking and how to read the results.

Point 8 is some additional information required especially for Devops or Sys Admins.

If you want to read the same blog on notion — https://www.notion.so/Published-Encrypted-Minio-Storage-Setup-6ff3e91640b84ba59a6466ac95656503

1. Concepts Overview:

Minio

Kubernetes Native,High Performance Object Storage

• Claims 183GB/s (Read) and 171GB/s (Write) speed on standard hardware
• Can be federated with other minio clusters spanning multiple datacenters
• Native to kubernetes
• MINIO S3 Gateway (Microsoft Azure uses it) and more compatible S3 API

Security Overview:

Master Key → generates External Key(EK)(Data Key) → derives Key Encryption Key(KEK) → encrypts Object Encryption Key(OEK) → stored as part of object metadata

OEK is used in Secure channel to encrypt 65536 bytes at a time with unique secret key derived from OEK and with unique nonce ⇒ PRF(OEK, part_no)

• OEK is generated and encrypted with KEK. Encrypted OEK is part of object metadata
• KEK is always derived from PRF(EK, IV, context_values). IV, context_values, Master Key ID and encrypted EK are part of object metadata
• EK is generated from master key from KMS

Ref: https://docs.min.io/docs/minio-security-overview.html

Hashicorp Vault

Seal/Unseal:

• Shamir seal
• This is the unseal process: the shards are added one at a time (in any order) until enough shards are present to reconstruct the key and decrypt the master key.

Lease/Renew/Revoke:

• Vault promises that the data will be valid for the given duration, or Time To Live (TTL).
• consumers of secrets need to check in with Vault routinely to either renew the lease (if allowed) or request a replacement secret
• To manually revoke → vault lease revoke
• To renew lease from current time → vault lease renew -increment=3600 my-lease-id
• Revoke with prefix → vault lease revoke -prefix aws/

Authentication:

• Client should be authenticated and gets a token with attached policies
• Enabling auth method → vault write sys/auth/my-auth type=userpass
• Follow docs

Tokens:

• Tokens maps to policies
• Root tokens are generated in 3 ways — init, using another root token and using generate-root
• Root tokens should be revoked as soon as initial setup/debug/exploration is done
• Generate new root token with key shards → vault operator generate-root
• Child tokens are revoked when parent is revoked, whereas orphan tokens can be created to make sure only that token is revoked but not the children.
• Token accessors are used to renew, revoke, lookup props and caps.
• Tokens can have TTLs except root(TTL=0 is infinite). Tokens TTLs can be renewed with vault token renew
• All Tokens have hard MAX TTL of 32 days. For long living connections use periodic tokens(renew them frequently type). Tokens can be renewed with token store roles, using appRole, have root token with auth/token/create endpoint

Policies:

• Refer — https://www.vaultproject.io/docs/concepts/policies
• Policy-Authorization Workflow
• Built in policies — default(attached with every token, cannot be deleted by can be excluded while token creation ) and root(available and cannot be deleted)
• Root tokens are created on initialization and should be revoked before putting in prod
• List policies → vault read sys/policy
• create policy → vault policy write policy-name policy-file.hcl
• update policy → vault write sys/policy/my-existing-policy policy=@updated-policy.json
• delete policy → vault delete sys/policy/policy-name
• Associating tokens with policy on create → vault token create -policy=dev-readonly -policy=logs

High availability mode:

• support storage backend to get high HA mode
• By default HA mode is turned. In general, the bottleneck of Vault is the data store itself, not Vault core. For example: to increase the scalability of Vault with Consul, you would generally scale Consul instead of Vault

GPG and keybase support:

• On initialisation, the unseal keys generated can be encrypted with GPG public keys of corresponding users and only private key of that user can be used to decrypt it. This is useful when the terminal output is eavesdropped.
• Below command ensures encryption with GPG keys

vault operator init -key-shares=3 -key-threshold=2 \\ -pgp-keys="jeff.asc,vishal.asc,seth.asc"

• Unsealing with GPG key → echo “wcBMA37…” | base64 — decode | gpg -dq

KES Server

• Ref: https://blog.min.io/introducing-kes/
• Responsible for cryptographic operations, thus only delegating master key storage to KMS
• Stateless and can scale very well
• Pros — Simple, secure(with TLS avoids username and password)

Pre-Requisites for Setup:

• Certificates for KMS and KES

2. Generate a Root CA certificate pair

1. Create a ca-config.json CA configuration file

{
  "CN": "MyCompany",
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "server": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "8760h"
      }
    }
  }
}

2. Create a ca-csr.json self signed certificate signing request file

## Content of ca-csr.json
{
	  "CN": "MyCompany",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names":[{
	    "C": "INDIA",
	    "ST": "MyState",
	    "L": "MyCity",
	    "O": "MyCompany",
	    "OU": "MyUnit"
	  }]
}

3. Generate certificates

$ ./cfssl gencert -initca ca-csr.json | ../cfssljson -bare ca

## Generates 3 files
- ca.pem # public certificate
- ca-key.pem # private key
- ca.csr # certificate signing request

4. You can also configure ca certificate at node level

• copy ca.pem to system trust store
• run update-ca-trust command

3. Hashicorp vault setup

Generate TLS certificates to access vault from KES: vault.sample-ns.svc.cluster.local

1. Create server.json file with supported domain names

{
    "CN": "MyCompany",
    "hosts": [
        "vault.sample-ns.svc.cluster.local",
        "vault.sample-ns.pod.cluster.local",
        "vault.sample-ns",
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names":[{
	    "C": "INDIA",
	    "ST": "MyState",
	    "L": "MyCity",
	    "O": "MyCompany",
	    "OU": "MyUnit"
	  }]
}

2. Run the command to generate certs

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

## Generates 3 files
- server.pem # public certificate
- server-key.pem # private certificate
- server.csr # certificate signing request

Create vault-config.json:

{
   "api_addr": "<https://0.0.0.0:8200>",
   "backend": {
     "file": {
       "path": "file"
     }
   },
  "default_lease_ttl": "168h",
  "max_lease_ttl": "720h",
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_cert_file": "server.pem",
      "tls_key_file": "server-key.pem",
      "tls_min_version": "tls12"
    }
  }
}

Start vault:

# Starts up vault server on https://0.0.0.0:8200
./vault server -config vault-config.json

Configure Vault for KES:

1. Create a kes-policy.hcl policy file for KES

path "kv/*" {
     capabilities = [ "create", "read", "delete" ]
}

2. Initialise Vault

$ export VAULT_SKIP_VERIFY=true

##########################

$ ./vault operator init

Unseal Key 1: JGDVZdF1f9UKYxYvIiW/tgfB24StDcTevJzzWX0K+TAj
Unseal Key 2: USjqCkON4/gn/oX2PzU1iz6STLuBohf14aP0GpzP+rge
Unseal Key 3: JEX0IJMOEMMwX2Fbld3IwkNLd1OqIU3OUQfSj7YQkfcL
Unseal Key 4: TzxJ5GJuHn4sA4NQDr/BWNKsXhBtO5FjqfJoV48cbYes
Unseal Key 5: bz+kUh5iw/Fakz9aUVP1UWJ+VvQDC3tLrjiSZ8LFkdgg

Initial Root Token: s.CG8JQskBbOVz43Vn9pvE7bgq

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

##########################

$ ./vault operator unseal JGDVZdF1f9UKYxYvIiW/tgfB24StDcTevJzzWX0K+TAj && ./vault operator unseal JEX0IJMOEMMwX2Fbld3IwkNLd1OqIU3OUQfSj7YQkfcL && ./vault operator unseal TzxJ5GJuHn4sA4NQDr/BWNKsXhBtO5FjqfJoV48cbYes

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       675f42c6-978d-85bb-10db-c6fb1f591743
Version            1.5.4
HA Enabled         false
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       675f42c6-978d-85bb-10db-c6fb1f591743
Version            1.5.4
HA Enabled         false
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.5.4
Cluster Name    vault-cluster-d954d2ef
Cluster ID      81d91296-59dd-4048-0969-309439d16b2e
HA Enabled      false

##########################

$ export VAULT_TOKEN=s.CG8JQskBbOVz43Vn9pvE7bgq
$ export VAULT_SKIP_VERIFY=true

$ ./vault secrets enable kv

# approle allows authentication and binds to a policy
$ ./vault auth enable approle

$ ./vault policy write kes-policy ./kes-policy.hcl

$ ./vault write auth/approle/role/kes-role token_num_uses=0  secret_id_num_uses=0  period=5m

$ ./vault write auth/approle/role/kes-role policies=kes-policy

$ ./vault read auth/approle/role/kes-role/role-id

$ ./vault write -f auth/approle/role/kes-role/secret-id

4. Minio KES setup

Generate TLS certificate to access KES from Minio:

1. Create server.json file with supported domain names

{
    "CN": "MyCompany",
    "hosts": [
        "minio-kes.sample-ns.svc.cluster.local",
        "minio-kes.sample-ns.pod.cluster.local",
        "minio-kes.sample-ns",
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names":[{
	    "C": "INDIA",
	    "ST": "MyState",
	    "L": "MyCity",
	    "O": "MyCompany",
	    "OU": "MyUnit"
	  }]
}

2. Run the command to generate certs:

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

## Generates 3 files
- server.pem # public certificate
- server-key.pem # private certificate
- server.csr # certificate signing request

Generate app identity for Minio using KES client:

$ ./kes tool identity new --key=app.key --cert=app.cert app

$ ./kes tool identity of app.cert

$ export APP_IDENTITY=$(./kes tool identity of app.cert)

Create server-config.yml:

address: 0.0.0.0:7373
root:    disabled  # We disable the root identity since we don't need it in this guide

tls:
  key:  server-key.pem
  cert: server.pem

policy:
  my-app:
    paths:
    - /v1/key/create/my-app*
    - /v1/key/generate/my-app*
    - /v1/key/decrypt/my-app*
    identities:
    - ${APP_IDENTITY}

keys:
  vault:
    endpoint: <https://vault.sample-ns:8200>
    approle:
      id:     "70bd11f5-2104-e4c0-8dbc-48f9066218ce" # Your AppRole ID
      secret: "3a06337c-0163-4c82-e81f-9ec9dce9cf5b" # Your AppRole Secret ID
      retry:  15s
    status:
      ping: 10s
    tls:
      ca: ca.pem # Since we use self-signed certificates

Start KES:

$ ./kes server --config=server-config.yml --auth=off

To access KES, If certificates are configured properly in minio. Hopefully “ — auth=off” option can be avoided

Verify KES to Vault interaction:

$ export KES_SERVER=https://minio-kes.sample-ns.svc.cluster.local:7373

$ export KES_CLIENT_CERT=app.cert

$ export KES_CLIENT_KEY=app.key

$ ./kes key create my-app-key -k

$ ./vault kv list kv

$ ./kes key derive my-app-key -k

$ ./vault kv get kv/my-app-key

5. Minio Standalone Setup

Minio Startup configuration:

$ export MINIO_KMS_KES_ENDPOINT=https://minio-kes.sample-ns
svc.cluster.local:7373
$ export MINIO_KMS_KES_KEY_FILE=app.key
$ export MINIO_KMS_KES_CERT_FILE=app.cert
$ export MINIO_KMS_KES_KEY_NAME=my-app-key
$ MINIO_ACCESS_KEY=minioadmin MINIO_SECRET_KEY=minioadmin ./minio server minio-data

# Place CA certs for minio communication to external services
$ cp ca.pem ~/.minio/certs/CAs/

Minio Client(mc):

$ ./mc alias set myminio [<http://127.0.0.1:9000>](<http://127.0.0.1:9000/>) minioadmin minioadmin

# Set policy to a bucket
$ ./mc policy set upload myminio/test

# Setup auto encryption on bucket
$ ./mc encrypt set sse-s3 myminio/test

# verify if encryption is enabled on a bucket
$ ./mc encrypt info myminio/bucket/

6. Minio Distributed Mode Setup

Generated template from https://min.io/download#/kubernetes

Below configuration is for 4 nodes with four drive paths(1 path on each no)

apiVersion: v1
kind: Service
metadata:
  name: minio
  labels:
    app: minio
spec:
  clusterIP: None
  ports:
    - port: 9000
      name: minio
  selector:
    app: minio
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
	namespace: sample-ns
  name: minio
spec:
  serviceName: minio
  replicas: 4
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        env:
        - name: MINIO_ACCESS_KEY
          value: "minioadmin"
        - name: MINIO_SECRET_KEY
          value: "minioadmin"
        image: minio/minio
        args:
        - server
        - http://minio-{0...3}.minio.sample-ns.svc.cluster.local/data
        ports:
        - containerPort: 9000
        # These volume mounts are persistent. Each pod in the PetSet
        # gets a volume mounted based on this field.
        volumeMounts:
        - name: data
          mountPath: /data
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      # Uncomment and add storageClass specific to your requirements below. Read more <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1>
      #storageClassName:
---
apiVersion: v1
kind: Service
metadata:
  name: minio-service
spec:
  type: LoadBalancer
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio

Observations of distributed erasure encoding tolerance to failures:

• If a file is deleted in more than N/2 nodes from a bucket, file is not recovered, otherwise tolerable until N/2 nodes.
• If a bucket is deleted in one node, request going to that replica node shows bucket is missing. Requests going through all other replica nodes, the bucket is visible.
• Looks like buckets are recoverable by directly copying missing buckets from other nodes
• Automatic healing happens once in 30 days. Manual healing is getting deprecated soon, hopefully we get knobs for automatic healing before that.
• If mount point is removed, Error: unformatted disk found is seen
• If mount point is restored, it is not recognised by the application, needed a restart of the pod. “Delete pod and wait for to scale back”

Storage class setup:

Storage classes “STANDARD” and “REDUCED_REDUNDANCY” can be set to minio or can also be sent from application using “x-amz-storage-class” or using “mc admin config”

Ref: https://github.com/minio/minio/tree/master/docs/erasure/storage-class

Issue with distributed mode: https://github.com/minio/minio/issues/9052

7. Minio Benchmarking using S3 benchmark

Setup

• 4 node cluster — one drive each(5GB PV)
• Node spec — 12 core 16GB RAM

Observations

Scenario 1

• Encryption enabled
• 100KB average object size

Run1:

$ ./s3-benchmark -a minioadmin -b new-test-bucket -s minioadmin -t 10 -u http://0.0.0.0:30100 -z 100K

Wasabi benchmark program v2.0

Parameters: url=http://0.0.0.0:30100, bucket=new-test-bucket, region=us-east-1, duration=60, threads=10, loops=1, size=100K

2020/10/29 11:45:42 WARNING: createBucket new-test-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.

status code: 409, request id: 164261F9CED6AF49, host id:

**Loop 1: PUT time 60.0 secs, objects = 17057, speed = 27.7MB/sec, 284.1 operations/sec. Slowdowns = 0**

**Loop 1: GET time 60.0 secs, objects = 43592, speed = 70.9MB/sec, 726.4 operations/sec. Slowdowns = 0**

**Loop 1: DELETE time 18.3 secs, 933.5 deletes/sec. Slowdowns = 0**

Run2:

$ ./s3-benchmark -a minioadmin -b new-test-bucket -s minioadmin -t 10 -u http://0.0.0.0:30100 -z 100K

Wasabi benchmark program v2.0

Parameters: url=http://0.0.0.0:30100, bucket=new-test-bucket, region=us-east-1, duration=60, threads=10, loops=1, size=100K

2020/10/29 11:54:41 WARNING: createBucket new-test-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.

status code: 409, request id: 16426277335B8DBB, host id:

**Loop 1: PUT time 60.0 secs, objects = 17507, speed = 28.5MB/sec, 291.7 operations/sec. Slowdowns = 0**

**Loop 1: GET time 60.0 secs, objects = 43073, speed = 70.1MB/sec, 717.8 operations/sec. Slowdowns = 0**

**Loop 1: DELETE time 18.3 secs, 958.3 deletes/sec. Slowdowns = 0**

Scenario 2:

• Encryption disabled
• 100KB average object size

Run1:

$ ./s3-benchmark -a minioadmin -b new-unencrypted-test-bucket -s minioadmin -t 10 -u http://0.0.0.0:30100 -z 100K

Wasabi benchmark program v2.0

Parameters: url=http://0.0.0.0:30100, bucket=new-unencrypted-test-bucket, region=us-east-1, duration=60, threads=10, loops=1, size=100K

2020/10/29 15:12:54 WARNING: createBucket new-unencrypted-test-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.

status code: 409, request id: 16426D483783E4E3, host id:

**Loop 1: PUT time 60.0 secs, objects = 18746, speed = 30.5MB/sec, 312.4 operations/sec. Slowdowns = 0**

**Loop 1: GET time 60.0 secs, objects = 59048, speed = 96.1MB/sec, 984.0 operations/sec. Slowdowns = 0**

**Loop 1: DELETE time 20.0 secs, 937.8 deletes/sec. Slowdowns = 0**

Run2:

$ ./s3-benchmark -a minioadmin -b new-unencrypted-test-bucket -s minioadmin -t 10 -u http://0.0.0.0:30100 -z 100K

Wasabi benchmark program v2.0

Parameters: url=http://0.0.0.0:30100, bucket=new-unencrypted-test-bucket, region=us-east-1, duration=60, threads=10, loops=1, size=100K

2020/10/29 15:16:02 WARNING: createBucket new-unencrypted-test-bucket error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.

status code: 409, request id: 16426D73F07F40A9, host id:

**Loop 1: PUT time 60.0 secs, objects = 19272, speed = 31.4MB/sec, 321.0 operations/sec. Slowdowns = 0**

**Loop 1: GET time 60.0 secs, objects = 52027, speed = 84.7MB/sec, 867.0 operations/sec. Slowdowns = 0**

**Loop 1: DELETE time 20.4 secs, 943.5 deletes/sec. Slowdowns = 0**

How do I read benchmarks?

Loop 1: PUT time 60.0 secs, objects = 17057, speed = 27.7MB/sec, 284.1 operations/sec. Slowdowns = 0

In 60 seconds, 17057 objects whose average size is 100KB is uploaded to minio. Speed of upload is coming around 27.7MB/sec which is average bandwidth utilised during this operation. Also an average of 284.1 files uploaded per second.

Loop 1: GET time 60.0 secs, objects = 43592, speed = 70.9MB/sec, 726.4 operations/sec. Slowdowns = 0

In 60 seconds, 43592 objects whose average size is 100KB is downloaded from minio. Speed of download is coming around 70.9MB/sec which is average bandwidth utilised during this operation. Also an average of 726.4 files downloaded per second.

Loop 1: DELETE time 18.3 secs, 933.5 deletes/sec. Slowdowns = 0

In 18.3 seconds, deletes were performed at a rate of 933.5 files per second.

8. Additional information:

1. For additional debugging, to log failed requests for minio

$ ./mc admin trace -v -e myminio

2. Root tokens generated as part of vault initialisation will not have expiry. It is always better to revoke the root token when the setup is finished. For creating a new root token when required, follow below steps

# vault operator generate-root -generate-otp
RgZjYVW2fIsl2avU5VNCpru6sZ

# vault operator generate-root -init -otp="RgZjYVW2fIsl2avU5VNCpru6sZ"
Nonce         c5cc8fb9-3226-71f2-865b-47d61bae4987
Started       true
Progress      0/3
Complete      false
OTP Length    26

# vault operator generate-root -otp="RgZjYVW2fIsl2avU5VNCpru6sZ"
Operation nonce: c5cc8fb9-3226-71f2-865b-47d61bae4987
Unseal Key (will be hidden):
Nonce       c5cc8fb9-3226-71f2-865b-47d61bae4987
Started     true
Progress    1/3
Complete    false

# vault operator generate-root -otp="RgZjYVW2fIsl2avU5VNCpru6sZ"
Operation nonce: c5cc8fb9-3226-71f2-865b-47d61bae4987
Unseal Key (will be hidden):
Nonce       c5cc8fb9-3226-71f2-865b-47d61bae4987
Started     true
Progress    2/3
Complete    false

# vault operator generate-root -otp="RgZjYVW2fIsl2avU5VNCpru6sZ"
Operation nonce: c5cc8fb9-3226-71f2-865b-47d61bae4987
Unseal Key (will be hidden):
Nonce            c5cc8fb9-3226-71f2-865b-47d61bae4987
Started          true
Progress         3/3
Complete         true
Encoded Token    IUk1JREjE1k8e0YLXDY0ZnljI3o+EDR5HGI

# vault operator generate-root -decode IUk1JREjE1k8e0YLXDY0ZnljI3o+EDR5HGI -otp RgZjYVW2fIsl2avU5VNCpru6sZ
s.oOHuDkZ25gnWB3L5m9NbAOo8

# export VAULT_TOKEN=s.oOHuDkZ25gnWB3L5m9NbAOo8

# vault token lookup
Key                 Value
---                 -----
accessor            laQoKv6pyhO3kpkA1ozmG8Ev
creation_time       1604397451
creation_ttl        0s
display_name        root
entity_id           n/a
expire_time         <nil>
explicit_max_ttl    0s
id                  s.oOHuDkZ25gnWB3L5m9NbAOo8
meta                <nil>
num_uses            0
orphan              true
path                auth/token/root
policies            [root]
ttl                 0s
type                service

3. When you need to do vault administration. Create an additional administrator app certificate as opposed to using the app certificate created for minio itself for better auditability.

# Create admin cert pair
kes tool identity new --key=minio-kes-admin.key --cert=minio-kes-admin.cert minio-kes-admin

# Generate app identity from cert
kes tool identity of minio-kes-admin.cert

# Configure required administration policies in KES server-config.yml file and reload KES service

4. Once a bucket is created with some content, if we enable encryption later on the bucket, existing files are not re-encrypted automatically.

5. mc admin heal is about to be deprecated and no sufficient alternative is in place when I checked last time.

A note on Automatic healing:

Healing is automatic on server side which runs on a continuous basis on a low priority thread, mc admin heal is deprecated and will be removed in future.

- Minio healing moved from once every day to once every month

- Option to disable self healing is under works

- https://github.com/minio/minio/issues/7892

- https://github.com/minio/minio/pull/7934

- https://github.com/minio/minio/pull/7868

6. While upgrading minio in distributed mode. Minio should be brought down one after another for maintenance and the instance should be immediately healed after being up before going for other instances. This is necessary, otherwise affects the redundancy factor

• Bring down minio
• Upgrade
• Bring up minio
• Run mc admin heal to recover the missing data
• Go for next node and upgrade only if the current one’s recovery is complete

8. Minio will not tell you the health status of disks. You should have separate alerting and monitoring in place to keep track of hardware health.

9. Note that there are other reasons but servers restarts that can cause nodes to become out of sync. For example, if there is a network failure that causes some of the nodes to not be reachable, writes will be less durable too. It is thus important to have good monitoring in place and respond accordingly. Minio itself will auto-heal the cluster every month if the administrator doesn’t trigger a heal themselves.

10. MINIO_KMS_AUTO_ENCRYPTION=on environment variable is about to get deprecated in favour of mc encrypt command which provides encryption at rest on specific buckets. However this deprecation doesn't sound reasonable to me as we don't see any suitable alternative that provides auto encryption for all buckets on creation.

11. Unsealing vault for every restart is a pain. Fortunately, Vault exposes different integrations to make this automatic — https://learn.hashicorp.com/collections/vault/auto-unseal

References:

Administration

• https://docs.wire.com/how-to/administrate/minio.html

Different certificate generation approaches

• https://kubernetes.io/docs/concepts/cluster-administration/certificates/
• https://gardener.cloud/documentation/guides/applications/https/
• https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/

Encrypted Minio Storage with KMS Setup was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

• clone the repository you wanted to use with google drive into your local work directory.

cd ~/work

git clone https://example.com/project-group/sample-repo.git

• Download google file stream app from google(Streams files on demand instead of making entire drive offline)
• Post installation, login with google email.
• Let’s say Google Drive is mounted on /Volumes/GoogleDrive folder on your machine.
• Create repo under /Volumes/GoogleDrive/Shared\ drives/Project\ Drive/work/ using the below command

git init — bare sample-repo

• In your local work directory, set git remote to google drive’s sample-repo folder and push all the branches, tags etc.

cd ~/work/sample-repo

git remote add origin file:///Volumes/GoogleDrive/Shared\ drives/Project\ Drive/work/sample-repo

git push origin — all

git push origin — tags

Sharing initialised repo with everyone else:

• Download google file stream app from google(Streams files on demand instead of making entire drive offline)
• Post installation, login with google email.
• Let’s say Google Drive is mounted on /Volumes/GoogleDrive folder on your machine.
• If you have sample-repo repo under ~/work already

cd ~/work/sample-repo

git remote add origin file:///Volumes/GoogleDrive/Shared\ drives/Project\ Drive/work/sample-repo

git pull -r origin master

Some common problems using git with google drive:

• Coordination is required when multiple people would like to push to master at the same time. Sometimes when one person pushes to master and due to some network problem, if drive is not able to replicate your local commit to everyone else. There would problems as google drive has its own way of resolving conflicts when it comes online.

Eg: If A is pushing to master. Publish the last known commit id to everyone in the group. If B wants to push another commit to master, B can check if the last known commit id mentioned by A is available locally before pushing.

Same rules apply to any branch where multiple people are working at the same time.

On a winter morning before the first light

I was going on my bike on a silent road in the early hours of the day. The road looks entirely abandoned as it was one of the most chilling days of winter and I was not expecting anyone coming on to the road. But I was very much ambivalent and kept an eye on both sides of the road as far I can see through the fog. I was also ready to apply the break and stop if at all someone or some animal shows up suddenly. All those thoughts were running in my subconscious mind.

As per human nature, you get to adapt to things based on your past experiences. Your mental alertness is a precautious state expecting something suddenly might happen such as a human standing in the middle of the road or a stray dog trying to run across or a vehicle coming in the opposite direction etc. Because everyone would have had these experiences one time or the other.

I wanted to add, that I was also observing lot of vehicles parked around and the sight resembles a lone ranger passing through a graveyard of vehicles. As I don’t see any people around, I am not expecting any of those vehicles to move at all.

Imagine a situation where one of the vehicle started moving onto the road to reach the owners front courtyard as it was given a command to do so. The car tries to drive by itself as it was an autonomous car. All of a sudden, that action of car might create a sense of panic in your mind. What the heck? Did I see someone coming around it? How is it moving? Oh..it could be an autonomous car.

You don’t feel this at all?

Let’s not take this very hypothetically instead, let’s try and understand the facts. It’s a simple theory. While we are going on road, our brain does lot of calculations about myriad of dynamic personas that can appear up on the road. Having the sense to control the speed, your readiness on the brakes etc are based on that. Most of those reactions are involuntary and are instinctive.

Along with the dynamic personas, there are some constants too like trees, street lights, road dividers etc. Since you never expect them to walk down the road like a zombie, your brain would always assign them with lesser weights(lesser probability of obstruction) in the equation. Similarly, if you are seeing vehicles parked on a road, you involuntarily treat them as constants unless you see someone coming out of the door to take a ride. But you may and can simply say, why not consider everything as a dynamic persona which can move by itself without the need for humans. But it simply increases the number of variables in your equation and the human brain can only do so much after all.

Didn’t made sense? Let’s take a simple example of you riding a bike inside a crowded market. The chances of you hitting someone is very likely unless you are damn slow and careful. Just because you have so many dynamic factors in the equation your chances of hitting are high and sometimes imminent.

Now don’t tell me that you add more computational power to yourself, it is simply the physical limitation of humans as of today.(Augmented vision/AI assisted driving might help though)

Adding on, incase of autonomous vehicles, the complexity of this equation further beefs up if you have to consider the type of the car and it’s manufacturer and the sensors they might have used and intelligence etc.

The scariest thing about these vendors are, they are all developing these self maneuvering technologies in silos without of much coordination between with each other fearing of revealing their so called patentable technology to competitors and may loose their position in the market. We wouldn’t be surprised if they come up with some crazy ideas to beat the competition. This further increases the unknowns in our equation.

Well, if you say, I am opposed to technological advancement, no I am not. All I am trying to do is to highlight the nuances and spin off a thought process, so that we can start fixing it collectively.

More situations!!

You could ask me, are you saying all of this just because you had a random thought that slipped into your mind on a bad winter morning?

No, that’s just the beginning. I have started thinking about it more aggressively because it is something which is evident to many people but non trivial to put it in words or express it. In fact I have put my best effort to explain them in a pictorial way.

The absence of human factor:

Below described situation in the picture might completely seem normal to anyone. A motorist trying to move on a road on which lot of vehicles clogged on both sides of it.

Had there been no concept of autonomous vehicles, person rides freely as long as he/she sees no person in and around these vehicles. Glad, this is still going to be the situation at least few years down the line. But this landscape is quickly changing.

Significance of human factor:

Putting myself in bikers shoes. As soon as I see someone in one of the car I would carefully adjust my navigation to circumvent the path of the car by slightly slowing down.

Here the car has become from being a static factor to dynamic factor and the presence of human was very evident from a distance and I would have enough time to think and plan the maneuver.

One more interesting fact here is, I am also dealing with an actual human being in the car. So it gives me the freedom to think that any human would take time to lay up the gear, turn the steering wheel, look at the road in the rear view mirror and move on to the road. This is a generic human behaviour anyone can anticipate and plan accordingly.

Autonomous driving vehicles:

In case of autonomous vehicles, this is all invisible, unknown and differs from car to car and vendor to vendor. Is this any better than the previous case involving human beings?

Above portrayed, is a much puzzled situation for anyone taking this road. 6 to 7 years down the line, you would most probably end up being in similar situation on our roads. Remember the bad winter morning I have started with and the corresponding prognosis will seem very much real and will come to life then.

I still don’t agree with you

Let’s say you don’t agree with all the points above. I will now talk about much real situation I had been in.

On my usual office commute way. Some part of the road is really wider. It is so wider that the width of road on the side I am passing by, is much wider than length of a bus. There are lot of new buildings being constructed and new paths that are getting linked to this road being instituted everyday. One fine day, all of a sudden I see a bus coming out of a gate exactly cutting my path.

Initially, I was very puzzled on seeing a bus coming out of the building. As the building was unoccupied until the previous day. Immediately after that, the next puzzle is, which direction would the bus go? Would it take a left turn and join my path or would cut across to take a U turn?

I have no clue about it and as I was approaching the bus more closer and closer, I have shifted my observation from the entire bus to the driver, which immediately helped. My speculations of how the bus would move are solved on observing the driver’s gestures and I have corrected my course accordingly.

The existence of an active human being driving the bus helped everyone heading that way to correct their course immeditely with less disruption. The nuance here is, the existence of an active human being controlling the bus and the person’s gesture helped me and everyone to take decision quickly.

The situation drawn in the picture looks pretty tidy with only two dynamic factors depicting just me and the bus. But the situation I had then was not so simple as this, the road is muddy and slippery because of constructions happening around with lot of traffic.

Can autonomous vehicles understand typical human gestures like a human would do?

If you look at the biker very closely. You can see the biker gave a gesture by looking right, which means person wants to go slight right and overtake the car before him/her. And you could have burst to laugh on imagining this, this is not a valid traffic rule/sign and how can we expect autonomous vehicles supposed to understand this?

Well if I am in the car following the biker, I would have definitely slowed down on biker’s gesture. The same would apply to more explicit gesture depicted below too.

This is very trivial human gesture that we do and observe everyday.

I wanted to ask, are the autonomous vehicles equipped to address this? Before I say I don’t know, I am really skeptical if manufacturers are really concerned about these subtleties!!

Conclusion

Technology is as disastrous as it could help human lives. If you follow autonomous vehicles developments closely, they actually makes the life of the people using the autonomous vehicles more safer and easier, simultaneously can push other people using non autonomous vehicles into various stages of dilemmas all along their daily commute.

I have always heard manufacturers and technologists talking about how safe is autonomous technology to people who use it, but very little of how accommodative it is for people who wish to still live with non autonomous choices. It is unfortunate that no one talks about it.

Why should we accept such a change if there are not going to cope up with our non autonomous world which we have been living.

Though we are living in a world racing towards autonomy, if someone chooses to not to use it, it should still be an easy choice to do so and this happens if this technology is ensured to be more humane for the non autonomous world.

When we all set to invite a change in the society. I think it should play well with existing rules.

One of the talks by Niek Timmers and Albert on compromising secure boot with various fault injection techniques was quite interesting to me. I wanna brief that here.

A regular boot sequence includes running a program from ROM which loads the first stage bootloader into SRAM from a flash storage. This code initializes DDR and loads second stage bootloader into DDR followed by kernel initialization etc.

External Flash storage is vulnerable to be tampered either by physical access or a software access. Hence the entire system control can be compromised if not taken proper care at the hardware storage layer.

A secure boot sequence ensures that an authentication mechanism in place by having a signature of the code that is about to be loaded into RAM and will be verified against at each and every stage of the boot process.

Secure boot flow:

Hardware → ROM → Bootloader → TEE bootloader → TEE OS → REE bootloader → REE OS → Apps.

Along with authentication, hardware root of trust, privilege escalation control, encryption of all code and proper hardware and software exploitation defenses ensures a better secure boot mechanism.

How does the fault injection actually works?

CPU always expects its supply voltage to be stable, if not, glitches can occur. This is exactly what can lead to a fault injection attack. Let’s say we have got a hardware that has capability to control supply voltage to the target processor and inject glitches at a high degree of precision. (Such hardwares do exist, I have seen them live). We can actually modify instructions that are currently getting executed. Isn’t cool??

Instruction skipping:

As we know external flash storage is vulnerable to attacks if the attacker gets physical access to it. In an encrypted secure boot design. Assume that the first stage bootloader is residing on the internal flash memory of the controller and whereas BL2 can still be on external flash. Now, it would be extremely hard for an attacker to modify BL1 without tampering the controller. BL1 has code to decrypt and verify the BL2 on SRAM before actually loading to DDR. Hence ensures BL2 to be valid.

How does the above setup be attacked? Can BL2 still be compromised? Yes.

If a glitch is introduced after BL2 is copied to SRAM before validation. It can potentially bypass the entire control flow until the verification step and directly jump to next set of instructions. Provided, the glitch is carefully timed and its duration is carefully chosen. This is called instruction skipping, where all the instructions in the control flow between load and verification will be corrupted and will have no impact on the state of execution.

How does an attacker determine the glitch timing?

Power analysis. True. An attacker can record the power consumption of the CPU and can easily determine various points of interest from the surges. Cryptographic operations typically consume more power while the execution is happening which will be very evident given the state of attacker’s familiarity on similar boards.

Instruction corruption:

Attack on ARM load instructions LDR and LDMIA

LDR — To copy a word from one memory address to a register.

LDMIA — To copy multiple words from a memory address to registers.

Load instructions are particularly an interesting target for an attacker primarily because of the instruction encoding scheme.

Controlling PC during the secure boot:

A PC is special register which holds the memory address of the next instruction to be executed. ARM instructions which can accept PC as an argument are prone to fault injection attack. Assume the attacker has physical access to the setup and was able to load malicious code and pointers into an external flash memory. Whenever a load operation is about to happen for a copy operation from a memory location to register, the instruction can be corrupted using a glitch tricking the processor to load the value into a program counter(PC) instead of register.

How is this possible?

Attacked instruction: LDR R3, [R0] → HEX equivalent of the given instruction is E5903000.

Instruction to be: LDR PC, [R0] → HEX equivalent of the given instruction is E590F000.

If we closely observe the two HEX values are differ only at the 3rd byte which is “30” and “F0” values through which R3 and PC are represented in the instruction. The corresponding binary representations for “30” and “F0” are “00110000” and “11110000” respectively, and a glitch introduced with precision can actually flip the first two bits of R3 to become PC. By this glitch, an attacker would be able make the CPU load the content of R0 into a Program counter(PC) instead of an intermediate register R3. This attack has almost no impact, unless the attacker has some kind of access to the storage and was able to inject some malicious code in it from which R0 is read.

The attack surface of LDMIA instruction seems to be higher than LDR based on experimental results, primarily because of the instruction encoding scheme

Attack on TEE(Trusted Execution Environment) after the boot:

More secure sensitive embedded devices have something called TEEs which separates themselves from Rich Execution Environment(REE). REE will have access to dedicated APIs to make calls to TEE for running some operations in a secure environment. TEE APIs usually copies the payload from REE and executes them internally. Assuming the attacker has full control over REE and was able to make calls to TEE with malicious payload. An attacker can again introduce a fault while the malicious code pointers are being copied into the internal registers. Through a successful instruction corruption can make them copy to program counter(PC) and gain access to the TEE.

How exactly a randomly corrupted instruction being helpful for attacker?

This answer is quite interesting and is more about learning the behavior/patterns of corruption for a given glitch amplitude, duration and timing.

How does an attacker determine the exact time, duration and amplitude of the glitch that can corrupt the instruction?

Without doing a practical experiment it is impossible to determine these values. A sample setup for the experiment would look like below

Along with the the above setup, a sample program is used which runs in a loop continuously executing the load instructions.

And with glitch VCC ranging between -1.4V to -1.0V, glitch length between 700ns to 1000ns and glitch delay between 30us to 35us and for a total of 10,000 iterations. Results of the experiments are as follows.

In the graphs, green dots(Expected) represents no affect on instructions. Yellow dots(Mute) represents the resets and red dots(Success) represents a successful glitch.

It is quite evident from the graphs, that the LDR instruction is quite less affected by the glitch compared to LDMIA instruction and hence the attack surface is larger for LDMIA instructions.

It is observed that power glitching is more difficult to be successful if it requires to have complex bit flips.

Countermeasures:

At hardware level:

• Having a variable clock speeds for the CPU would make the code execution flow less deterministic to the attacker.
• Sensors monitoring the environmental conditions and alert on unusual activity.

At software level:

• Introducing random delays inside code can also increase complexity of attack.
• Double checking conditions before branching and verify at regular intervals whether expected program executed.
• Redesign of instruction encoding that would maximize the hamming distance between valid to harmful instructions.

Conclusion:

Having all of the above countermeasures wouldn’t really make a system completely secure but reasonably harder enough for an attacker to counter them. All the attacks described are observed on ARM 32 bit architecture, but the principles described can be applied to many other architectures as well.

I would like to thank Niek Timmers @ riscure for reviewing this.

References:

• https://www.slideshare.net/tieknimmers/pew-pew-pew-designing-secure-boot-securely-134091830
• https://www.riscure.com/uploads/2017/09/Controlling-PC-on-ARM-using-Fault-Injection.pdf

Note: Some terms and definitions may have changed. EKS may have superseded ECS.

You may be a growing business and have to spin and scale your docker containers/services dynamically as your demand grows. All those containers that are created dynamically have to be load balanced, monitored for failures or restarts, autoscaled based on network traffic or based on cluster metrics like CPU and RAM utilization etc.

A container cluster is essentially a collection of docker containers running on multiple machines/servers. In our context of ECS, each and every cluster is specific to a particular docker image that is being run as multiple replicas for high availability. In other words, it is not a composition of different docker containers/services created from different docker images.

At first, we need to understand some AWS jargon before we jump start creating a container cluster.

Load balancers: AWS provides multiple load balancers that can be used based on the requirement.

Elastic load balancer(ELB) is one of the classic load balancer that balances load between EC2 instances/machines with services running on one single port (Eg: port 80 for web services). This is mostly used for non dockerized applications.

Application load balancer(ALB) is most recent edition to load balancer family that has more features like dynamic port mapping(multiple host ports can be dynamically mapped load balancer), path based routing etc. We have used ALB as our load balancer.

VPC(Virtual Private Cloud): If your cluster has to interact with multiple other services in the cloud, they all should share the same VPC. VPCs are spread across availability zones within a region. More info

Subnet: A subnet is again an isolation of resources by ip address range with in an availability zone and it cannot span across availability zones.

IAM(Identity and Access management): Your cluster should also share the same IAM roles as other services if you have any IAM roles defined for either of the services.

Security group: They are basically network ACL rules that are configured for EC2 instances that regulates the port communications in and out.

availability zones: AWS spread across the world in multiple regions and each region is again sub divided into multiple availability zones. Your instances can exist in multiple availability zones for high availability. More info

Cluster: A cluster is a logical grouping of EC2 instances that ECS uses to start containers. Each cluster resource(EC2 instance) will have a container agent that facilitates spinning of docker containers within it.

Task definition: A task definition is one important chunk of configuration for ECS cluster. It basically is a composition of your Dockerfile and docker-compose.yml. Your docker image, hard and soft limits for RAM and CPU, type of EC2 instance to use/cluster to use, Environment variables and host port mapping are important things that you can configure. In case of ALB, you should set host port to 0 to make ECS figure out the port mapping of your instances with ALB. More info

Service: A service allows you to run specified number of tasks of particular task definition on the cluster and all these tasks/containers are monitored for failures and restarted automatically by service scheduler. More info

— Prerequisites for creating an auto scaling group: Launch configuration, Target groups, ALB

Auto scaling groups: As ECS takes care of scaling the docker container instances on the available cluster resources. Auto scaling group can take care of scaling the cluster resources like the count of EC2 instances available in the cluster. You can attach load balancer to autoscaling group and define policies to auto scale based on the load balancer health checks. There are few policies available for scaling like policies listed here.

— You can either attach a load balancer or target groups to auto scaling group. A load balancer in turn can be attached a target group. More info

Launch configuration: An auto scaling group can be created from a launch configuration. It defines the type of EC2 instance, image on those instances, detailed cloudwatch monitoring, ip addresses, security groups, region, IAM roles and key pairs to login to those EC2 instances. More info.

— Once you have launch configuration, you can start creating a auto scaling group with it. More info

Target groups: Target groups are used by the load balancer to perform health checks and route traffic to them. It takes port, protocol, VPC and health check configuration.After a target group is created, you need to register some targets with load balancer to route traffic to specific targets.

ECS(Elastic Container Service): It is a service provided by AWS for dynamically scaling a docker container cluster across instances/regions/availability zones based on metrics from cloudwatch, Load balancers etc. More info

ECR(Elastic Container Registry): It is similar to docker dub where docker images can be stored. You can have multiple repositories and each can hold multiple images. All the repositories under one AWS account logically grouped as a registry. More info

CloudWatch: Each and every EC2 instance created from AWS has a daemon that constantly runs and collects statistics of the instance to estimate its health and also provide dashboard view of the various metrics like CPU, RAM, network in/out etc.

There are two ways to create a container cluster using ECS.

1. Using ECS first run wizard that walks through all the steps required to create a cluster like load balancer, task definition, auto scaling groups, target groups etc.
2. Manual way, where you create a target group, load balancer, launch configuration, auto scaling group, task definition, container service, container cluster, IAM roles(if needed), security groups, VPC, subnet, availability zones etc.

Both the approaches have advantages and disadvantages:

Option 1 advantages: Instantly spin up container cluster by just following the on screen steps without knowing much about security policies, auto scaling groups, load balancers etc. Also, each and every cluster created by this approach creates a cloud formation stack template. Having a template is easy to cleanup resources/left overs by just removing the template from cloud formation stack

Option 1 disadvantages: It is not for production clusters. Every time you create a task definition, service definition, cluster definition, load balancer with default names and default IAM roles and VPCs. You don’t have much control on the naming conventions.

Option 2 advantages: Much control on each every service. All the service definitions can be altered separately. Create services only once and reuse them.

Option 2 disadvantages: Need to have good understanding on all of the above mentioned jargon to create a running cluster.

Join our community Slack and read our weekly Faun topics ⬇

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

Understanding jargon of AWS ECS — Elastic Container Service was originally published in FAUN.dev() 🐾 on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Autosave notes”, it is a pretty important feature for any note taking app. Having autosave enabled and running in the background saves you from loosing any important notes you have taken. We would have felt autosave itself is a kind of backup feature for your notes.

Recently, I had been in a situation where the autosave feature of boostnote misbehaved and lost all my notes till date. It’s a big disappointment.

I have finished my work, wrote a beautiful document on the work I have done on boostnote and left for the day. Next morning, when I logged back in, I noticed I am seeing an older version of my notes. After a while, some more content missing from the same notes. I restarted the boostnote app with a hope to restore everything back, but unfortunately ended up being a clean slate. I have literally gone back in time and all my notes are actually gone.

I never thought that I would loose notes just like that. I did some googling and realized that boostnote stores all the notes in simple text files. The moment I saw that it is not backed by any sort of version control. I lost hope that my notes can be recovered at all by any means.

I should honestly agree, the user interface and markdown editor of boostnote is simply wonderful though. So, I could not give up on boostnote.

In spite of repenting(for not taking care of this before), I started writing a tool with the features that I felt would have been there in the first place if at all a tool exist.

It is a simple, auto commit application backed by git version control. It monitors for changes on a folder and makes a quick commit for you. It doesn’t commit, if there are no changes at all, thus preventing unnecessary commits in the history.

Initially I called it “boostnote-backup” and later felt that it can be a generic backup tool for any other note taking apps like orgzly etc.. and hence named it “backer-rs”.

It is actually opensourced out here. I would definitely love some pull requests, suggestions and feedback.

Currently, it is very basic version out there. We can add features like

• Making it run as a system service.
• Pushing to a remote repository after certain preconfigured time.
• Reading configuration from a file instead of just command line args.
• CI pipeline to setup Cross compiled binaries for windows, Mac, Linux and Android.
• Making it an installable package for the above platforms.
• Etc..

Long live note taker’s!!

My notes are my prized possession. It has my wonderful thoughts for the future and hard to remember facts of the past. — I made this up.

What is the most common protocol of internet. It is http. You open a browser enter a url and then you see content on your browser. How is that working? When you enter a url, your browser makes http get request to the site and fetches the content.

Likewise multiple methods are supported for http like post, put, delete etc.

How does http work internally?
Http is equipped with lot of headers to actually reach it's destination over the internet. Http runs on tcp protocol and every request makes a connection and closes it after the response.

How does WebSocket works?
WebSocket is a simple tcp based protocol which does the initial handshake by http and keeps a persistent tcp connection to the server.
Unlike http, it is a duplex connection that allows client and server push frames from either side. Hence allowing the client and server to communicate in more real-time.
Except the typical initial handhsake which will be in http, rest of the communication is headerless and lightweight.
WebSockets are supported on every browser thus allowing you to fetch data in real time without using traditional http polling mechanisms that works by pull approach where the client asks for any updates server has and makes and breaks connection for every single request, which is overhead for both client and servers.

Tcp connection is not a standard for browsers and hence having a protocol as light as tcp other than the traditional http is very useful.

Http has certain advantages like requests can be cached by intermediate proxy servers there by reducing the load on the actual server when serving the same response to multiple clients. However this kind of caching is not supported by websockets, but it is highly unusable for a scenario where websockets has to be used. When things change in real time, caching is of least importance.

WebSockets for IOT:
Given the need for visualizing lot of details on web dashboards in real time. Advent of cloud platforms for IOT etc. WebSockets would be a ideal choice for having lightweight communication between server and client.

A typical home IOT environment:
You might be having lot of devices at your home which needed to be visualized and controlled when you are away from your home or you might be lazy getting up from your couch to turn off something or you want your grandparents to have complete control of your home at their fingers and so on.
In a typical gateway based IOT architecture, your gateway is always connected to your server to see the current status of devices over the internet. Your server can send commands back to the gateway only if there is a two way communication. Doing the same thing over http will involve your gateway constantly polling for commands from server. WebSockets makes your life easier.
Let's say you wanted to offer your IOT system for all the residents of your apartment as a service. WebSocket would be a right choice in maintaining just one connection per gateway per flat. Otherwise, using http, your server will be flooded with lot of connections and disconnections for every poll per gateway per flat more than the data/commands to be sent across.

Real-time audio streaming environment:
I have built an application that allows people within a wifi network to talk from their mobile web browser. This requires a real low latency audio streaming from the browser itself. WebSockets are useful here as it needed binary data streaming in real time. Using a traditional http for each and every frame would impact latency as it makes and breaks connection for each and every frame, WebSockets on the other hand makes connection once and streams data on that connection continuously. Also, the same WebSocket can be used to push messages,notifications and even transpiled string asynchronously to all the clients from the server.

A self driving car, pizza delivery service, realtime ambulance guiding system to accident spot etc. There are many usecase where websockets can be or will be or being used. See if your next usecase needs a WebSocket.