If you’ve done one thing for 5 years, you will have things that you take for granted as a matter of habit and become the basis of daily decision making, which is neutral from specific technologies. In this post, I wrote down these things as practices of platform engineers who build platform for developers. The things I listed below are not special. I think you’ve heard or seen these things somewhere. These are just results from daily inputs or experiences with the team I’ve been working.

Since they are just practices, you won’t be able to use them for 5 or 10 years and they should be updated when the trend or the world changes. This is just a record of 2020.

• Immutable infrastructure: To provide a better product to customers faster, Continuous Delivery is one of the most important factors. A big part of making Continuous Delivery practice successful is to be able to push new changes without fear. The immutable infrastructure allows us to do deterministic deployment (safely rollouts and rollbacks). For successful continuous delivery and reliable delivery, all our components must be immutable. We don’t SSH to modify things in production. This practice must be also applied to configuration management or application source code as well.
• Environmental parity: We must keep the development and production of environmental parity as much as possible. We must enable developers (even us) to reproduce production issues in the development environment. Always test on development before production and grow confidence in production. And we must treat the development environment as production. If we break the development environment, it blocks the developer in testing and stalls agility.
• Declarative configuration: All infrastructure state or application state must be defined in “infrastructure as code” way. We should not do or depend on manual operations in production (and even in development). It leads to implicit knowledge and human errors. Infrastructure as code allows us to review the operation and automate it in CI. And the code must be declarative, not imperative. The declarative way enables us to build self-healing or autonomous components. It reduces our operation cost a lot. We should build tools based on “Infrastructure as data” way. We should not mix programmatic logic and data but clearly separate. The computation must be done by real programing language.
• Documentation as a product: Documentation is the minimum variable product of the platform. We must handle documentation as one of our products. This means the documentation must be always reviewed and maintained when something changes. When we release something new, we must provide the documentation together.
• Simple, consistent, and predictable: Developer experience (DX) we provide and operation workflows we prepare for our component should be simple, consistent, and predictable. How to work with component should be as simple to learn as possible and consistent across the multiple components (action to component A should be the same as component B). And action to the components should be always predictable. This means changes are always expected and visible before execution. The catastrophic changes should be always blocked or notified.
• Smart default but configurable: The tool we provide should be well abstracted and hide complex configuration (and underline complex infrastructure) as possible. Instead of asking developers to configure all but provide smart default which covers 80% of usage. To cover the rest of 20% of power users, open a well-designed control knob as well. We should consider both productivity and flexibility but the former should come first.
• Embrace failure: No software or infrastructure is perfect. Failures are always there. We should be embracing failures instead of focusing on preventing them. We must expect potential failures and actively be testing availability and reliability before going to production. Once we faced an incident, we must learn a lot from it and avoid the same failure again.
• Having a migration strategy, always: We don’t introduce new features without thinking about migration strategy from the existing ecosystem. Keeping multiple ways of doing the same things leads bad DX and becomes a burden for the operation (and even more we need to have a workaround for a legacy way to make it work with the new ecosystem). We don’t evaluate just introducing new features. Instead, we evaluate how much is used by developers (adaption rate). We don’t create a separate new platform for doing a new thing. There should be only one base platform.
• A platform for platforms: We can not support all workloads or requirements by a single platform. We must enable and help other teams to build their own specialized platform to support their own workflow top on our platform without involving the platform team.
• Operation by science, not by art: As a platform, we need to operate and maintain our components. The operation workflow or tool we design should be strongly backed by science and engineering, not art or individual knowledge or experience. We must handle it as a science. And we should avoid using all the time to react to the incident or manual operations. We must proactively be fixing the issue and the workflow and building tools. The time for proactive and reactive should be balanced well.
• Avoiding reinventing the wheel: Cloud providers provide great services (by using lots of costs and resources). The OSS ecosystem (especially cloud-native ecosystem) provides great software. We must utilize them first. We must be a good curator of them and utilize it for our platform. We must avoid reinventing someone else work (when reinventing we must balance the cost and benefit and resources we have). We must build something new, something special for our use cases, and something no one has worked before. And we must open to our jobs to the ecosystem as well.
• Standing on the shoulders of giants: As a microservices platform or microservices world, we are far behind from the giants. We must learn aggressively learn from what they succeeded or failed and the latest trends or research results (from conferences, papers, or meeting them). Our proposal and design must be based on these learnings. The originality must be on top of the existing knowledge. Then becoming the next giant and provide our knowledge to the world to support them.

At Mercari, we use Slack Bots to automate a lot of our tasks. For example, releases of our main API are automated with a bot, and over 10 releases per day occur on Slack across Mercari JP, US and UK.

At first, Slack Bots were mainly text-based, with the exception of being able to return an image, e.g., graph, as a response. However, bots have gotten much more useful since Slack started offering interactive messages, allowing us to add dropdown menus, buttons, and more!

Buttons have actually been available for a while, but using them meant you had to open up your bot to other teams, and also support OAuth–basically, there were quite a few hurdles to overcome. But once Slack started offering internal integrations, it became super simple to develop bots using interactive messages for use within your own team.

Mercari already uses multiple interactive Slack Bots, and we continue to develop more as new needs arise. In this article, I will explain how to write a bot with interactive messages. The sample code is all public (tcnksm/go-slack-interactive), so feel free to fork it and create a unique bot for yourself! (If you want to use Node, you might want to take a look at Slack’s sample Node app.)

Benefits of Interactive Messages

What’s so good about making a bot with interactive messages, you ask?? Because who doesn’t like clicking buttons!buttons are awesome!

With a single button, you can start a process and confirm that its requirements have been met. For really important processes, you can limit the number of people that can press the button and create an easy approval flow. If you use a menu, you can show the user a list of selections from the bot and get a preset response, which makes data validation easier compared to free-text input. Additionally, buttons are easier to use in the Slack mobile app.

It’s probably already obvious, but the main motivation behind making a bot is to collaborate with other team members in your channel. “If we’re all engineers, why not just use a command line tool?” you may ask, but our teammates are not all engineers. That’s what makes Slack bots great–they are accessible to everyone.

An Example at Mercari

At Mercari, we use interactive messages with a bot like the one below.

Kubernetes Deploy Bot

For Mercari US, we run some services on Kubernetes (GKE). (We generally use Kubernetes when making new microservices, too.) The Kubernetes Deploy Bot deploys a newly made docker image to our production cluster. When the user asks the bot to deploy, the bot shows the user a list of deployable images (tags) in a menu, and the selected one is deployed to the cluster.

In addition to this, this bot also makes a pull request by rewriting the Docker image version in Kubernetes manifest file, so we have ensured that the repository setting file and the actual image version on cluster is same.

Account Bot

SRE are in charge of creating creating VPNs and accounts to access designated servers. In the past, the SRE were manually logging into the server, but this is now automated with a bot. When a user makes an account creation request, the bot requests approval from an SRE, who clicks the “accept” button. Thus, while it is automated after the approval, we ensure that the necessary approvals are in place before anyone creates an account.

By the way, this bot works on Google App Engine, and when it receives an account creation request, it submits a job to Google Cloud Pub Sub. Workers that actually create accounts are running in each of our three regions (JP, US and UK), and each subscribes to the appropriate topics for each region.

Bot Example

Below are instructions on how to make a bot with interactive messages, including some sample code.

As an example, we’ll create @beerbot in order to order beer. (Unfortunately, it doesn’t really order beer. :( ) When you chat with the bot, it shows a menu with a list of the beers that you can order. You can then use a button to confirm your order, or cancel. See how it works here:

Slack App Preparation

Now we’ll get into actually making the bot. Interactive Message Bots need to be created as a Slack App. Firstly, we’ll make a new app.

Next, add a Bot user from the features panel and make Interactive Message active. At this point, it is necessary to set a specialized request URL. The results of a user’s action to an interactive message will be posted to this URL.

Lastly, we install the Slack app we made into the team Slack. From doing this, we can get a verification token to verify the bot’s User OAuth Access Token and Interactive Message request on the server side required for the bot to access the Slack API. This value is required to operate the Bot. With this, the preparation is finished.

Bot Development with Golang

Now, we will actually start writing the bot. It will need to

• Watch for Slack events and respond appropriately (Slack Event Listener)
• Receive the results of a user’s interaction with our menu and buttons (Interactive Handler)

Slack Event Listener

Firstly, we write the Slack Event Listener. In this example, it would respond to the message event “@beerbot hey” and show a menu to the user. For the Slack API Client, we use the nlopes/slack package. We write the below listener and watch for the necessary MessageEvent.

func (s *SlackListener) ListenAndResponse() { 
    // Start listening slack events
    rtm := s.client.NewRTM()
    go rtm.ManageConnection()
 
     // Handle slack events
     for msg := range rtm.IncomingEvents {
         switch ev := msg.Data.(type) {
         case *slack.MessageEvent:
             if err := s.handleMessageEvent(ev); err != nil {
                 log.Printf(“[ERROR] Failed to handle message: %s”, err)
             }
         }
     }
}

Next, we’ll write the function below to process the MessageEvent.

func (s *SlackListener) handleMessageEvent(ev *slack.MessageEvent) error

In the function, firstly we validate the MessageEvent. At the very least, we need to confirm the following:

• Whether a message comes from expected channel (You must limit channel where bot work. For example, bots that do releases should only work in the release channel).
• Whether the bot has been mentioned

Next, the actual thing said to the bot (MessageEvent.Msg.Text) needs to be parsed. We can think of this in the same way as when we are writing a command line tool. The parsed message can inform the menu or other elements shown. (In this example, it just responds to “hey”.)

Next we actually show the menu. We will use the Attachment field of the Slack Post Message API to show the menu. When written with Golang, it will be like the below.

var attachment = slack.Attachment{
    Text: “Which beer do you want? :beer:”,
    Color: “#f9a41b”,
    CallbackID: “beer”,
    Actions: []slack.AttachmentAction{
        {
            Name: actionSelect,
            Type: “select”,
            Options: []slack.AttachmentActionOption{
                {
                    Text: “Asahi Super Dry”,
                    Value: “Asahi Super Dry”,
                },
                {
                    Text: “Kirin Lager Beer”,
                    Value: “Kirin Lager Beer”,
                },
                {
                    Text: “Sapporo Black Label”,
                    Value: “Sapporo Black Label”,
                },
                {
                    Text: “Suntory Malt’s”,
                    Value: “Suntory Malts”,
                },
                {
                    Text: “Yona Yona Ale”,
                    Value: “Yona Yona Ale”,
                 },
             },
         },
         {
              Name: actionCancel,
              Text: “Cancel”,
              Type: “button”,
              Style: “danger”,
         },
     },
 }

Firstly, we select either a menu (“select” type) or button for AttachmentAction.Type . AttachmentAction.Options slice will appear in the menu options (beer brands in this case). All that’s left to do is post this in the channel that received the event with the Slack Post API. After doing so, the menu will appear as below.

Interactive Handler

Next, we’ll write the handler that receives the results of the interactive message. See the example handler below:

func (h interactionHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        log.Printf(“[ERROR] Invalid method: %s”, r.Method)
        w.WriteHeader(http.StatusMethodNotAllowed)
        return
    }
 
    buf, err := ioutil.ReadAll(r.Body)
    if err != nil {
        log.Printf(“[ERROR] Failed to read request body: %s”, err)
        w.WriteHeader(http.StatusInternalServerError)
        return
    }
 
    jsonStr, err := url.QueryUnescape(string(buf)[8:])
    if err != nil {
        log.Printf(“[ERROR] Failed to unescape request body: %s”, err)
        w.WriteHeader(http.StatusInternalServerError)
        return
    }
 
    var message slack.AttachmentActionCallback
    if err := json.Unmarshal([]byte(jsonStr), &message); err != nil {
        log.Printf(“[ERROR] Failed to decode json message from slack: %s”, jsonStr)
        w.WriteHeader(http.StatusInternalServerError)
        return
     }
 
    // Only accept message from slack with valid token
    if message.Token != h.verificationToken {
        log.Printf(“[ERROR] Invalid token: %s”, message.Token)
        w.WriteHeader(http.StatusUnauthorized)
        return
    }
 ...
}

The above examples are for the first half handler processing. (This will be the same for any interactive message) It is processing like the one shown below.

• Determining whether it is POST
• Unescape the payload
• Unmarshal the unescaped payload to slack.AttachmentActionCallback
• Check that the token of the message received matches the verification token issued when you registered the Slack App.

All we need to do now is to process the result of the user’s action. We’ll write this part like this;

func (h interactionHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 ...
    action := message.Actions[0]
    switch action.Name {
    case actionSelect:
        value := action.SelectedOptions[0].Value
        // Overwrite original drop down message.
        originalMessage := message.OriginalMessage
        originalMessage.Attachments[0].Text = fmt.Sprintf(“OK to order %s ?”, strings.Title(value))
        originalMessage.Attachments[0].Actions = []slack.AttachmentAction{
            {
                Name: actionStart,
                Text: “Yes”,
                Type: “button”,
                Value: “start”,
                Style: “primary”,
            },
            {
                Name: actionCancel,
                Text: “No”,
                Type: “button”,
                Style: “danger”,
            },
        }
        w.Header().Add(“Content-type”, “application/json”)
        w.WriteHeader(http.StatusOK)
        json.NewEncoder(w).Encode(&originalMessage)
        return
    case actionStart:
        title := “:ok: your order was submitted! yay!”
        responseMessage(w, message.OriginalMessage, title, “”)
        return
    case actionCancel:
        title := fmt.Sprintf(“:x: @%s canceled the request”, message.User.Name)
        responseMessage(w, message.OriginalMessage, title, “”)
        return
    default:
        log.Printf(“[ERROR] ]Invalid action was submitted: %s”, action.Name)
        w.WriteHeader(http.StatusInternalServerError)
        return
    }
}

At this point, we will do processing for each action received. For dividing up the actions, we will use the definitions in AttachmentAction.Name. This time, we defined the below three actions.

• actionSelect — Selecting a menu option
• actionStart — Confirming an order
• actionCancel — Canceling an order

For example, in the case of actionSelect, we determine their selection (beer brand) and confirm whether they really want to place an order with a button. In the event of actionCancel, the fact that the order was cancelled is returned as a response.

With Interactive Messages, the response to the user’s action overwrites their original message. This is to stop people from being able to press the button after they’ve sent the request. In order to do this, it is convenient to have the below function prepared. We delete the AttachmentAction of buttons etc. and return a response that overwrites the old message with a new one.

func responseMessage(w http.ResponseWriter, original slack.Message, title, value string) {
    original.Attachments[0].Actions = []slack.AttachmentAction{} // empty buttons
    original.Attachments[0].Fields = []slack.AttachmentField{
        {
            Title: title,
            Value: value,
            Short: false,
        },
    }
    w.Header().Add(“Content-Type”, “application/json”)
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(&original)
 }

It is important to show who executed the action in the action response. If you don’t do this, you’ll lose track of who initiated the action in the first place. With this bot, it has been made to allow people to know who pressed the button. (For help designing interactive buttons, see guidelines for building messages.)

Conclusion

In this article, I explained how to develop and write a Slack bot with interactive messages in Golang. I hope it helps you to create easy-to-use bots with interactive messages to automate time-consuming tasks! At Mercari, we are always looking for SREs who love automation and Golang. See our hiring page for more details!

Writing an Interactive Message Bot for Slack in Golang was originally published in Making Mercari on Medium, where people are continuing the conversation by highlighting and responding to this story.

Installation of Draft is quite easy (see documentation) and it’s done less than 10 min. But there are some missing parts for doing it on Google Container Engine (GKE) with Google Container Registry. Here is small notes.

(If you don’t have) Setup GKE cluster,

$ gcloud container clusters create ${CLUSTER} \
        --project ${PROJECT_ID} \
        --zone ${ZONE}

Draft uses a wildcard domain. Since this is just trial, it’s too much to prepare the domain. So in this time, just using xip.io. After you’ve installed the nginx-ingress controller, you can get an external IP address of it. If it’s, for example, 35.181.245.100, you can use 35.181.245.100.xip.io for your Draft wildcard domain.

Draft needs to push docker images to a container registry. On GKE, it’s easy to use Google Container Registry. To do it, you need some auth token setup.

First, get the access token,

 $ gcloud auth application-default print-access-token
yn89.GlxcBB8ylkd...

Then, format & base64 encode the token,

$ echo '{"registrytoken":"yn89.GlxcBB8ylkd..."}' | base64 -w 0
eyJyZWdpc3RyeXRva2VuIjoieW44OS5HbH...

Now preparation is done. Run the following command and install Draft!

$ draft init --set registry.url=gcr.io,registry.org=${PROJECT_ID},registry.authtoken=eyJyZWdpc3RyeXRva2VuIjo,basedomain=35.181.245.100.xip.io

That’s it. Now you can start using Draft! Easy! Yay!

reorx/httpstat ( and followed @davecheney’s golang implementation ) is getting hot recently ( there is also node implementation !) . It’s very useful to see these latency with intuitive way in console when something wrong happens on network.

While contributing the project, I thought it’s very useful if I can output the same information on my own golang HTTP client ( or my network connected CLI tool ) when debugging. So I wrote a small package to do this, https://github.com/tcnksm/go-httpstat .

Because it creates a httptrace powered `context.Context`, you just need to pass it to your `http.Request` by `withContext()` method ( So must be less code modification !).

The following is an example usage of this package (trace request to https://github.com ).

// Create a new HTTP request
req, err := http.NewRequest("GET", "https://github.com", nil)
if err != nil {
    log.Fatal(err)
}

// Create a httpstat powered context
var result httpstat.Result
ctx := httpstat.WithHTTPStat(req.Context(), &result)
req = req.WithContext(ctx)

// Send request by default HTTP client
client := http.DefaultClient
res, err := client.Do(req)
if err != nil {
    log.Fatal(err)
}

if _, err := io.Copy(ioutil.Discard, res.Body); err != nil {
    log.Fatal(err)
}
res.Body.Close()
end := time.Now()

// Show the results
log.Printf("DNS lookup: %d ms", int(result.DNSLookup/time.Millisecond))
log.Printf("TCP connection: %d ms", int(result.TCPConnection/time.Millisecond))
log.Printf("TLS handshake: %d ms", int(result.TLSHandshake/time.Millisecond))
log.Printf("Server processing: %d ms", int(result.ServerProcessing/time.Millisecond))
log.Printf("Content transfer: %d ms", int(result.ContentTransfer(time.Now())/time.Millisecond))

The result of this is like below,

2016/09/27 17:58:38 DNS lookup: 5 ms
2016/09/27 17:58:38 TCP connection: 180 ms
2016/09/27 17:58:38 TLS handshake: 461 ms
2016/09/27 17:58:38 Server processing: 296 ms
2016/09/27 17:58:38 Content transfer: 180 ms

The results are pretty much same as `httpstat` command. But there are some concerns about TLS handshaking time (see #7). Be care to use this in production as for now. I’ll continue to improve this.

If you find a something, please file a issue on Github.

In Go1.7, `testing` package will supports subtests and sub-benchmarks. With this feature, you can define tests/benchmarks in a test/benchmark function. It’s easy to create hierarchical tests or table-driven sub-benchmarks. It also provides a way to share common setup and tear-down code.

This feature may have less benefits on tests because normally table-driven tests is enough. It will improve your benchmarks codes a lot.

Let me show some sample codes. For example, when you want to take benchmark of function `Foo` with different configuration, before Go1.7, you need to prepare functions for each configuration.

// you need to prepare functions for each bench mark setting
func BenchmarkFoo1(b *testing.B) { benchFoo(b, 1) }
func BenchmarkFoo10(b *testing.B) { benchFoo(b, 10) }
func BenchmarkFoo100(b *testing.B) { benchFoo(b, 100) }

// helper function to run Foo with different config
func benchFoo(b *testing.B, config int) {
 for i := 0; i < b.N; i++ {
 Foo(base)
}

If you use Sub-benchmarks feature in Go1.7, you can write this benchmark like the following.

func BenchmarkFoo(b *testing.B) {
    cases := []struct {
         Config int
    }{
         {Config: 1},
         {Config: 10},
         {Config: 100},
    }

    for _, bc := range cases {
        b.Run(fmt.Sprintf(“%d”, bc.Config), func(b *testing.B) { benchFoo(b, bc.Base) })
    }
}

You can use table-driven approach! It’s simple and easy to read (Benchmark name will be Top level function name + first argument of `Run` method. e.g., `BenchmarkFoo/1`, `BenchmarkFoo/2`… )

If you check standard library changes in Go1.7, you can see it benefits from sub-benchmarks a lot. The followings are some of examples,

• https://go-review.googlesource.com/#/c/23428/
• https://go-review.googlesource.com/#/c/23429/
• https://go-review.googlesource.com/#/c/23420/

If we can not use DockerHub (Public registry), we need to build private one. Docker provide Docker image for private registry, so it’s very easy to start to it.

$ docker run -p 5000:5000 registry
$ docker push docker-private.com:5000/test-image:latest 

Easy, it’s done ! But it’s dangerous. Anyone who knows registry URL can push their own Docker images. So we need some authentication. Generally, we use Basic authentication because Docker client (docker login command) has function for it. Docker registry itself doesn’t have auth function, so we need to setup nginx or Appache in front of Docker registry as reverse proxy for Basic authentication.

In this case, we have some constrain (But, its reasonable):

• To Basic authentication, docker client ask us to use SSL
• Docker client verify secure certificate and we can’t allow insecure

If we use self-signed certification (for lazy), its’s a little hard and complicated steps. So I write procedures. For environment, we use Ubuntu for server, nginx for reverse proxy, client for OSX + boot2docker.

Server-side settings

We need 3 settings:

• nginx
• Password for Basic authentication
• Serf-signed certification

nginx

For reverse proxy, we use nginx. Docker registry provides example of nginx configuration in its repository (https://github.com/docker/docker-registry/tree/master/contrib/nginx). So we use it directly.

$ git clone https://github.com/docker/docker-registry 
$ cp docker-registry/contrib/nginx/nginx_1–3–9.conf /etc/nginx/conf.d/. 
$ cp docker-registry/contrib/nginx/docker-registry.conf /etc/nginx/.

password

We need to setup user and password for basic authentication.

$ htpasswd -bc /etc/nginx/docker-registry.htpasswd USERNAME PASSWORD 

Serf-signed certification

First, we need to initialise CA serial file and generate CA private and public keys:

$ echo 01 > ca.srl 
$ openssl genrsa -des3 -out ca-key.pem 2048 
$ openssl req -new -x509 -days 365 -key ca-key.pem -out ca.pem 

Now we have a CA, you can create a server key and certificate signing request (CSR).

$ openssl genrsa -des3 -out server-key.pem 2048 
$ openssl req -subj ‘/CN=<Your Hostname Here>’ -new -key server-key.pem -out server.csr 
$ openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem -out server-cert.pem 

We should remove the passphrase from server key:

$ openssl rsa -in server-key.pem -out server-key.pem 

Finally, we install server-key and server-cert:

$ cp server-cert.pem /etc/ssl/certs/docker-registry 
$ cp server-key.pem /etc/ssl/private/docker-registry 

Client-side settings

Docker client verify secure certificate and we can’t allow insecure connection like cURL’s -k option. So we need to make our CA valid one (There are some Issues to allow insecure connection, but not realized. See more #2687, #5817 ).

If you use boot2docker on OSX, you need to setup it not on OSX but on boot2docker-vm. Here, we use CA public key which we generate above step (https://github.com/boot2docker/boot2docker/issues/347).

$ boot2docker ssh
$ cat ca.pem >> /etc/ssl/certs/ca-certificates.crt
$ /etc/init.d/docker restart

Thats all, now we can login to our private registry.

$ docker login https://docker-private.com

References

• Running Docker with https
• Deploying your own Private Docker Registry | ActiveState
• https://github.com/docker/docker/pull/2687#issuecomment-50266315