Making concurrent HTTP requests in Go programming language

Why to use concurrent programming?

All high-level programming languages run on the single core of the machine. Go is the modern programming language that gives us freedom to write idiomatic code which utilizes all cores of the machine. It is a programming language to write applications those unleash the power of multi-cores with a C++ par performance. In this short article, we can see how to make parallel web requests in Go.

Problem

The problem here is to make HTTP requests for a set of URL and list them according to the order and time of executions.

If we use traditional high level programming languages Python or JavaScript, we can just execute one operation at a time i.e. sequential program execution happens. But using Go we can easily write concurrent programs to bring the parallelism. In this program, we compare concurrent Go with sequential languages.

Importing libraries

Let us import all libraries required for performing our operation

package main
import (
     "fmt"
     "io/ioutil"
     "os"
     "net/http"
     "time"
)

Now let us write a function called MakeRequest which actually

• “fmt” is for I/O
• “io/ioutil” is for reading the HTTP response body
• “os” is for accesing command line arguments
• “time” is for printing time data
• “net/http” is for making HTTP requests

Now let us define a function called MakeRequest which makes HTTP requests concurrently.

func MakeRequest(url string, ch chan<-string) {
  start := time.Now()
  resp, _ := http.Get(url)
  secs := time.Since(start).Seconds()
  body, _ := ioutil.ReadAll(resp.Body)
  ch <- fmt.Sprintf("%.2f elapsed with response length: %d %s", secs, len(body), url)
}

Here MakeRequest is taking two arguments URL and Channel ch of type string. We all know that go routines communicate through channels. Here we launch a go routine per URL. Whenever a request is finished that go routine writes a response message to channel. That channel is collected in main go routine and printed.

Now main function looks like this

func main() {
  start := time.Now()
  ch := make(chan string)
  for _,url := range os.Args[1:]{
      go MakeRequest(url, ch)
  }
  for range os.Args[1:]{
    fmt.Println(<-ch)
  }
  fmt.Printf("%.2fs elapsed\n", time.Since(start).Seconds())
}

Main function above has these simple steps.

1. Note down the start time
2. Create a channel ch
3. For each URL of command line arguments launch a go co-routine
4. For each URL read the channel for result
5. Calculate time difference between beginning and end

Now the total code looks in this way

// concurrent.go
package main
import (
     "fmt"
     "os"
     "net/http"
     "time"
     "io/ioutil"
)
func MakeRequest(url string, ch chan<-string) {
  start := time.Now()
  resp, _ := http.Get(url)
  secs := time.Since(start).Seconds()
  body, _ := ioutil.ReadAll(resp.Body)
  ch <- fmt.Sprintf("%.2f elapsed with response length: %d %s", secs, len(body), url)
}
func main() {
  start := time.Now()
  ch := make(chan string)
  for _,url := range os.Args[1:]{
      go MakeRequest(url, ch)
  }
  for range os.Args[1:]{
    fmt.Println(<-ch)
  }
  fmt.Printf("%.2fs elapsed\n", time.Since(start).Seconds())
}

Generate a Go build with the following command

$ go build concurrent.go

This generates an executable file called “concurrent” in the same directory.

Now let us compare Python3 sequential program Vs Go concurrent program.

I started a simple HTTP server on “http://localhost:8000". Now I will run both the programmes passing command line args. First goes Python3 inline program.

$ time python3 -c "import requests;print(len(requests.get('http://localhost:8000').text));print(len(requests.get('http://localhost:8000').text));print(len(requests.get('http://localhost:8000').text))"

Now let me run the concurrent Go executable

$ time ./concurrent http://localhost:8000 http://localhost:8000 http://localhost:8000

If we observe the time intervals of execution, go co-routines make requests in parallel. So the total program time will be equal to longest request-response time. But in the case of Python it will be a summation of all requests.

Time of execution

Go 1.6 -> Real 0.0s, User 0.0s, CPU 0.006s

Python 3 -> Real 0.18s, User 0.02s, CPU 0.197s

We can call any external URL apart from our dummy localhost and the execution trends will be similar. This shows how Go programs are working concurrently to reduce the time for giving task. Keep watching and more Go development articles are coming soon.