Open in app

Sign in

Write

Sign in

Custom TCP Application with Go

Dpinoagustin
6 min readNov 24, 2023

--

TCP is the protocol which all data transfer worldwide, HTTP and Websocket for example runs over TCP. Even the most common used DB like Mongo, Redis or Postgres are using TCP for runs their protocols.

So, code a custom TCP application is just create a super-protocol of TCP. The TCP + the application protocol.

Thanks to golang, the half of the work is done because there is a native package for this purpose: the “net” package.

Considerations

First, let’s take a look to some consideration when it’s working with raw TCP.

  1. Clients Management.
  2. Message Buffer Management.
  3. Application Custom Protocol.
  4. Server connection from Clients.

Because TCP only offers a protocol for transfer data, the taken and interpretation of that date is an application’s job. That’s why those consideration exists.

Server Side

The fundamental step is creating a server where the clients will can connect to. As it mentioned before, the package net has the all tool needed.

package server

func Listen(p int) error {  
  srv, err := net.Listen("tcp", "127.0.0.1:%d", p)
  if err != nil {
    return err
  }
  defer srv.Close()
  return nil
}

The function ‘Listen’ will listen to a port on 127.0.0.1 IP (the localhost). When the function ends release the taken port.

How is right now, the server cannot accept clients, so let’s code that.

package server

func Listen(p int) error {  
  srv, err := net.Listen("tcp", "127.0.0.1:%d", p)
  if err != nil {
    return err
  }
  defer srv.Close()
  
  // infinity loop
  for {
    cli, err := srv.Accept()
    if err != nil {
      return err
    }
  }

  return nil
}

Great, now the server can accept clients. The Accept method blocks the loop until a new client performs a connection.

What happen inside the Accept function is the TCP handshake. This consists in three steps.

  1. The client send a SYN to the server.
  2. The server accepts that SYN by responding to the client with a SYN-ACK.
  3. The client responses with an ACK.

Once those three steps are done, the connection is established.

SYN and ACK are TCP flags that form parts of the TCP itself where SYN menas Synchronize and ACK means Acknowledged.

Closing Connections

The server at the moment only accept clients but never does anything with them. It’s important to know that responsibility of closing connection is on the server. That’s means if the server takes a connection, after use it must close it.

For each new client connection a goroutine will be executed that handle the client.

package server

func Listen(p int) error {  
  srv, err := net.Listen("tcp", "127.0.0.1:%d", p)
  if err != nil {
    return err
  }
  defer srv.Close()
  
  // infinity loop
  for {
    cli, err := srv.Accept()
    if err != nil {
      return err
    }
    go handle(cli)
  }

  return nil
}

func handle(cli net.Conn) {
  defer cli.Close()
}

Excellent, now the client connections are safe!.

The net.Conn is an interface from net package that provides the method needed for manipulate that specific connection.

Reading Message Problem

When the connection with a client is established. Both client & server can share data. But big starting problem appears here. How to read the messages?

This is a problem because the message length is unknown. Typically in TCP, the message buffer is read by chunks of ’n’ bytes until there is no more bytes to read.

Another solution is send the message length as part of a message metadata. In HTTP for example this length is sent in a Header.

This kind of mechanism forms part of the application protocol rather than TCP itself.

Custom Application Protocol

The protocol of the custom application is just a set of rule that both client and server follow for understand each other.

The rules to follow for this application are.

  1. The TCP payload has to section: Length Message and Body Message.
  2. The Length Message are the first 2 bytes of the TCP payload.
  3. The Body Message has JSON format.
  4. Once the message is processed, the connection will close.

The second rule determines the max length of the Body Message, an integer of 2 bytes. From 0x0000 to 0xFFFF, in decimal base: 0 to 65535 bytes. 2 bytes int is a int16.

A message for this application will look like this.

LEN | BODY
0019 7B226D657373616765223A2268656C6C6F20776F726C64227D

LEN | BODY
25   {"message":"hello world"}

Coding the protocol

Well, let’s code the rules of this protocol called JSONP (JSON Protocol)

package jsonp

func ReadMessage(c net.Conn) []byte {
  l := make([]byte, 2) // takes the first two bytes: the Length Message.
  i, _ := c.Read(l)    // read the bytes.

  lm := binary.BigEndian.Uint16l(l[:i]) // convert the bytes into int16.
  
  b := make([]byte, lm) // create the byte buffer for the body.
  e, _ := c.Read(b)     // read the bytes.
  
  return b[:e]          // returns the body
}

The function above has the message reading rules. An import thing to mention the how the Length Message bytes are stored. In this case and also in the most cases for TCP, the bytes storing has the BigEndian format.

The BigEndian format stores the bytes starting from the right to the left. In other hand, the LittleEndian is the opposite.

len = {00, 01}

BigEndian(len) => 0001 => 1: int16
LittleEndian(len) => 0100 => 256: int16

Let’s continue with the message writing.

package jsonp

func WriteMessage(m []byte, c net.Conn) {
  l := make([]byte, 2)  // creates the Length Message buffer.
  binary.BigEndian.PutUint16(l, uint16(len(m))) // Converts len to bytes.
  c.Write(append(l, m...)) // send the message
}

So far so good. Now let’s create some json marshal/unmarshal functions for abstract from that logic.

package jsonp

func ReadJSON(c net.Conn, a any) error {
  msg := ReadMessage(c)
  return json.Unmarshal(msg, a)
}

func SendJSON(c net.Conn, a any) error {
  msg, err := json.Marshal(a)
  
  if err != nil {
    return err
  }

  WriteMessage(c, msg)
  return nil
}

Excellent, with all this, the protocol is done.

Handling Clients

The final part of the server needs some modifications to the original Listen function.

First, let’s add a type alias to the protocol.

package jsonp

type Handler func(c net.Conn)

This Handler is the context/callback where the messages are sending and receiving.

Now, in the server let’s implement the handler.

package server

func Listen(p int, h jsonp.Handler) error {  
  srv, err := net.Listen("tcp", "127.0.0.1:%d", p)
  if err != nil {
    return err
  }
  defer srv.Close()
  
  // infinity loop
  for {
    cli, err := srv.Accept()
    if err != nil {
      return err
    }
    
    go func(c net.Conn, hd jsonp.Handler) {
      defer c.Close()
      hd(c)
    }(cli, h)
  }

  return nil
}

With this modification, the application owns the client handling instead of the server. This last one only provides a safety context to use clients (by closing the connection after use it).

Client Side

The server side is done, so let’s continue with the Client Side. This is an easy part because is net package use the same interface for both side of the TCP socket. So, the half part of the work is done.

Server Connection

The connection to the server can be done by the net.Dial function. However, because the connection is alive only for sending one message after that the connection will close, the server connection is itself the message sending.

package client

func Connect(p int, h jsonp.Handler) error {
  conn, err := net.Dial("tcp", fmt.Sprintf("127.0.0.1:%d", p))
  if err != nil {
    return err
  }
  defer conn.Close()
  
  h(conn)
  
  return nil
}

This is all the client needs. The rest part is the handler but that is responsibility of the application itself not the connection or even the application protocol.

Hello World Example

Let’s make an example with all these stuffs.

Client

The client will send a “Hello World” message.

package main

type Message struct {
  Content string `json:"content"`
}

func main() {
  client.Connect(8080, func(c net.Conn) {
    jsonp.SendJSON(c, &Message{Content:"Hello World"})

    var res Message
    jsonp.ReadJSON(c, &res)

    fmt.Println(res.Content)
  })
}

Server

The server will do the same as the client but reading the message first then sending the “Hello World” as response.

package main

type Message struct {
  Content string `json:"content"`
}

func main() {
  server.Listen(8080, func(c net.Conn) {
    var res Message
    jsonp.ReadJSON(c, &res)

    fmt.Println(res.Content)

    jsonp.SendJSON(c, &Message{Content:"Hello World"})
  })
}

Conclusions

And that’s all. This article is not about of creating a new and better protocol than the existing but rather aims to see what happens under the hood of the TCP based Protocol.

With the concepts shown in this article, for example, the HTTP can be replicated .

Also for avoid connect and reconnect each time a message is sending (like HTTP does), the connection can be maintained alive for a while until the client send a some kind of command for close the socket.

Golang
Tcp
Go
Programming
Tutorial

--

--

Dpinoagustin

Written by Dpinoagustin

69 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams