How to apply clean architecture principles to Golang micro-services
A step by step example of how applying clean architecture in Golang can lead to more maintainable and testable code
As a freelancer, I spend a lot of my time between projects learning and experimenting with new technologies and concepts. A few years ago I got into Golang and it has since become the go-to language for a lot of my experiments.
Lately, I have been attempting to apply principles of clean architecture to Golang and I thought I would share how I’ve been doing it in my current personal project.
We’ll be building a simple micro-service based application. The first service (producer) will expose its API both via HTTP and RPC, the second (consumer) will only be a simple service that only listens for events on a message bus.
The sample code for this article is available on GitHub. Let’s get started…
The problem
Developing a micro-service architecture is always a challenge and there is quite a lot of boilerplate code to be written and repeated for every single service (parsing CLI flags, connecting to a DB, handling transport, etc.) plus it’s easy to end up with the whole logic into a single file.
Keeping that in mind, I set the following goals for the project:
- Don’t use an external micro-service frameworks
- Apply clean architecture principles
- Reduce boilerplate by implement reusable and self-contained components to allow for service composition
- Don’t try to be too smart and attempt to standardise similar sounding components (for instance avoid creating a unified interface to connect to any possible DB in existence)
Clean Architecture
Here’s a simple diagram showing the clean architecture conceptual model
In simple terms, each inner layer doesn’t know anything about the outer layers. So, entities don’t know anything about the use cases, use cases don’t know anything about how presenters will use them and so on…
For an in-depth reading about it refer to the post from Uncle Bob here.
Service and components
All the services are implemented around two simple abstractions.
type Component interface {
ID() string
Configure(service Service, cliCtx *cli.Context) error
DependsOn() []string
Flags() []cli.Flag
Initialize(wg *sync.WaitGroup, startedCh chan<- struct{}, shutdownCh <-chan struct{}, errCh chan<- error)
Logger() *zerolog.Logger
}
A component is a specialised unit responsible for:
- Processing the command line/environment arguments for itself
- Applying its own configuration
- Initialising and managing its own lifecycle
- Exposing functionalities (i.e. DB client), if required
type Service interface {
AddComponent(component Component, deps ...string) error
Bootstrap()
Component(id string) (Component, error)
DebugMode() bool
Handler() Component
Logger() *zerolog.Logger
Name() string
Shutdown()
}
A service is a simple implementation of the above interface and it wraps most of the boilerplate code and it is responsible for:
- Holding components and resolve dependencies between them
- Bootstrapping the micro-service
- Control the lifecycle of the application
You might have noticed the Handler() method in the service interface. The handler is the component at the heart of the service itself and it acts as a presenter in the clean architecture model. Its responsibility is to wire all other components (a poor man dependency injection) and to expose the business logic of the service. Notice I said “expose” and not “contain”, I will come back to it later in the article.
Each service follows the package structure convention below but not all packages are required for every service.
[serviceName]/
domain/
handler/
repository/
usecase/
main.go
Implementing a component
As we will be using NATS as a message broker between the services, we need to implement a component to connect to it. Here’s a snippet of the component implementation:
package natsbroker
import ( ... )
const (
Component = "nats-broker"
...
)
var cliFlags = []cli.Flag{ ... }
func Create(options ...nats.Option) framework.Component {
return &natsBroker{ natsOptions: options }
}
func Get(service framework.Service) (Broker, error) { ... }
type Broker interface {
Client() *nats.Conn
Publish(topic string, msg proto.Message) error
}
type natsBroker struct {
client *nats.Conn
logger *zerolog.Logger
natsUrl string
natsOptions []nats.Option
}...func (b *natsBroker) Configure(service framework.Service, cliCtx *cli.Context) error {
logger := service.Logger().With().Str("component", Component).Logger()
b.logger = &logger
natsUrl := cliCtx.String(flagNatsUrl)
if natsUrl == "" {
return fmt.Errorf("missing nats url")
}
b.natsUrl = natsUrl
return nil
}
func (b *natsBroker) Initialize(wg *sync.WaitGroup, startedCh chan<- struct{}, shutdownCh <-chan struct{}, errCh chan<- error) {
// make sure we notify when we are done
defer wg.Done()
b.logger.Debug().Msg("connecting...")
client, err := nats.Connect(b.natsUrl, b.natsOptions...)
if err != nil {
errCh <- err
return
}
b.client = client
b.logger.Info().Msg("connected") // Notify service of initialisation completed
close(startedCh)
doneCh := make(chan struct{}, 1)
stopFunc := func() {
b.logger.Debug().Msg("disconnecting...")
b.client.Close()
close(doneCh)
} // Wait for shutdown signal
<-shutdownCh
b.logger.Debug().Msg("shutdown signal received...")
stopFunc()
// Wait until stopFunc() completes
<-doneCh
b.logger.Info().Msg("disconnected")
}
As you can see, the component does only one thing. It holds the connection to NATS and exposes the client and a publish method.
The most important methods in the component are Configure() and Initialize(). The former handles the configuration flags from the command line and populates the configuration while the latter initialises the NATS client, notifies the service of success (or error) and waits for the shutdown notification from the service to close the client.
Implementing the consumer service
Now that we have our first component, let’s see how it all comes together by implementing the consumer service. This service is only listening for events coming from a message broker and it doesn’t expose any public API.
package main
import ( ... )const Name = "consumer"var (
GitCommit string
Version string
)
func main() {
service, err := framework.Create(Name, Version, GitCommit, handler.ServiceHandler())
if err != nil {
panic(err)
}
service.AddComponent(natsbroker.Create())
service.Bootstrap()
}
And that’s it, this is how we bootstrap a new service!
Let’s now have a look at the service handler and how it interacts with the component. As I mentioned earlier, the handler has to wire components and expose the business logic.
This is where the use case concept in clean architecture model comes in handy. A use case handles entities and wraps a unit of business logic, keeping it separate from the rest of the code. This separation makes the code cleaner, much easier to replace and easily testable.
For the sake of simplicity, we assume we are only printing a log line when we receive a new message.
package usecase
import ( ... )func NewHandleMessageUseCase() *handleMessageUseCase {
return &handleMessageUseCase{}
}
type HandleMessageFunc func(logger *zerolog.Logger, rawMsg *nats.Msg) error
type handleMessageUseCase struct {
}
func (uc *handleMessageUseCase) Execute(logger *zerolog.Logger, rawMsg *nats.Msg) error {
if rawMsg == nil {
return fmt.Errorf("the message cannot be NIL")
}
var message messaging.EventUserHello
if err := proto.Unmarshal(rawMsg.Data, &message); err != nil {
return fmt.Errorf("error deserializing user hello event")
}
logger.Info().Msgf("[%s] says: %s", message.GetName(), message.GetMessage())
return nil
}
In the real world, a use case might have references to repositories or other services. The important thing is wrapping the business logic to solve a specific problem in a single place.
Right… but how do we use it? Let’s go back to the service handler.
package handler
import ( ... )func ServiceHandler() framework.Component {
return &serviceHandler{}
}
type serviceHandler struct {
service framework.Service
logger *zerolog.Logger
handleMessage usecase.HandleMessageFunc
}...func (h *serviceHandler) DependsOn() []string {
return []string{natsbroker.Component}
}
func (h *serviceHandler) Initialize(...) {
// Notify shutdown complete on exit
defer wg.Done()
broker, err := natsbroker.Get(h.service)
if err != nil {
errCh <- err
return
}
h.handleMessage = usecase.NewHandleMessageUseCase().Execute
go h.monitorEvents(broker, shutdownCh)
close(startedCh)
// Wait for shutdown
<-shutdownCh
}
func (h *serviceHandler) monitorEvents(broker natsbroker.Broker, shutdownCh <-chan struct{}) {
// Listen for messages on the broker and dispatch
...
for {
select {
case msg := <-userHelloCh:
go h.handleMessage(h.logger, msg)
case <-shutdownCh:
h.logger.Info().Msg("stopped monitoring topics")
return
}
}
}
Similarly to the NATS component, the service handler configures and initialises itself and it can have its own CLI flags. As the handler depends on the broker, it has to explicitly declare that dependency:
func (h *serviceHandler) DependsOn() []string {
return []string{natsbroker.Component}
}
And this is how we use the use case to expose its logic in the handler:
type serviceHandler struct {
...
handleMessage usecase.HandleMessageFunc
}...
func (h *serviceHandler) Initialize(...) {
...
h.handleMessage = usecase.NewHandleMessageUseCase().Execute
...
}
func (h *serviceHandler) monitorEvents(...) {
...
for {
select {
case msg := <-userHelloCh:
go h.handleMessage(h.logger, msg)
case <-shutdownCh:
...
}
}
}
Neat, isn’t it? Let’s move on and implement a fully featured service
Implementing the producer service
The producer service is more complicated and it will have the following features:
- Expose an HTTP API
- Expose an RPC API
- Store the message in a DB
- Send the message to the consumer service through the message broker
Let’s assume we have implemented the HTTP, RCP and DB storage components (you can find the code on the GitHub repository), our producer service will look like this:
package main
import ( ... )const Name = "producer"
var (
GitCommit string
Version string
)
func main() {
service, err := framework.Create(Name, Version, GitCommit, handler.ServiceHandler())
if err != nil {
panic(err)
}
service.AddComponent(cloudstore.Create())
service.AddComponent(natsbroker.Create())
service.AddComponent(microhttp.Create(handler.InitHttpFunc), framework.HandlerComponent)
service.AddComponent(microrpc.Create(handler.InitRpcFunc), framework.HandlerComponent)
service.Bootstrap()
}
We are now composing the service with more components and adding an explicit dependency on the handler for the HTTP and RPC component.
Our use case to handle the request will require access to the message entity.
package domain
import ( ... )const MessageKind = "Message"
type Message struct {
ID *datastore.Key `datastore:"__key__"`
Name string
Message string
CreatedAt time.Time
}
func NewMessage(name string, message string) *Message {
key := datastore.IDKey(MessageKind, 0, nil)
return &Message{
ID: key,
Name: name,
Message: message,
}
}
and the repository:
package repository
import ( ... )func NewMessageRepository(logger *zerolog.Logger, cloudstore cloudstore.Store) MessageRepository {
return &messageRepository{
logger: logger,
cloudstore: cloudstore,
}
}
type MessageRepository interface {
GetAll() ([]domain.Message, error)
Store(*domain.Message) error
}
type messageRepository struct {
cloudstore cloudstore.Store
logger *zerolog.Logger
}
func (r *messageRepository) GetAll() ([]domain.Message, error) {
client := r.cloudstore.Client()
query := datastore.NewQuery(domain.MessageKind)
var result []domain.Message
if _, err := client.GetAll(context.Background(), query, &result); err != nil {
return nil, err
}
return result, nil
}
func (r *messageRepository) Store(entity *domain.Message) error {
client := r.cloudstore.Client()
if _, err := client.Put(context.Background(), entity.ID, entity); err != nil {
return err
}
return nil
}
We could avoid using a repository and let the use case talk directly to the datastore, but that would duplicate code and it would be prone to errors. A repository can be tested independently and mocked to test the use case without requiring a database instance.
For the first use case we want to store the message and publish an event
package usecase
import ( ... )func NewStoreAndPublishMessageUseCase(broker natsbroker.Broker, repo repository.MessageRepository) *storeAndPublishMessageUseCase {
return &storeAndPublishMessageUseCase{
broker: broker,
repo: repo,
}
}
type StoreAndPublishMessageFunc func(logger *zerolog.Logger, requestId string, msg *domain.Message) error
type storeAndPublishMessageUseCase struct {
broker natsbroker.Broker
repo repository.MessageRepository
}
func (uc *storeAndPublishMessageUseCase) Execute(logger *zerolog.Logger, requestId string, msg *domain.Message) error {
if msg == nil {
return fmt.Errorf("message cannot be NIL")
}
// Store message in datastore
if err := uc.repo.Store(msg); err != nil {
return fmt.Errorf("error storing message: %v", err)
}
// Publish message event to broker
event := &messaging.EventUserMessage{
RequestId: requestId,
Name: msg.Name,
Message: msg.Message,
}
if err := uc.broker.Publish(messaging.ChannelUserMessage, event); err != nil {
logger.Error().Err(err).Msg("error publishing event")
// We should still return even if the event has failed
}
return nil
}
The second use case retrieves all the messages from the DB
package usecase
import ( ... )
func NewGetMessagesUseCase(repo repository.MessageRepository) *getMessagesUseCase {
return &getMessagesUseCase{
repo: repo,
}
}
type GetMessagesUseCaseFunc func(logger *zerolog.Logger) ([]domain.Message, error)
type getMessagesUseCase struct {
repo repository.MessageRepository
}
func (uc *getMessagesUseCase) Execute(logger *zerolog.Logger) ([]domain.Message, error) {
logger.Debug().Msg("loading messages from repository")
messages, err := uc.repo.GetAll()
if err != nil {
return nil, fmt.Errorf("error loading messages from repository: %v", err)
}
return messages, nil
}
As you can see both use cases have dependencies on the repository and/or the message broker component. The service handler is responsible of making sure the dependencies are available and this is the relevant snippet from the service handler:
func (h *serviceHandler) Initialize(...) {
defer wg.Done()
broker, err := natsbroker.Get(h.service)
if err != nil {
errCh <- err
return
}
datastore, err := cloudstore.Get(h.service)
if err != nil {
errCh <- err
return
}
messageRepo := repository.NewMessageRepository(h.logger, datastore)
h.getMessages = usecase.NewGetMessagesUseCase(messageRepo).Execute
h.storeAndPublishMessage = usecase.NewStoreAndPublishMessageUseCase(broker, messageRepo).Execute
close(startedCh)
// We don't need to wait for the shutdown signal in this case.
// The HTTP and RPC components will do that for us
}
Now that everything is wired up we can access the use cases from the handler’s HTTP interface
package handler
import ( ... )
type httpPublishRequest struct {
Name string `json:"name,omitempty"`
Message string `json:"message,omitempty"`
}
type httpMessage struct {
Name string `json:"name,omitempty"`
Message string `json:"message,omitempty"`
CreatedAt time.Time `json:"createdAt,omitempty"`
}
func InitHttpFunc(service framework.Service, _ framework.Component, router *gin.Engine) error {
handler := service.Handler().(*serviceHandler)
router.GET("/messages", getMessagesHandler(handler))
router.POST("/publish", publishHandler(handler))
return nil
}
func getMessagesHandler(handler *serviceHandler) gin.HandlerFunc {
return func(c *gin.Context) {
logger := microhttp.RequestLogger(c)
messageEntities, err := handler.getMessages(logger)
if err != nil {
c.AbortWithError(http.StatusInternalServerError, err)
return
}
messages := make([]*httpMessage, 0)
for _, m := range messageEntities {
msg := &httpMessage{
Name: m.Name,
Message: m.Message,
CreatedAt: m.CreatedAt,
}
messages = append(messages, msg)
}
c.JSON(http.StatusOK, messages)
}
}
func publishHandler(handler *serviceHandler) gin.HandlerFunc {
return func(c *gin.Context) {
logger := microhttp.RequestLogger(c)
requestId := microhttp.RequestId(c)
var body httpPublishRequest
if err := c.ShouldBindWith(&body, binding.JSON); err != nil {
c.AbortWithStatus(http.StatusBadRequest)
return
}
message := domain.NewMessage(body.Name, body.Message)
if err := handler.storeAndPublishMessage(logger, requestId, message); err != nil {
logger.Error().Err(err).Msg("server error")
c.AbortWithStatus(http.StatusInternalServerError)
return
}
c.Status(http.StatusCreated)
}
}
and from the handler’s RPC interface:
package handler
import ( ... )func InitRpcFunc(service framework.Service, component framework.Component, server *grpc.Server) error {
RegisterProducerRPCServer(server, &rpcServer{
logger: component.Logger(),
handler: service.Handler().(*serviceHandler),
})
return nil
}
type rpcServer struct {
logger *zerolog.Logger
handler *serviceHandler
}
func (s *rpcServer) GetMessages(context.Context, *Empty) (*GetMessagesReply, error) {
messageEntities, err := s.handler.getMessages(s.logger)
if err != nil {
return nil, err
}
messages := make([]*Message, 0)
for _, m := range messageEntities {
createdAt, _ := ptypes.TimestampProto(m.CreatedAt)
msg := &Message{
Name: m.Name,
Message: m.Message,
CreatedAt: createdAt,
}
messages = append(messages, msg)
}
return &GetMessagesReply{
Messages: messages,
}, nil
}
func (s *rpcServer) PublishMessage(ctx context.Context, req *PublishMessageRequest) (*Empty, error) {
requestId := util.NewUUID()
message := domain.NewMessage(req.GetMessage().Name, req.GetMessage().Message)
if err := s.handler.storeAndPublishMessage(s.logger, requestId, message); err != nil {
s.logger.Error().Err(err).Msg("server error")
return nil, err
}
return &Empty{}, nil
}
Wrapping up
Thank you for reading this far! I didn’t think it was going to be this long. We have seen how easy is to apply clean architecture principles to a Golang application. Despite having to write a little more code for the use cases, I believe in the benefits of splitting the code in smaller units to make the application easily testable and more maintainable in the long run.
Have a look at the full code on GitHub and try it out!