A Pragmatic and Systematic Project Structure in Go

When embarking on a new Go project, the way you structure your codebase will have a lasting and hard-to-change effect throughout your project’s lifecycle. A well-thought-out structure makes it easier for new contributors to understand the project’s architecture and contribute effectively. In this article, I’ll introduce a pragmatic and systematic way to organize your Go projects, especially for microservices, with the following three objectives:

• Easy developer onboarding
• Simple structure
• Consistency

Background

Adhering to common standards in project layout helps maintain code hygiene and streamlines development efforts. One widely accepted standard within the Go community is the suggested project layout as documented in the Go project layout guide. This guide serves as a solid foundation, particularly for those new to Go, by providing a structured approach to organizing projects. However, it lacks further detailed guidance on how to organize your packages and create the package dependency tree. In order to solve this problem, we need to touch on two points first.

Firstly, Go’s package management is distinctively different from the ‘namespace’ concept found in object-oriented programming languages. In Go, packages encourage a layered API structure, where each layer offers a specific API to the layer above. This approach contrasts with namespaces, which primarily serve to group related code without necessarily enforcing a layered architecture.

Secondly, Go package structure is very strict about imports and does not allow circular imports. This leads engineering teams to consider very carefully when organizing their Go projects.

With this fresh knowledge in mind, we’re ready to analyze the suggested organization.

Suggested Project Organization

your-repository/
  /cmd
    /<program1>
      main.go
      /handlers
        handler.go
      /commands
        command.go
      /internal
        internal-logic.go
      /workers
        worker.go
  /internal
    /product
      product-feature1.go
    /auth
      auth.go
    /platform
      /<package1>
      /<package2>
      /<package3>
  /pkg
    /<package1>
    /<package2>
    /<package3>

/cmd

This folder contains all your programs and their main program files, meaning the ‘main’ package. The programs initiate from main.go file.

Each program under /cmd file contains specific types of execution-starting points depending on the application.

• If the application contains a web server, it’ll have /handlers for each endpoint.
• If the application contains a command-event structure, it’ll have /commands for each command type.
• If the application contains job producer-consumer logic, it’ll have /workers.

The /internal package under each program contains the specific use-case logic for that microservice. For example, you might create a specific use case from ordering, shipping and payment packages and create the ‘checkout’ use case.

/internal

Internal is a special package name in Go such that only packages in the same level can import from it and the package cannot be imported from other repositories. In the suggested organization, /internal is used to contain the business logic used in the /cmd programs.

Organize the main domain under /internal/domain or /internal/product and organize each subdomain in its own package under /internal. For example,

• /internal/product
• /internal/auth
• /internal/verification

However, there is one special folder under /internal which will contain the ‘enabler’ packages for the entire project.

/internal/platform

The platform folder contains all the enabler packages like database connections, cloud storage logic like s3, SMS notification sender, the in-memory cache logic, etc. The business logic and the use-case layer are built on top of the platform packages. (Thus, the name ‘platform’ comes from)

If the platform packages also needs to be used in other repositories, /pkg folder can be used instead of /internal/platform.

Since it is necessary to follow a ‘package’ mindset rather than a ‘namespace’ mindset, it is important to follow a strict dependency tree.

Dependency Tree

/cmd/<program>/main.go
         |
         |
        \|/
/cmd/<program>/handlers
/cmd/<program>/workers
/cmd/<program>/commands
         |
         |
        \|/
/cmd/<program>/internal
         |
         |
        \|/
/internal/product
/internal/<other packages>
         |
         |
        \|/
/internal/platform/<package>

The dependencies have a one-way flow from the main program down to program-specific entry points, to use-cases, and then to internal business logic, ending with the platform packages. This hierarchy ensures that dependencies are well-organized and maintainable.

The Algorithm for Project Layout

• Controller level concerns reside in /cmd/<service>/handler & commands under /cmd/<service>/commands
• Use-case level concerns are placed in /cmd/<service>/internal
• Core domain level concerns find their home in /internal/product/…
• Other domain-specific logic is located in /internal/<specific package>/…
• Foundational elements such as storage or database connections are in /internal/platform/… or /pkg/…

With a simple dependency flow and a consistent and lean folder structure, the suggested project organization aligns well with the initial motivations of easy onboarding, simplicity, and consistency.

Example Project Layout: E-commerce Order Microservice

Imagine structuring an order microservice for an e-commerce website:

/cmd
  /order
    main.go
    /handlers
      orders.go
    /internal
      order-workflow.go
/internal
  /auth
  /product (domain logic)
    order.go (model structs, etc.)
    order-validation.go (domain logic checks for orders)
  /platform
    /db
    /payment
    /storage
    /logging

In such a microservice,

• Order HTTP requests would be handled in /cmd/order/handlers/orders.go
• Order workflow would be orchestrated in /cmd/order/internal/order-workflow.go
• The specific business logic would be enforced on /internal/product/order.go
• The database connections would be supplied from /internal/platform/db
• The payment API would be supplied from /internal/platform/payment
• The logging API would be supplied from /internal/platform/logging

Testing Strategy

Developing a robust testing strategy is crucial for ensuring the reliability and maintainability of the application for its entire lifetime. Testing can generally be categorized into two main types: unit tests and integration tests. Both types play a vital role in the development cycle, helping to catch bugs early and facilitating smoother iterations.

Unit Tests

Unit tests are designed to test isolated pieces of code, usually functions or methods, to ensure they perform as expected. In the context of our structured Go project, unit tests are particularly important for the following areas:

• /cmd/<service>/internal/…: Testing at this level focuses on the use-cases that are specific to an individual service within your application. For example, if you have a payment processing service, you might write unit tests for the entire payment processing use-case.
• /internal/…: This is where the core domain logic of your application resides. Unit tests here might involve testing the functionality of domain models or services. For instance, in an e-commerce application, you would write tests for adding items to a shopping cart, calculating the total cost, or applying discounts.
• /internal/platform/…: Foundational components like database interactions, API clients, or utility functions need to be reliable since they are often used across the application. Unit tests might cover scenarios like ensuring a database connection handler returns the correct output or that an API client correctly handles different response statuses.

Example: Consider a function within /internal/product/ that discounts the price of a product if certain coupon codes are applied. A unit test for this function would assert that the function returns the correct product price on each test case as well as return an error when a non-valid case is tested.

Integration Tests

While unit tests focus on isolated parts of the application, integration tests verify how these parts work together. This type of testing is essential for identifying issues that might not be evident when components are tested in isolation. In the structured Go project, integration tests are critical for:

• /cmd/<service>/handler: These tests simulate real-world usage of your HTTP handlers to ensure that requests are processed correctly and that the appropriate responses are generated. For example, you might write integration tests to verify that your order creation endpoint correctly interacts with the underlying order processing logic and database layers to create a new order and return the correct response to the client.
• Other entry points: Beyond HTTP handlers, your application might include other types of entry points such as gRPC services, scheduled jobs, workers, commands, or event listeners (as mentioned in the suggested organization). Integration tests for these components ensure that they correctly interact with the rest of the application. For instance, a scheduled job that generates monthly reports would need tests to verify that it correctly retrieves data, generates the report, and stores or distributes it as expected.

Example: An integration test for the order creation endpoint (/cmd/order/handlers/orders.go) would involve sending a simulated HTTP request to create a new order and then verifying that:

• The request is correctly parsed.
• The order is successfully created in the database.
• The correct response (success status and order details) is returned to the client.
• Optionally, check if any side effects (like sending a confirmation email) are triggered as expected.

Conclusion

Adopting a structured approach to organizing your Go project can significantly enhance the development experience, making it easier to maintain and scale. The suggested project layout offers a clear and lean way to organize your codebase. Before embarking on a large-scale project, it’s advisable to try this approach on a smaller project to evaluate its effectiveness and make any necessary adjustments.

If you want to read more about optimizing container performance, you can read my colleague Umut Akkaya’s article here.