MicroService Patterns: Retry with Spring Boot

Part of the Resilience4J Article Series: If you haven’t read my other articles yet, please refer to the following links:
1. Circuit Breaker Pattern in Spring Boot
2. MicroService Patterns: Rate Limiting with Spring Boot

Resilience4j is an open-source library for building resilient applications, offering simple yet powerful tools and patterns. It enhances reliability and responsiveness, ensuring software gracefully recovers from errors, adapts to changing conditions, and maintains high performance. The library includes modules like Retry, Circuit Breaker, Rate Limiter, Bulkhead, and Time Limiter.

In this article, we’ll delve into:

1. Introducing the Retry module.
2. Practical Microservices Demonstration in a Spring Boot application. (The GitHub repository is linked at the end of the article.)
3. Testing the Retry module within the demo.

What is Resilience4j-Retry?

The Resilience4j Retry module is a handy tool for dealing with temporary failures in operations that might fail at first. It lets developers customize how retries work, like how many times to retry, how long to wait between retries, and when to retry based on specific errors.

Upon encountering a failure, the Retry module automatically initiates retries under the predefined policy. It intelligently handles timing between retries, facilitating options such as exponential or fixed delays, and even randomized intervals. This strategic approach prevents overwhelming the system or target service with repeated requests, ensuring smoother operation and better resilience.

MicroServices Demonstration

Our demonstration has 2 services named address-service and order-service. If you’ve previously explored the Circuit Breaker Pattern in Spring Boot, the scenarios in this demonstration may already be familiar to you.

Scenario

• Before making a purchase, shoppers desire to review the details of their order. As a result, they send a request to the order-service.
• order-service utilizes a postal code to call address-service for shipping address details.
• Upon receiving the shipping address details, order-service updates the order information and subsequently sends it back to the shopper.

Address Service

Let’s build address-service first since it is a dependent service.

Prerequisites

• Java 17
• Maven Wrapper
• Spring Boot 3+
• H2 Nosql

Defining Dependencies

Create the project as a Spring Boot project with the dependencies provided inside the POM file below. I have named it address-service.

https://github.com/buingoctruong/retry-pattern-spring-boot/blob/master/address-service/pom.xml

Model

@Builder
@Data
@AllArgsConstructor
@NoArgsConstructor
@Entity
@Table(name = "addresses")
public class Address {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Integer id;
    private String postalCode;
    private String state;
    private String city;
}

Repository

@Repository
public interface AddressRepository extends JpaRepository<Address, Integer> {
    Optional<Address> findByPostalCode(String postalCode);
}

Service

public interface AddressService {
    Address getAddressByPostalCode(String postalCode);
}

@Service
@RequiredArgsConstructor
public class AddressServiceImpl implements AddressService {
    private final AddressRepository addressRepository;
    public Address getAddressByPostalCode(String postalCode) {
        return addressRepository.findByPostalCode(postalCode)
                .orElseThrow(() -> new IllegalStateException("Address Not Found: " + postalCode));
    }
}

Controller

@RestController
@RequestMapping("addresses")
@RequiredArgsConstructor
public class AddressController {
    private final AddressService addressService;
    @GetMapping("/{postalCode}")
    public Address getAddressByPostalCode(@PathVariable("postalCode") String postalCode)
            throws SocketTimeoutException {
        return addressService.getAddressByPostalCode(postalCode);
    }
}

Data Setup

We might need some default data records for the database table.

A method annotated with @PostConstruct is what we need. Spring will call that method just after the initialization of bean properties and then populating data.

@Configuration
@RequiredArgsConstructor
public class DataSetup {
    private final AddressRepository addressRepository;
    @PostConstruct
    public void setupData() {
        addressRepository.saveAll(Arrays.asList(
                Address.builder().id(1).postalCode("1000001").state("Tokyo").city("Chiyoda")
                        .build(),
                Address.builder().id(2).postalCode("1100000").state("Tokyo").city("Taito").build(),
                Address.builder().id(3).postalCode("2100001").state("Kanagawa").city("Kawasaki")
                        .build()));
    }
}

Properties

server:
  port: 9090
spring:
  application:
    name: address-service
  jpa:
    database-platform: org.hibernate.dialect.H2Dialect
    hibernate:
      ddl-auto: create-drop
  datasource:
    url: jdbc:h2:mem:address-db
    username: admin
    password: 1111
    driverClassName: org.h2.Driver
  h2:
    console:
      enabled: true

We finished building address-service. Run and access the link http://localhost:9090/addresses/1000001, the expected response should be as below.

{
  "id": 1,
  "postalCode": "1000001",
  "state": "Tokyo",
  "city": "Chiyoda"
}

Order Service

When it comes to order-service, the most interesting part might be around configuring Retry and monitoring its status through Actuator. I will strive to provide a straightforward explanation, so let’s get started!

Prerequisites

• Java 17
• Maven Wrapper
• Spring Boot 3+
• Resilience4j
• Actuator

Defining Dependencies

Create the project as a Spring Boot project with the dependencies provided inside the POM file below. I have named it order-service.

https://github.com/buingoctruong/retry-pattern-spring-boot/blob/master/order-service/pom.xml

Model

public interface Type {
}

@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder
@Entity
@Table(name = "orders")
public class Order implements Type {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    Integer id;
    private String orderNumber;
    private String postalCode;
    private String shippingState;
    private String shippingCity;
}

@Data
public class Failure implements Type {
    private final String msg;
}

Repository

@Repository
public interface OrderRepository extends JpaRepository<Order, Integer> {
    Optional<Order> findByOrderNumber(String orderNumber);
}

Service

Here is a place where every logic is written.

The tricky part is how to call an external API. Fortunately, Spring has RestTemplate that can help us do that.

RestTemplate is a central spring class used to consume the web services for all HTTP methods. ( Remember we need to create a RestTemplate Bean, see in Setup part below)

public interface OrderService {
    Type getOrderByPostCode(String orderNumber);
}

@Service
@RequiredArgsConstructor
public class OrderServiceImpl implements OrderService {
    private static final String SERVICE_NAME = "order-service";
    private static final String ADDRESS_SERVICE_URL = "http://localhost:9090/addresses/";
    private final OrderRepository orderRepository;
    private final RestTemplate restTemplate;
    @Retry(name = SERVICE_NAME, fallbackMethod = "fallbackMethod")
    public Type getOrderByPostCode(String orderNumber) {
        Order order = orderRepository.findByOrderNumber(orderNumber)
                .orElseThrow(() -> new RuntimeException("Order Not Found: " + orderNumber));
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_JSON);
        HttpEntity<AddressDTO> entity = new HttpEntity<>(null, headers);
        try {
            ResponseEntity<AddressDTO> response = restTemplate.exchange(
                    (ADDRESS_SERVICE_URL + order.getPostalCode()), HttpMethod.GET, entity,
                    AddressDTO.class);
            Stream.ofNullable(response.getBody()).forEach(it -> {
                order.setShippingState(it.getState());
                order.setShippingCity(it.getCity());
            });
        } catch (HttpServerErrorException e) {
            System.out.println("Retry due to http server error at: " + Instant.now());
            throw e;
        } catch (ResourceAccessException e) {
            System.out.println("Retry due to resource access at: " + Instant.now());
            throw e;
        }
        return order;
    }

    private Type fallbackMethod(Exception e) {
        return new Failure("Address service is not responding properly");
    }
}

In this context, we use the “@Retry” annotation on the method. The “name” attribute set as “order-service”, ensures that the configurations of the “order-service” instance are applied to this method (for detailed configurations, check the Properties section below).

Additionally, we employ the “fallbackMethod” attribute to call a backup method if the maximum retry attempts are exceeded and an exception occurs. It’s crucial that both methods return the same data type, hence the use of a “Type interface” for consistent implementation in both model classes.

Setup

@Configuration
@RequiredArgsConstructor
public class AppConfig {
    private final OrderRepository orderRepository;
    @Bean
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }

    @PostConstruct
    public void setupData() {
        orderRepository.saveAll(Arrays.asList(
                Order.builder().id(1).orderNumber("0c70c0c2").postalCode("1000001").build(),
                Order.builder().id(2).orderNumber("7f8f9f15").postalCode("1100000").build(),
                Order.builder().id(3).orderNumber("394627b2").postalCode("2100001").build(),
                Order.builder().id(4).orderNumber("825364c9").postalCode("1001111").build()));
    }
}

Properties

Full application.yaml file: https://github.com/buingoctruong/retry-pattern-spring-boot/blob/master/order-service/src/main/resources/application.yaml

server:
  port: 1010
spring:
  application:
    name: order-service
  jpa:
    database-platform: org.hibernate.dialect.H2Dialect
    hibernate:
      ddl-auto: create-drop
  datasource:
    url: jdbc:h2:mem:order-db
    username: root
    password: 123
    driverClassName: org.h2.Driver
  h2:
    console:
      enabled: true
management:
  endpoints:
    web:
      exposure:
        include: "*"
resilience4j:
  retry:
    instances:
      order-service:
        # Maximum number of attempts (including the initial call as the first attempt)
        max-attempts: 3
        # Fixed wait duration between retry attempts
        wait-duration: 1s
        retry-exceptions:
          - org.springframework.web.client.HttpServerErrorException
        ignore-exceptions:
          - org.springframework.web.client.ResourceAccessException

Allow me to provide a brief introduction to resilience4j-retry configurations.

• max-attempts: This parameter specifies the maximum number of retry attempts, including the initial call as the first attempt.
• wait-duration: This parameter sets the fixed wait duration between retry attempts. In this example, the duration is set to 1 second.
• retry-exceptions: This is a list of exceptions that will trigger a retry if thrown by the method annotated with the Retry aspect. In this case, if a HttpServerErrorException is thrown, a retry will be attempted.
• ignore-exceptions: This is a list of exceptions that will be ignored, meaning they will not trigger a retry. In this case, if a ResourceAccessException is thrown, it will not trigger a retry.

In this demo, I use HttpServerErrorException and ResourceAccessException. You can also employ additional exceptions, including custom ones, to achieve clearer response types for various HTTP status codes. Error Handling for REST with Spring

I’m currently leveraging Simple Retry, Retrying on Exceptions, and a Fallback Method for this demonstration. However, there are additional significant features offered by Resilience4j Retry, including Conditional Retry, Backoff Strategies, and the ability to Act on Retry Events. I plan to cover these features in a separate article.

We finished building order-service. Run sequentially address-service and then order-service, access to the link http://localhost:1010/orders?orderNumber=0c70c0c2, the response should be as below

{
  "id": 1,
  "orderNumber": "0c70c0c2",
  "postalCode": "1000001",
  "shippingState": "Tokyo",
  "shippingCity": "Chiyoda"
}

Playing with Retry

The Resilience4j Retry module includes Actuator endpoints that give you valuable insights and metrics about the retry setups. These endpoints are handy for monitoring and controlling the retry functionality within the application.

1. http://localhost:1010/actuator/retries: Provide all the retry settings in the app. It gives a summary of each retry setup, including its name and how it’s configured.
2. http://localhost:1010/actuator/retryevents: Provides access to retry events that happened while the app ran. These events include details about successful retries, failed retries, and ignored errors. It helps keep track of retry behavior and understand how often retries occur.
3. http://localhost:1010/actuator/retryevents/{name}: Provide specific details about retry events for a chosen retry instance. Just by putting the instance name in the URL, you can see things like how many times it retried, what errors it encountered, and more.
4. http://localhost:1010/actuator/metrics/resilience4j.retry.calls: Provide metrics for the retry feature. It shows how many calls were made, how many succeeded, how many failed, and other important metrics. By keeping an eye on these numbers, you can see how well your retry settings are working.

After calling the API at http://localhost:1010/orders?orderNumber=394627b2, if we refresh the actuator link at http://localhost:1010/actuator/retryevents, we won’t find any retry events. This is because the system successfully retrieved the address information for order number 394627b2.

{
  "retryEvents": []
}

Upon accessing the API at http://localhost:1010/orders?orderNumber=825364c9 and subsequently refreshing the actuator link at http://localhost:1010/actuator/retryevents, we’ll notice retry events.

{
    "retryEvents": [
        {
            "retryName": "order-service",
            "type": "RETRY",
            "creationTime": "2024-04-07T21:57:17.496280+09:00[Asia/Tokyo]",
            "errorMessage": "org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : \"{\"timestamp\":\"2024-04-07T12:57:17.469+00:00\",\"status\":500,\"error\":\"Internal Server Error\",\"path\":\"/addresses/1001111\"}\"",
            "numberOfAttempts": 1
        },
        {
            "retryName": "order-service",
            "type": "RETRY",
            "creationTime": "2024-04-07T21:57:18.552293+09:00[Asia/Tokyo]",
            "errorMessage": "org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : \"{\"timestamp\":\"2024-04-07T12:57:18.544+00:00\",\"status\":500,\"error\":\"Internal Server Error\",\"path\":\"/addresses/1001111\"}\"",
            "numberOfAttempts": 2
        },
        {
            "retryName": "order-service",
            "type": "ERROR",
            "creationTime": "2024-04-07T21:57:19.601539+09:00[Asia/Tokyo]",
            "errorMessage": "org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : \"{\"timestamp\":\"2024-04-07T12:57:19.591+00:00\",\"status\":500,\"error\":\"Internal Server Error\",\"path\":\"/addresses/1001111\"}\"",
            "numberOfAttempts": 3
        }
    ]
}

The system made three attempts on encountering a HttpServerErrorException before eventually returning an error.

After shutting down the address-service, accessing the API at http://localhost:1010/orders?orderNumber=394627b2 will result in an error. Upon refreshing the actuator link at http://localhost:1010/actuator/retryevents, you’ll notice an error labeled as “IGNORED_ERROR” with type ResourceAccessException. This occurs because we’ve configured the system to skip retries when encountering a ResourceAccessException.

{
    "retryEvents": [
        ...,
        {
            "retryName": "order-service",
            "type": "IGNORED_ERROR",
            "creationTime": "2024-04-07T22:02:43.785254+09:00[Asia/Tokyo]",
            "errorMessage": "org.springframework.web.client.ResourceAccessException: I/O error on GET request for \"http://localhost:9090/addresses/2100001\": Connection refused",
            "numberOfAttempts": 0
        }
    ]
}

We have just explored the concept of retry and conducted a brief demonstration to observe its behavior.

Hope you can find something useful!

The completed source code can be found in this GitHub repository: https://github.com/buingoctruong/retry-pattern-spring-boot