riptide-failsafe

Riptide: Failsafe

Riptide: Failsafe adds Failsafe support to Riptide. It offers retries and a circuit breaker to every remote call.

Example

Http.builder().requestFactory(new HttpComponentsClientHttpRequestFactory())
    .plugin(new FailsafePlugin()
        .withPolicy(circuitBreaker)
        .withPolicy(new RetryRequestPolicy(retryPolicy)))
    .build();

Features

• seamlessly integrates Riptide with Failsafe

Dependencies

• Riptide Core
• Failsafe

Installation

Add the following dependency to your project:

<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>riptide-failsafe</artifactId>
    <version>${riptide.version}</version>
</dependency>

Configuration

The failsafe plugin will not perform retries nor apply circuit breakers unless they were explicitly configured:

Http.builder().requestFactory(new HttpComponentsClientHttpRequestFactory())
    .plugin(new FailsafePlugin()
        .withPolicy(
            new RetryRequestPolicy(
                RetryPolicy.<ClientHttpResponse>builder()
                    .withDelay(Duration.ofMillis(25))
                    .withDelayFn(new RetryAfterDelayFunction(clock))
                    .withMaxRetries(4)
                    .build())
                .withListener(myRetryListener))
        .withPolicy(
            CircuitBreaker.<ClientHttpResponse>builder()
                .withFailureThreshold(3, 10)
                .withSuccessThreshold(5)
                .withDelay(Duration.ofMinutes(1))
                .build()))
    .build();

Please visit the Failsafe readme in order to see possible configurations.

Retries

What retry config controls: The RetryPolicy controls timing, backoff, jitter, and limits. Enabling retry config does not make Riptide automatically retry arbitrary 4xx or 5xx responses.

How retries are triggered:

• Socket faults (e.g. read timeout) — Riptide retries automatically, but only for safe and idempotent methods. Non-idempotent requests are not retried on socket timeout unless you explicitly mark them idempotent.
• Connection faults (e.g. connection refused) — When using the Spring Boot auto-config path with transient-fault-detection.enabled: true, connection faults are retried for all methods (idempotent or not), because a connection that was never established means the request never reached the server. Without that auto-configured transient fault detection path, configure a retry policy for connection faults explicitly.
• Response-based retries — You must trigger these explicitly, either by calling retry() in your routing callback or by throwing RetryException. Receiving a 503 does not cause a retry by itself.

Retry-After and X-RateLimit-Reset headers influence the delay between retries once a retry has already been triggered; they do not make a response eligible for retry on their own.

Beware when using retryOn to retry conditionally on certain exception types. You'll need to register RetryException in order for the retry() route to work:

RetryPolicy.<ClientHttpResponse>builder()
    .handle(SocketTimeoutException.class)
    .handle(RetryException.class)
    .build();

By default, you can use RetryException in your routes to retry the request:

retryClient.get()
    .dispatch(
        series(), on(CLIENT_ERROR).call(
            response -> {
                if (specificCondition(response)) {
                    throw new RetryException(response); // we will retry this one
                }  else {
                    throw new AnyOtherException(response); // we wont retry this one
                }
            }
        )
    ).join()

To explicitly retry a 503 Service Unavailable:

http.get("/users/me")
    .dispatch(series(),
        on(SUCCESSFUL).call(User.class, this::greet),
        on(SERVER_ERROR).dispatch(status(),
            on(SERVICE_UNAVAILABLE).call(retry())),
        anySeries().call(problemHandling()))

Failsafe supports dynamically computed delays using a custom function.

Riptide: Failsafe offers implementations that understand:

• Retry-After (RFC 7231, section 7.1.3)
• X-RateLimit-Reset (RESTful API Guidelines)

Http.builder().requestFactory(new HttpComponentsClientHttpRequestFactory())
    .plugin(new FailsafePlugin()
        .withPolicy(RetryPolicy.<ClientHttpResponse>builder()
            .withDelayFn(new CompositeDelayFunction<>(Arrays.asList(
                new RetryAfterDelayFunction(clock),
                new RateLimitResetDelayFunction(clock)
            )))
            .withMaxDuration(Duration.ofSeconds(5))
            .build()))
    .build();

⚠️ Make sure you you specify a max duration otherwise any value coming from a server that is further ahead in the future will make your retry block practically forever.

Make sure you check out zalando/failsafe-actuator for a seamless integration of Failsafe and Spring Boot.

Timeout policy

You can use org.springframework.http.client.ClientHttpRequestFactory configuration to set up proper connection timeout, socket timeout and connection time to live. In addition you can use FailsafePlugin with dev.failsafe.Timeout policy to control the entire duration from sending the request to processing the response. See the use cases in the FailsafePluginTimeoutTest test.

Configuration example:

 Http.builder().requestFactory(new HttpComponentsClientHttpRequestFactory())
                .plugin(new FailsafePlugin()
                        .withPolicy(Timeout.of(Duration.ofSeconds(5))))
                .build();

Backup Requests

The BackupRequest policy implements the backup request pattern, also known as hedged requests:

Http.builder().requestFactory(new HttpComponentsClientHttpRequestFactory())
    .plugin(new FailsafePlugin()
        .withPolicy(new BackupRequest(1, SECONDS)))
    .build();

Custom executor

The withExecutor method allows to specify a custom ExecutorService being used to perform asynchronous executions and listen for callbacks:

Http.builder().requestFactory(new HttpComponentsClientHttpRequestFactory())
    .plugin(new FailsafePlugin()
        .withPolicy(
            CircuitBreaker.<ClientHttpResponse>builder()
                .withFailureThreshold(3, 10)
                .withSuccessThreshold(5)
                .withDelay(Duration.ofMinutes(1))
                .build())
        .withExecutor(Executors.newFixedThreadPool(2)))
    .build();

If no executor is specified, the default executor configured by Failsafe is used. See Failsafe DelegatingScheduler class, and also Failsafe documentation for more information.

Beware when specifying a custom ExecutorService:

1. The ExecutorService should have a core pool size or parallelism of at least 2 in order for timeouts to work
2. In general, it is not recommended to specify the same ExecutorService for multiple Http clients

Usage

Given the failsafe plugin was configured as shown in the last section: A regular call like the following will now be retried up to 4 times if the server did not respond within the socket timeout.

http.get("/users/me")
    .dispatch(series(),
        on(SUCCESSFUL).call(User.class, this::greet),
        anySeries().call(problemHandling()))

Handling certain technical issues automatically, like socket timeouts, is quite useful. But there might be cases where the server did respond, but the response indicates something that is worth retrying, e.g. a 409 Conflict or a 503 Service Unavailable. Use the predefined retry route that comes with the failsafe plugin:

http.get("/users/me")
    .dispatch(series(),
        on(SUCCESSFUL).call(User.class, this::greet),
        on(CLIENT_ERROR).dispatch(status(),
            on(CONFLICT).call(retry())),
        on(SERVER_ERROR).dispatch(status(),
            on(SERVICE_UNAVAILABLE).call(retry())),
        anySeries().call(problemHandling()))

Safe and Idempotent methods

Only safe and idempontent methods are retried by default. The following request methods can be detected:

• Standard HTTP method
• HTTP method override
• Conditional Requests
• Idempotency-Key header

You also have the option to declare any request to be idempotent by setting the respective request attribute. This is useful in situation where none of the options are above would detect it but based on the contract of the API you may know that a certain operation is in fact idempotent.

http.post("/subscriptions/{id}/cursors", subscriptionId)
    .attribute(MethodDetector.IDEMPOTENT, true)
    .header("X-Nakadi-StreamId", streamId)
    .body(cursors)
    .dispatch(series(),
        on(SUCCESSFUL).call(pass()),
        anySeries().call(problemHandling()))

In case those options are insufficient you may specify your own method detector:

Http.builder().requestFactory(new HttpComponentsClientHttpRequestFactory())
    .plugin(new FailsafePlugin()
        .withPolicy(retryPolicy)
        .withDecorator(new CustomIdempotentMethodDetector()))
    .build();

Getting Help

If you have questions, concerns, bug reports, etc., please file an issue in this repository's Issue Tracker.

Getting Involved/Contributing

To contribute, simply open a pull request and add a brief description (1-2 sentences) of your addition or change. For more details, check the contribution guidelines.

Credits and references

• Jeffrey Dean: Achieving Rapid Response Times in Large Online Services
• Jeffrey Dean and Luiz André Barroso: The Tail at Scale
• Uwe Friedrichsen: Patterns of Resilience - Fan Out, Quickest Reply