Module balance

Middleware that allows balancing load among multiple services.

In larger systems, multiple endpoints are often available for a given service. As load increases, you want to ensure that that load is spread evenly across the available services. Otherwise, clients could see spikes in latency if their request goes to a particularly loaded service, even when spare capacity is available to handle that request elsewhere.

This module provides the p2c middleware, which implements the “Power of Two Random Choices” algorithm. This is a simple but robust technique for spreading load across services with only inexact load measurements. Use this if the set of available services is not within your control, and you simply want to spread load among that set of services.

Examples

use tower::balance::p2c::Balance;
use tower::load::Load;
use tower::{Service, ServiceExt};
use futures_util::pin_mut;

async fn spread<Req, S: Service<Req> + Load>(svc1: S, svc2: S, reqs: impl Stream<Item = Req>)
where
    S::Error: Into<tower::BoxError>,
    S::Metric: std::fmt::Debug,
{
    // Spread load evenly across the two services
    let p2c = Balance::new(tower::discover::ServiceList::new(vec![svc1, svc2]));

    // Issue all the requests that come in.
    // Some will go to svc1, some will go to svc2.
    pin_mut!(reqs);
    let mut responses = p2c.call_all(reqs);
    while let Some(rsp) = responses.next().await {
        // ...
    }
}

Modules

• error
  Error types for the tower::balance middleware.
• p2c
  This module implements the “Power of Two Random Choices” load balancing algorithm.