Load balancers make horizontal scaling and high availability possible. You put several servers behind one address, and the load balancer spreads requests across them. When a server stops answering its health check, the load balancer takes it out of rotation until it recovers, so users never reach it.
Layer 4 vs. layer 7
- Layer 4 (transport). Routes connections by IP address and port, over TCP or UDP, without reading the request. It is fast and works with any protocol. AWS Network Load Balancer is an example.
- Layer 7 (application). Reads HTTP requests, including host, path, headers and cookies, so it can send
/apito one pool and/staticto another, terminate TLS and rewrite headers. NGINX, HAProxy, Envoy and AWS Application Load Balancer work at this layer.
Common algorithms
- Round robin: each server gets the next request in turn. Weighted round robin sends more to bigger servers.
- Least connections: sends the request to the server with the fewest active connections, which suits long or uneven requests.
- Hash-based: hashes the client IP or a key so the same client keeps reaching the same server (session affinity). Consistent hashing limits reshuffling when servers are added or removed.
- Random and least response time are offered by some products.
Types of load balancer
- Hardware appliances in data centers, such as F5 BIG-IP.
- Software you run yourself: NGINX, HAProxy, Envoy.
- Cloud-managed: the AWS Elastic Load Balancing family, Google Cloud Load Balancing, Azure Load Balancer and Application Gateway.
- DNS-based or global load balancing, which spreads users across regions.
Inside Kubernetes, a Service load-balances traffic across pods, and an Ingress controller or gateway handles traffic from outside the cluster.
What to monitor
- Request rate and active connections
- Latency at the load balancer and from each backend
- Error rates, split into errors from backends and errors the load balancer generates itself:
502 Bad Gateway(bad response from a backend),503 Service Unavailable(no healthy backend) and504 Gateway Timeout(backend too slow) - Healthy vs. unhealthy backend count
- Uneven distribution, where one backend gets far more traffic than the others
- TLS certificate expiry on HTTPS listeners
Load balancer monitoring with Atatus
Infrastructure monitoring includes plugins for NGINX and HAProxy, and NGINX monitoring collects connection and request metrics and parses access and error logs into fields such as status code and URL path. Uptime monitoring checks the public endpoint and its SSL certificate from several locations, and APM shows how backend response times change behind the load balancer.