How to force Nginx to use Upstream even if it fails?

Using Nginx as a reverse proxy in a containerized environment might be challenging.

Photo by Nathan Dumlao on Unsplash

Consider the following cluster topology:

Consider you have the following scenario

Your cluster serves content for web1.com and web2.com.

Requests for web1.com are failing due to an internal problem in the container.

However, everything is fine with the web2.com’s containers.

Because the requests for web1.com are failing, Nginx will mark the upstream IP addresses as failed and cease sending requests to those IP addresses.

Nginx will show the “502 Bad Gateway” message to the users.

Web2.com can no longer be reached either.

Because of a problem with the web1.com container, it is no longer possible to reach web2.com. Because they are both behind the same IP addresses, and Nginx has marked those IP addresses as failed.

We need to force Nginx to send network traffic to upstream IP addresses no matter what the state of those IP addresses is.

We can force Nginx using these upstream parameters:

fail_timeout=0
max_fails=0

This is an example of a Nginx configuration:

upstream mycluster {
    server 10.10.0.1:8080 fail_timeout=0 max_fails=0;
    server 10.10.0.2:8080 fail_timeout=0 max_fails=0;
}
server {
    listen 80;
    server_name _;
    location / {
        proxy_pass http://mycluster;
        proxy_set_header Host $http_host;
    }
}

These settings are dangerous. Even if the web1.com container is unable to accept new connections, it will continue to receive requests.

P.S.: This post is based on an actual production problem.