nginx Reverse Proxy Server Forwarding 502 Bad Gateway

[hans345hamburg]

Hello,

I don’t know, if this is the correct forum. I am working in a proxmox environment, setting up a ngnix reverse proxy (192.168.178.103) forwarding requests via https to a nginx backend server (192.168.178.105). On the backend server shellinabox is installed. Request from the internet are encrypted via a Letsentcrypt certificate. For the encryption to the backend server I use a self-signed certificate.

When I want to open the next-shell.example.com I get an 502 Bad Gateway error
On the reverse proxy are the following configs

HttpGateway

server {

listen 80 default_server;

listen [::]:80 default_server;

server_name nextcloud.example.com shellinabox.example.com netdata.example.com px.example.com proxy-shell.example.com next-shell.example.com 192.168.178.103;

root /var/www;


location ^~ /.well-known/acme-challenge {

default_type text/plain;

root /var/www/letsencrypt;
}

   location / {

 return 301 https://$host$request_uri;

   }

}

next-shell.example.com

server {

listen 443 ssl 
server_name next-shell.example.com;

       # SSL configuration

       # RSA certificates
       ssl_certificate /etc/letsencrypt/next-shell.example.com/rsa/fullchain.pem;

ssl_certificate_key /etc/letsencrypt/next-shell.example.com/rsa/key.pem;
# ECC certificates

ssl_certificate /etc/letsencrypt/next-shell.example.com/ecc/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/next-shell.example.com/ecc/key.pem;


#
# SSL Configuration
#

# Not using TLSv1 will break:
# Android <= 4.4.40 IE <= 10 IE mobile <=10
# Removing TLSv1.1 breaks nothing else!
ssl_protocols TLSv1.2 TLSv1.3;

# SSL ciphers: RSA + ECDSA

# Two certificate types (ECDSA, RSA) are needed.
ssl_ciphers 'TLS-CHACHA20-POLY1305-SHA256:TLS-AES-256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384';


# Diffie-Hellman parameter for DHE ciphersuites, recommended 4096 bits
ssl_dhparam /etc/nginx/dhparams/dhparams.pem;

# Use multiple curves.
ssl_ecdh_curve secp521r1:secp384r1;

# Server should determine the ciphers, not the client
ssl_prefer_server_ciphers on;

# SSL session handling
ssl_session_timeout 1d;

ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;


# DNS resolver
resolver 192.168.178.1;



#
# Header configuration
#  

# HSTS (ngx_http_headers_module is required) In order to be recoginzed by SSL test, there must be an index.hmtl in the server's root
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload;" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Robots-Tag none always;
add_header X-Download-Options noopen always;
add_header X-Permitted-Cross-Domain-Policies none always;
add_header Referrer-Policy no-referrer always;

add_header X-Frame-Options "SAMEORIGIN" always;

# Disable FLoC
add_header Permissions-Policy "interest-cohort=()";

# Remove X-Powered-By, which is an information leak
fastcgi_hide_header X-Powered-By;


location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_ssl_certificate /etc/selfcerts/stern-example-cert-chain.pem;
proxy_ssl_certificate_key /etc/selfcerts/stern-example-key.pem;
proxy_ssl_verify off;
proxy_pass https://192.168.178.105:4200;
}
}

On the backend server there is the following config

next-shell.example.com

server {

listen 192.168.178.105:4200;
server_name next-shell.example.com;

#
# Header configuration
#  

# HSTS (ngx_http_headers_module is required) In order to be recoginzed by SSL test, there must be an index.hmtl in the server's root
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload;" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;

add_header X-Robots-Tag none always;

add_header X-Download-Options noopen always;

add_header X-Permitted-Cross-Domain-Policies none always;
add_header Referrer-Policy no-referrer always;
add_header X-Frame-Options "SAMEORIGIN" always;

# Disable FLoC
add_header Permissions-Policy "interest-cohort=()";

# Remove X-Powered-By, which is an information leak
fastcgi_hide_header X-Powered-By;


ssl_certificate /etc/selfcerts/stern-example-cert-chain.pem;

ssl_certificate_key /etc/selfcerts/stern-example-key.pem;


location / {
rewrite ^/shellinabox/(.*) /$1 break;
proxy_pass http://127.0.0.1:4200;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 350;

proxy_connect_timeout 350;


}
}

When I try to open the page there is this error in the nginx error log

[error] 1103#1103: *1 SSL_do_handshake() failed (SSL: error:0A00010B:SSL routines::wrong version

number) while SSL handshaking to upstream, client: 95.116.52.151, server: next-shell.example.com, request: "GET /f

avicon.ico HTTP/2.0", upstream: "https://192.168.178.105:4200/favicon.ico", host: "next-shell.example.com"

Has anybody an idea?

[LHammonds]

Why do you have both RSA and ECC certificates? Doesn't the server only utilize the most-recent certificate?

When you created the external Let's Encrypt certificate, I am certain you used a domain name. But how did you create the internal certificate? Do you use the internal IP address of the destination server or did you use an internal DNS name?

The "502 Bad Gateway" error is generally a server communication problem. Make sure your software firewall ports are open on the defined ports you are using (port 4200 for backend, 443 for proxy). The next thing I'd try is removing the encryption between the proxy and the backend and see if it works which would indicate the issue is in the SSL configuration.

I have not used httpgateway so I cannot recommend anything specific.

LHammonds

[TheFu]

Which versions of Ubuntu and nginx are you using?

$ nginx -v
nginx version: nginx/1.14.0 (Ubuntu)

I have a setup for 18.04, but will be moving to 20.04 soon.
I'm not seeing an 'upstream' stanza, which is how I do it. BTW, I don't do encryption between the reverse-proxy and backend servers.

# #########################################
upstream nc-proxy {
        server 172.22.22.33:80;
}
server {
   listen   443 ssl ;
   server_name  nc.example.com;
   include includes/log_format ;

   access_log  /var/log/nginx/nc.access.log;
   error_log  /var/log/nginx/nc.error.log;

   index  index.php index.html ;
   ssl_certificate  /etc/nginx/ssl/nc.example.com/fullchain.pem;
   ssl_certificate_key  /etc/nginx/ssl/nc.example.com/key.pem;
   ssl_trusted_certificate  /etc/nginx/ssl/nc.example.com/cert.pem;

   # Allow and deny all
   include includes/allow_lan_only;

   # Let's Encrypt webroot
   include includes/letsencrypt-webroot;

....
   location / {
       try_files index.html @nc-proxy ;
    }
    location @nc-proxy {
       proxy_set_header  X-Real-IP  $remote_addr;
       proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header  Host $host;
       proxy_cache       one;
       proxy_pass http://nc-proxy;
     }
}

Let's Encrypt certbot didn't work until I changed to acme.sh for management. Probably my ignorance. With acme.sh, I use the stand-alone cert request/update. For about 20 websites - I keep all the certs separate - it takes less than 90 seconds. During that time, nginx is offline. Only static websites can be updated while nginx is online, IME.

LE requires access from at least 3 different parts of the world before they issue certs. I had restrictive firewall rules and LE refused to issue any certs. Because LE moves the IPs around, it wasn't possible to just allow those specific IPs, or at least I wasn't able to easily figure a solution for private servers. In the end, I disable the firewall while the certs are being updated.