When I open my rails app, I get a 504-Gateway timeout error.
Result of tail /var/log/nginx/error.log:
upstream timed out (110: Connection timed out) while reading response header from upstream, client: 122.169.153.194, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "162.243.204.173"
2014/01/27 12:38:05 [error] 621#0: *63 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 122.169.153.194, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "162.243.204.173"
2014/01/27 13:05:53 [error] 621#0: *73 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 122.169.153.194, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "162.243.204.173"
2014/01/27 13:16:28 [error] 621#0: *77 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 122.169.153.194, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "162.243.204.173"
2014/01/27 13:18:49 [error] 621#0: *80 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 122.169.153.194, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "162.243.204.173"
Then, the content of tail /home/unicorn/log/unicorn.log:
* Restarting Unicorn web server unicorn [ OK ]
root#liquorexam:/home/rails# tail /home/unicorn/log/unicorn.log
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/lib/unicorn /http_server.rb:303:in `join'
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/unicorn-4.7.0/bin/unicorn:126:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.0.0-p353/bin/unicorn:23:in `load'
/usr/local/rvm/gems/ruby-2.0.0-p353/bin/unicorn:23:in `<main>'
/usr/local/rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `<main>'
E, [2014-01-27T13:16:14.982896 #20809] ERROR -- : reaped #<Process::Status: pid
21135 exit 1> worker=0
I, [2014-01-27T13:16:14.983194 #20809] INFO -- : worker=0 spawning...
I, [2014-01-27T13:16:14.988711 #21141] INFO -- : worker=0 spawned pid=21141
I, [2014-01-27T13:16:15.019707 #21141] INFO -- : Refreshing Gem list
Your backend unicorn worker process is crashing, but you only including the bottom part of the stack trace. The complete stack trace in the unicorn.log file will show you what in your application code is crashing. I would guess bad middleware or a malformed response.
Related
I have set up API Umbrella in my Ubuntu 20 cloud vm.
Try to access but got 502 Bad gateway like in here:
Obviously, the routing is a failure for some reason.
The output of /var/log/api-umbreall/nginx/current is the following:
2022-09-01T06:08:19.57992 starting nginx...
2022-09-01T06:08:27.48168 2022/09/01 06:08:27 [error] 319#0: *13 [lua] elasticsearch_setup.lua:106: create_aliases(): failed to create elasticsearch index: Unsuccessful response: {"error":{"root_cause":[{"type":"index_already_exists_exception","reason":"already exists","index":"api-umbrella-logs-v1-2022-09"}],"type":"index_already_exists_exception","reason":"already exists","index":"api-umbrella-logs-v1-2022-09"},"status":400}, context: ngx.timer
2022-09-01T06:21:45.17055 2022/09/01 06:21:45 [warn] 318#0: *39756 using uninitialized "x_api_umbrella_request_id" variable while logging request, client: 192.241.213.X, server: mydomain.city, request: "GET / HTTP/1.1", host: "150.230.240.y:443"
2022-09-01T06:32:42.70713 2022/09/01 06:32:42 [error] 318#0: *72162 connect() failed (111: Connection refused) while connecting to upstream, client: 185.14.196.Z, server: mydomain.city, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:14009/", host: "mydomain.city"
I'm trying to open ws connection, but get the error:
val openConnection = exec( ws("Connect -> WS").wsName("user").connect("wss://socket.develop.test.com?access_token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9"))
Then, I get:
09:49:53.343 [DEBUG] i.g.h.c.i.DefaultHttpClient - Installing SslHandler for wss://socket.develop.test.com?access_token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9
09:49:53.381 [DEBUG] i.n.h.s.u.InsecureTrustManagerFactory - Accepting a server certificate: CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co
09:49:53.418 [DEBUG] i.n.h.c.h.w.WebSocketClientHandshaker13 - WebSocket version 13 client handshake key: YndmXwIGgZseWyRlmXBoyw==, expected response: H2YGNn6p+DyLyudnc1JCluHjj7E=
09:49:53.419 [DEBUG] i.g.h.c.i.WebSocketHandler - ctx.write msg=DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: EmptyByteBufBE)
GET ?access_token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9
HTTP/1.1
accept: */*
upgrade: websocket
connection: upgrade
sec-websocket-key: YndmXwIGgZseWyRlmXBoyw==
host: socket.develop.test.com
origin: https://socket.develop.test.com
sec-websocket-version: 13
09:49:53.423 [DEBUG] i.n.h.s.SslHandler - [id: 0xfa53644a, L:/192.168.150.134:54082 - R:socket.develop.test.com/78.47.16.48:443] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
09:49:53.450 [DEBUG] i.g.h.c.i.WebSocketHandler - Read msg=HttpObjectAggregator$AggregatedFullHttpResponse(decodeResult: success, version: HTTP/1.1, content: CompositeByteBuf(ridx: 0, widx: 163, cap: 163, components=1))
HTTP/1.1 400 Bad Request
Server: openresty/1.15.8.1
Date: Tue, 10 Mar 2020 06:49:53 GMT
Content-Type: text/html
Content-Length: 163
Connection: close
09:49:53.452 [DEBUG] i.g.h.c.i.WebSocketHandler - Crash
io.netty.handler.codec.http.websocketx.WebSocketHandshakeException: Invalid handshake response getStatus: 400 Bad Request
at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:267)
In gatling.conf, I set also:
enableSni = false
useOpenSsl = false
enableHostnameVerification = false
If I try to open connection with any WS clients like SmartWebsocketClient - everything is ok, but by means of gatling I can't to open connection. Any ideas to fix it?
wss://domain?access_token... is not a valid URL. The path component must start with a / and not a ?. This means that the request is invalid which can explain the response of 400 Bad Request.
The URL must be at least wss://domain/?access_token... (i.e. a / before the ?) but maybe the rest of the URL is unexpected by the server too. Please check the actual requirements of the server.
This is a bug in Netty that doesn't properly compute WebSocket handshake request url when path is empty.
I've just contributed a patch: https://github.com/netty/netty/pull/10095.
Sibling issue in Gatling: https://github.com/gatling/gatling/issues/3876
I'm hosting a Ghost blog using Digital Ocean. My droplet is Ubuntu Ghost 0.8.0 on 14.04.
Yesterday I successfully installed a TLS/SSL certificate from LetsEncrypt in order to enable HTTPS. The site was working fine then and this morning.
Today I uploaded a new Ghost theme and restarted Ghost in order to access it. I now get the response 502 Bad Gateway when I try to access the site.
Each request for the site adds an instance of the following errors to mynginx error log.
Would someone walk me through what these 2 error messages mean? I'd really appreciate it.
Please note that I've substituted my actual domain name with example.com
2016/06/16 17:28:45 [error] 8125#0: *13 connect() failed (111: Connection refused) while connecting to upstream, client: 98.247.253.8, server: example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:2368/favicon.ico", host: “example.com”, referrer: "https://example.com/“
2016/06/16 17:30:14 [error] 8125#0: *18 connect() failed (111: Connection refused) while connecting to upstream, client: 98.247.253.8, server: example.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:2368/", host: “example.com”
I get this error when I try to use nginx as a reverce proxy server for apache. I guess I have something wrong with nginx side of configuration because when I access the web-host with port I use for apache, everything works nice.
So, here is my nginx configuration file for host:
server{
listen 80;
server_name my-site;
root /usr/share/nginx/www;
index index.html index.htm;
location /static {
alias /var/www/my-site/myapp/static;
}
location /media {
alias /var/www/my-site/myapp/media;
}
location / {
proxy_pass http://my-site:8000;
}
}
Here's my error log for nginx, it wasn't any useful for me though:
2014/02/22 13:58:37 [error] 1398#0: *10 upstream timed out (110: Connection timed out) while connecting to upstream, client: XXX.XXX.XXX.XXX, server: my-site, request: "GET / HTTP/1.1", upstream: "http://XXX.XXX.XXX.XXX:8000/", host: "my-site"
2014/02/22 14:01:40 [error] 1398#0: *14 upstream timed out (110: Connection timed out) while connecting to upstream, client: XXX.XXX.XXX.XXX, server: my-site, request: "GET / HTTP/1.1", upstream: "http://XXX.XXX.XXX.XXX:8000/", host: "my-site"
2014/02/22 14:10:29 [error] 1609#0: *1 upstream timed out (110: Connection timed out) while connecting to upstream, client: XXX.XXX.XXX.XXX, server: my-site, request: "GET / HTTP/1.1", upstream: "http://XXX.XXX.XXX.XXX:8000/", host: "my-site"
2014/02/22 19:07:59 [error] 647#0: *176 upstream timed out (110: Connection timed out) while connecting to upstream, client: XXX.XXX.XXX.XXX, server: my-site, request: "GET / HTTP/1.1", upstream: "http://XXX.XXX.XXX.XXX:8000/", host: "my-site"
2014/02/23 13:40:35 [error] 647#0: *650 upstream timed out (110: Connection timed out) while connecting to upstream, client: XXX.XXX.XXX.XXX, server: my-site, request: "GET / HTTP/1.1", upstream: "http://XXX.XXX.XXX.XXX:8000/", host: "my-site"
I've read somewhere that it may happen when load is too big, but in my case this is definitely not it.
Is your Apache bound to port 8000?
Have you made sure that you can access port 8000 with the external IP address associated to your domain? You can do this by retrieving a page with the http://my-site:8000 URL on the server.
You could try using
proxy_pass 127.0.0.1:8000;
instead of my-site:8000;
This way you don't need the ability to connect to your site via your external IP.
It appears that only one passenger instance serves requests, the others just serve 502 errors. This causes an intermittent error pattern because only the requests that are directed to the second instance fail.
~$ rvmsudo passenger-status
----------- General information -----------
max = 4
count = 2
active = 0
inactive = 2
Waiting on global queue: 0
----------- Application groups -----------
/u/apps/pixie.strd6.com/current:
App root: /u/apps/pixie.strd6.com/current
* PID: 3179 Sessions: 0 Processed: 121 Uptime: 3m 57s
* PID: 3762 Sessions: 0 Processed: 0 Uptime: 2s
This happened after updating to Rails 3.1.0 rc5
2011/07/27 21:37:37 [error] 3125#0: *608 upstream prematurely closed connection while reading response header from upstream, client: 68.226.71.148, server: pixieengine.com, request: "GET /chats/recent HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "pixieengine.com", referrer: "http://pixieengine.com/projects/426/ide"
2011/07/27 21:38:31 [error] 3125#0: *596 upstream prematurely closed connection while reading response header from upstream, client: 76.102.14.57, server: pixieengine.com, request: "GET /chats/recent HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "pixieengine.com", referrer: "http://pixieengine.com/pixel-editor"
2011/07/27 21:39:12 [error] 3125#0: *576 upstream prematurely closed connection while reading response header from upstream, client: 68.8.173.234, server: pixieengine.com, request: "GET /chats/recent HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "pixieengine.com", referrer: "http://pixieengine.com/community/forums/1"
2011/07/27 21:39:12 [error] 3125#0: *687 upstream prematurely closed connection while reading response header from upstream, client: 201.231.103.247, server: pixieengine.com, request: "GET /chats/active_users HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "pixieengine.com", referrer: "http://pixieengine.com/projects/demo/ide"
2011/07/27 21:39:12 [error] 3125#0: *686 upstream prematurely closed connection while reading response header from upstream, client: 201.231.103.247, server: pixieengine.com, request: "GET /chats/recent HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "pixieengine.com", referrer: "http://pixieengine.com/projects/demo/ide"
I solved this by switching to Unicorn.
I wasn't actually able to figure out how to fix passenger, but I was able to verify that it broke because of the transition from Rails 3.0.9 to 3.1.0.rc5.
The temporary fix is to use PassengerSpawnMethod conservative in passenger, which disables forking running processes.