I am trying to send logs from Filebeat to Logstash trought Nginx. I have each Elasticsearch and Logstash behind nginx reverse proxy which they are on same server. When I send logs from Filebeat directly to the Elasticsearch proxy it works fine, however when I try send them to the Logstash nothing happend and no errors are generated.
Log goes: Filebeat -> NGINX_PROXY_FQDN -> ES_SERVER_FQDN:5044 -> ES_SERVER_FQDN:9200
Filebeat.yml
output.logstash:
hosts: ["NGINX_PROXY_FQDN:443"]
protocol: https
ssl.certificate: "/etc/filebeat/certs/nginxproxy.crt"
ssl.key: "/etc/filebeat/certs/nginxproxy.key"
ssl.certificate_authorities: /etc/filebeat/certs/ca.pem
Nginx, Logstash reverse proxy
server {
listen 443;
listen [::]:443;
ssl on;
ssl_certificate certs/nginxproxy.crt;
ssl_certificate_key certs/nginxproxy.key;
ssl_trusted_certificate certs/ca.pem;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
server_name NGINX_PROXY_FQDN;
location / {
proxy_pass https://ES_SERVER_FQDN:5044;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Proxy-Connection "upgrade";
proxy_set_header Host $host;
}
}
Logstash.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/logstash/certs/nginxproxy.crt"
ssl_key => "/etc/logstash/certs/nginxproxy.key"
ssl_certificate_authorities => ["/etc/logstash/certs/ca.pem"]
ssl_verify_mode => "force_peer"
}
}
output {
elasticsearch {
hosts => ["https://ES_SERVER_FQDN:9200"]
index => "%{[#metadata][beat]}-%{[#metadata][version]}-%{+YYYY.MM.dd}"
user => "*****"
password => "*****"
ssl => true
cacert => "/etc/logstash/certs/ca.pem"
ilm_enabled => false
}
}
Related
I'm deploying an OKE kubernetes cluster which uses crio as container runtime. I have already configured a reverse proxy nginx for a private registry (with nexus3, and outside the cluster) and tried and succeed locally pulling, pushing and running containers only with docker login <registry.com>.
However when I try to pull those docker images within the cluster it fails with the following message:
Exec commands:
crioctl pull <domain.com.br>/imageName:tag --creds user:pass
error creating build container: Error initializing source : error pinging docker registry : invalid status code from registry 404 (Not Found)
Following the nginx.conf:
worker_processes 2;
events {
worker_connections 1024;
}
http {
client_max_body_size 0;
error_log /var/log/nginx/error.log warn;
access_log /dev/null;
proxy_intercept_errors off;
proxy_send_timeout 120;
proxy_read_timeout 300;
upstream nexus {
server nexus-registry:8081;
}
upstream registry {
server nexus-registry:5000;
}
server {
listen 80;
server_name <my-domain>;
keepalive_timeout 5 5;
proxy_buffering off;
location ~ /.well-known/acme-challenge{
allow all;
root /usr/share/nginx/html/letsencrypt;
}
location / {
return 301 https://<my-domain>$request_uri;
}
}
server {
listen 443 ssl;
server_name <my-domain>;
ssl on;
server_tokens off;
ssl_certificate /etc/nginx/ssl/live/<my-domain>/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/live/<my-domain>/privkey.pem;
ssl_dhparam /etc/nginx/dhparam/dhparam-2048.pem;
ssl_buffer_size 8k;
ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;
location / {
#redirect to docker registry
if ($http_user_agent ~ docker ) {
proxy_pass http://registry;
}
proxy_pass http://nexus;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}
The problem was because of the client which was trying to pull images from nexus registry. I had to add cri-o with docker so it worked.
if ($http_user_agent ~ (docker|cri-o) )
I tried to configure a Websocket proxy on my Nginx server, but unfortunately, I don't get it working. I have read various forms but cannot get out of it. I think it has something to do between the client connection to the server. Local om my pc is everything working fine
client code:
var port = new osc.WebSocketPort({
url: "wss://circusfamilyprojects.nl/"
});
server code:
var wss = new WebSocket.Server({
port: 8083
});
This is my configuration in Nginx
# custom code for hop by hop headers
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
#custom code for connection with websocket this is the port on the server
upstream websocket {
server 178.62.209.37:8083;
}
server {
listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/circusfamilyprojects.nl/fullchain.pem; # managed by Cert$
ssl_certificate_key /etc/letsencrypt/live/circusfamilyprojects.nl/privkey.pem; # managed by Ce$
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
root /var/www/html/vue/cfdomotica/server/public;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
server_name circusfamilyprojects.nl www.circusfmailyprojects.nl; # managed by Certbot
location / {
proxy_pass http://websocket;
proxy_set_header HOST $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass_request_headers on;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
}
server {
if ($host = www.circusfamilyprojects.nl) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = circusfamilyprojects.nl) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 8080 ;
listen [::]:8080 ;
server_name circusfamilyprojects.nl;
return 404; # managed by Certbot
}
In my browser I get the message "Status Code: 426 Upgrade Required" I hope someone can help me, thanks in advance!
I solved it my self by separating a server block for port 443 and a server block for the WebSocket client. I did the server block for the client on port 8086.
I have nginx configured to forward traffic to an express server and want to force SSL by redirecting traffic directed at http:// to https://. I've done extensive googling on the subject however can not get this to work. HTTPS is working fine but so is HTTP - the redirect does not appear to be having any effect.
server {
listen 80;
listen [::]:80;
server_name my.domain www.my.domain;
return 301 https://my.domain$request_uri;
}
server {
listen 443 ssl default_server;
listen [::]:443 ssl;
server_name my.domain www.my.domain;
ssl_certificate /home/user/my_domain.crt;
ssl_certificate_key /home/user/my_domain.key;
ssl on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/mydomain.access.log;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass https://localhost:3000;
proxy_read_timeout 90;
proxy_redirect https://localhost:3000 https://my.domain;
}
}
i have also the same problem so I apply these to server to accept only one request which will decide for both http and https
server {
listen 80 ;
listen [::]:80 ;
listen 443 ssl http2 ;
listen [::]:443 ssl http2 ;
server_name example.com www.example.com;
#------ ssl certificates and other config --------
set $https_redirect 0;
#if request came from port 80/http
if ($server_port = 80) {
set $https_redirect 1;
}
# or if the requested host came with www
if ($host ~ '^www\.') {
set $https_redirect 1;
}
#then it will redirects
if ($https_redirect = 1) {
return 301 https://example.com$request_uri;
}
}
by using this I have only server block to hanlde any request
I am attempting to setup SSL with Nginx running on a docker swarm but have run into an issue. Everything looks correct but any request I make just hangs until I get a 502. I made sure to expose port 443 in my compose file. Here is the nginx error I am getting:
*7 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 10.255.0.2, server: subdomain.mysite.com, request: "GET /api-v1/user-login HTTP/2.0", upstream: "https://10.0.0.6:5051/api-v1/user-login", host: "subdomain.mysite.com"
Here is the relevant piece of my nginx default.conf:
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_protocols SSLv3 TLSv1;
upstream siteStage {
ip_hash;
server siteStage:5051;
}
server {
listen 443 ssl http2 ;
server_name subdomain.mysite.com;
ssl on;
ssl_certificate /path/provided.crt;
ssl_certificate_key /path/client.key;
ssl_client_certificate /path/ca.crt;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_verify_client off;
location / {
proxy_ssl_certificate /etc/ssl/client.pem;
proxy_ssl_certificate_key /etc/ssl/client.key;
proxy_ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
proxy_ssl_ciphers HIGH:!aNULL:!MD5;
proxy_ssl_session_reuse on;
proxy_pass https://siteStage/;
}
}
It turns out it was my nginx config. Here is how I finally got it to work:
# No upstream
server {
listen 80;
listen 443 ssl default_server;
server_name subdomain.mysite.com;
ssl on;
ssl_certificate /path/provided.crt;
ssl_certificate_key /path/client.key;
if ($scheme = http) {
return 301 https://$server_name$request_uri;
}
location / {
proxy_pass http://siteStage:5051/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
}
}
I have this nginx configuration
server {
listen 80;
server_name app.com www.app.com;
rewrite ^ https://$server_name$request_uri? permanent;
}
server {
listen 443;
server_name app.com www.app.com;
ssl on;
ssl_certificate /etc/nginx/ssl/app.crt;
ssl_certificate_key /etc/nginx/ssl/server.key;
location = /favicon.ico {
root /opt/myapp/app/programs/web.browser/app;
access_log off;
expires 1w;
}
location ~* "^/[a-z0-9]{40}\.(css|js)$" {
root /opt/myapp/app/programs/web.browser;
access_log off;
expires max;
}
location ~ "^/packages" {
root /opt/myapp/app/programs/web.browser;
access_log off;
}
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
}
}
and deployed to ec2 using mup with normal settings
It is deployed and i can access the site app.com
But the https://app.com is not working
as In the config file all the requests are rewriting to https
What is happening here
I can access the site when I enter app.com that means it is
forwarding app.com ad https://app.com
I cannot access https://app.com that means nginx is not working
Which of the above two scenarios are true?
I'm out of options. I checked with ssl checkers they are showing that ssl certificate is not installed.
then why my app is working when enter app.com?
Now Meteor Up has the built in SSL Support. No more hard work.
Just add the SSL certificates and the key and do mup setup.
We use stud to terminate SSL
I am not NGINX knowledgeable but looking at my working production configs I see a number of parameters you have not included in yours.
In particular you may need the following at the top in order to proxy websocket connections:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
My 443 server also includes the following in addition to what you already have:
server {
ssl_stapling on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
add_header Strict-Transport-Security "max-age=31536000;";
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto http;
proxy_set_header X-Nginx-Proxy true;
proxy_redirect off;
}
}
Finally I would try commenting out your location directives for bug checking. The issue should not be with your SSL certificate, it should still allow you to visit (with a warning) for a self-signed or misconfigured certificate. Hope this helps.