Racking my brain on what I may be missing here... I upgraded to version 3.15.1 and unable to listen on TCP or UDP. Oddly enough, I am able to listen on TCP and communicate over TLS. I have read and re-read the manual (https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-sources-network.html). Running on CentOS 7.
Here is relevant portions of my config:
...
# never listens and have tried using udp() and tcp() sources as
# well as breaking each out to their individual source defs
source s_net {
network(ip(0.0.0.0) port(6514) transport("tcp"));
network(ip(0.0.0.0) port(514) transport("udp"));
};
# works perfectly
source s_net_tls {
network(
ip(0.0.0.0)
port(22514)
transport("tls")
tls(
key-file('/path/to/foo.key')
cert-file('/path/to/foo.pem')
peer-verify(optional-untrusted)
ssl-options(no-sslv2, no-sslv3, no-tlsv1)
)
);
};
...
Output of netstat:
$ netstat -tulna | grep -e 514 -e 6514 -e 22514
tcp 0 0 0.0.0.0:22514 0.0.0.0:* LISTEN
$
Here is the stdout:
$ sudo syslog-ng -Fedv; syslog-ng-ctl verbose --set=on && syslog-ng-ctl trace --set=on && syslog-ng-ctl debug --set=on
... [ OMITTING UNNECESSARY LINES ] ...
[2018-06-23T20:46:47.406200] Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='libafsocket.so', module='afsocket'
[2018-06-23T20:46:47.406363] Registering candidate plugin; module='afsocket', context='source', name='unix-stream'
[2018-06-23T20:46:47.406371] Registering candidate plugin; module='afsocket', context='destination', name='unix-stream'
[2018-06-23T20:46:47.406375] Registering candidate plugin; module='afsocket', context='source', name='unix-dgram'
[2018-06-23T20:46:47.406421] Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram'
[2018-06-23T20:46:47.406427] Registering candidate plugin; module='afsocket', context='source', name='tcp'
[2018-06-23T20:46:47.406432] Registering candidate plugin; module='afsocket', context='destination', name='tcp'
[2018-06-23T20:46:47.406440] Registering candidate plugin; module='afsocket', context='source', name='tcp6'
[2018-06-23T20:46:47.406444] Registering candidate plugin; module='afsocket', context='destination', name='tcp6'
[2018-06-23T20:46:47.406448] Registering candidate plugin; module='afsocket', context='source', name='udp'
[2018-06-23T20:46:47.406452] Registering candidate plugin; module='afsocket', context='destination', name='udp'
[2018-06-23T20:46:47.406456] Registering candidate plugin; module='afsocket', context='source', name='udp6'
[2018-06-23T20:46:47.406460] Registering candidate plugin; module='afsocket', context='destination', name='udp6'
[2018-06-23T20:46:47.406464] Registering candidate plugin; module='afsocket', context='source', name='syslog'
[2018-06-23T20:46:47.406468] Registering candidate plugin; module='afsocket', context='destination', name='syslog'
[2018-06-23T20:46:47.406474] Registering candidate plugin; module='afsocket', context='source', name='network'
[2018-06-23T20:46:47.406478] Registering candidate plugin; module='afsocket', context='destination', name='network'
[2018-06-23T20:46:47.406482] Registering candidate plugin; module='afsocket', context='source', name='systemd-syslog'
... [ OMITTING UNNECESSARY LINES ] ...
[2018-06-23T20:46:47.409282] Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int'
... [ OMITTING UNNECESSARY LINES ] ...
**[2018-06-23T20:46:47.438730] Accepting connections; addr='AF_INET(0.0.0.0:22514)'**
[2018-06-23T20:46:47.439056] Module loaded and initialized successfully; module='linux-kmsg-format'
[2018-06-23T20:46:47.439307] Running application hooks; hook='1'
[2018-06-23T20:46:47.439315] Running application hooks; hook='3'
[2018-06-23T20:46:47.439328] syslog-ng starting up; version='3.15.1'
I am not new to syslog-ng and hoping it's something simple I have overlooked - Hoping a few more eyes find what mine are missing.
The issue was although I defined the source driver, I had overlooked it not be referenced in the defined log path. I had been bouncing back and forth through too many configs and somehow missed creating the reference for the input source. It's working now.
...
log {
source(s_net);
...
}
Output of netstat:
$ netstat -tulna | grep -e 514 -e 6514 -e 22514
tcp 0 0 0.0.0.0:22514 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6514 0.0.0.0:* LISTEN
udp 0 0 0.0.0.0:514 0.0.0.0:*
$
Related
Background
I have a machine in production running an elixir application (no access to iex, only to erl) and I am tasked with running an analysis on why we are consuming so much CPU. The idea here would be to launch observer, check the processes tab and see the processes with the most reductions.
How am I connecting?
To connect I am following a tutorial from a blog:
https://sgeos.github.io/elixir/erlang/observer/2016/09/16/elixir_erlang_running_otp_observer_remotely.html 1
Their instructions are as follows:
launch the app in the production machine with a cookie and a name
from local run: ssh user#public_ip "epmd -names" to get the name of the app and the port used
from local create a ssh tunnel to the remote machine: ssh -L 4369:user#public_ip:4369 -L 42877:user#public_ip:42877 user#public_ip (4369 is the epmd port by default, 42877 is the port of the app)
from local connect to the remote machine using the node's name: erl -name "user#app_name" -setcookie "mah_cookie" -hidden -run observer
Problem
And now in theory I should be able to use observer on the machine. Instead however I am greeted with the following error:
Protocol ‘inet_tcp’: register/listen error: epmd_close
So, after scouring the dark side of internet, I decided to use sudo journalctl -f to check all the logs of the machine and I found this:
channel 3: open failed: administratively prohibited: open failed
my_app_name sshd[8917]: error: connect_to flame#99.999.99.999: unknown host (Name or service not known)
/scripts/watchdog.sh")
my_app_name CRON[9985]: pam_unix(cron:session): session closed for user flame
Where:
erlang -name: my_app_name
machine user: flame
machine public ip: 99.999.99.999 (obviously not real)
so it tells me, unknown host ?? I am confused since 99.999.99.999 is the public IP of the machine itself!
Questions
What am I doing wrong?
I read that in older versions of erlang I can’t monitor a machine with observer if they are in different networks (which is the case, because I want to monitor this machine from my localhost) but I didn’t find any information regarding this in modern days.
If this is in fact impossible, what alternatives do I have?
Solution
After 3 days of non-stop searching, I finally found something that works.
To summarize I am putting it here everything I did.
All steps in local machine:
get the ports from the remote server:
> ssh remote-user#remote-ip "epmd -names"
epmd: up and running on port 4369 with data:
name super_duper_app at port 43175
create a ssh tunel with the ports:
ssh remote-user#remote-ip -L4369:localhost:4369 -L43175:localhost:43175
On another terminal in your local machine, run a iex terminal with the cookie the app in your remote server is using. Then connect to it and start observer:
iex --name observer#127.0.0.1 --cookie super_duper_cookie
Node.connect :"super_duper_app#127.0.0.1"
> true
:observer.start
With observer started, select the machine from the Nodes menu.
Possible setbacks
If you have tried this and it didn't work there are a few things you can check for:
Check if the EPMD port on your local machine is free, if not, kill the process using it and free it.
Check your ssh tunneling keys and configurations for permissions. As #Roberto Aloi pointed out this link can be useful: https://unix.stackexchange.com/questions/14160/ssh-tunneling-error-channel-1-open-failed-administratively-prohibited-open
I am try to setup a rabbitmq cluster in my Mac OS X 10.11.6(single machine).
It is fail for me to start a second instance of rabbitmq even i try ways on internet. what command i used is following:
export RABBITMQ_NODE_PORT=5672
export RABBITMQ_NODENAME=rabbit#localhost
rabbitmq-server -detached
export RABBITMQ_NODE_PORT=5673
export RABBITMQ_NODENAME=rabbit2#localhost
rabbitmq-server -detached
Only the first instance can be started.
here's the error_log from rabbitmq and i save it to github:
link for error log
Apart from the AMQP port (5672). RabbitMQ also binds the port 15672 for the management API. In the error log you can see that the two instances of rabbitMQ are trying to bind that same port, so you have to bind a different management port in each instance
I'm trying to setup a three-node Aerospike cluster on Ubuntu 14.04. Apart from the IP address/name, each machine is identical. I installed Aerospike and the management console, per the documentation, on each machine.
I then edited the network/service and network/heartbeat sections in /etc/aerospike/aerospike.conf:
network {
service {
address any
port 3000
access-address 10.0.1.11 # 10.0.1.12 and 10.0.1.13 on the other two nodes
}
heartbeat {
mode mesh
port 3002
mesh-seed-address-port 10.0.1.11 3002
mesh-seed-address-port 10.0.1.12 3002
mesh-seed-address-port 10.0.1.13 3002
interval 150
timeout 10
}
[...]
}
When I sudo service aerospike start on each of the nodes, the service runs but it's not clustered. If I try to add another node in the management console, it informs me: "Node 10.0.1.12:3000 cannot be monitored here as it belongs to a different cluster."
Can you see what I'm doing wrong? What changes should I make to aerospike.conf, on each of the nodes, in order to setup an Aerospike cluster instead of three isolated instances?
Your configuration appears correct.
Check if you are able to open a TCP connection over ports 3001 and 3002 from each host to the rest.
nc -z -w5 <host> 3001; echo $?
nc -z -w5 <host> 3002; echo $?
If not I would first suspect firewall configuration.
Update 1:
The netcat commands returned 0 so let's try to get more info.
Run and provide the output of the following on each node:
asinfo -v service
asinfo -v services
asadm -e info
Update 2:
After inspecting the output in the gists, the asadm -e "info net" indicated that all nodes had the same Node IDs.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Fqdn Ip Client Current HB HB
. Id . . Conns Time Self Foreign
h *BB9000000000094 hadoop01.woolford.io:3000 10.0.1.11:3000 15 174464730 37129 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Fqdn Ip Client Current HB HB
. Id . . Conns Time Self Foreign
h *BB9000000000094 hadoop03.woolford.io:3000 10.0.1.13:3000 5 174464730 37218 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Fqdn Ip Client Current HB HB
. Id . . Conns Time Self Foreign
h *BB9000000000094 hadoop02.woolford.io:3000 10.0.1.12:3000 5 174464731 37203 0
Number of rows: 1
The Node ID is constructed with the fabric port (port 3001 in hex) followed by the MAC address in reverse byte order. Another flag was that the "HB Self" was non-zero and is expected to be zero in a mesh configuration (in a multicast configuration this will also be non-zero since the nodes will receive their own heartbeat messages).
Because all of the Node IDs are the same, this would indicate that all of the MAC address are the same (though it is possible to change the node IDs using rack aware). Heartbeats that appear to have originated from the local node (determined by hb having the same node id) are ignored.
Update 3:
The MAC addresses are all unique, which contradicts previous conclusions. A reply provided the interface name being used, em1, which is not an interface name Aerospike looks for. Aerospike looks for interfaces named either eth#, bond#, or wlan#. I assume since the name wasn't one of the expected three this caused the issue with the MAC addresses; if so I would suspect the following warning exists in the logs?
Tried eth,bond,wlan and list of all available interfaces on device.Failed to retrieve physical address with errno %d %s
For such scenarios the network-interface-name parameter may be used to instruct Aerospike which interface to use for node id generation. This parameter also determines which interface's IP address should be advertised to the client applications.
network {
service {
address any
port 3000
access-address 10.0.1.11 # 10.0.1.12 and 10.0.1.13 on the other two nodes
network-interface-name em1 # Needed for Node ID
}
Update 4:
With the 3.6.0 release, these device names will be automatically discovered. See AER-4026 in release notes.
What ports does RabbitMQ Server use or need to have open on the firewall for a cluster of nodes?
My /usr/lib/rabbitmq/bin/rabbitmq-env is set below which I'm assuming are needed (35197).
SERVER_ERL_ARGS="+K true +A30 +P 1048576 \
-kernel inet_default_connect_options [{nodelay,true}] \
-kernel inet_dist_listen_min 35197 \
-kernel inet_dist_listen_max 35197"
I haven't touched the rabbitmq.config to set a custom tcp_listener so it should be listening on the default 5672.
Here are the relevant netstat lines:
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 728/epmd
tcp 0 0 0.0.0.0:35197 0.0.0.0:* LISTEN 5126/beam
tcp6 0 0 :::5672 :::* LISTEN 5126/beam
My questions are:
for other nodes to be able to connect to the cluster, do all 3 ports 4369, 5672 and 35197 need to be open?
Why isn't 5672 running on tcp and not just tcp6?
PORT 4369: Erlang makes use of a Port Mapper Daemon (epmd) for resolution of node names in a cluster. Nodes must be able to reach each other and the port mapper daemon for clustering to work.
PORT 35197
set by inet_dist_listen_min/max Firewalls must permit traffic in this range to pass between clustered nodes
RabbitMQ Management console:
PORT 15672 for RabbitMQ version 3.x
PORT 55672 for RabbitMQ pre 3.x
Make sure that the rabbitmq_management plugin is enabled, otherwise you won't be able to access management console on those ports.
PORT 5672 RabbitMQ main port (AMQP)
PORT 5671 TLS-encrypted AMQP (if enabled)
For a cluster of nodes, they must be open to each other on 35197, 4369 and 5672.
For any servers that want to use the message queue, only 5672 (or possibly 5671) is required.
What ports is RabbitMQ using?
Default: 5672, the manual has the answer. It's defined in the RABBITMQ_NODE_PORT variable.
https://www.rabbitmq.com/configure.html#define-environment-variables
The number might be differently if changed by someone in the rabbitmq configuration file:
vi /etc/rabbitmq/rabbitmq-env.conf
Ask the nmap if it can see it:
sudo nmap -p 1-65535 localhost
Starting Nmap 5.51 ( http://nmap.org ) at 2014-09-19 13:50 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00041s latency).
PORT STATE SERVICE
443/tcp open https
5672/tcp open amqp
15672/tcp open unknown
35102/tcp open unknown
59440/tcp open unknown
Oh look, 5672, and 15672
Ask netstat if it can see it:
netstat -lntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:15672 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:55672 0.0.0.0:* LISTEN
tcp 0 0 :::5672 :::* LISTEN
Oh look 5672.
lsof to see ports:
eric#dev ~$ sudo lsof -i | grep beam
beam.smp 21216 rabbitmq 17u IPv4 33148214 0t0 TCP *:55672 (LISTEN)
beam.smp 21216 rabbitmq 18u IPv4 33148219 0t0 TCP *:15672 (LISTEN)
use nmap from a different machine, find out if 5672 is open:
sudo nmap -p 5672 10.0.1.71
Starting Nmap 5.51 ( http://nmap.org ) at 2014-09-19 13:19 EDT
Nmap scan report for 10.0.1.71
Host is up (0.00011s latency).
PORT STATE SERVICE
5672/tcp open amqp
MAC Address: 0A:40:0E:8C:75:6C (Unknown)
Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds
Try to connect to a port manually with telnet, 5671 is CLOSED:
telnet localhost 5671
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Try to connect to a port manually with telnet, 5672 is OPEN:
telnet localhost 5672
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Check your firewall:
sudo cat /etc/sysconfig/iptables
It should tell you what ports are made open:
-A INPUT -p tcp -m tcp --dport 5672 -j ACCEPT
Reapply your firewall:
sudo service iptables restart
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Flushing firewall rules: [ OK ]
iptables: Unloading modules: [ OK ]
iptables: Applying firewall rules: [ OK ]
To find out what ports rabbitmq uses:
$ epmd -names
Outputs:
epmd: up and running on port 4369 with data:
name rabbit at port 25672
Run these as root:
lsof -i :4369
lsof -i :25672
More about epmd options.
Port Access
Firewalls and other security tools may prevent RabbitMQ from binding to a port. When that happens, RabbitMQ will fail to start. Make sure the following ports can be opened:
4369: epmd, a peer discovery service used by RabbitMQ nodes and CLI tools
5672, 5671: used by AMQP 0-9-1 and 1.0 clients without and with TLS
25672: used by Erlang distribution for inter-node and CLI tools communication and is allocated from a dynamic range (limited to a single port by default, computed as AMQP port + 20000). See networking guide for details.
15672: HTTP API clients and rabbitmqadmin (only if the management plugin is enabled)
61613, 61614: STOMP clients without and with TLS (only if the STOMP plugin is enabled)
1883, 8883: (MQTT clients without and with TLS, if the MQTT plugin is enabled
15674: STOMP-over-WebSockets clients (only if the Web STOMP plugin is enabled)
15675: MQTT-over-WebSockets clients (only if the Web MQTT plugin is enabled)
Reference doc:
https://www.rabbitmq.com/install-windows-manual.html
Check \AppData\Roaming\RabbitMQ\log. If you have a log file there search for started TCP listener on [::].
It should tell you on which port rabbitMq is running. Look for the last entry.
Disclaimer: not the best way to find port, as there might be a chance the port has been changed in the mean time.
I am setting up a rabbitmq cluster and ran into an issue during the one step in the process. Its straight out of the rabbitmq clustering guide.
root#celery:~# rabbitmqctl status
Status of node celery#celery ...
[{pid,20410},
{running_applications,[{rabbit,"RabbitMQ","2.5.1"},
{os_mon,"CPO CXC 138 46","2.2.4"},
{sasl,"SASL CXC 138 11","2.1.8"},
{mnesia,"MNESIA CXC 138 12","4.4.12"},
{stdlib,"ERTS CXC 138 10","1.16.4"},
{kernel,"ERTS CXC 138 10","2.13.4"}]},
{os,{unix,linux}},
{erlang_version,"Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
{memory,[{total,25296704},
{processes,9680280},
{processes_used,9662720},
{system,15616424},
{atom,1099393},
{atom_used,1082732},
{binary,89768},
{code,11606637},
{ets,726848}]}]
...done.
root#celery:~# rabbitmqctl cluster_status
Cluster status of node celery#celery ...
[{nodes,[{disc,[celery#celery]}]},{running_nodes,[celery#celery]}]
...done.
root#celery:~# rabbitmqctl stop_app
Stopping node celery#celery ...
...done.
root#celery:~# rabbitmqctl reset
Resetting node celery#celery ...
...done.
root#celery:~# rabbitmqctl cluster worker1#worker1
Clustering node celery#celery with [worker1#worker1] ...
Error: {failed_to_cluster_with,[worker1#worker1],
"Mnesia could not connect to some nodes."}
What are the possible reasons one node wouldn't be able to connect to another?
Here's the guide I'm following: http://www.rabbitmq.com/clustering.html
I jumped into the #rabbitmq channel on freenode. Here's the discussion that followed:
14:29 shakakai: hey all, i'm having a little issue with clustering rabbitmq http://stackoverflow.com/questions/6948624/mnesia-cant-connect-to-another-node
14:30 shakakai: has anyone run into that problem before?
14:30 daysmen has left IRC (Read error: Connection reset by peer)
14:30 antares_: shakakai: make sure that epmd is running on every node
14:30 antares_: shakakai: and that port it uses (4369) is open in your firewall
14:31 |Blaze|: shakakai: is your dns correct? Can you ping worker1 from celery and celery from worker1
14:31 shakakai: |Blaze|: hmm...i'll check
14:31 daysmen has joined (~quassel#host-84-13-157-50.opaltelecom.net)
14:32 shakakai: |Blaze|: this is where I'm a little confused, the rabbitmq nodename is worker1#worker1 but the fqdn to ping the box is "ping worker1.mydomain.com"
14:33 |Blaze|: can you "ping worker1"
14:34 shakakai: |Blaze|: no
14:34 |Blaze|: k, you'll need to fix that
14:34 hyperboreean has left IRC (Ping timeout: 250 seconds)
14:37 shakakai: |Blaze|: gotcha, so I setup a hosts file and i should be good
14:37 |Blaze|: yup
14:37 |Blaze|: in both directions
TL;DR
Make sure you can ping the rabbit nodename from each of the boxes you are clustering. If you can't, setup a hosts file for each rabbit nodename.
I installed the Docker RabbitMQ also encountered similar problems in the process.
The main reason is /var/lib/RabbitMQ/mnesia/rabbit/cluster_nodes.config configuration file on errors cannot be connected to.
Mnesia is a distributed, soft real-time database management system written in the Erlang programming language
There are several ways to repair this problem:
Fix the configure file，using the correct cluster node name, from the log we see that our Node name is rabbit#cb43449d5d72
// log info
...
rabbitmq | Starting broker...2019-11-27 16:18:22.621 [info] <0.304.0>
rabbitmq | node : rabbit#cb43449d5d72
...
// This is the wrong configuration file:
$ cat ./mnesia/rabbit/cluster_nodes.config
{[rabbit#cb43449d5d72,rabbit#dc3288264c34],[rabbit#dc3288264c34]}.
// Update it with correctly config node name, and restart RabbitMQ server:
$ cat ./mnesia/rabbit/cluster_nodes.config
{[rabbit#cb43449d5d72],[rabbit#cb43449d5d72]}.
The simplest way is to remove the mnesia directory and configure the correct node name, which like rabbit#my-rabbit, in /etc/hosts is 127.0.0.1 my-rabbit, after the operation, you should see the following configuration details
$ find . -name cluster_nodes.config
./mnesia/rabbit/cluster_nodes.config
./mnesia/rabbit#my-rabbit/cluster_nodes.config
$ cat ./mnesia/rabbit#my-rabbit/cluster_nodes.config
{['rabbit#my-rabbit'],['rabbit#my-rabbit']}.
There are several things to check before you can get the cluster to work well:
0) Ensure you are running the exact same rabbitmq version on each node
1) set up network until you are able to ping each server from each other
2) cookies - You have to get the exact same erlang cookie in the .erlang.cookie file on each server
One trick is useful is to try this command from one node to see if you can reach another one from rabbitmq
rabbitmqctl eval 'net_adm:ping(rabbit#othernode).'
this should say Pang if it's nok or pong if it's ok
be careful to not forget the dot close to the end of the eval expression.
I got it working fine after several hours of unsuccessful trials.
3) Bear in mind that there may be an issue when restarting a node of a cluster if this node was not the last that was stop - it wont start before the last that stop was restarted.
When all the above (0 to 2) are correct, 3 may well be the root cause of your problem...
Hope this help,
cheers,
jb
One thing I've read is that the erlang cookie needs to be on all cluster nodes so that they can communicate. i believe it lives in /var/lib/rabbitmq/.erlang.cookie