I went through the section "Password protecting the JMX connector" on: http://activemq.apache.org/jmx.html
I have ActiveMQ 5.6 installed.
In the activemq.xml file, I updated the following:
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
And in bin/activemq, I updated the following:
Before:
# ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=11099 "
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote"
After:
ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=11099 "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote"
I uncommented the lines for ACTIVEMQ_SUNJMX_START.
After I uncommented them, I tried starting ActiveMQ, the server never came up.
Nothing showed up in the activemq.log file.
When I comment them back and start the server, it starts with no issue.
In order to isolate the issue, I uncommented the first line and tried starting, it never started.
Looks like it is expecting something else when I uncomment ACTIVEMQ_SUNJMX_START.
I made sure that jmx.password and jmx.access exist under the conf folder.
Can you please let me know if I am missing something?
Thanks for your time!
The reason why the ActiveMQ server started is that the file permission for jmx.password got to be:
chmod u=rw,go= jmx.password
I found this on: http://fusesource.com/docs/broker/5.5/security/JMX-AmqConnector-Authent.html
After updating permission, it worked fine.
Related
I have the problem that my ECS logs (awslogs driver) are not working as expected. In Cloudwatch I'm only seeing the server startup logs & not the useful logs from the apache (/var/log/apache2/error.log & /var/log/apache2/access.log)
I have a docker multicontainer setup with one container running the apache server & the other container running PHP-FPM. My container logs on cloudwatch look like this:
Apache-Container:
23:35:39 *** Running /etc/my_init.d/02_init.sh...
23:35:39 Starting Apache
23:35:39 * Starting Apache httpd web server apache2
23:35:39 /usr/sbin/apache2ctl: 87: ulimit: error setting limit (Operation not permitted)
23:35:39 Setting ulimit failed. See README.Debian for more information.
23:35:40 *** Running /etc/rc.local...
23:35:40 *** Booting runit daemon...
23:35:40 *** Runit started as PID 225
23:35:40 Oct 25 22:35:40 apache-container syslog-ng[231]: syslog-ng starting up; version='3.5.6'
2019-10-26 00:17:01
Oct 25 23:17:01 apache-container CRON[947]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
...
07:35:16 tail: '/var/log/syslog' has been replaced; following new file
...
FPM-Container:
...
10:25:23 172.x.x.x - 27/Okt/2019:09:25:23 +0000 "GET /app.php" 200
10:25:25 172.x.x.x - 27/Okt/2019:09:25:24 +0000 "GET /app.php" 200
...
I've checked various forums & online resources. As I understood it right I just need to symlink my logs to STDOUT/STDERR or even better to /proc/self/fd/1 & /proc/self/fd/2 like this:
ln -sf /dev/stdout /var/log/apache2/access.log
ln -sf /dev/stderr /var/log/apache2/error.log
I've tried to link the logs in my .Dockerfile via the RUN command & also during runtime, but no success. I see that my logs are showing up correctly in the log files before linking them. I've also tried things like echo "test stderr logs" >> /dev/stderr or echo "test stdout logs" >> /dev/stdout inside & outside the containers, but nothing showing up in the cloudwatch logs. When I try docker logs MY_DOCKER_CONTAINER_ID I get: Error response from daemon: configured logging driver does not support reading.
Maybe I'm missing some basic knowledge here. I see that syslog is in my environment/base image (maybe i need to merge syslog & apache logs?) and that the PHP-FPM-container is logging 200's but only to the app.php even though I would like to know the exact path of the accessed url.
You need to specify in your docker-compose used by ECS to use the cloudwatch logging driver like so:
version: '2'
services:
myapp:
build:
context: .
logging:
driver: awslogs
options:
awslogs-group: "/my/log/group"
awslogs-region: "us-west-2"
awslogs-stream-prefix: some-prefix
This should cause /dev/stdout and /dev/stderr to appear in CloudWatch. You can find more information on the logging driver options on the Docker page.
Hey thx for the responses. If I remember it right, the problem was, that all of my output was redirected to syslog & there was a misconfiguration in my syslog config.
Because of limitations, we're forced to use Windows to host and manage our Zookeeper/SolrCloud cluster.
We're using 3 Windows Server 2016 servers, in Microsoft Azure, with an Azure Load Balancer in front of it.
I was able to install and configure everything, but Zookeeper isn't communicating in SSL with SolrCloud, so none of our API calls are working when creating things like new collections, etc.
I've followed the Zookeeper documentation (https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide), but all of it is for Linux systems. I've adapted it to the best of my knowledge, but it's just not working.
Here's what I did:
Added the following to zkCli.cmd : set CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location=C:/solr-7.2.1/server/etc/wildcard_sidlee_cloud.pfx -Dzookeeper.ssl.keyStore.password=somepassword -Dzookeeper.ssl.trustStore.location=C:/solr-7.2.1/server/etc/wildcard_sidlee_cloud.pfx -Dzookeeper.ssl.trustStore.password=somepassword"
Added %CLIENT_JVMFLAGS% to the Java call in zkCli.cmd
Added the following to zkServer.cmd: set SERVER_JVMFLAGS="-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=C:/solr-7.2.1/server/etc/wildcard_sidlee_cloud.pfx -Dzookeeper.ssl.keyStore.password=somepassword -Dzookeeper.ssl.trustStore.location=C:/solr-7.2.1/server/etc/wildcard_sidlee_cloud.pfx -Dzookeeper.ssl.trustStore.password=somepassword"
Added %SERVER_JVMFLAGS% to the Java call in zkServer.cmd
Modified clientPort=2181 in zoo.cfg to secureClientPort=2181
Zookeeper service "starts" but there's actually nothing happening. If I start zkServer.cmd manually, it fails with the error:
java.lang.NumberFormatException: For input string: "-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=C:/solr-7.2.1/server/etc/wildcard_sidlee_cloud.pfx -Dzookeeper.ssl.keyStore.password=somepassword -Dzookeeper.ssl.trustStore.location=C:/solr-7.2.1/server/etc/wildcard_sidlee_cloud.pfx -Dzookeeper.ssl.trustStore.password=somepassword"
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at org.apache.zookeeper.server.ServerConfig.parse(ServerConfig.java:63)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:103)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
I'm really at a loss here, and don't know where to go from here!
Thanks in advance for the help!!
Iv'e encountered the same error and figured it out:
When you added: %SERVER_JVMFLAGS% to the Java call in zkServer.cmd, you must of added it to the end of the line. Try Adding it next to all the arguments. before the " %* " at the end of the line.
That worked for me.
call %JAVA% "-Dzookeeper.log.dir=%ZOO_LOG_DIR%" "-Dzookeeper.root.logger=%ZOO_LOG4J_PROP%" "%SERVER_JVMFLAGS%" "-Dzookeeper.log.file=%ZOO_LOG_FILE%" "-XX:+HeapDumpOnOutOfMemoryError" "-XX:OnOutOfMemoryError=cmd /c taskkill /pid %%%%p /t /f" -cp "%CLASSPATH%" %ZOOMAIN% "%ZOOCFG%" %*
I'm trying to use JMX with activeMQ for monitoring so far I've been using this and this as a reference but so far I'm unable to connect to jmx remotely and also I don't see any mention of jmx url in activemq logs. I'm wondering if there is another way to make sure jmx is working? is it supposed to be indicated in activemq logs?
PS I'm using jdk1.7 and activeMQ 5.14.2.
Thanks in advance!
EDIT
I set useJmx="true" in my activemq.xml file:
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="primary" useJmx="true" dataDirectory="${activemq.data}">
I tried two steps:
FIRST
I tried changing management context from createConnector="false" to :
<managementContext>
<managementContext createConnector="true" connectorPort="1099"/>
</managementContext>
FOR FIRST TIME THE PORT IS OPEN AND ACTIVEMQ RUNS FINE AND JMX URL GETS REPORTED IN LOGS ALTHOUGH I CAN NOT CONNECT IT TO IT REMOTLEY BUT IM ASSUMING ITS WORKING
SECOND
I reverted back the changes I made for managmentContext and I tried setting:
ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_BASE}/jmx.password -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_BASE}/jmx.access"
in bin/activemq script and I set a username in conf/jmx.access file as:
admin readwrite
And also have set a password in conf/jmx.password:
admin activemq
NOW ACTIVEMQ IS NOT RUNNINT AT ALL BUT IT WILL RUN IF I SET
AUTHENTICATE=FALSE AND DELETE JMX.ACCESS AND JMX.PASSWORD
CONFIGURATION IN BIN/ACTIVEMQ FILE BUT I NEED USER NAME AND PASSWORD
FOR SECURITY REASONS
I found this post which has the exact same issue as mine. any ideas?
Password authentication for remote monitoring is enabled by default. To disable it, set the following system property when you start the JVM:
-Dcom.sun.management.jmxremote.authenticate=false like you done in second test but you need to add system property -Dcom.sun.management.jmxremote
Try to add these jvm param to env file and update host ip
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=X.X.X.X
UPDATE
SO, to resume, i think that the FIRST step you tried is the best, for making it working these are the steps :
revert all jmx env file changes, like this :
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099 "
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote"
<broker useJmx="true" ...
<managementContext>
<managementContext createConnector="true" connectorPort="1099" />
</managementContext>
verify that in AMQ logs you have
INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi |
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
NOTE : Assuming that 10.10.10.16 is the IP of AMQ host.
try to connect with jconsole from another machine than AMQ host with url "service:jmx:rmi:///jndi/rmi://10.10.10.16:1099/jmxrmi" without user/pwd.
if you cannot connect, try like this :
<managementContext>
<managementContext createConnector="true" connectorPort="1099" connectorHost="10.10.10.16" />
</managementContext>
verify that in AMQ logs you have
INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://10.10.10.16:1099/jmxrmi |
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
retry to connect, step 4
at this step normally you can connect with jconsole.
if you want to add security and authorizations, use this :
<managementContext>
<managementContext createConnector="true" connectorPort="1099" connectorHost="10.10.10.16" >
<property xmlns="http://www.springframework.org/schema/beans" name="environment">
<map xmlns="http://www.springframework.org/schema/beans">
<entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.password.file"
value="${activemq.conf}/jmx.password"/>
<entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.access.file"
value="${activemq.conf}/jmx.access"/>
</map>
</property>
</managementContext>
</managementContext>
Please try these steps and let me know in which one you fails to connect and provide error message from jconsole.
A couple troubleshooting steps:
Start jconsole or visualvm on the same system and connect using the "pid" attach method. Browse the MBeans and confirm org.apache.activemq beans are present
Run netstat -na and confirm ports 1099 (and 44444) are in LISTEN
Look at logs and confirm you do not have any "java.net.BindException: Address already in use.." messages that indicate a port conflict with an already running Java process.
Edit bin/env to configure JMX (this disables requiring SSL, sets the port to 1099 and disables requiring username and password.
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099 "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote "
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"`
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
I am getting the following WARN message while I start my host which is one of the Host Controller (HC) that is attached to the Domain Controller(DC).
[Server:server-two] 14:06:13,822 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 33) JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.
And my host-slave.xml has the following config...
<server-identities>
<!-- Replace this with either a base64 password of your own, or use a vault with a vault expression -->
<secret value="c2xhdmVfdXNlcl9wYXNzd29yZA=="/>
</server-identities>
I hope this config is the reason...... maybe I didn't understand..... but I couldn't find node identifier property rather this is the default secret value which I hope could be the cause of this WARN message.
However, I didn't mention HC to lookup host-slave.xml..... the command which I ran to start my HC is.....
[host-~-\-\-\bin]$./domain.sh -Djboss.domain.master.address=nnn.nn.nn.88 -b nnn.nn.nn.89 -bmanagement nnn.nn.nn.89 &
nnn.nn.nn.88 is my DC
Else please advise what's cause of the WARN message.
And please let me know the implication of this WARN message and advise us on the required config to overcome and sort out any consecutive consequences that would've been bound for this WARN.
I'm new to wildfly, and noticed this warning when I started it standalone from eclipse (I'm doing the following tutorial: https://wwu-pi.github.io/tutorials/lectures/eai/020_tutorial_jboss_project.html)
The fix was to add a node-identifier to the core-environment in the subsystem:
<subsystem xmlns="urn:jboss:domain:transactions:2.0">
<core-environment node-identifier="meindertwillemhoving">
<process-id>
<uuid/>
</process-id>
</core-environment>
<recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
</subsystem>
This is in file [wildfly]\standalone\configuration\standalone.xml.
This is the same answer as https://developer.jboss.org/message/880136#880136
According to WFLY-10541 if you are using WildFly v14.0.0 or newer you can pass the following to the startup script to set the transaction node identifier:
-Djboss.tx.node.id=<some-unique-id>
Setting the node identifier to an unique value is only required for proper handling of XA Transactions.
You can set it as follows in your XML configuration:
<subsystem xmlns="urn:jboss:domain:transactions:6.0">
<core-environment node-identifier="${jboss.tx.node.id}">
It needs to be a unique value up to 23 bytes long.
More about this here: http://www.mastertheboss.com/jboss-server/jboss-configuration/configuring-transactions-jta-using-jboss-as7-wildfly
Building on #kaptan's answer I added the following to the bottom of
bin/standalone.conf:
JAVA_OPTS="$JAVA_OPTS -Djboss.tx.node.id=`hostname -f`
This way I don't have to remember to add the "-Djboss.tx.node.id=" when running up wildfly by hand.
For this <server-identities> is not the issue. In fact, it shouldn't be touched at all.
When JBoss is started in domain mode by domain.sh, by default there will be three servers server-one server-two server-three. When you are running one more HC attached to the DC.... the defaulted server which is in auto-start mode will get clash when we start HC attaching to DC,- by the following command.
[host-~-\-\-\bin]$./domain.sh -Djboss.domain.master.address=nnn.nn.nn.88 -b nnn.nn.nn.89 -bmanagement nnn.nn.nn.89 &
Or by having the host configuration at HC (default host.xml... until unless we choose a different one....).
<domain-controller>
<remote host="${jboss.domain.master.address:nnn.nn.nn.88}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>
<domain-controller>
In order to solve this, we need to turn auto-start to false..... And we need to create a new server-group...... To that group we need to add dc-created-server and hc-created-server..... we can choose the appropriate same profile either full-ha or full for both created servers across DC and HC.
SO when we start the group by configuring the required HEAP size including permgen space... You could start both DC and HC.... and in DC you could see both of your-created-servers are started in the created server-group.
DC- Domain Controller
HC- Host Controller
To deploy you need to upload .ear or web-archive in the Application Console. You cannot place it in the deployments folder as how you do in standalone mode with .dodeploy file.
If you upload the same .ear next version do the Replace option instead of the Remove & Add option in the upload process.
I'm using Graylog2 server as my application log server. But couldn't connect apache log to graylog2. Is there any guide to send apache log to graylog2 server or can someone help me to solve this ?
I put this at the bottom of my /etc/rsyslog.conf on Ubuntu 14.04
# Apache access file:
$ModLoad imfile
$InputFileName /var/log/apache2/access.log
$InputFileTag apache-access:
$InputFileStateFile stat-apache-access
$InputFileSeverity info
$InputRunFileMonitor
#Apache Error file:
$InputFileName /var/log/apache2/error.log
$InputFileTag apache-errors:
$InputFileStateFile stat-apache-error
$InputFileSeverity error
$InputRunFileMonitor
$InputFilePollInterval 10
if $programname == 'apache-access' then #10.11.11.33:514
if $programname == 'apache-errors' then #10.11.11.33:514
where 10.x.x.x is my Graylog2 server.
There will be a GELF module for Apache soon. Until that is released I can recommend using Logstash to parse and forward the Apache log files. You could even send in the log lines to "Raw/Plaintext" inputs in Graylog2 using tail and netcat.