How to make Orion Context Broker works with HTTPS notification? - ssl

I want to enable https for notifications. The Orion Context Broker version 1.7.0 is installed in Ubuntu 16.04. To start, the following command is being used:
sudo /etc/init.d/contextBroker start -logAppend -https -key /path/to/orion.key -cert /path/to/orion.crt
The answer is:
[ ok ] Starting contextBroker (via systemctl): contextBroker.service.
The status is:
sudo systemctl status contextBroker.service
contextBroker.service - LSB: Example initscript
Loaded: loaded (/etc/init.d/contextBroker; bad; vendor preset: enabled)
Active: active (exited) since Tue 2017-04-04 12:56:13 BRT; 14s ago
Docs: man:systemd-sysv-generator(8)
Process: 8312 ExecStart=/etc/init.d/contextBroker start (code=exited, status=0/SUCCESS)
Apr 04 12:56:13 fiware-ubuntu systemd[1]: Starting LSB: Example initscript...
Apr 04 12:56:13 fiware-ubuntu contextBroker[8312]: contextBroker
Apr 04 12:56:13 fiware-ubuntu contextBroker[8312]: /path/bin/contextBroker
Apr 04 12:56:13 fiware-ubuntu systemd[1]: Started LSB: Example initscript.
Another approach is running Orion as:
sudo /path/bin/contextBroker -logLevel DEBUG -localIp x.y.z.t -https -key /path/to/orion.key -cert /path/to/orion.crt
The log follows:
time=2017-04-04T18:37:58.881Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=contextBroker.cpp[1705]:main | msg=Orion Context Broker is running
time=2017-04-04T18:37:58.887Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongoConnectionPool.cpp[205]:mongoConnect | msg=Successful connection to database
time=2017-04-04T18:37:58.887Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=connectionOperations.cpp[681]:setWriteConcern | msg=Database Operation Successful (setWriteConcern: 1)
time=2017-04-04T18:37:58.887Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=connectionOperations.cpp[724]:getWriteConcern | msg=Database Operation Successful (getWriteConcern)
time=2017-04-04T18:37:58.888Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=connectionOperations.cpp[626]:runCollectionCommand | msg=Database Operation Successful (command: { buildinfo: 1 })
...
time=2017-04-04T18:37:58.897Z | lvl=FATAL | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=rest.cpp[1720]:restStart | msg=Fatal Error (error starting REST interface)
It is not working...

If you run Orion as a service (as recommended) then command line parameters have to be configured in the /etc/sysconfig/contexBroker file. The file is explained in this piece of documentation.
Note the BROKER_EXTRA_OPS variable at the end of the file. This is used to include CLI parameters that are not set using any other option, as the ones related with HTTPS you are using. Thus, it should be a matter of setting BROKER_EXTRA_OPS in this way:
BROKER_EXTRA_OPS="-logAppend -https -key /path/to/orion.key -cert /path/to/orion.crt"
Then start the service using:
sudo /etc/init.d/contextBroker start
(Note that no parameter is added after 'start')
You can check that Orion is running with the right parameters using ps ax | grep contextBroker.
Finally, regarding the error Fatal Error (error starting REST interface) it apperas when Orion, for some reason, is not able to start the listening server for the REST API. Typically this is due to some other process (maybe a forgotten instance of Orion) runs listening at the same port. Use sudo netstat -ntpld | grep 1026 to know which other process could be listening at that part (assuming that 1026 is the port in which you are trying to run Orion, of course).

Related

have_ssl on MariaDB stays DISABLED, can't enable SSL on Debian and YaSSL

I am trying to set up MariaDB with TLS on a Raspberry 3, running dietpi with LAMP installed. Maria DB 10.1.23
I followed the official docu on mariadb.com and also tried the linked guideline.
However, when I look at my variables, have_ssl stays DISABLED.
MariaDB [(none)]> SHOW VARIABLES LIKE '%ssl%';
+---------------------+--------------------------------+
| Variable_name | Value |
+---------------------+--------------------------------+
| have_openssl | NO |
| have_ssl | DISABLED |
| ssl_ca | /etc/mysql/ssl/ca-cert.pem |
| ssl_capath | |
| ssl_cert | /etc/mysql/ssl/server-cert.pem |
| ssl_cipher | |
| ssl_crl | |
| ssl_crlpath | |
| ssl_key | /etc/mysql/ssl/server-key.pem |
| version_ssl_library | YaSSL 2.4.2 |
+---------------------+--------------------------------+
10 rows in set (0.01 sec)
This is my /etc/mysql/mariadb.conf.d/50-server.cnf
#
ssl-ca=/etc/mysql/ssl/ca-cert.pem
ssl-cert=/etc/mysql/ssl/server-cert.pem
ssl-key=/etc/mysql/ssl/server-key.pem
#
# Accept only connections using the latest and most secure TLS protocol version.
# ..when MariaDB is compiled with OpenSSL:
# ssl-cipher=TLSv1.2
# ..when MariaDB is compiled with YaSSL (default in Debian):
ssl=on
When trying openssl s_client I get the following result:
openssl s_client -state -nbio -debug -connect 127.0.0.1:3306 | grep "ssl"
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:error in SSLv3/TLS write client hello
1995634080:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:252:
openssl s_client -connect 127.0.0.1:3306
CONNECTED(00000003)
1995855264:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:252:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 176 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1542233581
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
I also tried to connect from another Raspi with this result:
root#DietPi:/etc/mysql/ssl# openssl s_client -state -nbio -debug -connect 192.168.1.89:3306 | grep "^ssl"
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
SSL_connect:error in SSLv2/v3 read server hello A
548036626192:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:782:
My research so far was not very successful. No solutions worked or fit to my environment. I also tried to install analyze-ssl.pl, but the installation of cpan App::cpanminus failed with
Writing MYMETA.yml and MYMETA.json
MIYAGAWA/App-cpanminus-1.7044.tar.gz
/usr/bin/perl Makefile.PL INSTALLDIRS=site -- OK
Running make for M/MI/MIYAGAWA/App-cpanminus-1.7044.tar.gz
MIYAGAWA/App-cpanminus-1.7044.tar.gz
make -- NOT OK
I will try to install analyze-ssl.pl tomorrow or open another thread. Just wanted to explain what I tried.
I would be happy if someone could help me solve my riddle.
Thanks!
Markus
you can try to convert you PKCS#8 key generate by default by openssl to PKCS#1 key for mariadb
openssl rsa -in pcks#8.key -out pkcs#1.key
after this operation mariadb say
MariaDB [(none)]> SHOW VARIABLES LIKE '%ssl%';
+---------------------+-----------------------------------------------+
| Variable_name | Value |
+---------------------+-----------------------------------------------+
| have_openssl | NO |
| have_ssl | YES |
| ssl_ca | /etc/mysql/ssl/mariaDB-CA.pem |
| ssl_capath | |
| ssl_cert | /etc/mysql/ssl/server.pem |
| ssl_cipher | |
| ssl_crl | |
| ssl_crlpath | |
| ssl_key | /etc/mysql/ssl/server.key |
| version_ssl_library | YaSSL 2.4.4 |
+---------------------+-----------------------------------------------+
David
Had the exact same problem. The solution by #David didn't work for me, but this helped:
https://serverfault.com/questions/950451/mariadb-mysql-shows-have-ssl-disabled-when-certs-are-installled
Turns out that the mysql user needs to own the certificate files.

nova-compute service state is down

I have an all-in-one-setup with my controller and compute services running on the same node.all my nova and other dependent services are up and running. However, when i try to launch an instance the state of the nova-compute process becomes down. Because of this the instance is stuck in spawning state.
> [root#localhost nova(keystone_admin)]# nova service-list
> +----+------------------+-----------------------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status |
> State | Updated_at | Disabled Reason |
> +----+------------------+-----------------------+----------+---------+-------+----------------------------+-----------------+ | 6 | nova-cert | localhost.localdomain | internal | enabled |
> up | 2016-11-04T07:24:32.000000 | - | | 7 |
> nova-consoleauth | localhost.localdomain | internal | enabled | up
> | 2016-11-04T07:24:32.000000 | - | | 8 | nova-scheduler
> | localhost.localdomain | internal | enabled | up |
> 2016-11-04T07:24:33.000000 | - | | 9 | nova-conductor
> | localhost.localdomain | internal | enabled | up |
> 2016-11-04T07:24:33.000000 | - | | 11 | nova-compute
> | localhost.localdomain | nova | enabled | **down** |
> 2016-11-04T06:43:03.000000 | - | | 12 | nova-console
> | localhost.localdomain | internal | enabled | up |
> 2016-11-04T07:24:32.000000 | - |
====
[root#localhost nova(keystone_admin)]# systemctl status
openstack-nova-compute.service -l ● openstack-nova-compute.service -
OpenStack Nova Compute Server Loaded: loaded
(/usr/lib/systemd/system/openstack-nova-compute.service; enabled;
vendor preset: disabled) Active: active (running) since Fri
2016-11-04 12:08:54 IST; 49min ago Main PID: 37586 (nova-compute)
CGroup: /system.slice/openstack-nova-compute.service
└─37586 /usr/bin/python2 /usr/bin/nova-compute
Nov 04 12:08:46 localhost.localdomain systemd[1]: Starting OpenStack
Nova Compute Server... Nov 04 12:08:53 localhost.localdomain
nova-compute[37586]: Option "verbose" from group "DEFAULT" is
deprecated for removal. Its value may be silently ignored in the
future. Nov 04 12:08:53 localhost.localdomain nova-compute[37586]:
Option "notification_driver" from group "DEFAULT" is deprecated. Use
option "driver" from group "oslo_messaging_notifications". Nov 04
12:08:54 localhost.localdomain systemd[1]: Started OpenStack Nova
Compute Server.
========
The status for the nova compute process is perfectly fine.
My rabbitmq service is also running
FYI,
[root#localhost nova(keystone_admin)]# systemctl status
rabbitmq-server ● rabbitmq-server.service - RabbitMQ broker Loaded:
loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled;
vendor preset: disabled) Drop-In:
/etc/systemd/system/rabbitmq-server.service.d
└─limits.conf
Active: active (running) since Thu 2016-11-03 12:32:08 IST; 24h ago
Main PID: 1835 (beam.smp) CGroup:
/system.slice/rabbitmq-server.service
├─1835 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home
/var/lib/rabbitmq --...
├─1964 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
├─5873 inet_gethost 4
└─5875 inet_gethost 4
Nov 04 12:13:12 localhost.localdomain rabbitmq-server[1835]:
{user,<<"guest">>, Nov 04 12:13:12 localhost.localdomain
rabbitmq-server[1835]: [administrator], Nov 04 12:13:12
localhost.localdomain rabbitmq-server[1835]:
rabbit_auth_backend_internal,...}, Nov 04 12:13:12
localhost.localdomain rabbitmq-server[1835]: <<"/">>, Nov 04 12:13:12
localhost.localdomain rabbitmq-server[1835]: [{<<...>>,...},{...}],
Nov 04 12:13:12 localhost.localdomain rabbitmq-server[1835]:
<0.14812.0>,<0.14816.0>]}}, Nov 04 12:13:12 localhost.localdomain
rabbitmq-server[1835]: {restart_type,intrinsic}, Nov 04 12:13:12
localhost.localdomain rabbitmq-server[1835]: {shutdown,4294967295},
Nov 04 12:13:12 localhost.localdomain rabbitmq-server[1835]:
{child_type,worker}]}]}} Nov 04 12:13:12 localhost.localdomain
rabbitmq-server[1835]: function_clause
=======
[root#localhost nova(keystone_admin)]# netstat -anp | grep 5672 | grep
37586 tcp 0 0 10.1.10.22:55628 10.1.10.22:5672
ESTABLISHED 37586/python2 tcp 0 0 10.1.10.22:56204
10.1.10.22:5672 ESTABLISHED 37586/python2 tcp 0 0 10.1.10.22:56959 10.1.10.22:5672 ESTABLISHED 37586/python2
===== 37586 is the nova-compute process id.
I have checked the logs for nova-compute, nova-api and nova-conductor and there are no errors.
I have checked the nova scheduler logs and there are some errors stating refused to connect to rabbitmq and the database service.
**
2016-11-03 12:24:50.930 2092 ERROR nova.servicegroup.drivers.db
DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't
connect to MySQL server on '10.1.10 .22' ([Errno 111] ECONNREFUSED)")
2016-11-03 12:24:53.811 2092 ERROR oslo.messaging._drivers.impl_rabbit
[-] AMQP server on 10.1.10.22:5672 is unreachable: [Errno 111]
ECONNREFUSED. Trying again in
**
16 seconds.
=======
Can someone suggest what should i do to handle it.
As i am on the same node, why are these services not reachable?
If nova-compute is down, there are two possible reasons:
a. nova-compute is actually down
b. it cannot communicate with rabbit, or nova-conductor cannot communicate with rabbit.
As far as I can see in your logs, you have issue with rabbit: "10.1.10.22:5672 is unreachable". Check if you have rabbit listening on this IP/port. Check if you can connect to rabbit from compute host. I usually use nc 10.1.10.22 5672 to see if there are connection or not.
Check if nova settings for rabbit are correct. Example of correct settings:
[DEFAULT]
rpc_backend=rabbit
rabbit_host=rabbitmq-ip-here
rabbit_port=5672
rabbit_hosts=$rabbit_host:$rabbit_port
rabbit_use_ssl=false
rabbit_userid=guest
rabbit_password=guest
rabbit_login_method=AMQPLAIN
rabbit_virtual_host=/compute
Check logs in the /var/log/nova/*.log
Enable debug=true in the [DEFAULT] section of nova.conf

Red Hat 7 with JBoss A-MQ 6.1: management console is not accessible

We're having the same problem as this OP: JBoss ActiveMQ on Red Hat - Unable to access AMQ Web Console, however that post doesn't indicate how he solved the problem.
We're trying to evaluate JBoss A-MQ, and have installed RHEL 7 with A-MQ 6.1 according to their installation guide. Everything works locally, e.g. the sample producer/consumer tests succeed in the Verifying the Installation step.
However, we cannot access the management console, even after configuring the remote user. We did have to add a JAVA_OPTIONS entry to setenv to override the default IPv6 sockets, and netstat shows that the service is now listening on the tcp socket instead of tcp6:
File /opt/jboss-a-mq-6.1.0.redhat-379/bin/setenv now contains:
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
export JAVA_OPTS
# netstat -paunt | grep 8181
tcp 0 0 0.0.0.0:8181 0.0.0.0:* LISTEN 10698/java
We can ping the box from other systems, however attempting to open a telnet session on port 8181 times out. The management console URLs we've tried are:
http://172.16.100.110:8181/hawtio
http://172.16.100.110:8181
but neither work. What are we missing?
Here are entries from amq.log containing hawt, which seem like it is starting things up correctly:
$ grep hawt amq.log
2014-09-09 11:32:35,778 | INFO | FelixStartLevel | HttpServiceFactoryImpl | .internal.HttpServiceFactoryImpl 35 | 98 - org.ops4j.pax.web.pax-web-runtime - 3.0.6 | Binding bundle: [io.hawt.hawtio-web [146]] to http service
2014-09-09 11:32:35,860 | INFO | pool-10-thread-1 | ConfigManager | io.hawt.system.ConfigManager 32 | 146 - io.hawt.hawtio-web - 1.2.0.redhat-379 | Configuration will be discovered via system properties
2014-09-09 11:32:35,863 | INFO | pool-10-thread-1 | JmxTreeWatcher | io.hawt.jmx.JmxTreeWatcher 63 | 146 - io.hawt.hawtio-web - 1.2.0.redhat-379 | Welcome to hawtio 1.2-redhat-379 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2014-09-09 11:32:35,865 | INFO | pool-10-thread-1 | UploadManager | io.hawt.jmx.UploadManager 40 | 146 - io.hawt.hawtio-web - 1.2.0.redhat-379 | Using file upload directory: /opt/jboss-a-mq-6.1.0.redhat-379/data/tmp/uploads
2014-09-09 11:32:35,868 | INFO | pool-10-thread-1 | AuthenticationFilter | io.hawt.web.AuthenticationFilter 84 | 146 - io.hawt.hawtio-web - 1.2.0.redhat-379 | Starting hawtio authentication filter, JAAS realm: "karaf" authorized role: "admin" role principal classes: ""
2014-09-09 11:32:35,965 | INFO | FelixStartLevel | HttpServiceFactoryImpl | .internal.HttpServiceFactoryImpl 35 | 98 - org.ops4j.pax.web.pax-web-runtime - 3.0.6 | Binding bundle: [io.hawt.hawtio-karaf-terminal [148]] to http service
2014-09-09 11:32:35,987 | INFO | pool-10-thread-2 | ConfigManager | io.hawt.system.ConfigManager 32 | 148 - io.hawt.hawtio-karaf-terminal - 1.2.0.redhat-379 | Configuration will be discovered via system properties
2014-09-09 11:32:35,988 | INFO | pool-10-thread-2 | AuthenticationFilter | io.hawt.web.AuthenticationFilter 84 | 148 - io.hawt.hawtio-karaf-terminal - 1.2.0.redhat-379 | Starting hawtio authentication filter, JAAS realm: "karaf" authorized role: "admin" role principal classes: ""
2014-09-09 11:32:35,996 | WARN | FelixStartLevel | ConfigFacade | io.hawt.config.ConfigFacade 23 | 149 - io.hawt.hawtio-maven-indexer - 1.2.0.redhat-379 | No ConfigFacade constructed yet so using default configuration for now
2014-09-09 11:32:36,015 | INFO | pool-10-thread-2 | LoginServlet | io.hawt.web.LoginServlet 55 | 148 - io.hawt.hawtio-karaf-terminal - 1.2.0.redhat-379 | hawtio login is using default HttpSession timeout
2014-09-09 11:32:36,128 | INFO | pool-10-thread-1 | hawtio-web - 146} | lipse.jetty.util.log.JavaUtilLog 75 | 90 - org.eclipse.jetty.aggregate.jetty-all-server - 8.1.14.v20131031 | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed
2014-09-09 11:32:36,191 | INFO | pool-10-thread-1 | LoginServlet | io.hawt.web.LoginServlet 55 | 146 - io.hawt.hawtio-web - 1.2.0.redhat-379 | hawtio login is using default HttpSession timeout
2014-09-09 11:32:39,227 | INFO | de startup timer | MavenIndexerFacade | maven.indexer.MavenIndexerFacade 98 | 149 - io.hawt.hawtio-maven-indexer - 1.2.0.redhat-379 | Storing maven index files in local directory: /opt/jboss-a-mq-6.1.0.redhat-379/data/mavenIndexer
2014-09-09 11:32:39,621 | INFO | de startup timer | MavenIndexerFacade | maven.indexer.MavenIndexerFacade 148 | 149 - io.hawt.hawtio-maven-indexer - 1.2.0.redhat-379 | Updating the maven indices. This may take a while, please be patient...
2014-09-09 11:32:39,852 | INFO | de startup timer | MavenIndexerFacade | maven.indexer.MavenIndexerFacade 185 | 149 - io.hawt.hawtio-maven-indexer - 1.2.0.redhat-379 | Completed updating 2 maven indices.
OMG, it was the firewall on the local server.
After spending the day scouring the web for ideas, trying to figure out what was wrong with my configuration, in desperation I finally thought I should try disabling the firewall and see what happens. So I shut it down with the command:
sudo service firewalld stop
and suddenly I could access the management console! I tried the default URL, and it worked:
http://172.16.100.110:8181
d'oh!

Master/slave using Oracle

we're using the so called JDBC Master Slave architecture with Oracle DB. We have 2 nodes and each one has one Broker started. We start the Broker1 (on node1) and it becomes the MASTER obtaining the lock over the tables. Then we start the Broker2 on the node2 and this one starts as SLAVE. WE can see in the log of Slave broker that it's trying to obtain the lock every 10sec, but it fails:
2013-06-12 19:32:38,714 | INFO | Default failed to acquire lease. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
2013-06-12 19:32:48,720 | INFO | Default Lease held by Default till Wed Jun 12 19:32:57 UTC 2013 | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
Everything works fine and then in one moment in SLAVE's log we see that it suddenly becomes the MASTER:
2013-06-13 00:38:11,262 | INFO | Default Lease held by Default till Thu Jun 13 00:38:17 UTC 2013 | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
2013-06-13 00:38:11,262 | INFO | Default failed to acquire lease. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
...
2013-06-13 00:38:21,314 | INFO | Default, becoming the master on dataSource: org.apache.commons.dbcp.BasicDataSource#9c6a99d | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
2013-06-13 00:38:21,576 | INFO | Apache ActiveMQ 5.8.0 (Default, ID:corerec3-49774-1371083901328-0:1) is starting | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:21,692 | WARN | Failed to start jmx connector: Cannot bind to URL [rmi://localhost:1616/jmxrmi]: javax.naming.NameAlreadyBoundException: jmxrmi [Root exception is java.rmi.AlreadyBoundException: jmxrmi]. Will restart management to re-create jmx connector, trying to remedy this issue. | org.apache.activemq.broker.jmx.ManagementContext | JMX connector
2013-06-13 00:38:21,700 | INFO | Listening for connections at: tcp://corerec3:61617?transport.closeAsync=false | org.apache.activemq.transport.TransportServerThreadSupport | main
2013-06-13 00:38:21,700 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
2013-06-13 00:38:21,701 | INFO | Apache ActiveMQ 5.8.0 (Default, ID:corerec3-49774-1371083901328-0:1) started | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:21,701 | INFO | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:21,701 | ERROR | Memory Usage for the Broker (512 mb) is more than the maximum available for the JVM: 245 mb | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:22,157 | INFO | Web console type: embedded | org.apache.activemq.web.WebConsoleStarter | main
2013-06-13 00:38:22,292 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
2013-06-13 00:38:22,353 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
while the MASTER's log shows no change from what it usually outputs...
So, it seems that somehow SLAVE obtains the lock (due to hmm... for example connection loss between master and the DB), but if we don't restart the brokers we start losing messages...
The problem is that in the producers' log we can see that it successfully sends the messages to the QueueX, but we don't see the consumer's taking them from the queue...
If we go to the DB and query _ACTIVEMQ_MSGS_ table we see that the messages are unprocessed.
It looks as if the broker (Producers are connected to) has the lock and inserts the messages into the DB and the brokers Clients are consuming from doesn't have the lock and can't consult the tables...
I don't know if all this makes much sense, but I surely hope someone might shed some light upon this one...
I didn't want to saturate the post with the configuration details, but if you need specific details like failover config, IPs, ports etc. I will post it...

Why does ActiveMQ restart automatically and how do i prevent it?

We've been using AMQ 5.5.1 in production for several months. Occasionally, we observe that the broker decides to refresh itself with no outside trigger. When this happens, our queue senders fail until the broker is back online some 10 minutes later. I cannot find any information or settings that would cause this behavior .. and let me control it.
Is this normal for the broker to recycle on its own like this? If so, what things would cause it?
2012-12-11 11:02:11,603 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1#f6ac0b: startup date [Tue Dec 11 11:02:11 EST 2012]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain
2012-12-11 11:02:13,806 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | WrapperSimpleAppMain
2012-12-11 11:02:13,821 | INFO | PListStore:D:\Tools\ActiveMQ\apache-activemq-5.5.1\bin\win32\..\..\data\localhost\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | WrapperSimpleAppMain
2012-12-11 11:02:13,868 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[D:\Tools\ActiveMQ\apache-activemq-5.5.1\bin\win32\..\..\data\kahadb] | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2012-12-11 11:02:16,618 | INFO | KahaDB is version 3 | org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain
2012-12-11 11:02:16,697 | INFO | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain
I found that the wrapper exe process was forcing the restart.
I was able to see in the wrapper.log (windows service) that the process was being restarted because the JVM was not responding. So this is not an issue with the broker auto-restarting per se.. it was an issue with the broker JVM somehow becoming hung (separate problem).
Here are the wrapper log entries for those interested:
ERROR | wrapper | 2012/12/11 11:01:58 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR | wrapper | 2012/12/11 11:01:58 | JVM did not exit on request, terminated
STATUS | wrapper | 2012/12/11 11:02:04 | Launching a JVM...