I'm using the 7.40.0 Cumulocity Java Client SDK to subscribe to real-time notifications. I'm able to get notifications but after leaving the client running for a while, it seems to stop listening and I'm no longer able to retrieve notifications.
Has anyone encountered this scenario?
WARN 2017-01-03 08:44:49,727 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.ConnectionHeartBeatWatcher: canceling the long poll request because of inactivity
DEBUG 2017-01-03 08:44:49,727 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.MessageExchange: canceling [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, channel=/meta/connect, id=16, connectionType=long-polling}]
DEBUG 2017-01-03 08:44:49,727 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.MessageExchange: stopping heartbeat watcher [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, channel=/meta/connect, id=16, connectionType=long-polling}]
DEBUG 2017-01-03 08:44:50,728 [pool-21-thread-1] com.cumulocity.sdk.client.notification.CumulocityLongPollingTransport.1365985548: sending messages [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, advice={timeout=0}, channel=/meta/connect, id=19, connectionType=long-polling}]
DEBUG 2017-01-03 08:44:50,728 [pool-21-thread-1] com.cumulocity.sdk.client.notification.MessageExchange: starting heartbeat watcher [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, advice={timeout=0}, channel=/meta/connect, id=19, connectionType=long-polling}]
DEBUG 2017-01-03 08:44:51,710 [CumulocityLongPollingTransport-scheduler-1] com.cumulocity.sdk.client.notification.MessageExchange: wait for response headers [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, advice={timeout=0}, channel=/meta/connect, id=19, connectionType=long-polling}]
DEBUG 2017-01-03 08:44:51,710 [CumulocityLongPollingTransport-scheduler-1] com.cumulocity.sdk.client.notification.MessageExchange: recived response headers [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, advice={timeout=0}, channel=/meta/connect, id=19, connectionType=long-polling}]
DEBUG 2017-01-03 08:44:51,710 [CumulocityLongPollingTransport-scheduler-1] com.cumulocity.sdk.client.notification.MessageExchange: getting heartbeants POST https://msc.cumulocity.com//cep/realtime returned a response status of 200 OK
DEBUG 2017-01-03 08:44:51,710 [CumulocityLongPollingTransport-scheduler-1] com.cumulocity.sdk.client.notification.MessageExchange: new messages recived
DEBUG 2017-01-03 08:44:51,711 [CumulocityLongPollingTransport-scheduler-1] com.cumulocity.sdk.client.notification.MessageExchange: Received messages [{channel=/meta/connect, id=19, advice={reconnect=retry, interval=0, timeout=5400000}, successful=true}]
DEBUG 2017-01-03 08:44:51,711 [CumulocityLongPollingTransport-scheduler-1] com.cumulocity.sdk.client.notification.MessageExchange: stopping heartbeat watcher [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, advice={timeout=0}, channel=/meta/connect, id=19, connectionType=long-polling}]
DEBUG 2017-01-03 08:44:51,711 [pool-21-thread-1] com.cumulocity.sdk.client.notification.CumulocityLongPollingTransport.1365985548: sending messages [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, channel=/meta/connect, id=23, connectionType=long-polling}]
DEBUG 2017-01-03 08:44:51,711 [pool-21-thread-1] com.cumulocity.sdk.client.notification.MessageExchange: starting heartbeat watcher [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, channel=/meta/connect, id=23, connectionType=long-polling}]
DEBUG 2017-01-03 08:56:51,711 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.MessageExchange: canceling [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, channel=/meta/connect, id=23, connectionType=long-polling}]
DEBUG 2017-01-03 08:56:51,711 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.MessageExchange: stopping heartbeat watcher [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, channel=/meta/connect, id=23, connectionType=long-polling}]
DEBUG 2017-01-03 08:56:52,711 [pool-21-thread-1] com.cumulocity.sdk.client.notification.CumulocityLongPollingTransport.1365985548: sending messages [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, advice={timeout=0}, channel=/meta/connect, id=26, connectionType=long-polling}]
DEBUG 2017-01-03 08:56:52,712 [pool-21-thread-1] com.cumulocity.sdk.client.notification.MessageExchange: starting heartbeat watcher [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, advice={timeout=0}, channel=/meta/connect, id=26, connectionType=long-polling}]
DEBUG 2017-01-03 08:56:53,659 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.MessageExchange: wait for response headers [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, advice={timeout=0}, channel=/meta/connect, id=26, connectionType=long-polling}]
DEBUG 2017-01-03 08:56:53,659 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.MessageExchange: recived response headers [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, advice={timeout=0}, channel=/meta/connect, id=26, connectionType=long-polling}]
DEBUG 2017-01-03 08:56:53,659 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.MessageExchange: getting heartbeants POST https://msc.cumulocity.com//cep/realtime returned a response status of 200 OK
DEBUG 2017-01-03 08:56:53,660 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.MessageExchange: new messages recived
DEBUG 2017-01-03 08:56:53,661 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.MessageExchange: Received messages [{channel=/meta/connect, id=26, advice={reconnect=retry, interval=0, timeout=5400000}, successful=true}]
DEBUG 2017-01-03 08:56:53,661 [CumulocityLongPollingTransport-scheduler-0] com.cumulocity.sdk.client.notification.MessageExchange: stopping heartbeat watcher [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, advice={timeout=0}, channel=/meta/connect, id=26, connectionType=long-polling}]
DEBUG 2017-01-03 08:56:53,661 [pool-21-thread-1] com.cumulocity.sdk.client.notification.CumulocityLongPollingTransport.1365985548: sending messages [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, channel=/meta/connect, id=28, connectionType=long-polling}]
DEBUG 2017-01-03 08:56:53,661 [pool-21-thread-1] com.cumulocity.sdk.client.notification.MessageExchange: starting heartbeat watcher [{clientId=4b11irnlptb55g4rs3hauu1ctjpy, channel=/meta/connect, id=28, connectionType=long-polling}]
This kind of scenario can happen when you have intermediate network equipment that is configured to drop or cut network traffic for long lasting connections. Examples are corporate HTTP proxies, but also mobile network providers may have such configurations.
This is difficult to troubleshoot, often you have to go on low-level wireshark or network protocol means. We had success with the following mechanisms:
Use a different network, e.g., direct DSL line instead of corporate Intranet.
Raise a ticket with your IT or the network provider to not cut connections too early or not drop packets.
If you are running Cumulocity yourself, you can configure it to send server-side heartbeats more frequently. That makes intermediate network equipment regard the connection as live, but at the expense of additional traffic for all clients.
Use the MQTT API and set a higher ping rate.
Sorry for no easy answers here :-)
I have experienced the same. Apparently in our case some middlebox dropped the long-polling TCP connection before the server sent the keepalive packet (by default after 10 minutes).
Futhermore, when testing with 7.26 version the Java reference agent we saw that if the agent did not receive the server-side keepalives, it stopped re-establishing the long-polling connection after three attempts. Haven't checked if this has changed in later releases.
We run our own instance of Cumulocity, and are able to configure the server side keepalive timer. We set the keepalive timer to 3 minutes which solved the issue.
Related
I am running flink job on yarn, we use "fink run" in command line to submit our job to yarn, one day we had an exception on flink job, as we didn't enable the flink restart strategy so it simply failed, but eventually we found that the job status was "SUCCEED" from the yarn application list, which we expect to be "FAILED".
Flink CLI log:
06/12/2018 03:13:37 FlatMap (getTagStorageMapper.flatMap)(23/32) switched to CANCELED
06/12/2018 03:13:37 GroupReduce (ResultReducer.reduceGroup)(31/32) switched to CANCELED
06/12/2018 03:13:37 FlatMap (SubClassEDFJoinMapper.flatMap)(29/32) switched to CANCELED
06/12/2018 03:13:37 CHAIN DataSource (SubClassInventory.AvroInputFormat.createInput) -> FlatMap (SubClassInventoryMapper.flatMap)(27/32) switched to CANCELED
06/12/2018 03:13:37 GroupReduce (OutputReducer.reduceGroup)(28/32) switched to CANCELED
06/12/2018 03:13:37 CHAIN DataSource (SubClassInventory.AvroInputFormat.createInput) -> FlatMap (BIMBQMInstrumentMapper.flatMap)(27/32) switched to CANCELED
06/12/2018 03:13:37 GroupReduce (BIMBQMGovCorpReduce.reduceGroup)(30/32) switched to CANCELED
06/12/2018 03:13:37 FlatMap (BIMBQMEVMJoinMapper.flatMap)(32/32) switched to CANCELED
06/12/2018 03:13:37 Job execution switched to status FAILED.
No JobSubmissionResult returned, please make sure you called ExecutionEnvironment.execute()
2018-06-12 03:13:37,625 INFO org.apache.flink.yarn.YarnClusterClient - Sending shutdown request to the Application Master
2018-06-12 03:13:37,625 INFO org.apache.flink.yarn.YarnClusterClient - Start application client.
2018-06-12 03:13:37,630 INFO org.apache.flink.yarn.ApplicationClient - Notification about new leader address akka.tcp://flink#ip-10-97-46-149.tr-fr-nonprod.aws-int.thomsonreuters.com:45663/user/jobmanager with session ID 00000000-0000-0000-0000-000000000000.
2018-06-12 03:13:37,632 INFO org.apache.flink.yarn.ApplicationClient - Sending StopCluster request to JobManager.
2018-06-12 03:13:37,633 INFO org.apache.flink.yarn.ApplicationClient - Received address of new leader akka.tcp://flink#ip-10-97-46-149.tr-fr-nonprod.aws-int.thomsonreuters.com:45663/user/jobmanager with session ID 00000000-0000-0000-0000-000000000000.
2018-06-12 03:13:37,634 INFO org.apache.flink.yarn.ApplicationClient - Disconnect from JobManager null.
2018-06-12 03:13:37,635 INFO org.apache.flink.yarn.ApplicationClient - Trying to register at JobManager akka.tcp://flink#ip-10-97-46-149.tr-fr-nonprod.aws-int.thomsonreuters.com:45663/user/jobmanager.
2018-06-12 03:13:37,688 INFO org.apache.flink.yarn.ApplicationClient - Successfully registered at the ResourceManager using JobManager Actor[akka.tcp://flink#ip-10-97-46-149.tr-fr-nonprod.aws-int.thomsonreuters.com:45663/user/jobmanager#182802345]
2018-06-12 03:13:38,648 INFO org.apache.flink.yarn.ApplicationClient - Sending StopCluster request to JobManager.
2018-06-12 03:13:39,480 INFO org.apache.flink.yarn.YarnClusterClient - Application application_1528772982594_0001 finished with state FINISHED and final state SUCCEEDED at 1528773218662
2018-06-12 03:13:39,480 INFO org.apache.flink.yarn.YarnClusterClient - YARN Client is shutting down
2018-06-12 03:13:39,582 INFO org.apache.flink.yarn.ApplicationClient - Stopped Application client.
2018-06-12 03:13:39,583 INFO org.apache.flink.yarn.ApplicationClient - Disconnect from JobManager Actor[akka.tcp://flink#ip-10-97-46-149.tr-fr-nonprod.aws-int.thomsonreuters.com:45663/user/jobmanager#182802345].
Flink job manager Log:
FlatMap (BIMBQMEVMJoinMapper.flatMap) (32/32) (67a002e07fe799c1624a471340c8cf9d) switched from CANCELING to CANCELED.
Try to restart or fail the job Flink Java Job at Tue Jun 12 03:13:17 UTC 2018 (1086cedb3617feeee8aace29a7fc6bd0) if no longer possible.
Requesting new TaskManager container with 8192 megabytes memory. Pending requests: 1
Job Flink Java Job at Tue Jun 12 03:13:17 UTC 2018 (1086cedb3617feeee8aace29a7fc6bd0) switched from state FAILING to FAILED.
Could not restart the job Flink Java Job at Tue Jun 12 03:13:17 UTC 2018 (1086cedb3617feeee8aace29a7fc6bd0) because the restart strategy prevented it.
Unregistered task manager ip-10-97-44-186/10.97.44.186. Number of registered task managers 31. Number of available slots 31
Stopping JobManager with final application status SUCCEEDED and diagnostics: Flink YARN Client requested shutdown
Shutting down cluster with status SUCCEEDED : Flink YARN Client requested shutdown
Unregistering application from the YARN Resource Manager
Waiting for application to be successfully unregistered.
Can anybody help me understand why does yarn say my flink job was "SUCCEED"?
The reported application status in Yarn does not reflect the status of the executed job but the status of the Flink cluster since this is the Yarn application. Thus, the final status of the Yarn application only depends on whether the Flink cluster finished properly or not. Differently said, if a job fails, then it does not necessarily mean that the Flink cluster failed. These are two different things.
Problem solved: check bottom of posting
I cant connect to websockets. Port 1883 works fine.
This is the output from MQTT.fx:
2017-01-21 07:46:26,293 INFO --- BrokerConnectorController : onConnect
2017-01-21 07:46:26,294 INFO --- ScriptsController : Clear console.
2017-01-21 07:46:26,295 INFO --- MqttFX ClientModel : MqttClient with ID MQTT_FX_Client_Websocket assigned.
2017-01-21 07:46:36,314 ERROR --- MqttFX ClientModel : Error when connecting
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:146) ~[org.eclipse.paho.client.mqttv3-1.1.0.jar:?]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_112]
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(Unknown Source) ~[?:1.8.0_112]
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:65) ~[org.eclipse.paho.client.mqttv3-1.1.0.jar:?]
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:107) ~[org.eclipse.paho.client.mqttv3-1.1.0.jar:?]
... 1 more
2017-01-21 07:46:36,315 ERROR --- MqttFX ClientModel:Please verify your Settings (e.g. Broker Address, Broker Port & Client ID) and the user credentials!
org.eclipse.paho.client.mqttv3.MqttException: Connection lost
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:146) ~[org.eclipse.paho.client.mqttv3-1.1.0.jar:?]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_112]
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(Unknown Source) ~[?:1.8.0_112]
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:65) ~[org.eclipse.paho.client.mqttv3-1.1.0.jar:?]
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:107) ~[org.eclipse.paho.client.mqttv3-1.1.0.jar:?]
... 1 more
2017-01-21 07:46:36,321 INFO --- ScriptsController : Clear console.
2017-01-21 07:46:36,322 ERROR --- BrokerConnectService : Connection lost
I made a Telnet test to the server and port and get a blank terminal.
I guess that means there is a connection because otherwise i would have a "connect failed". The message log plugin shows nothing and there is nothing in the log file.
Its Debian and HiveMQ 3.2.2.
JAVA_OPTS: -Djava.net.preferIPv4Stack=true -XX:-UseSplitVerifier -noverify -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/hivemq/heap-dump.hprof
-------------------------------------------------------------------------
2017-01-21 07:37:02,159 INFO - Starting HiveMQ Server
2017-01-21 07:37:02,176 INFO - HiveMQ version: 3.2.2
2017-01-21 07:37:02,178 INFO - HiveMQ home directory: /opt/hivemq
2017-01-21 07:37:02,226 INFO - Log Configuration was overridden by /opt/hivemq/conf/logback.xml
2017-01-21 07:37:12,013 INFO - Loaded Plugin HiveMQ JMX Metrics Reporting Plugin - v3.0.0
2017-01-21 07:37:12,014 INFO - Loaded Plugin HiveMQ MQTT Message Log Plugin - v3.0.0
2017-01-21 07:37:12,014 INFO - Loaded Plugin HiveMQ Sys Topic Plugin - v3.0.0
2017-01-21 07:37:12,014 INFO - Loaded Plugin HiveMQ JVM Metrics Plugin - v3.1.0
2017-01-21 07:37:12,038 INFO - JMX Metrics Reporting started.
2017-01-21 07:37:12,099 INFO - JMX Metrics Reporting started.
2017-01-21 07:37:12,109 INFO - Starting TCP listener on address 192.168.0.12 and port 1883
2017-01-21 07:37:12,131 INFO - Starting Websocket listener on address 192.168.0.12 and port 8000
2017-01-21 07:37:12,139 INFO - Started TCP Listener on address 192.168.0.12 and on port 1883
2017-01-21 07:37:12,139 INFO - Started Websocket Listener on address 192.168.0.12 and on port 8000
2017-01-21 07:37:12,140 INFO - Started HiveMQ in 9967ms
2017-01-21 07:37:12,142 INFO - No valid license file found. Using evaluation license, restricted to 25 connections.
EDIT
Ok it's a Nginx problem because without SSL my JavaScript site works.
It should be possible to use SSL for http and none secure for the websockets as i can see here:
Nginx MQTT websocket proxy 1
Nginx MQTT websocket proxy 2
I tried there configuration but had no luck.
Solution:
There is no need for an Nginx proxy with HiveMQ.
The Problem is that Firefox does not accept the self signed certificate for websockets. Setting "network.websocket.allowInsecureFromHTTPS" to true will make it work. In Chrome you get a message about JavaScript security and you can accept it. Since i only use Firefox and there is no message it took hours to find out what was wrong. Also the paho onFailure function didn't show up.
unfortunately websocket support has not been implemented into MQTT.fx yet.
You will have to use a different MQTT-client, if you insist on connecting via websocket.
Cheers,
Florian, from the HiveMQ Team.
I am using Akka.Remote to communicate between a server-side service application and multiple desktop client applications. The clients send a request message to the server (using Akka.net) and waits for the server to reply with a response message. The client applications are transient, meaning that they often connect to the server, stay connected for some time, disconnect and then reconnect again.
The problem I encountered is that sometimes when a client disconnects from the server actor (by shutting down its ActorSystem) and then reconnects back to the server, it does not receive any replies from the server for some time. After a few minutes the communication works without any problems. I found out that this issue occurs when the server sends a reply to a client that has disconnected during the request and is no longer reachable. The server cannot deliver the response message and it somehow marks the client endpoint as invalid.
In the log (on the server side) I am getting the following messages when the client is disconnected.
[DEBUG] 2016-01-21 13:04:58.6151 received AutoReceiveMessage <Terminated>: [akka.tcp://qb#client:8090/user/qb] - ExistenceConfirmed=True ServerActor
[DEBUG] 2016-01-21 13:04:58.6550 Stopped Akka.Remote.Transport.ProtocolStateActor
[ INFO] 2016-01-21 13:04:58.6550 Quarantined address [akka.tcp://qb#client:8090] is still unreachable or has not been restarted. Keeping it quarantined. Akka.Event.DummyClassForStringSources
[DEBUG] 2016-01-21 13:04:58.6725 Stopped Akka.Remote.ReliableDeliverySupervisor
[DEBUG] 2016-01-21 13:04:58.6725 no longer watched by [akka://myservice/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fqb%40client%3a8090-2] Akka.Remote.EndpointWriter
[DEBUG] 2016-01-21 13:04:58.6725 Disassociated [akka.tcp://myservice#server:8081] <- akka.tcp://qb#client:8090 Akka.Remote.EndpointWriter
[DEBUG] 2016-01-21 13:04:58.6725 Stopped Akka.Remote.EndpointWriter
And then when the client attempts to reconnect, I get:
[DEBUG] 2016-01-21 13:05:15.5883 ConnectResponse [akka.tcp://qb#client:8090/user/qb] ServerActor
[DEBUG] 2016-01-21 13:05:16.0467 Started (Akka.Remote.Transport.ProtocolStateActor) Akka.Remote.Transport.ProtocolStateActor
[DEBUG] 2016-01-21 13:05:16.0467 Stopped Akka.Remote.Transport.ProtocolStateActor
[ WARN] 2016-01-21 13:05:16.0467 AssociationError [akka.tcp://myservice#server:8081] -> akka.tcp://qb#client:8090: Error [Invalid address: akka.tcp://qb#client:8090] [] Akka.Remote.EndpointWriter
[ INFO] 2016-01-21 13:05:16.0467 Quarantined address [akka.tcp://qb#client:8090] is still unreachable or has not been restarted. Keeping it quarantined. Akka.Event.DummyClassForStringSources
[DEBUG] 2016-01-21 13:05:16.0643 Stopped Akka.Remote.ReliableDeliverySupervisor
[DEBUG] 2016-01-21 13:05:16.0711 no longer watched by [akka://myservice/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fqb%40client%3a8090-4] Akka.Remote.EndpointWriter
[DEBUG] 2016-01-21 13:05:16.0711 Disassociated [akka.tcp://myservice#server:8081] -> akka.tcp://qb#client:8090 Akka.Remote.EndpointWriter
[DEBUG] 2016-01-21 13:05:16.0711 Stopped Akka.Remote.EndpointWriter
[DEBUG] 2016-01-21 13:05:16.0867 received AutoReceiveMessage <Terminated>: [akka://myservice/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fqb%40client%3a8090-4] - ExistenceConfirmed=True Akka.Remote.EndpointManager
[DEBUG] 2016-01-21 13:05:16.0867 Terminated [akka.tcp://qb#client:8090/user/qb] ServerActor
I suspect that this behavior is a feature of Akka.net, however, I need to implement my system so that clients can disconnect and then reconnect back to the server without the need to wait. Is there any way to disable the quarantine mechanism or to gracefully close the client endpoint on the server so that the client endpoint doesn't get quarantined?
[ INFO] 2016-01-21 13:04:58.6550 Quarantined address [akka.tcp://qb#client:8090] is still unreachable or has not been restarted. Keeping it quarantined. - that says it all. The node was quarantined which requires a restart of the actor system.
However, IMHO - just upgrade to Akka.NET 1.0.6, which we released on Monday. We made the remoting policy manager much less brittle than it has been historically.
I'm using the AMQP endpoint and have got it working (with one caveat mentioned here [Possible bug with Mule AMQP transport 3.6.2 community) in other places in my Mule application.
I'm trying to build a reliable SMTP queue but having problems.
I made the assumption that if the SMTP connector couldn't perform, it would throw an exception, so by setting message acknowledgement to MANUAL, and catching SMTP exceptions and rejecting (and requeuing) the message in the exception strategy the system should deal with SMTP outages.
My code (slightly simplified) looks like this:
<amqp:connector name="connector.amqp.mule.default" ackMode="MANUAL" doc:name="AMQP Connector" validateConnections="true" host="${amiab-rabbitmq-hostname}" port="${amiab-rabbitmq-portnumber}" password="${amiab-rabbitmq-password}" username="${amiab-rabbitmq-username}"/>
<smtp:connector name="smtpConnector" contentType="text/html" doc:name="SMTPConnector"/>
<flow name="utility-smtpFlow">
<amqp:inbound-endpoint exchangeName="AMQP.DEFAULT.EXCHANGE" queueName="utilitySMTP" exchangeType="direct" responseTimeout="10000" doc:name="AMQP-0-9" queueDurable="true"/>
<logger message="Recevied SMTP Message from Queue" level="INFO" doc:name="Logger"/>
<smtp:outbound-endpoint host="${smtp-hostname}" user="${smtp-username}" password="{smtp-password}" to="#[flowVars.smtpTo]" subject="Testing Message" responseTimeout="10000" doc:name="SMTP"/>
<amqp:acknowledge-message doc:name="AMQP-0-9 Acknowledge Message"/>
<catch-exception-strategy doc:name="Catch Exception Strategy">
<logger message="SMTP Error, Requeuing" level="INFO" doc:name="Logger"/>
<amqp:reject-message requeue="true" doc:name="AMQP-0-9 Reject Message"/>
</catch-exception-strategy>
When my SMTP connector throws an exception, the exception strategy is correctly instigated, but then AMQP throws an error as shown below:
INFO 2015-06-18 16:32:22,383 [HTTP_Listener_AMIAB.worker.01] org.mule.transport.service.DefaultTransportServiceDescriptor: Loading default outbound transformer: org.mule.transport.amqp.internal.transformer.ObjectToAmqpMessage
INFO 2015-06-18 16:32:22,383 [HTTP_Listener_AMIAB.worker.01] org.mule.transport.service.DefaultTransportServiceDescriptor: Loading default response transformer: org.mule.transport.amqp.internal.transformer.ObjectToAmqpMessage
INFO 2015-06-18 16:32:22,387 [HTTP_Listener_AMIAB.worker.01] org.mule.lifecycle.AbstractLifecycleManager: Initialising: 'connector.amqp.mule.default.dispatcher.936721322'. Object is: Dispatcher
INFO 2015-06-18 16:32:22,389 [HTTP_Listener_AMIAB.worker.01] org.mule.lifecycle.AbstractLifecycleManager: Starting: 'connector.amqp.mule.default.dispatcher.936721322'. Object is: Dispatcher
INFO 2015-06-18 16:32:22,446 [[amiab-esb-utility].utility-smtpFlow.stage1.02] org.mule.api.processor.LoggerMessageProcessor: Recevied SMTP Message from Queue
INFO 2015-06-18 16:32:22,457 [[amiab-esb-utility].smtpConnector.dispatcher.01] org.mule.transport.service.DefaultTransportServiceDescriptor: Loading default outbound transformer: org.mule.transport.email.transformers.ObjectToMimeMessage
INFO 2015-06-18 16:32:22,481 [[amiab-esb-utility].smtpConnector.dispatcher.01] org.mule.lifecycle.AbstractLifecycleManager: Initialising: 'smtpConnector.dispatcher.404056326'. Object is: SmtpMessageDispatcher
ERROR 2015-06-18 16:32:22,568 [[amiab-esb-utility].smtpConnector.dispatcher.01] org.mule.exception.CatchMessagingExceptionStrategy:
********************************************************************************
Message : Unable to connect to mail transport.
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. mail.lyrical.co.uk (java.net.UnknownHostException)
java.net.AbstractPlainSocketImpl:178 (null)
2. Unknown SMTP host: mail.lyrical.co.uk (javax.mail.MessagingException)
com.sun.mail.smtp.SMTPTransport:1704 (http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/mail/MessagingException.html)
3. Unable to connect to mail transport. (org.mule.api.endpoint.EndpointException)
org.mule.transport.email.SmtpMessageDispatcher:67 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/endpoint/EndpointException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.net.UnknownHostException: mail.lyrical.co.uk
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
INFO 2015-06-17 15:25:47,190 [[amiab-esb-utility].smtpConnector.dispatcher.01] org.mule.api.processor.LoggerMessageProcessor: SMTP Error, Requeuing
WARN 2015-06-17 15:25:47,193 [amqpReceiver.01] org.mule.transport.amqp.internal.endpoint.receiver.MessageReceiverConsumer: Received shutdown signal for consumer tag: amq.ctag-NZ96mnj_oscnpJKAPy16ng, the message receiver will try to restart.
com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=90)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:478) ~[?:?]
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315) ~[?:?]
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144) ~[?:?]
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91) ~[?:?]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:550) ~[?:?]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]
INFO 2015-06-17 15:25:47,195 [amqpReceiver.01] org.mule.lifecycle.AbstractLifecycleManager: Stopping: 'null'. Object is: MultiChannelMessageSubReceiver
INFO 2015-06-17 15:25:47,195 [amqpReceiver.01] org.mule.transport.amqp.internal.endpoint.receiver.MultiChannelMessageSubReceiver: Connecting clusterizable message receiver
INFO 2015-06-17 15:25:47,195 [amqpReceiver.01] org.mule.lifecycle.AbstractLifecycleManager: Starting: 'null'. Object is: MultiChannelMessageSubReceiver
INFO 2015-06-17 15:25:47,196 [amqpReceiver.01] org.mule.transport.amqp.internal.endpoint.receiver.MultiChannelMessageSubReceiver: Starting clusterizable message receiver
INFO 2015-06-17 15:25:47,203 [amqpReceiver.01] org.mule.transport.amqp.internal.endpoint.receiver.MultiChannelMessageSubReceiver: Started subscription: amq.ctag-sCCRCnOWKyREyip26pSYDA on channel: AMQChannel(amqp://guest#127.0.0.1:5672/,3)
Have I missed something obvious, or am I misunderstanding how this should be used? Very grateful if anyone can point me in the right direction.
As you have noted in your comment, the smtp:outbound-endpoint being a one-way endpoint, its dispatch operation is performed in another thread, thus it occurs in parallel with the execution of the flow. Therefore, amqp:acknowledge-message gets called anyway, independently of the success or failure of the smtp:outbound-endpoint operation.
You can get around this issue by setting processingStrategy="synchronous" on the flow: this will force all the endpoint interactions to be performed on the inbound thread.
Shameless plug: this is discussed in details in chapter 11.2 of Mule in Action, Second Edition.
I am using mulestudio.
I wish to insert the values in jms queue using mule studio .But i have done all required changes but queue is not creating in activemq
I am using activemq-5.8.0 version even i added jar file
my config is
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns:jms="http://www.mulesoft.org/schema/mule/jms" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.4.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd
http://www.mulesoft.org/schema/mule/jms http://www.mulesoft.org/schema/mule/jms/current/mule-jms.xsd">
<jms:activemq-connector name="jmsConnector"
brokerURL="tcp://localhost:61616"
maxRedelivery="30"
disableTemporaryReplyToDestinations="true"
acknowledgementMode="CLIENT_ACKNOWLEDGE"
persistentDelivery="true" doc:name="Active MQ" password="admin" username="admin" validateConnections="true">
</jms:activemq-connector> <flow name="Acivemqdemo_flow" doc:name="Acivemqdemo_flow">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8087" doc:name="HTTP"/>
<jms:outbound-endpoint queue="queue" connector-ref="jmsConnector" doc:name="JMS">
<jms:transaction action="NONE"/>
</jms:outbound-endpoint>
</flow>
</mule>
i have done as per docs but unable get the expected queue
my error log is
INFO 2014-01-03 16:20:09,890 [main] org.mule.lifecycle.AbstractLifecycleManager: Starting connector: jmsConnector
INFO 2014-01-03 16:20:09,911 [main] org.mule.lifecycle.AbstractLifecycleManager: Starting model: _muleSystemModel
INFO 2014-01-03 16:20:09,912 [main] org.mule.construct.FlowConstructLifecycleManager: Starting flow: Acivemqdemo_flow
INFO 2014-01-03 16:20:09,913 [main] org.mule.processor.SedaStageLifecycleManager: Starting service: Acivemqdemo_flow.stage1
INFO 2014-01-03 16:20:09,919 [main] org.mule.transport.http.HttpConnector: Registering listener: Acivemqdemo_flow on endpointUri: http://localhost:8087
INFO 2014-01-03 16:20:09,924 [main] org.mule.transport.service.DefaultTransportServiceDescriptor: Loading default response transformer: org.mule.transport.http.transformers.MuleMessageToHttpResponse
INFO 2014-01-03 16:20:09,938 [main] org.mule.lifecycle.AbstractLifecycleManager: Initialising: 'null'. Object is: HttpMessageReceiver
INFO 2014-01-03 16:20:09,940 [main] org.mule.transport.http.HttpMessageReceiver: Connecting clusterizable message receiver
ERROR 2014-01-03 16:20:09,942 [main] org.mule.exception.DefaultSystemExceptionStrategy:
********************************************************************************
Message : Address already in use (java.net.BindException)
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. Address already in use (java.net.BindException)
java.net.PlainSocketImpl:-2 (null)
2. Address already in use (java.net.BindException) (org.mule.transport.ConnectException)
org.mule.transport.http.HttpConnectionManager:73 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/transport/ConnectException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:353)
at java.net.ServerSocket.bind(ServerSocket.java:336)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
INFO 2014-01-03 16:20:09,944 [main] org.mule.exception.DefaultSystemExceptionStrategy: Exception caught is a ConnectException, attempting to reconnect...
INFO 2014-01-03 16:20:09,944 [main] org.mule.lifecycle.AbstractLifecycleManager: Stopping connector: connector.http.mule.default
INFO 2014-01-03 16:20:09,945 [main] org.mule.lifecycle.AbstractLifecycleManager: Stopping: 'null'. Object is: HttpMessageReceiver
ERROR 2014-01-03 16:20:09,946 [main] org.mule.transport.http.HttpConnector: null
java.lang.NullPointerException
at org.mule.transport.http.HttpConnector.disconnect(HttpConnector.java:542)
at org.mule.transport.http.HttpMessageReceiver.doDisconnect(HttpMessageReceiver.java:52)
at org.mule.transport.AbstractTransportMessageHandler.disconnect(AbstractTransportMessageHandler.java:251)
at org.mule.transport.AbstractConnector.disconnect(AbstractConnector.java:1656)
at org.mule.exception.AbstractSystemExceptionStrategy.handleReconnection(AbstractSystemExceptionStrategy.java:97)
at org.mule.exception.AbstractSystemExceptionStrategy.handleException(AbstractSystemExceptionStrategy.java:59)
at org.mule.exception.AbstractSystemExceptionStrategy.handleException(AbstractSystemExceptionStrategy.java:77)
at org.mule.lifecycle.AbstractLifecycleManager.invokePhase(AbstractLifecycleManager.java:148)
at org.mule.construct.FlowConstructLifecycleManager.fireStartPhase(FlowConstructLifecycleManager.java:95)
at org.mule.construct.AbstractFlowConstruct.start(AbstractFlowConstruct.java:140)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.mule.lifecycle.phases.DefaultLifecyclePhase.applyLifecycle(DefaultLifecyclePhase.java:225)
at org.mule.lifecycle.RegistryLifecycleManager$RegistryLifecycleCallback.onTransition(RegistryLifecycleManager.java:276)
at org.mule.lifecycle.RegistryLifecycleManager.invokePhase(RegistryLifecycleManager.java:155)
at org.mule.lifecycle.RegistryLifecycleManager.fireLifecycle(RegistryLifecycleManager.java:126)
at org.mule.registry.AbstractRegistryBroker.fireLifecycle(AbstractRegistryBroker.java:80)
at org.mule.registry.MuleRegistryHelper.fireLifecycle(MuleRegistryHelper.java:120)
at org.mule.lifecycle.MuleContextLifecycleManager$MuleContextLifecycleCallback.onTransition(MuleContextLifecycleManager.java:94)
at org.mule.lifecycle.MuleContextLifecycleManager$MuleContextLifecycleCallback.onTransition(MuleContextLifecycleManager.java:90)
at org.mule.lifecycle.MuleContextLifecycleManager.invokePhase(MuleContextLifecycleManager.java:72)
at org.mule.lifecycle.MuleContextLifecycleManager.fireLifecycle(MuleContextLifecycleManager.java:64)
at org.mule.DefaultMuleContext.start(DefaultMuleContext.java:255)
at org.mule.module.launcher.application.DefaultMuleApplication.start(DefaultMuleApplication.java:147)
at org.mule.module.launcher.application.ApplicationWrapper.start(ApplicationWrapper.java:107)
at org.mule.module.launcher.DefaultMuleDeployer.deploy(DefaultMuleDeployer.java:48)
at org.mule.tooling.server.application.ApplicationDeployer.main(ApplicationDeployer.java:127)
INFO 2014-01-03 16:20:09,948 [main] org.mule.transport.http.HttpConnector: Disconnected: HttpConnector
{
name=connector.http.mule.default
lifecycle=stop
this=4f980c26
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=true
supportedProtocols=[http]
serviceOverrides=<none>
}
INFO 2014-01-03 16:20:09,948 [main] org.mule.transport.http.HttpMessageReceiver: Connecting clusterizable message receiver
ERROR 2014-01-03 16:20:09,948 [main] org.mule.retry.notifiers.ConnectNotifier: Failed to connect/reconnect: HttpConnector
{
name=connector.http.mule.default
lifecycle=stop
this=4f980c26
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=false
supportedProtocols=[http]
serviceOverrides=<none>
}
. Root Exception was: null. Type: class java.lang.NullPointerException
ERROR 2014-01-03 16:20:09,951 [main] org.mule.exception.DefaultSystemExceptionStrategy: null
INFO 2014-01-03 16:20:09,958 [main] org.mule.module.management.agent.WrapperManagerAgent: This JVM hasn't been launched by the wrapper, the agent will not run.
INFO 2014-01-03 16:20:09,969 [main] org.mule.module.management.agent.JmxAgent: Attempting to register service with name: Mule.acivemqjms:type=Endpoint,service="Acivemqdemo_flow",connector=connector.http.mule.default,name="endpoint.http.localhost.8087"
INFO 2014-01-03 16:20:09,969 [main] org.mule.module.management.agent.JmxAgent: Registered Endpoint Service with name: Mule.acivemqjms:type=Endpoint,service="Acivemqdemo_flow",connector=connector.http.mule.default,name="endpoint.http.localhost.8087"
INFO 2014-01-03 16:20:09,970 [main] org.mule.module.management.agent.JmxAgent: Registered Connector Service with name Mule.acivemqjms:type=Connector,name="connector.http.mule.default.1"
INFO 2014-01-03 16:20:09,970 [main] org.mule.module.management.agent.JmxAgent: Registered Connector Service with name Mule.acivemqjms:type=Connector,name="jmsConnector.1"
INFO 2014-01-03 16:20:09,972 [main] org.mule.module.launcher.application.DefaultMuleApplication: Reload interval: 3000
INFO 2014-01-03 16:20:09,974 [main] org.mule.DefaultMuleContext:
**********************************************************************
* Application: acivemqjms *
* OS encoding: UTF-8, Mule encoding: UTF-8 *
* *
* Agents Running: *
* Clustering Agent *
* JMX Agent *
**********************************************************************
INFO 2014-01-03 16:20:09,974 [main] org.mule.module.launcher.MuleDeploymentService:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Started app 'acivemqjms' +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
will you help for this and how to consume the message from mule give me any example doc
the mule application cannot start it cannot connect to the http port 8087
it is most likely because you already have an instance running.
Try in order of esculation:
hit the red button in the console window
restart eclipse studio
or if you need to close the eclipse studio, check if you have any java processes running other than activemq
hope that helps.
In Ubuntu use this kill the process and release the port.
fuser -k 80/tcp
replace 80 with port number you need to release.
In Mac, find and kill the PID with netstat.
From your exception java.net.BindException: Address already in use , it seems your inbound address, port is already used by some other endpoint, either by some other Mule endpoint or any other external application.
Solution is, please check once if the port or address is used by other application.
To check, use the command netstat in the following ways to check your port from command prompt :- Command line for looking at specific port