I am encountering the error when trying out my app. It works fine on my local instance as well as on a stand-alone Mule Server, but not on CloudHub. Any reason why this problem occurs?
Update
The app doesn't have any issues running from Studio on my local machine. The problem only occurred after I deployed it to CloudHub. I am not able to access the app from CloudHub and encounter a 405 Not Allowed (nginx) error page.
CloudHub logs shows the app has started successfully as far as I understand it. See the CloudHub deployment logs below:
19:29:02.272 | 07/10/2015 | SYSTEM | Deploying application to 1 workers.
19:29:05.450 | 07/10/2015 | SYSTEM | Provisioning CloudHub worker...
19:29:07.277 | 07/10/2015 | SYSTEM | Deploying application to 1 workers.
19:29:08.325 | 07/10/2015 | SYSTEM | Provisioning CloudHub worker...
19:29:25.696 | 07/10/2015 | SYSTEM | Starting CloudHub worker at 52.74.74.41 ...
19:29:32.862 | 07/10/2015 | SYSTEM | Starting CloudHub worker at 54.169.12.90 ...
19:30:49.027 | 07/10/2015 | SYSTEM | Worker(54.169.12.90): Starting your application...
19:30:50.715 | 07/10/2015 | INFO |
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Initializing app 'myapp' +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
19:30:50.958 | 07/10/2015 | INFO | Monitoring enabled: true
19:30:50.958 | 07/10/2015 | INFO | Registering ping flow injector...
19:30:50.975 | 07/10/2015 | INFO | Creating the PersistentQueueManager. Asynchronous VM queues will be persistent.
19:30:50.984 | 07/10/2015 | INFO | About to create the PersistentQueueManager with environment: scope: 55824160e4b0f5ecab93e207_559f73dfe4b02b744ec0f626 bucket: ch-persistent-queues-us-east-prod accessKey: AKIAIYBZZZI7M2PAT4WQ region: ap-southeast-1 privateKey: no
19:30:53.393 | 07/10/2015 | INFO | Apply ap-southeast-1 to the client com.amazonaws.services.sqs.AmazonSQSClient#5c6c69b6
19:30:53.393 | 07/10/2015 | INFO | New sqsClient is created com.amazonaws.services.sqs.AmazonSQSClient#5c6c69b6
19:30:53.412 | 07/10/2015 | INFO | Apply ap-southeast-1 to the client com.amazonaws.services.s3.AmazonS3Client#4a1e0237
19:30:53.469 | 07/10/2015 | INFO | New s3Client is created com.mulesoft.ch.queue.sqs.clients.S3Client#20d271f6
19:30:53.470 | 07/10/2015 | INFO | Done with SQSQueueManagementService() instance
19:30:53.505 | 07/10/2015 | INFO | Initialising RegistryBroker
19:30:53.760 | 07/10/2015 | INFO | Refreshing org.mule.config.spring.MuleArtifactContext#44b5763: startup date [Fri Jul 10 11:30:53 UTC 2015]; root of context hierarchy
19:30:57.438 | 07/10/2015 | WARN | Schema warning: Use of element <response-builder> is deprecated. HTTP transport is deprecated and will be removed in Mule 4.0. Use HTTP module instead..
19:30:57.630 | 07/10/2015 | WARN | Schema warning: Use of element <header> is deprecated. HTTP transport is deprecated and will be removed in Mule 4.0. Use HTTP module instead..
19:30:58.537 | 07/10/2015 | INFO | Using files for tx logs /opt/mule/mule-3.6.2-R43/./.mule/myapp/queue-tx-log/tx1.log and /opt/mule/mule-3.6.2-R43/./.mule/myapp/queue-tx-log/tx2.log
19:30:58.538 | 07/10/2015 | INFO | Using files for tx logs /opt/mule/mule-3.6.2-R43/./.mule/myapp/queue-xa-tx-log/tx1.log and /opt/mule/mule-3.6.2-R43/./.mule/myapp/queue-xa-tx-log/tx2.log
19:30:58.632 | 07/10/2015 | INFO | Initialising model: _muleSystemModel
19:30:59.749 | 07/10/2015 | INFO | Initialising flow: myappFlow
19:30:59.749 | 07/10/2015 | INFO | Initialising exception listener: org.mule.exception.DefaultMessagingExceptionStrategy#4fe27195
19:30:59.856 | 07/10/2015 | INFO | Initialising service: myappFlow.stage1
19:31:00.389 | 07/10/2015 | INFO | Registering the queue=55824160e4b0f5ecab93e207_559f73dfe4b02b744ec0f626_2D1531125C052F65252D3BF0500990 with display name=seda.queue(myappFlow.stage1)
19:31:00.402 | 07/10/2015 | INFO | Configured Mule using "org.mule.config.spring.SpringXmlConfigurationBuilder" with configuration resource(s): "[ConfigResource{resourceName='/opt/mule/mule-3.6.2-R43/apps/myapp/myapp.xml'}]"
19:31:00.403 | 07/10/2015 | INFO | Configured Mule using "org.mule.config.builders.AutoConfigurationBuilder" with configuration resource(s): "[ConfigResource{resourceName='/opt/mule/mule-3.6.2-R43/apps/myapp/myapp.xml'}]"
19:31:00.430 | 07/10/2015 | INFO | Apply ap-southeast-1 to the client com.amazonaws.services.sqs.AmazonSQSClient#1c5e47e5
19:31:00.430 | 07/10/2015 | INFO | New sqsClient is created com.amazonaws.services.sqs.AmazonSQSClient#1c5e47e5
19:31:00.448 | 07/10/2015 | INFO | Apply ap-southeast-1 to the client com.amazonaws.services.s3.AmazonS3Client#7b8419b2
19:31:00.468 | 07/10/2015 | INFO | New s3Client is created com.mulesoft.ch.queue.sqs.clients.S3Client#2c611f62
19:31:00.473 | 07/10/2015 | INFO | Done with SQSQueueManagementService() instance
19:31:00.504 | 07/10/2015 | INFO |
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Starting app 'myapp' +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
19:31:00.561 | 07/10/2015 | INFO | Starting ResourceManager
19:31:00.561 | 07/10/2015 | INFO | Starting persistent queue manager
19:31:00.562 | 07/10/2015 | INFO | Started ResourceManager
19:31:00.574 | 07/10/2015 | INFO | Successfully opened CloudHub ObjectStore
19:31:00.590 | 07/10/2015 | INFO |
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ DevKit Extensions (0) used in this application +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
19:31:00.591 | 07/10/2015 | INFO | Starting model: _muleSystemModel
19:31:00.594 | 07/10/2015 | INFO | Starting flow: myappFlow
19:31:00.595 | 07/10/2015 | INFO | Starting service: myappFlow.stage1
19:31:00.881 | 07/10/2015 | WARN | HTTP transport is deprecated and will be removed in Mule 4.0. Use HTTP module instead.
19:31:00.947 | 07/10/2015 | INFO | Initialising connector: connector.http.mule.default
19:31:01.156 | 07/10/2015 | INFO | No suitable connector in application, new one is created and used.
19:31:01.158 | 07/10/2015 | INFO | Connected: HttpConnector
{
name=connector.http.mule.default
lifecycle=initialise
this=5f68f6a9
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=true
supportedProtocols=[http]
serviceOverrides=<none>
}
19:31:01.158 | 07/10/2015 | INFO | Starting: HttpConnector
{
name=connector.http.mule.default
lifecycle=initialise
this=5f68f6a9
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=true
supportedProtocols=[http]
serviceOverrides=<none>
}
19:31:01.159 | 07/10/2015 | INFO | Starting connector: connector.http.mule.default
19:31:01.169 | 07/10/2015 | INFO | Initialising flow: ____ping____flow_____76326334
19:31:01.169 | 07/10/2015 | INFO | Initialising exception listener: org.mule.exception.DefaultMessagingExceptionStrategy#66551924
19:31:01.186 | 07/10/2015 | INFO | Initialising service: ____ping____flow_____76326334.stage1
19:31:01.380 | 07/10/2015 | INFO | Starting flow: ____ping____flow_____76326334
19:31:01.381 | 07/10/2015 | INFO | Starting service: ____ping____flow_____76326334.stage1
19:31:01.387 | 07/10/2015 | INFO | Registering listener: ____ping____flow_____76326334 on endpointUri: http://localhost:7557/____ping____
19:31:01.459 | 07/10/2015 | INFO | Loading default response transformer: org.mule.transport.http.transformers.MuleMessageToHttpResponse
19:31:01.542 | 07/10/2015 | INFO | Initialising: 'null'. Object is: HttpMessageReceiver
19:31:01.559 | 07/10/2015 | INFO | Connecting clusterizable message receiver
19:31:01.563 | 07/10/2015 | WARN | Localhost is being bound to all local interfaces as specified by the "mule.tcp.bindlocalhosttoalllocalinterfaces" system property. This property may be removed in a future version of Mule.
19:31:01.575 | 07/10/2015 | INFO | Starting clusterizable message receiver
19:31:01.575 | 07/10/2015 | INFO | Starting: 'null'. Object is: HttpMessageReceiver
19:31:01.644 | 07/10/2015 | INFO | Listening for requests on http://localhost:8085
19:31:01.645 | 07/10/2015 | INFO | Mule is embedded in a container already launched by a wrapper.Duplicates will not be registered. Use the org.tanukisoftware.wrapper:type=WrapperManager MBean instead for control.
19:31:01.675 | 07/10/2015 | INFO | Attempting to register service with name: Mule.myapp:type=Endpoint,service="____ping____flow_____76326334",connector=connector.http.mule.default,name="endpoint.http.localhost.7557.ping"
19:31:01.676 | 07/10/2015 | INFO | Registered Endpoint Service with name: Mule.myapp:type=Endpoint,service="____ping____flow_____76326334",connector=connector.http.mule.default,name="endpoint.http.localhost.7557.ping"
19:31:01.686 | 07/10/2015 | INFO | Registered Connector Service with name Mule.myapp:type=Connector,name="connector.http.mule.default.1"
19:31:01.689 | 07/10/2015 | INFO |
**********************************************************************
* Application: myapp *
* OS encoding: /, Mule encoding: UTF-8 *
* *
* Agents Running: *
* DevKit Extension Information *
* Batch module default engine *
* Clustering Agent *
* JMX Agent *
**********************************************************************
19:31:01.754 | 07/10/2015 | INFO | Registering the queue=55824160e4b0f5ecab93e207_559f73dfe4b02b744ec0f626_515E160FC244EB5157E8E7C8B162E0 with display name=seda.queue(____ping____flow_____76326334.stage1)
19:31:02.561 | 07/10/2015 | SYSTEM | Worker(54.169.12.90): Your application has started successfully.
19:31:09.295 | 07/10/2015 | INFO | Mule system health monitoring started for your application.
19:31:59.537 | 07/10/2015 | SYSTEM | Worker(52.74.74.41): Starting your application...
19:32:01.200 | 07/10/2015 | INFO |
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Initializing app 'myapp' +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
19:32:01.525 | 07/10/2015 | INFO | Creating the PersistentQueueManager. Asynchronous VM queues will be persistent.
19:32:01.526 | 07/10/2015 | INFO | About to create the PersistentQueueManager with environment: scope: 55824160e4b0f5ecab93e207_559f73dfe4b02b744ec0f626 bucket: ch-persistent-queues-us-east-prod accessKey: AKIAIYBZZZI7M2PAT4WQ region: ap-southeast-1 privateKey: no
19:32:07.054 | 07/10/2015 | INFO |
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Stopping app 'myapp' +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
19:32:07.056 | 07/10/2015 | INFO | Stopping flow: ____ping____flow_____807729792
19:32:07.056 | 07/10/2015 | INFO | Removing listener on endpointUri: http://localhost:7557/____ping____
19:32:07.057 | 07/10/2015 | INFO | Stopping: 'null'. Object is: HttpMessageReceiver
19:32:07.059 | 07/10/2015 | INFO | Disposing: 'null'. Object is: HttpMessageReceiver
19:32:07.059 | 07/10/2015 | INFO | Stopping service: ____ping____flow_____807729792.stage1
19:32:12.028 | 07/10/2015 | SYSTEM | Worker(52.74.74.41): Your application has started successfully.
19:32:15.041 | 07/10/2015 | INFO | Mule system health monitoring started for your application.
19:32:33.488 | 07/10/2015 | INFO | Stopping flow: myappFlow
19:32:33.489 | 07/10/2015 | INFO | Stopping service: myappFlow.stage1
19:32:37.061 | 07/10/2015 | SYSTEM | Application was updated successfully with zero downtime. The new version of your application has been launched and the old version has been stopped.
19:32:37.545 | 07/10/2015 | SYSTEM | Your application is started.
I found the solution to my problem. From the deployment logs:
19:31:01.644 | 07/10/2015 | INFO | Listening for requests on http://localhost:8085
The host should have been the default All interfaces [0.0.0.0], and the port 8085 is not allowed on CloudHub. And I must use the default port 8081.
I was running multiple Mule apps on my local machine at the same time, that's why I had used a different port for this app, and I wasn't aware that port 8085 is not allowed on CloudHub.
Thank you all for the response.
for the port please try using ${http.port} or ${https.port} as per your requirement when deployed to Cloud hub.
It worked for me.
Related
I have the following code which handles some websocket messages:
Flux.create(sink -> {
handler//.log("handler")
.doOnNext(s -> log.info("propagating: {}", s))
.doOnNext(sink::next)
.doOnError(sink::error)
.onErrorComplete() // this allows you to silence the error and let the upstream subscriber handle it.
.publishOn(Schedulers.boundedElastic())
.subscribeOn(Schedulers.boundedElastic(), false)
.subscribe();
})
.take(10)
.onErrorResume(e -> Exceptions.unwrap(e)
.getClass()
.isAssignableFrom(Errors.NativeIoException.class),
t -> {
log.error("Websocket connection error: {}", t.getMessage());
log.debug("{}", t.fillInStackTrace().toString());
// the error is handled and hidden by retryWhen.
return Mono.error(t);
})
.retryWhen(RetryBackoffSpec.backoff(10000, Duration.ofSeconds(3))
.maxBackoff(Duration.ofSeconds(3))
.transientErrors(true)
.doBeforeRetry((s) -> log.error("Retrying connection to {}", "abc"))
.doAfterRetry(s -> log.error("Attempt {}/{} to restore connection to {} failed with {}", s.totalRetriesInARow(), 10000, "abc", s.failure().getMessage()))
);
Every now and then the connection drops and that's why there's a retryWhen operator in the pipe. As you can see above, I am printing messages to the console which inform about the connection drop and how many times it retries.
I, however, am not able to figure out how to print a "recover" message (i.e. Connection to X restored). Am I missing something from the docs or am I expected to write a custom RetryBackoffSpec that does it?
Example log output:
03:11:34.205 10-02-2023 | INFO | boundedElastic-6 | org.example.chaos.simulation.NetworkDisruptionTest | propagating: {"timestamp":1675991494202,"hello":"world!"}
03:11:34.703 10-02-2023 | INFO | boundedElastic-6 | org.example.chaos.simulation.NetworkDisruptionTest | propagating: {"timestamp":1675991494702,"hello":"world!"}
03:11:35.205 10-02-2023 | INFO | boundedElastic-6 | org.example.chaos.simulation.NetworkDisruptionTest | propagating: {"timestamp":1675991495203,"hello":"world!"}
03:11:35.704 10-02-2023 | INFO | boundedElastic-6 | org.example.chaos.simulation.NetworkDisruptionTest | propagating: {"timestamp":1675991495703,"hello":"world!"}
03:11:46.746 10-02-2023 | ERROR | boundedElastic-6 | org.example.chaos.simulation.NetworkDisruptionTest | Websocket connection error: recvAddress(..) failed: Connection timed out
03:11:46.749 10-02-2023 | ERROR | boundedElastic-6 | org.example.chaos.simulation.NetworkDisruptionTest | Retrying connection to abc
03:11:49.752 10-02-2023 | ERROR | parallel-3 | org.example.chaos.simulation.NetworkDisruptionTest | Attempt 0/10000 to restore connection to abc failed with recvAddress(..) failed: Connection timed out
03:11:52.763 10-02-2023 | ERROR | boundedElastic-5 | org.example.chaos.simulation.NetworkDisruptionTest | Retrying connection to abc
03:11:55.764 10-02-2023 | ERROR | parallel-4 | org.example.chaos.simulation.NetworkDisruptionTest | Attempt 1/10000 to restore connection to abc failed with connection timed out: /172.25.0.2:8090
03:11:58.772 10-02-2023 | ERROR | boundedElastic-3 | org.example.chaos.simulation.NetworkDisruptionTest | Retrying connection to abc
03:12:01.773 10-02-2023 | ERROR | parallel-5 | org.example.chaos.simulation.NetworkDisruptionTest | Attempt 2/10000 to restore connection to abc failed with connection timed out: /172.25.0.2:8090
--- A message such as "Connection to abc has been restored." is expected to appear here.
I'm trying to run the monitoring service in openstack, but I'm receiving this error:
~$ monasca metric-list
Authorization failed: The resource could not be found. (HTTP 404)
When I check the log file this is why I found:
2016-09-20 10:27:36.357 27771 WARNING keystonemiddleware.auth_token [-] Fetch revocation list failed, fallback to online validation.
2016-09-20 10:27:36.376 27771 ERROR keystonemiddleware.auth_token [-] Bad response code while validating token: 403
2016-09-20 10:27:36.377 27771 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "You are not authorized to perform the requested action, identity:validate_token.", "code": 403, "title": "Forbidden"}}
2016-09-20 10:27:36.377 27771 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Failed to fetch token data from identity server
This is the services, projects, users , role and endpoints in keystone
+----------------------------------+----------+------------+
| ID | Name | Type |
+----------------------------------+----------+------------+
| 1c38cf31124d404783561793fc1fb7f0 | monasca | monitoring |
| 1eb72109ea604b6e8f2bd264787ca370 | keystone | identity |
+----------------------------------+----------+------------+
+----------------------------------+---------+
| ID | Name |
+----------------------------------+---------+
| 733a0a1369f94f6ab31b8875ef19e0ee | service |
| 9e732f1a2aca48e098daf62bb230f85e | monasca |
| f2df2111f893434f83fda7d5bd6cac4a | admin |
+----------------------------------+---------+
+----------------------------------+---------------+
| ID | Name |
+----------------------------------+---------------+
| 3a1b8582a11f4e07b3a21e84e9fb7c23 | monasca-user |
| 559752237e824d81a6133494b63c5789 | monasca-agent |
| 5bcf19af4e8e4067a5679e6a0f2f88f1 | admin |
+----------------------------------+---------------+
+----------------------------------+---------------+
| ID | Name |
+----------------------------------+---------------+
| 1679c1c099b543db96ac4412be21b15a | admin |
| 6ca31578625c49568085284dee72e4b8 | monasca-agent |
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
| a3267f589e7342ceaedef57ea9e4aac2 | monasca-user |
+----------------------------------+---------------+
+----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
| 3fbfc68e9f894e47846b896c6c8d3f3e | RegionOne | keystone | identity | True | internal | http://controller:5000/v2.0 |
| 470043a7f6364add902548df6fb7b60e | RegionOne | monasca | monitoring | True | public | http://localhost:8082/v2.0 |
| 9e68606b37084cbeb95106ff1bede0cb | RegionOne | monasca | monitoring | True | internal | http://localhost:8082/v2.0 |
| b4273c72671e4fac99e7d2bc6334156c | RegionOne | monasca | monitoring | True | admin | http://localhost:8082/v2.0 |
| d27bb34d619443658ca745b9fee1c967 | RegionOne | keystone | identity | True | admin | http://controller:35357/v2.0 |
| f736ebca8ac24b78bdf1dff60ac86ab1 | RegionOne | keystone | identity | True | public | http://controller:5000/v2.0 |
+----------------------------------+-----------+--------------+--------------+---------+-----------+------------------------------+
this is the keystone section in my api.conf file:
[keystone_authtoken]
identity_uri = http://controller:35357
auth_uri = http://controller:5000/v3
admin_password = PASSWORD
admin_user = monasca-user
admin_tenant_name = monasca
cafile =
certfile =
keyfile =
insecure = false
file to get the token for the identity service
export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=PASSWORD
export OS_AUTH_URL=http://controller:35357/v3
export OS_IDENTITY_API_VERSION=3
and for the monitoring service
export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_NAME=monasca
export OS_TENANT_NAME=monasca
export OS_USERNAME=monasca-user
export OS_PASSWORD=PASSWORD
export OS_AUTH_URL=http://controller:35357/
export OS_IDENTITY_API_VERSION=3
I can find what is wrong with the configuration
Are you able to run the following command?
openstack token issue
Why do you have localhost in endpoint?
you can use --debug option for monasca commands to get more details
I think the problem is that you register keystone endpoint with version specified while in env var the version is another one. it is recommended that not include version number in endpoint, especially the keystone one. Please check install manuals and follow its instructions
Have been testing out GCE and the load balancing capabilities - however have been seeing some unexpected results.
The trial configuration involves 2 instances acting as DNS resolvers in a target pool with a 3rd test instance. There is also a http server running on the hosts. No health check scripts have been added.
DNS request to individual instance public IP (from ANY) - OK
HTTP request to individual instance public IP (from ANY) - OK
HTTP request to load balance IP (from ANY) - OK
DNS request to load balance IP (from an instance in the target pool) - OK
DNS request to load balance IP (from an instance in the same network - but not in the target pool) - NOK
DNS request to load balance IP (other) - NOK
I can see in the instance logs that the DNS request arrive for all cases and are distributed evenly - though the replies don't seem to get back to the originator.
The behavior seems unexpected. I've played with the session affinity with similar results - though the default behavior is the most desired option.
Have hit a wall. Are there some ideas to try?
Information on the setup:
$ gcutil listhttphealthchecks
+------+------+------+
| name | host | port |
+------+------+------+
$ gcutil listtargetpools
+----------+-------------+
| name | region |
+----------+-------------+
| dns-pool | us-central1 |
+----------+-------------+
$ gcutil listforwardingrules
+---------+-------------+-------------+
| name | region | ip |
+---------+-------------+-------------+
| dns-tcp | us-central1 | 8.34.215.45 |
+---------+-------------+-------------+
| dns-udp | us-central1 | 8.34.215.45 |
+---------+-------------+-------------+
| http | us-central1 | 8.34.215.45 |
+---------+-------------+-------------+
$ gcutil getforwardingrule dns-udp
+---------------+----------------------------------+
| name | dns-udp |
| description | |
| creation-time | 2013-12-28T12:28:05.816-08:00 |
| region | us-central1 |
| ip | 8.34.215.45 |
| protocol | UDP |
| port-range | 53-53 |
| target | us-central1/targetPools/dns-pool |
+---------------+----------------------------------+
$ gcutil gettargetpool dns-pool
+------------------+-------------------------------+
| name | dns-pool |
| description | |
| creation-time | 2013-12-28T11:48:08.896-08:00 |
| health-checks | |
| session-affinity | NONE |
| failover-ratio | |
| backup-pool | |
| instances | us-central1-a/instances/dns-1 |
| | us-central1-b/instances/dns-2 |
+------------------+-------------------------------+
[#dns-1 ~]$ curl "http://metadata/computeMetadata/v1/instance/network-interfaces/?recursive=true" -H "X-Google-Metadata-Request: True"
[{"accessConfigs":[{"externalIp":"162.222.178.116","type":"ONE_TO_ONE_NAT"}],"forwardedIps":["8.34.215.45"],"ip":"10.240.157.97","network":"projects/763472520840/networks/default"}]
[#dns-2 ~]$ curl "http://metadata/computeMetadata/v1/instance/network-interfaces/?recursive=true" -H "X-Google-Metadata-Request: True"
[{"accessConfigs":[{"externalIp":"8.34.215.162","type":"ONE_TO_ONE_NAT"}],"forwardedIps":["8.34.215.45"],"ip":"10.240.200.109","network":"projects/763472520840/networks/default"}]
$ gcutil getfirewall dns2
+---------------+------------------------------------+
| name | dns2 |
| description | Allow the incoming service traffic |
| creation-time | 2013-12-28T10:35:18.185-08:00 |
| network | default |
| source-ips | 0.0.0.0/0 |
| source-tags | |
| target-tags | |
| allowed | tcp: 53 |
| allowed | udp: 53 |
| allowed | tcp: 80 |
| allowed | tcp: 443 |
+---------------+------------------------------------+
The instances are CentOS and have their iptables firewalls disabled.
Reply from instance in target pool
#dns-1 ~]$ nslookup test 8.34.215.45 | grep answer
Non-authoritative answer:
#dns-1 ~]$
Reply from other instance in target pool
#dns-2 ~]$ nslookup test 8.34.215.45 | grep answer
Non-authoritative answer:
#dns-2 ~]$
No reply from instance not in the target pool on the load balanced IP. However it gets a reply from all other interfaces
#dns-3 ~]$ nslookup test 8.34.215.45 | grep answer
#dns-3 ~]$
#dns-3 ~]$ nslookup test 8.34.215.162 | grep answer
Non-authoritative answer:
#dns-3 ~]$ nslookup test 10.240.200.109 | grep answer
Non-authoritative answer:
#dns-3 ~]$ nslookup test 10.240.157.97 | grep answer
Non-authoritative answer:
#dns-3 ~]$ nslookup test 162.222.178.116 | grep answer
Non-authoritative answer:
-- Update --
Added a health check so that the instances wouldn't be marked as UNHEALTHY. However got the same result.
$ gcutil gettargetpoolhealth dns-pool
+-------------------------------+-------------+--------------+
| instance | ip | health-state |
+-------------------------------+-------------+--------------+
| us-central1-a/instances/dns-1 | 8.34.215.45 | HEALTHY |
+-------------------------------+-------------+--------------+
| us-central1-b/instances/dns-2 | 8.34.215.45 | HEALTHY |
+-------------------------------+-------------+--------------+
-- Update --
Looks like the DNS service is not responding with the same IP that the request came in on. This is for sure be the reason it doens't appear to be responding.
0.000000 162.222.178.130 -> 8.34.215.45 DNS 82 Standard query 0x5323 A test.internal
2.081868 10.240.157.97 -> 162.222.178.130 DNS 98 Standard query response 0x5323 A 54.122.122.227
Looks like the DNS service is not responding with the same IP that the request came in on. This is for sure be the reason it doens't appear to be responding.
0.000000 162.222.178.130 -> 8.34.215.45 DNS 82 Standard query 0x5323 A test.internal
2.081868 10.240.157.97 -> 162.222.178.130 DNS 98 Standard query response 0x5323 A 54.122.122.227
Is it possible to view RabbitMQ message contents directly from the command line?
sudo rabbitmqctl list_queues lists the queues.
Is there any command like sudo rabbitmqctl list_queue_messages <queue_name>?
You should enable the management plugin.
rabbitmq-plugins enable rabbitmq_management
See here:
http://www.rabbitmq.com/plugins.html
And here for the specifics of management.
http://www.rabbitmq.com/management.html
Finally once set up you will need to follow the instructions below to install and use the rabbitmqadmin tool. Which can be used to fully interact with the system.
http://www.rabbitmq.com/management-cli.html
For example:
rabbitmqadmin get queue=<QueueName> requeue=false
will give you the first message off the queue.
Here are the commands I use to get the contents of the queue:
RabbitMQ version 3.1.5 on Fedora linux using https://www.rabbitmq.com/management-cli.html
Here are my exchanges:
eric#dev ~ $ sudo python rabbitmqadmin list exchanges
+-------+--------------------+---------+-------------+---------+----------+
| vhost | name | type | auto_delete | durable | internal |
+-------+--------------------+---------+-------------+---------+----------+
| / | | direct | False | True | False |
| / | kowalski | topic | False | True | False |
+-------+--------------------+---------+-------------+---------+----------+
Here is my queue:
eric#dev ~ $ sudo python rabbitmqadmin list queues
+-------+----------+-------------+-----------+---------+------------------------+---------------------+--------+----------+----------------+-------------------------+---------------------+--------+---------+
| vhost | name | auto_delete | consumers | durable | exclusive_consumer_tag | idle_since | memory | messages | messages_ready | messages_unacknowledged | node | policy | status |
+-------+----------+-------------+-----------+---------+------------------------+---------------------+--------+----------+----------------+-------------------------+---------------------+--------+---------+
| / | myqueue | False | 0 | True | | 2014-09-10 13:32:18 | 13760 | 0 | 0 | 0 |rabbit#ip-11-1-52-125| | running |
+-------+----------+-------------+-----------+---------+------------------------+---------------------+--------+----------+----------------+-------------------------+---------------------+--------+---------+
Cram some items into myqueue:
curl -i -u guest:guest http://localhost:15672/api/exchanges/%2f/kowalski/publish -d '{"properties":{},"routing_key":"abcxyz","payload":"foobar","payload_encoding":"string"}'
HTTP/1.1 200 OK
Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
Date: Wed, 10 Sep 2014 17:46:59 GMT
content-type: application/json
Content-Length: 15
Cache-Control: no-cache
{"routed":true}
RabbitMQ see messages in queue:
eric#dev ~ $ sudo python rabbitmqadmin get queue=myqueue requeue=true count=10
+-------------+----------+---------------+---------------------------------------+---------------+------------------+------------+-------------+
| routing_key | exchange | message_count | payload | payload_bytes | payload_encoding | properties | redelivered |
+-------------+----------+---------------+---------------------------------------+---------------+------------------+------------+-------------+
| abcxyz | kowalski | 10 | foobar | 6 | string | | True |
| abcxyz | kowalski | 9 | {'testdata':'test'} | 19 | string | | True |
| abcxyz | kowalski | 8 | {'mykey':'myvalue'} | 19 | string | | True |
| abcxyz | kowalski | 7 | {'mykey':'myvalue'} | 19 | string | | True |
+-------------+----------+---------------+---------------------------------------+---------------+------------------+------------+-------------+
I wrote rabbitmq-dump-queue which allows dumping messages from a RabbitMQ queue to local files and requeuing the messages in their original order.
Example usage (to dump the first 50 messages of queue incoming_1):
rabbitmq-dump-queue -url="amqp://user:password#rabbitmq.example.com:5672/" -queue=incoming_1 -max-messages=50 -output-dir=/tmp
If you want multiple messages from a queue, say 10 messages, the command to use is:
rabbitmqadmin get queue=<QueueName> ackmode=ack_requeue_true count=10
This is how it looks on front interface avalable on http://localhost:15672 :
If you don't want the messages requeued, just change ackmode to ack_requeue_false.
you can use RabbitMQ API to get count or messages :
/api/queues/vhost/name/get
Get messages from a queue. (This is not an HTTP GET as it will alter the state of the queue.) You should post a body looking like:
{"count":5,"requeue":true,"encoding":"auto","truncate":50000}
count controls the maximum number of messages to get. You may get fewer messages than this if the queue cannot immediately provide them.
requeue determines whether the messages will be removed from the queue. If requeue is true they will be requeued - but their redelivered flag will be set.
encoding must be either "auto" (in which case the payload will be returned as a string if it is valid UTF-8, and base64 encoded otherwise), or "base64" (in which case the payload will always be base64 encoded).
If truncate is present it will truncate the message payload if it is larger than the size given (in bytes).
truncate is optional; all other keys are mandatory.
Please note that the publish / get paths in the HTTP API are intended for injecting test messages, diagnostics etc - they do not implement reliable delivery and so should be treated as a sysadmin's tool rather than a general API for messaging.
http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_1_3/priv/www/api/index.html
a bit late to this, but yes rabbitmq has a build in tracer that allows you to see the incomming messages in a log. When enabled, you can just tail -f /var/tmp/rabbitmq-tracing/.log (on mac) to watch the messages.
the detailed discription is here http://www.mikeobrien.net/blog/tracing-rabbitmq-messages
Our apache tomcat server is printing random nullpointer exceptions. Normally you would see a stacktrace..
INFO | jvm 1 | srvmain | 2012/03/30 13:44:43.733 | SEVERE: Servlet.service() for servlet frontend threw exception
INFO | jvm 1 | srvmain | 2012/03/30 13:44:43.733 | java.lang.NullPointerException
INFO | jvm 1 | srvmain | 2012/03/30 13:44:46.139 | Mar 30, 2012 1:44:46 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO | jvm 1 | srvmain | 2012/03/30 13:44:46.139 | SEVERE: Servlet.service() for servlet frontend threw exception
INFO | jvm 1 | srvmain | 2012/03/30 13:44:46.139 | java.lang.NullPointerException
INFO | jvm 1 | srvmain | 2012/03/30 13:44:47.998 | Mar 30, 2012 1:44:47 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO | jvm 1 | srvmain | 2012/03/30 13:44:47.998 | SEVERE: Servlet.service() for servlet frontend threw exception
INFO | jvm 1 | srvmain | 2012/03/30 13:44:47.998 | java.lang.NullPointerException
INFO | jvm 1 | srvmain | 2012/03/30 13:44:50.623 | Mar 30, 2012 1:44:50 PM org.apache.catalina.core.StandardWrapperValve invoke
You are going to have to restart the server if you want complete stack traces. You need to set the following JVM option:
XX:-OmitStackTraceInFastThrow