We have a situation where we have a single RabbitMQ node in US-East, with producers in other zones (Ireland, Sydney etc). We are seeing huge performance hits when queueing from other zones. Sydney -> US-East queue is 1s to queue a message, whereas queuing Sydney -> Sydney is 50ms. It seems a lot of the time is spent creating the channel and declaring the queue.
What options do we have to improve the performance? Could we look at some sort of distributed RabbitMQ cluster, with a node in each region? Would that help us?
Here's the code we are using to test:
var queueConnection = amqp.connect("OUR amqp servers in each region")
var queueName = "test-queue"
var queueMessage = function(message) {
return queueConnection.then(function(conn) {
return conn.createChannel()
}).then(function(ch) {
var queue = ch.assertQueue(queueName, { durable: false });
return queue.then(function() {
ch.sendToQueue(queueName, new Buffer(JSON.stringify(message)), { deliveryMode: true });
return ch.close()
});
})
};
Promise.map(_.range(0, 10), function(item) {
var timedQueueMessage = timely.promise(queueMessage)
return timedQueueMessage({ name: "Dom" }).then(function(res) {
console.log("Completed in " + timedQueueMessage.time + "ms")
})
}, { concurrency: 10 }).done(process.exit)
For these use cases you should look at Federation or Shovel.
This page explains the pros and cons of each of the distributed options offered by RabbitMQ: http://www.rabbitmq.com/distributed.html
Related
I need a help from somebody experienced. I've built 2 microservices recently (let's call them Amber and Boris) which are communicating between each other using ClientProxy and REDIS. From time to time, when Amber is asking for data from Boris, it gets timeout Error.
This is Amber config:
constructor(companyName: string, userId: number) {
this.companyName = companyName;
this.userId = userId;
this.client = ClientProxyFactory.create({
transport: Transport.REDIS,
options: {
retryAttempts: 0,
retryDelay: 0,
url: 'redis://<some_url>:<some_port>,
},
});
}
Then request-response:
private async sendRequest(pattern: string, payload?: object): Promise<any[]> {
payload = payload || {};
try {
const result = await this.client.send(
{ type: pattern },
{ userId: this.userId, companyName: this.companyName, ...payload}
)
.pipe(
timeout(30000),
map((response: any) => { // Success...
return response;
}),
catchError((error) => { // Error...
return throwError(error);
}),
)
.toPromise();
return result;
} catch (err) {
Logger.error('Couldn\'t get data from Boris service: ' + err.message)
}
}
Then on Boris service, I have basically just Controller set with #MessagePattern and I'm just returning data:
#MessagePattern({type: 'getAvailableCases'})
findAll(#Payload() data: object): Promise<object> {
this.assignPayload(data);
return this.getData();
}
Important to say, Boris service is doing queries to database in order to return data. But on db side seems there is no problem.
What I'm interested in the most is:
whether I have ClientProxy set up properly
whether I have answer processing set up properly with pipe() and toPromise(), as I'm not well familiarized with ClientProxy and RxJs.
Thank you a hundred times for any response!
Turned out, the ClientProxy wasn't releasing connections to Redis after the communication is done. This way, the number of connections was increasing until there was no connection left.
The solution is to close connection after the data are returned:
this.client.close();
I'm trying to recieve a Message from the RabbitMq-Que with the Stomp-plugin. This works fine but the Problem is that I get every Message from the Queue. So if the Queue has 13 Messages, I get 13. The Key is that I just want to get 1 Message and after sending Ack or Nack the next one. Do somebody got any Idea how to get only one Message? Thanks for Help.
Here the code I got:
GetMessage()
{
this.GetRabbitMqClient().then((client)=>
{
var headers ={ack:'client', 'x-max-priority': '10'};
var subscription = client.subscribe("/queue/TestQue",this.messageCallback,headers);
});
}
private messageCallback = function(Message :IMessage)
{
console.log(Message.body);
setTimeout(()=> {Message.ack();},100000000000000);
}
private GetRabbitMqClient( ):Promise<Client> {
var promise = new Promise<Client>((resolve,reject)=>{
var client = new Client(
{
brokerURL: "ws://localhost:15674/ws",
connectHeaders:
{
login: "guest",
passcode: "guest"
},
// debug: function (str) {
// console.log(str);
// },
reconnectDelay: 5000,
heartbeatIncoming: 4000,
heartbeatOutgoing: 4000
});
client.onConnect = function (frame) {
resolve(client);
};
client.onStompError = function (frame) {
// Will be invoked in case of error encountered at Broker
// Bad login/passcode typically will cause an error
// Complaint brokers will set `message` header with a brief message. Body may contain details.
// Compliant brokers will terminate the connection after any error
reject(frame);
console.log('Broker reported error: ' + frame.headers['message']);
console.log('Additional details: ' + frame.body);
};
client.activate();
});
return promise;
}
I found the solution:
you just need to set in the headers the attribute:
'prefetch-count':'1'
I'm following the example here: https://www.w3.org/TR/webrtc/#simple-peer-to-peer-example
I've modified the code because I only need one-way streaming:
var configuration = null; //{ "iceServers": [{ "urls": "stuns:stun.example.org" }] };
var peerConnection;
var outboundPeerStream = null;
var outboundPeerStreamSessionId = null;
var createPeerConnection = function () {
if (peerConnection)
return;
peerConnection = new RTCPeerConnection(configuration);
// send any ice candidates to the other peer
peerConnection.onicecandidate = function (event) {
signalrModule.sendClientNotification(JSON.stringify({ "candidate": event.candidate }));
};
// let the "negotiationneeded" event trigger offer generation
peerConnection.onnegotiationneeded = peerStreamingModule.sendOfferToPeers;
// once remote track arrives, show it in the remote video element
peerConnection.ontrack = function (event) {
var inboundPeerStream = event.streams[0];
remoteStreamHelper.pushStreamToDom(inboundPeerStream, foo);
}
}
// this gets called either on negotiationNeeded and every 30s to ensure all peers have the offer from the stream originator
peerStreamingModule.sendOfferToPeers = function () {
peerConnection.createOffer().then(function (offer) {
return peerConnection.setLocalDescription(offer);
}).then(function () {
// send the offer to the other peer
signalrModule.sendClientNotification(JSON.stringify({ "desc": peerConnection.localDescription}));
}).catch(logger.internalLog);
};
// this gets called by the stream originator when the stream is available to initiate streaming to peers
peerStreamingModule.initializeWithStream = function (outboundStream, sessionId) {
outboundPeerStream = outboundStream;
outboundPeerStreamSessionId = sessionId;
createPeerConnection();
peerConnection.addStream(outboundStream);
//peerStreamingModule.sendOfferToPeers(); I don't think I need this...
}
peerStreamingModule.handleP2PEvent = function (notification) {
if (!peerConnection)
createPeerConnection();
if (notification.desc) {
var desc = notification.desc;
// if we get an offer, we need to reply with an answer
if (desc.type == "offer") {
peerConnection.setRemoteDescription(desc).then(function () {
return peerConnection.createAnswer();
}).then(function (answer) {
return peerConnection.setLocalDescription(answer);
}).then(function () {
signalrModule.sendClientNotification(JSON.stringify({ "desc": peerConnection.localDescription, "sessionId": sessionManager.thisSession().deviceSessionId() }), app.username());
}).catch(logger.internalLog);
} else if (desc.type == "answer") {
peerConnection.setRemoteDescription(desc).catch(logger.internalLog);
} else {
logger.internalLog("Unsupported SDP type. Your code may differ here.");
}
} else
pc.addIceCandidate(notification.candidate).catch(logger.internalLog);
}
This seems to be working, but I'm stumped with two parts:
1) WebRTC - Failed to set remote answer sdp: Called in wrong state: STATE_INPROGRESS - this is appearing in my logs from time to time - am I doing something wrong in the above that is causing this?
2) Am I correctly implementing sendOfferToPeers and initializeWithStream? I'm afraid that the sendOfferToPeers getting triggered on interval from the originator isn't how the spec is intended to be used; my goal is to ensure that all peers eventually receive an offer no matter when they join or whether or not they're facing connectivity issues that drop the original offer / negotiation.
// this gets called either on negotiationNeeded and every 30s to ensure all peers have the offer
You can't send the same offer to multiple peers. It's peer-to-peer, not peer-to-peers. One-to-many requires at minimum a connection per participant, and probably a media server to scale.
Also, SDP is not for discovery. The offer/answer exchange is a fragile state-machine negotiation between two end-points only, to set up a single connection.
You should solve who's connecting with whom ahead of establishing the WebRTC connection.
I have two server, ServerA and ServerB. They share the same hangfire database.
I have two jobs, JobA and JobB.
On ServerA, I use:
RecurringJob.AddOrUpdate(
"JobA",
() => new JobA().Execute(),
this._configuration.Schedule, queue: "A");
On ServerB, I use:
RecurringJob.AddOrUpdate(
"JobB",
() => new JobB().Execute(),
this._configuration.Schedule, queue: "B");
The problem is that each jobs is "Enqueued" in the "Job" table, they are never executed.
If I remove the queue override in the "AddOrUpdate" method, jobs are executed (obviously without queue configured).
Something missing? How configure recurring jobs with queue configuration?
Code was missing...
ServerA:
var options = new BackgroundJobServerOptions
{
Queues = new[] { "A" }
};
this._backgroundJobServer = new BackgroundJobServer(options);
ServerB:
var options = new BackgroundJobServerOptions
{
Queues = new[] { "B" }
};
this._backgroundJobServer = new BackgroundJobServer(options);
Solution - might help someone with similar issue:
app.UseHangfireServer(new BackgroundJobServerOptions
{
// queue name must be in lowercase
Queues = new[] { "qname" } //This will setup the server to only process qname queues
});
RecurringJob.AddOrUpdate(
() => new JobB().Execute(),
Cron.Hourly(5),
null,
"qname");
Hi I am developing a website as my chrome extension.the extension periodically checks the output of a server file and then works according to the output of the server file.Now what I need is that presently it checks the output of the server file every 5 min.So what I need is that when the output of the server file changes at any time,at that moment I have to do some operations.How can I do this?
Here is my background.js
var myNotificationID = null;
var oldChromeVersion = !chrome.runtime;
chrome.windows.onCreated.addListener(function (){
updateIcon();
});
chrome.tabs.onCreated.addListener(function (){
updateIcon();
});
chrome.tabs.onUpdated.addListener(function (){
updateIcon();
});
function getGmailUrl() {
return "http://calpinemate.com/";
}
function isGmailUrl(url) {
return url.indexOf(getGmailUrl()) == 0;
}
chrome.windows.onCreated.addListener(function (){
updateIcon();
});
function onInit() {
updateIcon();
if (!oldChromeVersion) {
chrome.alarms.create('watchdog',{periodInMinutes:5,delayInMinutes: 0});
}
}
function onAlarm(alarm) {
if (alarm && alarm.name == 'watchdog') {
onWatchdog();
}
else {
updateIcon();
}
}
function onWatchdog() {
chrome.alarms.get('refresh', function(alarm) {
if (alarm) {
console.log('Refresh alarm exists. Yay.');
}
else {
updateIcon();
}
});
}
if (oldChromeVersion) {
updateIcon();
onInit();
}
else {
chrome.runtime.onInstalled.addListener(onInit);
chrome.alarms.onAlarm.addListener(onAlarm);
}
It is the updateIcon() which reads the server file.And when there is change in the output of server file,I have to call the updateIcon() itself.Presently only in every 5 min,the output is checked and updated in extension.But I need it to happen at the moment the status of server file changes.Anyone please help me.
In short,what I need is that when the output of API changes at any time at that time I have to call updateIcon().
Here is my updateIcon()
function updateIcon(){
var urlPrefix = 'http://www.calpinemate.com/employees/attendanceStatus/';
var urlSuffix = '/2';
var req = new XMLHttpRequest();
req.addEventListener("readystatechange", function() {
if (req.readyState == 4) {
if (req.status == 200) {
var item=req.responseText;
if(item==1){
.....//something
}
else{
// do something
}
else {
alert("ERROR: status code " + req.status);
}
}
});
var url = urlPrefix + encodeURIComponent(localStorage.username) + urlSuffix;
req.open("GET", url);
req.send(null);
}
I need to periodically monitor the API for output.
You could use a WebSocket to maintain a persistent, two-way channel between the server and your extension.
But, considering that PHP does not have native support for WebSockets (i.e. you would need to use an external library) and the fact that the interaction will be infrequent (only at log-in/-out), it might be unnecessary overhead.
I suggest you communicate the login-status directly from your web-page to your extension. For a more detailed description of the process, see my answer to one similar question of yours.
UPDATE:
Since it turned out you don't have control of the domain you need to "monitor", there is unfortunately no other option (that I know of) than polling at frequent intervals on that server-resource.
If you are using a non-persistent background-page (which is advisable due to its resource-friendliness) you must use the chrome.alarms API, which allows at most 1 triggering per minute: (in order to reduce the load on the user's machine - note: to help debug your extension, this limit is no imposed on unpacked extensions).
If you decide to use a persistent background-page, you can use setInterval() with an arbitrarily smal period (in milliseconds):
setInterval(function() {
/* Check up on the server */
...
}, 30000); // <-- trigger every 30.000 milliseconds (== 30 seconds)