Hangfire - Recurring job with specified queue name - hangfire

I have two server, ServerA and ServerB. They share the same hangfire database.
I have two jobs, JobA and JobB.
On ServerA, I use:
RecurringJob.AddOrUpdate(
"JobA",
() => new JobA().Execute(),
this._configuration.Schedule, queue: "A");
On ServerB, I use:
RecurringJob.AddOrUpdate(
"JobB",
() => new JobB().Execute(),
this._configuration.Schedule, queue: "B");
The problem is that each jobs is "Enqueued" in the "Job" table, they are never executed.
If I remove the queue override in the "AddOrUpdate" method, jobs are executed (obviously without queue configured).
Something missing? How configure recurring jobs with queue configuration?

Code was missing...
ServerA:
var options = new BackgroundJobServerOptions
{
Queues = new[] { "A" }
};
this._backgroundJobServer = new BackgroundJobServer(options);
ServerB:
var options = new BackgroundJobServerOptions
{
Queues = new[] { "B" }
};
this._backgroundJobServer = new BackgroundJobServer(options);

Solution - might help someone with similar issue:
app.UseHangfireServer(new BackgroundJobServerOptions
{
// queue name must be in lowercase
Queues = new[] { "qname" } //This will setup the server to only process qname queues
});
RecurringJob.AddOrUpdate(
() => new JobB().Execute(),
Cron.Hourly(5),
null,
"qname");

Related

Nestjs - redis communication between 2 microservices with ClientProxy

I need a help from somebody experienced. I've built 2 microservices recently (let's call them Amber and Boris) which are communicating between each other using ClientProxy and REDIS. From time to time, when Amber is asking for data from Boris, it gets timeout Error.
This is Amber config:
constructor(companyName: string, userId: number) {
this.companyName = companyName;
this.userId = userId;
this.client = ClientProxyFactory.create({
transport: Transport.REDIS,
options: {
retryAttempts: 0,
retryDelay: 0,
url: 'redis://<some_url>:<some_port>,
},
});
}
Then request-response:
private async sendRequest(pattern: string, payload?: object): Promise<any[]> {
payload = payload || {};
try {
const result = await this.client.send(
{ type: pattern },
{ userId: this.userId, companyName: this.companyName, ...payload}
)
.pipe(
timeout(30000),
map((response: any) => { // Success...
return response;
}),
catchError((error) => { // Error...
return throwError(error);
}),
)
.toPromise();
return result;
} catch (err) {
Logger.error('Couldn\'t get data from Boris service: ' + err.message)
}
}
Then on Boris service, I have basically just Controller set with #MessagePattern and I'm just returning data:
#MessagePattern({type: 'getAvailableCases'})
findAll(#Payload() data: object): Promise<object> {
this.assignPayload(data);
return this.getData();
}
Important to say, Boris service is doing queries to database in order to return data. But on db side seems there is no problem.
What I'm interested in the most is:
whether I have ClientProxy set up properly
whether I have answer processing set up properly with pipe() and toPromise(), as I'm not well familiarized with ClientProxy and RxJs.
Thank you a hundred times for any response!
Turned out, the ClientProxy wasn't releasing connections to Redis after the communication is done. This way, the number of connections was increasing until there was no connection left.
The solution is to close connection after the data are returned:
this.client.close();

RabbitMQ StompJs recieving just one Message

I'm trying to recieve a Message from the RabbitMq-Que with the Stomp-plugin. This works fine but the Problem is that I get every Message from the Queue. So if the Queue has 13 Messages, I get 13. The Key is that I just want to get 1 Message and after sending Ack or Nack the next one. Do somebody got any Idea how to get only one Message? Thanks for Help.
Here the code I got:
GetMessage()
{
this.GetRabbitMqClient().then((client)=>
{
var headers ={ack:'client', 'x-max-priority': '10'};
var subscription = client.subscribe("/queue/TestQue",this.messageCallback,headers);
});
}
private messageCallback = function(Message :IMessage)
{
console.log(Message.body);
setTimeout(()=> {Message.ack();},100000000000000);
}
private GetRabbitMqClient( ):Promise<Client> {
var promise = new Promise<Client>((resolve,reject)=>{
var client = new Client(
{
brokerURL: "ws://localhost:15674/ws",
connectHeaders:
{
login: "guest",
passcode: "guest"
},
// debug: function (str) {
// console.log(str);
// },
reconnectDelay: 5000,
heartbeatIncoming: 4000,
heartbeatOutgoing: 4000
});
client.onConnect = function (frame) {
resolve(client);
};
client.onStompError = function (frame) {
// Will be invoked in case of error encountered at Broker
// Bad login/passcode typically will cause an error
// Complaint brokers will set `message` header with a brief message. Body may contain details.
// Compliant brokers will terminate the connection after any error
reject(frame);
console.log('Broker reported error: ' + frame.headers['message']);
console.log('Additional details: ' + frame.body);
};
client.activate();
});
return promise;
}
I found the solution:
you just need to set in the headers the attribute:
'prefetch-count':'1'

How to getQueryResults() of 1 job after createQueryJob() successfully in Google BigQuery?

I'm using google cloud big query service with nodejs client version 1.0x . I created a job successfully by function createQueryJob(). After that, I used an event listen when a callback create job response with getQueryResults() such as:
const options = {
query: sqlQuery,
useLegacySql: true,
dryRun: true
};
// this.bigquery is an constructor.
// this.bigquery = new BigQuery({
// projectId: this.projectId,
// keyFilename: this.keyFile,
// });
this.bigquery.createQueryJob(options, function (err, job) {
if (!err) {
// job id such as 731bf23b-5044-4842-894b-4d9f77485d9b
function manualPaginationCallback(err, rows, nextQuery, apiResponse) {
if (nextQuery) {
job.getQueryResults(nextQuery, manualPaginationCallback);
} else {
return Promise.resolve(rows);
}
}
return job.getQueryResults({
maxResults: 100000,
autoPaginate: false,
// timeoutMs : 60000
}, manualPaginationCallback);
}
});
But It throw an error
{"error":{"code":404,"message":"Not found: Job
[myProjectId]:731bf23b-5044-4842-894b-4d9f77485d9b","errors":[{"message":"Not
found: Job
[myProjectId]:731bf23b-5044-4842-894b-4d9f77485d9b","domain":"global","reason":"notFound"}],"status":"NOT_FOUND"}}
refrence
https://cloud.google.com/nodejs/docs/reference/bigquery/1.0.x/BigQuery#createQueryJob
https://cloud.google.com/nodejs/docs/reference/bigquery/1.0.x/Job#getQueryResults
What wrong's with me? Any help. Thank you!
You're setting the dryrun option in your request, which only validates the job but doesn't actually run the query. Dryrun jobs don't persist, which is why you get not found on the subsequent request.

WebRTC - Failed to set remote answer sdp: Called in wrong state: STATE_INPROGRESS

I'm following the example here: https://www.w3.org/TR/webrtc/#simple-peer-to-peer-example
I've modified the code because I only need one-way streaming:
var configuration = null; //{ "iceServers": [{ "urls": "stuns:stun.example.org" }] };
var peerConnection;
var outboundPeerStream = null;
var outboundPeerStreamSessionId = null;
var createPeerConnection = function () {
if (peerConnection)
return;
peerConnection = new RTCPeerConnection(configuration);
// send any ice candidates to the other peer
peerConnection.onicecandidate = function (event) {
signalrModule.sendClientNotification(JSON.stringify({ "candidate": event.candidate }));
};
// let the "negotiationneeded" event trigger offer generation
peerConnection.onnegotiationneeded = peerStreamingModule.sendOfferToPeers;
// once remote track arrives, show it in the remote video element
peerConnection.ontrack = function (event) {
var inboundPeerStream = event.streams[0];
remoteStreamHelper.pushStreamToDom(inboundPeerStream, foo);
}
}
// this gets called either on negotiationNeeded and every 30s to ensure all peers have the offer from the stream originator
peerStreamingModule.sendOfferToPeers = function () {
peerConnection.createOffer().then(function (offer) {
return peerConnection.setLocalDescription(offer);
}).then(function () {
// send the offer to the other peer
signalrModule.sendClientNotification(JSON.stringify({ "desc": peerConnection.localDescription}));
}).catch(logger.internalLog);
};
// this gets called by the stream originator when the stream is available to initiate streaming to peers
peerStreamingModule.initializeWithStream = function (outboundStream, sessionId) {
outboundPeerStream = outboundStream;
outboundPeerStreamSessionId = sessionId;
createPeerConnection();
peerConnection.addStream(outboundStream);
//peerStreamingModule.sendOfferToPeers(); I don't think I need this...
}
peerStreamingModule.handleP2PEvent = function (notification) {
if (!peerConnection)
createPeerConnection();
if (notification.desc) {
var desc = notification.desc;
// if we get an offer, we need to reply with an answer
if (desc.type == "offer") {
peerConnection.setRemoteDescription(desc).then(function () {
return peerConnection.createAnswer();
}).then(function (answer) {
return peerConnection.setLocalDescription(answer);
}).then(function () {
signalrModule.sendClientNotification(JSON.stringify({ "desc": peerConnection.localDescription, "sessionId": sessionManager.thisSession().deviceSessionId() }), app.username());
}).catch(logger.internalLog);
} else if (desc.type == "answer") {
peerConnection.setRemoteDescription(desc).catch(logger.internalLog);
} else {
logger.internalLog("Unsupported SDP type. Your code may differ here.");
}
} else
pc.addIceCandidate(notification.candidate).catch(logger.internalLog);
}
This seems to be working, but I'm stumped with two parts:
1) WebRTC - Failed to set remote answer sdp: Called in wrong state: STATE_INPROGRESS - this is appearing in my logs from time to time - am I doing something wrong in the above that is causing this?
2) Am I correctly implementing sendOfferToPeers and initializeWithStream? I'm afraid that the sendOfferToPeers getting triggered on interval from the originator isn't how the spec is intended to be used; my goal is to ensure that all peers eventually receive an offer no matter when they join or whether or not they're facing connectivity issues that drop the original offer / negotiation.
// this gets called either on negotiationNeeded and every 30s to ensure all peers have the offer
You can't send the same offer to multiple peers. It's peer-to-peer, not peer-to-peers. One-to-many requires at minimum a connection per participant, and probably a media server to scale.
Also, SDP is not for discovery. The offer/answer exchange is a fragile state-machine negotiation between two end-points only, to set up a single connection.
You should solve who's connecting with whom ahead of establishing the WebRTC connection.

RabbitMQ latency issues across geographical distance

We have a situation where we have a single RabbitMQ node in US-East, with producers in other zones (Ireland, Sydney etc). We are seeing huge performance hits when queueing from other zones. Sydney -> US-East queue is 1s to queue a message, whereas queuing Sydney -> Sydney is 50ms. It seems a lot of the time is spent creating the channel and declaring the queue.
What options do we have to improve the performance? Could we look at some sort of distributed RabbitMQ cluster, with a node in each region? Would that help us?
Here's the code we are using to test:
var queueConnection = amqp.connect("OUR amqp servers in each region")
var queueName = "test-queue"
var queueMessage = function(message) {
return queueConnection.then(function(conn) {
return conn.createChannel()
}).then(function(ch) {
var queue = ch.assertQueue(queueName, { durable: false });
return queue.then(function() {
ch.sendToQueue(queueName, new Buffer(JSON.stringify(message)), { deliveryMode: true });
return ch.close()
});
})
};
Promise.map(_.range(0, 10), function(item) {
var timedQueueMessage = timely.promise(queueMessage)
return timedQueueMessage({ name: "Dom" }).then(function(res) {
console.log("Completed in " + timedQueueMessage.time + "ms")
})
}, { concurrency: 10 }).done(process.exit)
For these use cases you should look at Federation or Shovel.
This page explains the pros and cons of each of the distributed options offered by RabbitMQ: http://www.rabbitmq.com/distributed.html