Can we create an infinite job/task (kind like an Azure Worker role) with Hangfire? - hangfire

I would like to know if we can create an infinite job/task (kind like an Azure Worker role) with Hangfire. I would like to queue emails in an Azure Queue using an Hangfire scheduled job (every 4 hours) and then run an infinite Hangfire fire and forget job/task (when Website start) to process (dequeue) each email and send it to Amazon SES every 200 milliseconds (5 emails per second). The infinite job needs to be up all the time in order to process new queue emails.
So, performance wise is it okay to do that and then how to manage potential errors that could stop the infinite job and how to restart it if needed. In the infinite loop job should I also create new fire and forget task for each email in order have the possibility make other retry attempt for each email. My Hangfire server will be hosted in an Azure Website.
Finally, I'm doing all this because Amazon SES cannot queue emails. My Amazon AWS subscription authorizes me to send 15 emails per second.
Hope it's clear,
Thank you,

Not sure why you would want to use Hangfire for this purpose, but Azure WebJobs seem to be quite a nice fit where it can process messages in Azure Queue out of the box.
https://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-webjobs-sdk-storage-queues-how-to/#trigger
You can then enqueue a Hangfire job when the queue is filled with something.
public static void ProcessQueueMessage([QueueTrigger("emailqueue")] string email, TextWriter logger)
{
BackgroundJobClient.Enqueue(() => SendEmail(email));
}

Related

How to know the last job in a queue

I have a group of of jobs that need to be processed.
Some may take 10 min, some may take 1h.
Now I need to know what is the last job executed because at the end of that group of jobs I need to fire another message.
The message queue in this case is RabbitMQ.
Is there a way I can accomplish this with only RabbitMQ?
What would be a good strategy for this task?
Thats strategy you can use with any messaging system.
I assume you have group of workers listening to single queue with jobs "jobs queue" to be processed. Now you can have service lets call it Manager witch duplicates this queue and saves all no finished messages. Now when worker finishes the job it send acknowledgment message to the Manager. Manager for example discards all finished jobs and stores only running once. (If you want to take in to account passable failures it can track that too).
When Manager have no more messages it publishes message to the "all messages in the group done topic". Now publishers can listen to the topic and fire new job messages to the "job queue".
Of course in simple case you can have one producer witch could be the Manager in the same time.
Example RabbitMQ implementation.
Now to implement this in RabbitMQ you can for example create one FanoutExchange (for producer to send messages to) and two queues jobsQueue (to send jobs to workers) and jobTrackingQueue (to send jobs to Manager for tracking jobs). Now you create second FonoutExchange (for Manager to send task done messges to) you create unnamed queue per producer who wants to know if all messges are done.

Restoring job-queue between telegram-bot restarts

I've built a small telegram bot using python-telegram-bot.
When a conversation is started,I add a periodical job to the job queue and then message back to the user every X minutes.
Problem is when my bot goes offline (maintenance, failures, etc), the jobqueue is lost and clients do not receive updates anymore unless they send /start again
I could maybe store all chat_ids in a persistent queue and restore them at startup but how do I send a message without responding to an update ?
you have lots of options. at first you need to store all chat_ids. you can do it in database or simple text file.
then you need a trigger in order to start sending messages. I'm not familiar with your technology but i just create simple service in order to do it.

Webjob Errors since move to Azure Sql v12

Each morning at 4am a Scheduled Task creates around 500 messages on a message queue. Each message is an id of an email to send. Each message is picked up and a url created and sent via await HttpClient.GetAsync(url) The url target then creates and sends the email. This has worked well for months.
I've just upgraded to SQL Azure v12 and all is now not well.
The very first messages to be processed (after 2 minutes running time) threw a
"System.Threading.Tasks.TaskCanceledException"
I'm also seeing
"System.Data.Entity.Core.EntityException: The underlying provider
failed on Open. ---> System.InvalidOperationException: Timeout
expired. The timeout period elapsed prior to obtaining a connection
from the pool. This may have occurred because all pooled connections
were in use and max pool size was reached."
and a couple of
"The timeout period elapsed prior to completion of the operation or
the server is not responding. This failure occurred while attempting
to connect to the routing destination. The duration spent while
attempting to connect to the original server was - [Pre-Login]
initialization=6; handshake=426; [Login] initialization=0;
authentication=0;"
The webjob that sends the request to the api is awaiting a response. I'm wondering if because it's async, while awaiting the response the thread is freed to go off and processes another queue item - and therefore creates another api request, essentially this hits the api again and again until there are so many requests being processed by the api all the theads are in use - and that I might be better off NOT making the webjob async because then it (the 'trapped' thread) would send a request only after the first request completes? Is that right? edit: actually the IIS logs suggest that the api requests are not all happening at once. So my question is "what should I look at next? Are these common SQL v12 errors or is the recent upgrade a red-herring?"
just to clarify, the webjob that fires in response to the queue message simply does:
using (HttpClient client = new HttpClient())
{
response = await client.GetAsync(url);
}
and hits the web api of an Always On standard tier azure website. Database DTU % is about 25% while this happens.
edit:
"Gateway no longer provides retry logic in V12 Before version V12,
Azure SQL Database had a gateway that acted as a proxy to buffer all
interactions between the database and your client program. The gateway
provided automated retry logic for some transient errors.
V12 eliminated the gateway. Now your program must more fully handle
transient errors."
adding DbConfiguration class for SqlAzureExecutionStrategy. Will so how it runs tonight.
Adding the EF retry config class fixed the transient errors. The cancelled tasks are a different issue (new question)
//https://msdn.microsoft.com/en-us/data/jj680699
public class SqlAzureConfiguration : DbConfiguration
{
public SqlAzureConfiguration()
{
this.SetExecutionStrategy("System.Data.SqlClient", () => new SqlAzureExecutionStrategy());
}
}
and in web.config (because I have multiple contexts)
<entityFramework codeConfigurationType="Abc.DataService.SqlAzureConfiguration, Abc.DataService">

WCF & Azure: Make a worker role wait till it receives data from another worker role

I am quite new to WCF. I followed a tutorial on how to use Internal Endpoints (WCF) to make a Role to Role communication. Link for the tutorial
They actually create multiple instance of a worker role and poke each other.
The code is
foreach (var ep in endPoints)
{
IService1 worker = WorkerRole.factory.CreateChannel(new EndpointAddress(string.Format("net.tcp://{0}/Service1", ep.IPEndpoint)));
try
{
Trace.WriteLine(worker.SayHello(currentInstance.Id.ToString()), "Information");
((ICommunicationObject)worker).Close();
}
catch (Exception e)
{
Trace.TraceError("unable to poke worker role instance '{0}'. {1}", ep.RoleInstance.Id, e.Message);
((ICommunicationObject)worker).Abort();
}
}
But I want to make a worker role wait till it is being poked by other worker role. Say for example, there are 3 worker roles. The worker role 2 and worker role 3 should wait till it is being poked by worker role 1.
Can anyone tell me how to do that.
To do this diectly see http://blog.structuretoobig.com/post/2010/02/03/Windows-Azure-Role-Communication.aspx, using the Azure Service Bus see http://blogs.msdn.com/b/appfabriccat/archive/2010/09/30/implementing-reliable-inter-role-communication-using-windows-azure-appfabric-service-bus-observer-pattern-amp-parallel-linq.aspx , or you can use the Azure Queue to communicate between roles, or you could use the Azure Caching Service ( http://msdn.microsoft.com/en-us/library/windowsazure/gg278356.aspx )
I think I would architect this slightly differently.
Instead of having worker roles exposing WCF Endpoints and sending messages between them it may be neater to use queues.
Messages can be posted to queues and picked up and processed by other worker roles. This introduces a certain amount of durability in that if a worker role that is supposed to receive a message is down for any reason it can carry on processing those messages on its queue when it comes back up. Also any unhandled exceptions that happen whilst processing a message will mean that the message reappears on the queue after a certain timeout period. If your app/site really takes off you can add additional instances of those worker roles to process the messages on the queues more quickly.
Therefore by using queues you have a certain amount of additional durability and its easier to scale up later.
There is a good intro to using queues on the developer fusion website

What is a delayed background job? How does it work?

I am using Heroku to host a small app. It's running a screenscraper using Mechanize and Nokogiri for every search request, and this takes about 3 seconds to complete.
Does the screenscraper block anyone else who wants to access the app at that moment? In other words, is it in holding mode for only the current user or for everyone?
If you have only one heroku dyno, then yes it is the case that other users would have to wait in line.
Background jobs are for cases like yours where there is some heavy processing to be done. The process running rails doesn't do the hard work up-front, instead it triggers a background job to do it and responds quickly, freeing itself up to respond to other requests.
The data processed by the background job is viewed later - perhaps in a few requests time, or whenever the job is done, loaded in by javascript.
Definitely, because a dyno is single threaded if it's busy scrapping then other requests will be queued until the dyno is free or ultimately timed out if they hang for more than 30 seconds.
Anything that relies on an external service would be best run through a worker via DJ - even sending an email, so your controller puts the message into the queue and returns the user to a screen and then the mail is picked up by DJ and delivered so the user doesn't have to wait for the mailing process to complete.
Install the NewRelic gem to see what your queue length is doing
John.
If you're interested in a full service worker system we're looking for beta testers of our Heroku appstore app SimpleWorker.com. It's integrated tightly into Heroku and offers a number of advantages over DJ.
It's as simple as a few line of code to send your work off to our elastic worker cloud and you can take advantage of massive parallel processing because we do not limit the number of workers.
Shoot me a message if you're interested.
Chad