I'm after some advice on polling an external web service every 30 secs from a Domino server side action.
A quick bit of background...
We track the location of cars thru the TomTom api. We now have a requirement to show this in our web app, overlayed onto a map (google, bing, etc.) and mashed up with other lat long data from our application. Think of it as dispatching calls to taxis and we want to assign those calls to the taxis (...it's not taxis\ calls, but it is similar process). We refresh the dispatch controllers screens quite aggressively, so they can see the status of all the objects and assign to the nearest car. If we trigger the pull of data from the refresh of the users screen, we get into some tricky controlling server side, else we will hit the max allowable requests per minute to the TomTom api.
Originally I was going to schedule an agent to poll the web service, write to a cached object in our app, and the refreshing dispatch controllers screen pulls the data from our cache....great, except, user requirement is our cache must be updated every 30secs. I can create a program doc that runs every 1 min, but still not aggressive enough.
So we are currently left with: our .net guy will create a service that polls TomTom every 30secs, and we retrieve from his service, or I figure out a way to do in Domino. It would be nice to do in Domino database, and not some stand alone java app or .net, to keep as much of the logic as possible in one system (Domino).
We use backing beans heavily in our system. I will be testing this later today I hope, but would this seem like a sensible route to go down..?:
Spawning threads in a JSF managed bean for scheduled tasks using a timer
...or are their limitations I am not aware of, has anyone tackled this before in Domino or have any comments?
Thanks in advance,
Nick
Check out DOTS (Domino OSGi Tasklet Service): http://www.openntf.org/internal/home.nsf/project.xsp?action=openDocument&name=OSGI%20Tasklet%20Service%20for%20IBM%20Lotus%20Domino
It allows you to define background Java tasks on a Domino server that have all the advantages of agents (can be scheduled or triggered) with none of the performance or maintenance issues.
If you cache the data in a bean (application or session scoped). Have a date object that contains the last refreshed date. When the data is requested, check last cached date against current time. If it's more than/equal to 30 seconds, refresh data.
A way of doing it would be to write a managed bean which is created in the application scope ( aka there can only be one..). In this managed bean you take care of the 30sec polling of the webservice by good old java webservice implementation and a java thread which you start at the creation of your managed-bean something like
public class ServicePoller{
private static myThread = null;
public ServicePoller(){
if(myThread == null){
myThread = new ServicePollThread();
(new Thread(myThread)).start());
}
}
}
class ServicePollThread implements Runnable(){
private hashMap yourcache = null;
public ServicePollThread(){
}
public void run(){
while(running){
doPoll();
Thread.sleep(4000);
}
}
....
}
This managed bean will then poll every 30 seconds the webservice and save it's findings in a hashmap or some other managed-bean classes. This way you dont need to run an agent or something like that and you achieve when you use the dispatch screen to retrieve data from the cache.
Another option would be to write an servlet ( that would be possible with the extlib but I cant find the information right now ) which does the threading and reading the service for you. Then in your database you should be able to read the cache of the servlet and use it wherever you need.
As Tim said DOTS or as jjtbsomhorst said a thread or an Eclipse job.
I've created a video describing DOTS: http://www.youtube.com/watch?v=CRuGeKkddVI&list=UUtMIOCuOQtR4w5xoTT4-uDw&index=4&feature=plcp
Next Monday I'll publish a sample how to do threads and Eclipse jobs. Here is a preview video: http://www.youtube.com/watch?v=uYgCfp1Bw8Q&list=UUtMIOCuOQtR4w5xoTT4-uDw&index=1&feature=plcp
Related
In the apache brooklyn web interface we would like to display some content for the sytsem managers. The content is too long to be served as a simple sensor value.
Our idea was to create a task and write the content into the output stream of the task, and then offer the REST based URL to the managers like this:
/v1/activities/{task}/stream/stdout (Of course the link masked with some nice text)
The stream and task is created like this:
LOG.info("{} Creating Activity for ClusterReport Feed", this);
activity = Tasks.builder().
displayName("clusterReportFeed").
description("Output for the Cluster Report Feed").
body(new Runnable() {
#Override
public void run() {
//DO NOTHING
}
}).
parallel(true).
build();
LOG.info("{} Task Created with Id: " + activity.getId(), this);
Entities.submit(server, activity).getUnchecked();
The task seems to be created and the interraction works perfectly fine.
However when I want to access the tasks output stream from my browser using a prepared URL I get the error that the task does not exist.
Our idea is that we are not in the right Management/Execution Context. The Web page is running in an other context compared to the entities and their sensors. How can we put a task so that it's visible for the web consoles context also.
Is it possible to write the content into a file and then offer it for download via Jetty(brooklyns web server)? That would be a much simpler way.
Many tasks in Brooklyn default to being transient - i.e. they are deleted shortly after they complete (things like effector invocations are by default non-transient).
You can mark your task as non-transient using the code below in your use of the task builder:
.tag(BrooklynTaskTags.NON_TRANSIENT_TASK_TAG)
However, note that (as of Brooklyn version 0.9.0) tasks are kept in-memory using soft references. This means the stdout of the task will likely be lost at some point in the future, when that memory is needed for other in-memory objects.
For your use-case, would it make sense to have this as an effector result perhaps?
Or could you write to an object store such as S3 instead? The S3-approach would seem best to me.
For writing it to a file, care must be taken when used with Brooklyn high-availability. Would you write to a shared volume?
If you do write to a file, then you'd need to provide a web-extension so that people can access the contents of that file. As of Brooklyn 0.9.0, you can add your own WARs in code when calling BrooklynLauncher (which calls BrooklynWebServer).
I've been using MVC since version 2, and lately I have come across a project where all of the controller actions are 'async', returning Tasks, and I am trying to understand why somebody would do this.
The View Model for each view is built via an async call to an API. I understand that in order to use the await keyword one must use an async method (and return a Task), but surely without the View Model then the view will fail. There is no choice but to wait for the API to build my View Model.
public async Task<ActionResult> MyCar()
{
return View(await MyAPI.BuildMyCarViewModel());
}
For what reason would controller actions need to be asynchronous?
Let's assume that your part of code
MyAPI.BuildMyCarViewModel()
needs for execution 15 seconds. Then let's assume, that you have 10 000 users, which in range of 2 seconds decided to load some model. And then assume that you don't use caching ( for the sake of example ).
IIS by default has pool of threads 5000.
In described case application pool of IIS will be busy with 5000 threads which will translate into awaiting of your 5000 users for 5 seconds, and other 5000 users will wait until code finish executing. But with async/await .Net will generate state machine, and threads will be executed till moment of awaiting, and then threads will be released for making another useful job. And as soon as
MyAPI.BuildMyCarViewModel()
will return results, other threads or the same threads will return you result. And as outcome application pool of IIS will not be exhausted quickly for long running tasks and your users will receive response much faster, then without usage await/async. If to put simply, await/async gives you possibility to avoid thread pool exhausting quickly for long running fragments of code.
I have an MSDN article on the topic of async ASP.NET. In summary, the benefit is that the request does not take up a thread for the duration of the request. This allows your web app to scale if your backend can scale.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
Today we have built a console application for running the scheduled tasks for our ASP.NET website. But I think this approach is a bit error prone and difficult to maintain. How do you execute your scheduled task (in an windows/IIS/ASP.NET environment)
Update:
Examples of tasks:
Sending email from an email-queue in the database
Removing outdated objects from the database
Retrieving stats from Google AdWords and fill a table in the database.
This technique by Jeff Atwood for Stackoverflow is the simplest method I've come across. It relies on the "cache item removed" callback mechanism build into ASP.NET's cache system
Update: Stackoverflow has outgrown this method. It only works while the website is running but it's a very simple technique that is useful for many people.
Also check out Quartz.NET
All of my tasks (which need to be scheduled) for a website are kept within the website and called from a special page. I then wrote a simple Windows service which calls this page every so often. Once the page runs it returns a value. If I know there is more work to be done, I run the page again, right away, otherwise I run it in a little while. This has worked really well for me and keeps all my task logic with the web code. Before writing the simple Windows service, I used Windows scheduler to call the page every x minutes.
Another convenient way to run this is to use a monitoring service like Pingdom. Point their http check to the page which runs your service code. Have the page return results which then can be used to trigger Pingdom to send alert messages when something isn't right.
Create a custom Windows Service.
I had some mission-critical tasks set up as scheduled console apps and found them difficult to maintain. I created a Windows Service with a 'heartbeat' that would check a schedule in my DB every couple of minutes. It's worked out really well.
Having said that, I still use scheduled console apps for most of my non-critical maintenance tasks. If it ain't broke, don't fix it.
I've found this to be easy for all involved:
Create a webservice method such as DoSuchAndSuchProcess
Create a console app that calls this webmethod.
Schedule the console app in the task scheduler.
Using this methodology all of the business logic is contained in your web app, but you have the reliability of the windows task manager, or any other commercial task manager to kick it off and record any return information such as an execution report. Using a web service instead of posting to a page has a bit of an advantage because it's easier to get return data from a webservice.
Why reinvent the wheel, use the Threading and the Timer class.
protected void Application_Start()
{
Thread thread = new Thread(new ThreadStart(ThreadFunc));
thread.IsBackground = true;
thread.Name = "ThreadFunc";
thread.Start();
}
protected void ThreadFunc()
{
System.Timers.Timer t = new System.Timers.Timer();
t.Elapsed += new System.Timers.ElapsedEventHandler(TimerWorker);
t.Interval = 10000;
t.Enabled = true;
t.AutoReset = true;
t.Start();
}
protected void TimerWorker(object sender, System.Timers.ElapsedEventArgs e)
{
//work args
}
Use Windows Scheduler to run a web page.
To prevent malicous user or search engine spiders to run it, when you setup the scheduled task, simply call the web page with a querystring, ie : mypage.aspx?from=scheduledtask
Then in the page load, simply use a condition :
if (Request.Querystring["from"] == "scheduledtask")
{
//executetask
}
This way no search engine spider or malicious user will be able to execute your scheduled task.
This library works like a charm
http://www.codeproject.com/KB/cs/tsnewlib.aspx
It allows you to manage Windows scheduled tasks directly through your .NET code.
Additionally, if your application uses SQL SERVER you can use the SQL Agent to schedule your tasks. This is where we commonly put re-occurring code that is data driven (email reminders, scheduled maintenance, purges, etc...). A great feature that is built in with the SQL Agent is failure notification options, which can alert you if a critical task fails.
I'm not sure what kind of scheduled tasks you mean. If you mean stuff like "every hour, refresh foo.xml" type tasks, then use the Windows Scheduled Tasks system. (The "at" command, or via the controller.) Have it either run a console app or request a special page that kicks off the process.
Edit: I should add, this is an OK way to get your IIS app running at scheduled points too. So suppose you want to check your DB every 30 minutes and email reminders to users about some data, you can use scheduled tasks to request this page and hence get IIS processing things.
If your needs are more complex, you might consider creating a Windows Service and having it run a loop to do whatever processing you need. This also has the benefit of separating out the code for scaling or management purposes. On the downside, you need to deal with Windows services.
If you own the server you should use the windows task scheduler. Use AT /? from the command line to see the options.
Otherwise, from a web based environment, you might have to do something nasty like set up a different machine to make requests to a certain page on a timed interval.
I've used Abidar successfully in an ASP.NET project (here's some background information).
The only problem with this method is that the tasks won't run if the ASP.NET web application is unloaded from memory (ie. due to low usage). One thing I tried is creating a task to hit the web application every 5 minutes, keeping it alive, but this didn't seem to work reliably, so now I'm using the Windows scheduler and basic console application to do this instead.
The ideal solution is creating a Windows service, though this might not be possible (ie. if you're using a shared hosting environment). It also makes things a little easier from a maintenance perspective to keep things within the web application.
Here's another way:
1) Create a "heartbeat" web script that is responsible for launching the tasks if they are DUE or overdue to be launched.
2) Create a scheduled process somewhere (preferrably on the same web server) that hits the webscript and forces it to run at a regular interval. (e.g. windows schedule task that quietly launches the heatbeat script using IE or whathaveyou)
The fact that the task code is contained within a web script is purely for the sake of keeping the code within the web application code-base (the assumption is that both are dependent on each other), which would be easier for web developers to manage.
The alternate approach is to create an executable server script / program that does all the schedule work itself and run the executable itself as a scheduled task. This can allow for fundamental decoupling between the web application and the scheduled task. Hence if you need your scheduled tasks to run even in the even that the web app / database might be down or inaccessible, you should go with this approach.
You can easily create a Windows Service that runs code on interval using the 'ThreadPool.RegisterWaitForSingleObject' method. It is really slick and quite easy to get set up. This method is a more streamlined approach then to use any of the Timers in the Framework.
Have a look at the link below for more information:
Running a Periodic Process in .NET using a Windows Service:
http://allen-conway-dotnet.blogspot.com/2009/12/running-periodic-process-in-net-using.html
We use console applications also. If you use logging tools like Log4net you can properly monitor their execution. Also, I'm not sure how they are more difficult to maintain than a web page, given you may be sharing some of the same code libraries between the two if it is designed properly.
If you are against having those tasks run on a timed basis, you could have a web page in your administrative section of your website that acts as a queue. User puts in a request to run the task, it in turn inserts a blank datestamp record on MyProcessQueue table and your scheduled task is checking every X minutes for a new record in MyProcessQueue. That way, it only runs when the customer wants it to run.
Hope those suggestions help.
One option would be to set up a windows service and get that to call your scheduled task.
In winforms I've used Timers put don't think this would work well in ASP.NET
A New Task Scheduler Class Library for .NET
Note: Since this library was created, Microsoft has introduced a new task scheduler (Task Scheduler 2.0) for Windows Vista. This library is a wrapper for the Task Scheduler 1.0 interface, which is still available in Vista and is compatible with Windows XP, Windows Server 2003 and Windows 2000.
http://www.codeproject.com/KB/cs/tsnewlib.aspx
I am trying to write a control panel to
Inform about certain KPIS
Enable the user to init certain requests / jobs by pressing a button that then runs a stored proc on the DB or sets a specific setting etc
So far, so good, except I would like to run some bigger jobs where the length of time that the job is running for is unknown and could run over both the script timeout period AND the time the user is willing to wait for a response.
What I want is a "fire and forget" process so the user hits the button and even if they kill the page or turn off their phone they know the job has been initiated and WILL complete.
I was looking into C# BeginExecuteNonQuery which is an async call to the query so the proc is fired but the control doesn't have to wait for a response from it to carry on. However I don't know what happens when the page/app that fired it is shut.
Also I was thinking of some sort of Ajax command that fires the code in a page behind the scenes so the user doesn't know about it running but then again I believe if the user shuts the page down the script will die and the command will die on the server as well.
The only way for certain I know of is a "queue" table where jobs are inserted into this table then an MS Agent job comes along every minute or two checking for new inserts and then runs the code if there is any. That way it is all on the DB and only a DB crash will destroy it. It won't help with multiple jobs waiting to be run concurrently that both take a long time but it's the only thing I can be sure of that will ensure the code is run at all.
Any ideas?
Any language is okay.
Since web browsers are unconnected, requests from them always take the full amount of time. The governing factor isn't what the browser does, but how long the web site itself will allow an action to continue.
IIS (and in general, web servers) have a timeout period for requests, where if the work being done takes simply too long, the request is terminated. This would involve abruptly stopping whatever is taking so long, such as a database call, running code, and so on.
Simply making your long-running actions asynchronous may seem like a good idea, however I would recommend against that. The reason is that in ASP and ASP.Net, asynchronously-called code still consumes a thread in a way that blocks other legitimate request from getting through (in some cases you can end up consuming two threads!). This could have performance implications in non-obvious ways. It's better to just increase the timeout and allow the synchronously blocking task to complete. There's nothing special you have to do to make such a request complete fully, it will occur even if the sender closes his browser or turns off his phone immediately after (presuming the entire request was received).
If you're still concerned about making certain work finish, no matter what is going on with the web request, then it's probably better to create an out-of-process server/service that does the work and to which such tasks can be handed off. Your web site then invokes a method that, inside the service, starts its own async thread to do the work and then immediately returns. Perhaps it also returns a request ID, so that the web page can check on the status of the requested work later through other methods.
You may use asynchronous method and call the query from this method.
Your simple method can be changed in to a asynch method in the following manner.
Consider that you have a Test method to be called asynchronously -
Class AsynchDemo
{
public string TestMethod(out int threadId)
{
//call your query here
}
//create a asynch handler delegate:
public delegate string AsyncMethodCaller(out int threadId);
}
In your main program /or where you have to call the Test Method:
public static void Main()
{
// The asynchronous method puts the thread id here.
int threadId;
// Create an instance of the test class.
AsyncDemo ad = new AsyncDemo();
// Create the delegate.
AsyncMethodCaller caller = new AsyncMethodCaller(ad.TestMethod);
// Initiate the asychronous call.
IAsyncResult result = caller.BeginInvoke(
out threadId, null, null);
// Call EndInvoke to wait for the asynchronous call to complete,
// and to retrieve the results.
string returnValue = caller.EndInvoke(out threadId, result);
Console.WriteLine("The call executed on thread {0}, with return value \"{1}\".",
threadId, returnValue);
}
From my experience a Classic ASP or ASP.NET page will run until complete, even if the client disconnects, unless you have something in place for checking that the client is still connected and do something if they are not, or a timeout is reached.
However, it would probably be better practice to run these sorts of jobs as scheduled tasks.
On submitting your web page could record in a database that the task needs to be run and then when the scheduled task runs it checks for this and starts the job.
Many web hosts and/or web control panels allow you to create scheduled tasks that call a URL on schedule.
Alternately if you have direct access to the web server you could create a scheduled task on the server to call a URL on schedule.
Or, if ASP.NET, you can put some code in global.asax to run on a schedule. Be aware though, if your website is set to stop after a certain period of inactivity then this will not work unlesss there is frequent continuous activity.
I see several other questions about load testing web services. But as far as I can tell those are all synchronous load testing tools. (Meaning they send a ton of requests but the go one at a time.)
I am looking for a tool where I can say, "I want 100 requests to be launched at the exact same time".
Now, I am new to the whole load testing thing, so it is possible that those tools are asynchronous and I am just missing it.
Anyway, in short my question is: Is there a good tool for load testing WCF Web Services asynchronously (ie lots of threads).
In general, I recommend you look at soapUI, for anything to do with testing web services. They do have load testing features in the Professional edition (I haven't used these yet).
In addition, they've just entered beta with a loadUI product. If it's anywhere near as good as the parent product, then it's worth a hard look.
you can use the Visual Studio load testing agent components to run on multiple client machines and that will allow you to run as asynchronously as you have machines to load.
There is a licence requirement for using this feature.
There are no tools that will allow you to apply a load at exactly the same instant (i.e. within milliseconds), but this is not necessary to load test an application correctly.
For most needs a single load test server running Visual Studio Ultimate edition will be more than enough to get an understand of how your webservice performs under load.
Visual Studio and most other tools I imagine will apply load in an asynchronous manner, but I think in your view you want to apply a set load all at once.
This is not really necessary as in practice load is not applied to a service in this manner.
The best bet for services expecting high load is to load your service until a given number of "requests per second" is reached. Finding what level your application should expect is a bit trickier, but involves figuring out roughly how many users you would expect and the amount they will be using it over a given period.
The other test to do is to setup a load test harness and run the load up until either the webservice starts to perform badly or the test harness runs out of "oomph" and cannot create any more load.
For development time you can use NLoad (http://nload.github.io)
to run load tests on your development machine or testing environment.
For example
public class MyTest : ITest
{
public void Initialize()
{
// Initialize your test, e.g., create a WCF client, load files, etc.
}
public void Execute()
{
// Send http request, invoke a WCF service or whatever you want to load test.
}
}
Then create, configure and run a load test:
var loadTest = NLoad.Test<MyTest>()
.WithNumberOfThreads(100)
.WithDurationOf(TimeSpan.FromMinutes(5))
.WithDeleyBetweenThreadStart(TimeSpan.Zero)
.OnHeartbeat((s, e) => Console.WriteLine(e.Throughput))
.Build();
var result = loadTest.Run();