I am use Web Polygraph load testing tool to make rapid http requests as it is reliable, low resource consumption, and has good reporting. However, I cannot find any settings to tell Web Polygraph to run for a certain amount of time. I want to be able to have accurate reporting instead of potential misses caused by sending a kill signal to the process.
I have been reading through web polygraph's help pages and can see that the requests per second is configurable, but am not seeing support for request duration time.
I have the config file as such (I think this is where the option would go, likely in the Robot configuration):
Content SimpleContent = {
size = exp(1KB); // response sizes distributed exponentially
cachable = 100%;
};
Server S1 = {
kind = "S101";
contents = [ SimpleContent ];
direct_access = contents;
addresses = ['X.X.X.X' ];
};
// a primitive robot
Robot R1 = {
kind = "R101";
req_rate = 100/sec;
interests = [ "foreign" ];
foreign_trace = "/home/x/trace.urls";
pop_model = { pop_distr = popUnif(); };
recurrence = 100% / SimpleContent.cachable;
origins = S1.addresses;
addresses = ['X.X.X.X' ];
};
I am expecting to be able to set some duration, say 40min, where I am able to have the R1 robot request 100 pages per second for 40 minutes.
I got an answer from the Web Polygraph support. For future reference, this can be accomplished through the Phase and Goal objects, as well as using the Schedule function with them. Here is a snipbit of the email I got back:
See the goal field inside the Phase object:
http://www.web-polygraph.org/docs/reference/pgl/types.html#type:docs/reference/pgl/types/Goal
http://www.web-polygraph.org/docs/reference/pgl/types.html#type:docs/reference/pgl/types/Phase
Do not forget to schedule() your phases:
http://www.web-polygraph.org/docs/reference/pgl/calls.html
Many workloads that are distributed with Polygraph include Phase
schedules. To see examples, search for "goal" in workloads/
Related
In django application I need to call an external rabbitmq, running on a windows server and using some application there, where the django app runs on a linux server.
I'm currently able to add a task to the queue by using the celery send_task:
app.send_task('tasks', kwargs=self.get_input(), queue=Queue('queue_async', durable=False))
My settings looks like:
CELERY_BROKER_URL = CELERY_CONFIG['broker_url']
BROKER_TRANSPORT_OPTIONS = {"max_retries": 3, "interval_start": 0, "interval_step": 0.2, "interval_max": 0.5}
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_DEFAULT_QUEUE = 'celery'
CELERY_TASK_RESULT_EXPIRES = 3600
CELERY_RESULT_BACKEND = 'rpc://'
CELERY_CREATE_MISSING_QUEUES = True
What I'm not sure about is how I can get and parse the response, since the send_task only returns a key?
If you want to store results of your task , you could use this parameter result_backend or CELERY_RESULT_BACKEND depending on the version of celery you're using.
Complete list of Configuration options can be found here (search for result_backend on this page) => https://docs.celeryproject.org/en/stable/userguide/configuration.html
Many options are available to store results - SQL DBs , NoSQL DBs, Elasticsearch, Memcache, Redis, etc,etc . Choose as per your project stack.
Thanks that helped for the understanding. So since I want to further process the answer I use rpc as already defined in the config I had in the example.
What I found usefull was this example, because most python celery examples assume that the consumer is the same application, that describes the interaction to a Java app Celery-Java since it gives a good example on how to request from python side.
Therefore my implementation is now:
result = app.signature('tasks', kwargs=self.get_input(), queue=Queue('queue_async', durable=False)).delay().get()
which waits and parses the result.
In my WPF application I’m trying to use off-line map functionality. Right now my feature service is configured for data sync and I’m able to create data replica on server and download local copy of geodatabase.
gdbSyncTask = await GeodatabaseSyncTask.CreateAsync(_featureServiceUri);
Envelope extent = new Envelope(xmin, ymin, xmax, ymax, new SpatialReference(wkidStart));
GenerateGeodatabaseParameters generateParams = await _gdbSyncTask.CreateDefaultGenerateGeodatabaseParametersAsync(extent);
_generateGdbJob = _gdbSyncTask.GenerateGeodatabase(generateParams, _gdbPath);
_generateGdbJob.JobChanged += GenerateGdbJobChanged;
_generateGdbJob.ProgressChanged += ((object sender, EventArgs e) =>
{
UpdateProgressBar();
});
_generateGdbJob.Start();
After initial synchronization, I’m able to successfully work with map in off-line mode. This includes operations like adding new geometries or editing existing polygons inside local DB.
However, when I’m trying to synchronize changes back to server – I’m getting no results.
To perform data synchronization with local database – I’m using the following code:
SyncGeodatabaseParameters parameters = new SyncGeodatabaseParameters()
{
GeodatabaseSyncDirection = SyncDirection.Bidirectional,
RollbackOnFailure = false
};
Geodatabase gdb = await Geodatabase.OpenAsync(this.GetGdbPath());
foreach (GeodatabaseFeatureTable table in gdb.GeodatabaseFeatureTables)
{
long id = table.ServiceLayerId;
SyncLayerOption option = new SyncLayerOption(id);
option.SyncDirection = SyncDirection.Bidirectional;
parameters.LayerOptions.Add(option);
}
_gdbSyncTask = await GeodatabaseSyncTask.CreateAsync(_featureServiceUri);
SyncGeodatabaseJob job = _gdbSyncTask.SyncGeodatabase(parameters, gdb);
job.JobChanged += SyncJob_JobChanged;
job.ProgressChanged += SyncJob_ProgressChanged;
job.Start();
Everything goes well. The synchronization ends with status “Succeeded”. The messages logged by the SyncGeodatabaseJob are like on the screen below:
However – when I open edited feature layer from server inside map web client I cannot found any of my local changes. In the serve database I can also see that no new records were created during synchronization.
Interesting think is that when I open “Replica” data inside web I can see the following information:
Replica Server Gen: 2
Creation Date: 2018/02/07 10:49:54 UTC
Last Sync Date: 2018/02/07 10:49:54 UTC
The “Last Sync Data” is equal to replica “Creation date” However, in the replica log in ArcMap I can see the following information:
Can anyone can tell me how should I interpret above described situation? Am I missing some steps in my code? Or maybe some configuration feature is missing on the server? It looks like data modifications are successfully pushed back to replica on server but after that replica is not synchronized with server database (should it work automatically?).
I’m a “fresh” person regarding ArcGis development so any help will be appreciated
Thanks for all the answers. It occurred that there is versioning enabled on the server database and the offline, versioned changes was not reconciled to the server.
After running reconcile/post script (http://desktop.arcgis.com/en/arcmap/10.3/manage-data/geodatabases/automate-reconcile-post-after-sync.htm) off-line changes started to be visibile to other system users.
The code looks ok on fast look so I would assume that there is something going on in the setup.
What do you get back from the sync operation after the sync has completed? Note that you can just use await syncJob.GetResultsAsync to start the job and wait the results.
How is the Feature Service set up on the server? Please refer https://enterprise.arcgis.com/en/server/latest/publish-services/linux/prepare-data-for-offline-use.htm for the different ways to set these things.
I'm trying to use akka-streams and akka-http and the alpakka library to download/upload files to Amazon S3. I am seeing two issues which might be related...
I can only download very small files, the largest one 8kb.
I can't upload larger files. It fails with the message
Error during processing of request: 'Substream Source has not been
materialized in 5000 milliseconds'. Completing with 500 Internal
Server Error response. To change default exception handling behavior,
provide a custom ExceptionHandler.
akka.stream.impl.SubscriptionTimeoutException:
Substream Source has not been materialized in 5000 milliseconds
Here are my routes
pathEnd {
post {
fileUpload("attachment") {
case (metadata, byteSource) => {
val writeResult: Future[MultipartUploadResult] = byteSource.runWith(client.multipartUpload("bucketname", key))
onSuccess(writeResult) { result =>
complete(result.location.toString())
}
}
}
}
} ~
path("key" / Segment) {
(sourceSystem, sourceTable, sourceId) =>
get {
val result: Future[ByteString] =
client.download("bucketname", key).runWith(Sink.head)
onSuccess(result) {
complete(_)
}
}
}
Trying to download a file of say 100KB will end up fetching a truncated version of the file usually of size around 16-25Kb
Any help appreciated
Edit: For the download issue, I took Stefano's suggestion and got
[error] found : akka.stream.scaladsl.Source[akka.util.ByteString,akka.NotUsed]
[error] required: akka.http.scaladsl.marshalling.ToResponseMarshallable
This made it work
complete(HttpEntity(ContentTypes.`application/octet-stream`, client.download("bucketname", key).runWith(Sink.head)))
1) On the download issue: by calling
val result: Future[ByteString] =
client.download("bucketname", key).runWith(Sink.head)
you are streaming all the data from S3 into memory, and then serve the result.
Akka-Http as streaming support that allows you to stream bytes straight from a source, without buffering them all in memory. More info on this can be found in the docs. Practically, this means the complete directive can take a Source[ByteString, _], as in
...
get {
complete(client.download("bucketname", key))
}
2) On the upload issue: you can try to tweak Akka HTTP akka.http.server.parsing.max-content-length setting:
# Default maximum content length which should not be exceeded by incoming request entities.
# Can be changed at runtime (to a higher or lower value) via the `HttpEntity::withSizeLimit` method.
# Note that it is not necessarily a problem to set this to a high value as all stream operations
# are always properly backpressured.
# Nevertheless you might want to apply some limit in order to prevent a single client from consuming
# an excessive amount of server resources.
#
# Set to `infinite` to completely disable entity length checks. (Even then you can still apply one
# programmatically via `withSizeLimit`.)
max-content-length = 8m
Resulting code to test this would be something along the lines of:
withoutSizeLimit {
fileUpload("attachment") {
...
}
}
My use case varies for this, but in general i'm trying to collect a bunch of elements and then apply _.map() to each. The problem is that this series of .getAttribute() calls can cause a test that works locally to fail against a remote server like sauce/android.
One example: collecting all <div class='article'><a href='articles/{id}'> on a page and then getting the hrefs. it might look something like this, and this approach will work until i test on a mobile (android) sauce environment. then I get a timeout.
Is it possible this is an issue related to my android environment capabilities? To piling up so many requests? I've tried scaling my test down from using 75 articles to only 45 and i've upped the timeout to 60s and still the mobile test fails. locally with chromedriver is fine, chrome desktop + sauce is fine.
Not my actual test but an approximation of the code i'm talking about:
/// ... return this.remote
.findAllByTagName('div.article a')
.then(function (articles) {
var promises = articles.map(function(article) {
return article.getAttribute('href');
});
Promise.all(promises)
.then(function (hrefs) {
uniques = _.uniq(hrefs);
assert(hrefs.length === uniques.length);
});
});
Since you're seeing a timeout error, I'd suggest continuing to increase the test timeout until the test passes. The mobile testing environments on Sauce are both slower to initialize and slower to operate than the desktop environments, so it's quite possible that a test with many requests is simply very slow.
One way to speed things up would be to use an execute block to gather the references, like:
.then(function (articles) {
return this.parent.execute(function (articles) {
return articles.map(function (node) {
return node.getAttribute('href');
});
}, [ articles ]);
})
In the above snippet, the articles element array is passed as an argument to the execute block. The remote WebDriver will deserialize the element references into actual DOM elements that can be operated on in the execute code. This is significantly more efficient than using individual getAttribute requests for each element since only a single request will be made to the remote browser.
I've implemented a jobs queue a few days ago and I've been experiencing problems with duplication, I'm currently working with Redis and followed the Laravel's official tutorial.
In my case, whenever someone goes to the homepage, a job is sent to the queue, lets take this example:
HomeController's index() :
public function index()
{
if(/*condition*/){
//UpdateServer being the job
$this->dispatch(new UpdateServer());
}
}
Since this task takes about 10 seconds to complete, if there's n requests to my homepage while the task is being processed, there will be n more of the same job in queue, resulting in unexpected results in my Database.
So my question is, is there any way to know if a certain job is already in queue?
I know it's an old question but I find myself coming back here again and again from Google so I wanted to give it an answer. I wanted an easy way to view the jobs in the queue inside my Laravel application on a dashboard and used the following code.
$thejobs = array();
// Get the number of jobs on the queue
$numJobs = Redis::connection()->llen('queues:default');
// Here we select details for up to 1000 jobs
$jobs = Redis::connection()->lrange('queues:default', 0, 1000);
// I wanted to clean up the data a bit
// you could use var_dump to see what it looks like before this
// var_dump($jobs);
foreach ($jobs as $job) {
// Each job here is in json format so decode to object form
$tmpdata = json_decode($job);
$data = $tmpdata->data;
// I wanted to just get the command so I stripped away App\Jobs at the start
$command = $this->get_string_between($data->command, '"App\Jobs\\', '"');
$id = $tmpdata->id;
// Could be good to see the number of attempts
$attempts = $tmpdata->attempts;
$thejobs[] = array($command, $id, $attempts);
}
// Now you can use the data and compare it or check if your job is already in queue
I don't recommend doing this, especially on page load such as the index page like the op has done. Most likely you need to rethink the way you are doing things if you need to have this code to check if a job is running.
The answer is specific to queues running Redis.
I know it's very old question, but I'm answering it for future Google users.
Since Laravel 8 there is the "Unique Jobs" feature - https://laravel.com/docs/8.x/queues#unique-jobs.
For anyone wondering why
Queue::size('queueName');
is not the same size as
Redis::llen('queues:queueName');
is because Laravel uses 3 records to count the size of a queue, so if you want the true number of jobs in the queue you must do:
Redis::lrange('queues:queueName', 0, -1);
Redis::zrange('queues:queueName:delayed', 0, -1);
Redis::zrange('queues:queueName:reserved', 0, -1);
Now you can evaluate if your desired input is in one of those queues and act according.
You can do it in jobs handle function and skip work if another same job is scheduled
public function handle()
{
$queue = \DB::table(config('queue.connections.database.table'))->orderBy('id')->get();
foreach ($queue as $job){
$payload = json_decode($job->payload,true);
if($payload['displayName'] == self::class && $job->attempts == 0){
// same job in queue, skip
return ;
}
}
// do the work
}