Ok, so I have control over the below parameters in Apache JMeter:
Number of Threads (users)
Ramp-up period (in seconds)
How do I test for varying sampling rate and not varying user addition rate? Even a fixed sampling rate would do.
Thanks in advance :)
Got my answer :)
Sampling rate can be limited by setting timer parameter.
Right click on your test plan, then Add, then Timers.
There are a varieties of timers to cater to various needs. I used constant timer in my case.
Particular timer you need is: http://code.google.com/p/jmeter-plugins/wiki/ThroughputShapingTimer
Constant Throughput Timer is excellent to regulate the number of hits without changing the threads and/or delays.
http://jmeter.apache.org/usermanual/component_reference.html#Constant_Throughput_Timer
It can also be controlled from outside (not easy).
Related
I would like to know whether there is a recommended way of measuring execution time in Tensorflow Federated. To be more specific, if one would like to extract the execution time for each client in a certain round, e.g., for each client involved in a FedAvg round, saving the time stamp before the local training starts and the time stamp just before sending back the updates, what is the best (or just correct) strategy to do this? Furthermore, since the clients' code run in parallel, are such a time stamps untruthful (especially considering the hypothesis that different clients may be using differently sized models for local training)?
To be very practical, using tf.timestamp() at the beginning and at the end of #tf.function client_update(model, dataset, server_message, client_optimizer) -- this is probably a simplified signature -- and then subtracting such time stamps is appropriate?
I have the feeling that this is not the right way to do this given that clients run in parallel on the same machine.
Thanks to anyone can help me on that.
There are multiple potential places to measure execution time, first might be defining very specifically what is the intended measurement.
Measuring the training time of each client as proposed is a great way to get a sense of the variability among clients. This could help identify whether rounds frequently have stragglers. Using tf.timestamp() at the beginning and end of the client_update function seems reasonable. The question correctly notes that this happens in parallel, summing all of these times would be akin to CPU time.
Measuring the time it takes to complete all client training in a round would generally be the maximum of the values above. This might not be true when simulating FL in TFF, as TFF maybe decided to run some number of clients sequentially due to system resources constraints. In practice all of these clients would run in parallel.
Measuring the time it takes to complete a full round (the maximum time it takes to run a client, plus the time it takes for the server to update) could be done by moving the tf.timestamp calls to the outer training loop. This would be wrapping the call to trainer.next() in the snippet on https://www.tensorflow.org/federated. This would be most similar to elapsed real time (wall clock time).
I want to have a function like if the calculation time get too long, we abort routing calculation and submit the best solution at the point of time. Is there such a function in optaplanner ?
For example in a GUI application, you would start solving on a background (worker) thread. In this scenario you can stop solver asynchronously by calling solver.terminateEarly() from another thread, typically the UI thread when you click a stop button.
If this is not what you're looking for, read on.
Provided that by calculation you actually mean the time spent solving, you have several options how to stop solver. Besides asynchronous termination described in the first paragraph, you can use synchronous termination:
Use time spent termination if you know how much time you want dedicate to solving beforehand.
Use unimproved time spent termination if you want to stop solving if the solution doesn't improve for a specified amount of time.
Use best score termination if you want to stop solving after a certain score has been reached.
Synchronous termination is defined before starting the solver and it's done either by XML solver configuration or using the SolverConfig API. See OptaPlanner documentation for other termination conditions.
Lastly, in case you're talking about score calculation and it takes too long to calculate score for a single move (solution change) then you're most certainly doing something wrong. For OptaPlanner to be able to search the solution space effectively, the score calculation must be fast (at least 1000 calculations per second).
For example in vehicle routing problem, driving time or road distances must be known at the time when you start solving. You shouldn't slow down score calculation with a heavy computation that can be done beforehand.
can you give some.explanation on how to interpret deviation vs.throughput?
Is 1000++ deviation result means a poor performance of web under test?
And how can you also say that the web under test performs good? Is it base on.throughput result? How?
And what listener is the best for tracing the load/performance of a thousand users
Lastly is that possible to check the cpu/ram usage of the server while pwrforming the test
Standard deviation quantifies how much response time varies around its mean, or average. It is not advisable to judge the system performance based on Standard Deviation. In reality this gives how much system is fluctuating. Deviations should be minimum i.e. less than 5%.
Thoughput is defined as number of requests processed per second.
It is better to use Throughput as a factor to judge system/application performance. Higher throughput means good system performance but again this depends on your choice. For some critical systems low response time is more required than high throughput. Throughput simply states that how much concurrent transactions your system can process per second which might mean high response time. if response time increases beyond a certain limit then that system is considered for performance tuning.
some systems Throughput means
You can either use Summary Report or Aggregate Report listeners.
cpu/ram usage, you can use "jp#gc - Perfmon Metrics collector" from Jmeter Plugin.
Hope this will help.
I'm working on an IRC bot in VB.net 2012. I know I know, but it's not a botnet, lol
I wanted to ask your advise on how to manage to many timers. Right now I have a timer that rewards points at a user specified interval, a timer that autosaves those points at a user specified interval. I also have one that displays advertisements, and broadcast a collection of responses at specified intervals.
I feel like it's getting out of hand and would love to know if there is a way I could do all of these thing with a single or at least less timers.
Please keep in mind I learn as I go and don't always understand all the terminology, but I am a quick learner if you have the patience to explain. :)
Thank you.
Yes, of course you can do them with a single timer. In fact, that is what you should do—timers are a limited resource, and there's hardly ever a reason for a single application to use more than one of them.
What you do is create a single timer that ticks at the most frequent interval required by all of your logic. Then, inside of that timer's Tick event handler, you set/check flags that indicate how much time has elapsed. Depending on which interval has expired, you perform the appropriate action and update the state of the flags. By "flags", I mean module-level variables that keep track of the various intervals you want to track—the ones you're tracking now with different timers.
It is roughly the same way that you keep track of time using a clock. You don't use separate clocks for every task you want to time, you use a single clock that ticks every second. You operate off of displacements from this tick—60 of these ticks is 1 minute, 3600 of these ticks is 1 hour, etc.
However, I strongly recommend figuring out a way to do as many of these things as possible in response to events, rather than at regular intervals tracked by a timer. You could, for example, reward points in response to specific user actions. This type of polling is very resource-intensive and rarely the best solution.
I want to record Cpu usage ,cpu time ,VM size in notepad per minute for any particular process(not for all.Is there any way to this,because i work as a performance/stress tester and its my duty to take the cpu performance after at particular time and the script takes more time so it is some time inconvenient to me take the all reading
please suggest.
thank u.
Use performance monitor, if the Windows is the system that you are working on. It has all kinds of log options, and will do what you need.
Performance monitor gives you the option of recording the performance data for particular process. Look under 'process'...