I'm writing a benchmark for computing the time taken for serialization/de-serialization for OpenFast
The question is when I'm computing the time taken, should I also take into account the time taken for parsing and setting the values for fields and also the same while de-serialization.
I'm writing it in java so what accuray is good Nano or Milli? Are there any issues in nano second computation that I should be worried about?
Thanks.
From Effective Java, Edition 2, Item 69:
For interval
timing, always use System.nanoTime in preference to System.currentTime-
Millis. System.nanoTime is both more accurate and more precise, and it is not
affected by adjustments to the system’s real-time clock.
Related
I would like to know whether there is a recommended way of measuring execution time in Tensorflow Federated. To be more specific, if one would like to extract the execution time for each client in a certain round, e.g., for each client involved in a FedAvg round, saving the time stamp before the local training starts and the time stamp just before sending back the updates, what is the best (or just correct) strategy to do this? Furthermore, since the clients' code run in parallel, are such a time stamps untruthful (especially considering the hypothesis that different clients may be using differently sized models for local training)?
To be very practical, using tf.timestamp() at the beginning and at the end of #tf.function client_update(model, dataset, server_message, client_optimizer) -- this is probably a simplified signature -- and then subtracting such time stamps is appropriate?
I have the feeling that this is not the right way to do this given that clients run in parallel on the same machine.
Thanks to anyone can help me on that.
There are multiple potential places to measure execution time, first might be defining very specifically what is the intended measurement.
Measuring the training time of each client as proposed is a great way to get a sense of the variability among clients. This could help identify whether rounds frequently have stragglers. Using tf.timestamp() at the beginning and end of the client_update function seems reasonable. The question correctly notes that this happens in parallel, summing all of these times would be akin to CPU time.
Measuring the time it takes to complete all client training in a round would generally be the maximum of the values above. This might not be true when simulating FL in TFF, as TFF maybe decided to run some number of clients sequentially due to system resources constraints. In practice all of these clients would run in parallel.
Measuring the time it takes to complete a full round (the maximum time it takes to run a client, plus the time it takes for the server to update) could be done by moving the tf.timestamp calls to the outer training loop. This would be wrapping the call to trainer.next() in the snippet on https://www.tensorflow.org/federated. This would be most similar to elapsed real time (wall clock time).
I have been using Pusher for some time now. I always assumed "Real time" meant "instantaneous". Lately I have step into this article: https://en.wikipedia.org/wiki/Real-time_computing, and a sentence grab my attention:
"Real-time programs must guarantee response within specified time
constraints"
They give an example based on audio processing:
"Consider an audio DSP example; if a process requires 2.01 seconds to
analyze, synthesize, or process 2.00 seconds of sound, it is not
real-time. However, if it takes 1.99 seconds, it is or can be made
into a real-time DSP process."
My questions:
1. This definition only applies to hardware/electronic devices or can be applied to software too?
2. If applies to software, does it apply to remote services like Pusher?
3. What is the time constraint for pusher to be considered "Real time"?
4. What is the time constraint for other services like WebRTC, Firebase?
Sorry for the lengthly post that doesn't specifically answer your question, but I hope it will make you better undestand where the "real time" definition comes from.
Yes, it is an understandable confusion that "real time" means "instantaneous". But if you really start to think about it you will soon find out that "instantaneous" is difficult to define.
What does instantaneous mean? 0 (zero) seconds response time (as in 0 sec 0 ms 0 ns 0 ps) from the time of the command to the time of the response is physically impossible. We can then try to say that instantaneous would mean that the command-response time is perceived instantaneously, i.e. it would not be seen as a delay. But then... what exactly does "perceived instantaneously" mean? Perceived by humans? Ok, that is good, we are getting somewhere. Human eye and the brain image processing are a very very complex machine and it does not really simply work in fps, but we can use data to approximate some. A human eye can "perceive an image flashed on the screen for 1/250th of a second". That would be 0.004 seconds or 250 fps. So by this approximation a graphical program would be real time if it has a response time < 0.004 sec or would run faster than #250 fps. But we know that in practice games are perceived smooth by most people at just 60 fps, or 0.01666 seconds. So now we have two different answers. Can we somehow justify them both? Yes. We can say in theory realtime would mean 0.004 seconds, but in practice 0.01666 seconds is enough.
We could be happy and stop here, but we are on a journey of discovery. So lets think further. Would you want a "real time" avionic automation system to have 0.01666 seconds response time? Would you deem acceptable a 0.01666 seconds response time for a "real time" nuclear plant system? Would an oil control system where a valve takes physically 15 seconds to close be defined as "real time" if the command-completion time is 0.0166 seconds? The answer to all these questions is most definitely no. Why? Answer that and you answer why "real time" is defined as it is: "Real-time programs must guarantee response within specified time constraints".
I am sorry, I am not familiar at all with "Pusher", but I can answer your first question and part of your second one: "real times" can be applied to any system that needs to "react" or respond to some form of input. Here "system" is more generic than you might think. A brain would qualify, but in the context of engineering means the whole stack: hardware + software.
This definition only applies to hardware/electronic devices or can be applied to software too?
It applies to software too. Anything that has hard time constraints. There are real-time operating systems, for example, and even a real-time specification for Java.
If applies to software, does it apply to remote services like Pusher?
Hard to see how, if a network is involved. More probably they just mean 'timely', or maybe it's just a sloppy way of saying 'push model', as the name implies. Large numbers of users on this site seem to think that 'real-time' means 'real-world'. In IT it means a system that is capable of meeting hard real-time constraints. The Wikipedia definition you cited is correct but the example isn't very satisfactory.
What is the time constraint for pusher to be considered "Real time"?
The question is malformed. The real question is whether Pusher can actually meet hard real-time constraints at all, and only then what their minimum value might be. It doesn't seem at all likely without operating system and network support.
What is the time constraint for other services like WebRTC, Firebase?
Ditto.
Most interpretations of the term "real-time " refer to the traditional static type, often referred to as "hard real-time." Although there is not much of a consensus on the meanings of the terms "hard real-time" and "soft real-time," I provide definitions, based on scientific first principles, of these and other essential terms in Introduction to Fundamental Principles of Dynamic Real-Time Systems.
We are in the case of using a SQL database for a single node storage of roughly 1 hour of high frequency metrics (several k inserts a second). We quickly ran into I/O issues which proper buffering would not simply handle, and we are willing to put time into solving the performance issue.
I suggested to switch to a specialised database for handling time series, but my colleague stayed pretty skeptical. His argument is that the gain "out of the box" is not guaranteed as he knows SQL well and already spent time optimizing the storage, and we in comparison do not have any kind of TSDB experience to properly optimize it.
My intuition is that using a TSDB would be much more efficient even with an out of box configuration but I don't have any data to measure this, and internet benchs such as InfluxDB's are nowhere near trustable. We should run our own, except we can't affoard to loose time in a dead end or a mediocre improvement.
What would be, in my use case but very roughly, the performance gap between relational storage and TSDB, when it comes to single node throughput ?
This question may be bordering on a software recommendation. I just want to point one important thing out: You have an existing code base so switching to another data store is expensive in terms of development costs and time. If you have someone experienced with the current technology, you are probably better off with a good-faith effort to make that technology work.
Whether you switch or not depends on the actual requirements of your application. For instance, if you don't need the data immediately, perhaps writing batches to a file is the most efficient mechanism.
Your infrastructure has ample opportunity for in-place growth -- more memory, more processors, solid-state disk (for example). These might meet your performance needs with a minimal amount of effort.
If you cannot make the solution work (and 10k inserts per second should be quite feasible), then there are numerous solutions. Some NOSQL databases relax some of the strict ACID requirements of traditional RDBMSs, providing faster throughout.
How to convert Microsecond value into CPU cycle calculation, anybody know how to convert this ?
In multicore processors and modern architectures, it's not easy to come with a definition of 'spent' CPU cycles associated to a time interval consumed/elapsed in your program (I wonder SQL procedure/query according to your tags), like in the old pre-superscalar architectures.
I recommend you to find other metrics, procedures or special tools for measuring your performance. I'm not an user of SQL Server, but all professional grade DBMS systems offer profiling tools for tuning DB/application performance.
Read this post and the user comments, it's related to the function QueryPerformanceCounter of Windows for approaching some kind of 'cycles' accounting. The discussion lights up why your cycle-based approach is not correct, from my point of view.
https://blogs.msdn.microsoft.com/oldnewthing/20080908-00/?p=20963/
I want to record Cpu usage ,cpu time ,VM size in notepad per minute for any particular process(not for all.Is there any way to this,because i work as a performance/stress tester and its my duty to take the cpu performance after at particular time and the script takes more time so it is some time inconvenient to me take the all reading
please suggest.
thank u.
Use performance monitor, if the Windows is the system that you are working on. It has all kinds of log options, and will do what you need.
Performance monitor gives you the option of recording the performance data for particular process. Look under 'process'...