Imagine T1(n) and T2(n) are running times of P1 and P2 programs, and
T1(n) ∈ O(f(n))
T2(n) ∈ O(g(n))
What is the amount of T1(n)+T2(n), when P1 is running along side P2?
The Answer is O(max{f(n), g(n)}) but why?
When we think about Big-O notation, we generally think about what the algorithm does as the size of the input n gets really big. A lot of times, we can fall back on some sort of intuition with math. Consider two functions, one that is O(n^2) and one that is O(n). As n gets really large, both algorithms increases without bound. The difference is, the O(n^2) algorithm grows much, MUCH faster than O(n). So much, in fact, that if you combine the algorithms into something that would be O(n^2+n), the factor of n by itself is so small that it can be ignored, and the algorithm is still in the class O(n^2).
That's why when you add together two algorithms, the combined running time is in O(max{f(n), g(n)}). There's always one that 'dominates' the runtime, making the affect of the other negligible.
The Answer is O(max{f(n), g(n)})
This is only correct if the programms run independently of each other. Anyhow, let's assume, this is the case.
In order to answer the why, we need to take a closer look at what the BIG-O-notation represents. Contrary to the way you stated it, it does not represent time but an upperbound on the complexity.
So while running both programms might take more time, the upperbound on the complexity won't increase.
Lets considder an example: P_1 calculates the the product of all pairs of n numbers in a vector, it is implemented using nested loops, and therefore has a complexity of O(n*n). P_2 just prints the numbers in a single loop and therefore has a complexity of O(n).
Now if we run both programms at the same time, the nested loops of P_1 are the most 'complex' part, leaving the combination with a complexity of O(n*n)
Related
I have read many explanations of amortized analysis and how it differs from average-case analysis. However, I have not found a single explanation that showed how, for a particular example for which both kinds of analysis are sensible, the two would give asymptotically different results.
The most wide-spread example of amortized running time analysis shows that appending an element to a dynamic array takes O(1) amortized time (where the running time of the operation is O(n) if the array's length is an exact power of 2, and O(1) otherwise). I believe that, if we consider all array lengths equally likely, then the average-case analysis will give the same O(1) answer.
So, could you please provide an example to show that amortized analysis and average-case analysis may give asymptotically different results?
Consider a dynamic array supporting push and pop from the end. In this example, the array capacity will double when push is called on a full array and halve when pop leaves the array size 1/2 of the capacity. pop on an empty array does nothing.
Note that this is not how dynamic arrays are "supposed" to work. To maintain O(1) amortized complexity, the array capacity should only halve when the size is alpha times the capacity, for alpha < 1/2.
In the bad dynamic array, when considering both operations, neither has O(1) amortized complexity, because alternating between them when the capacity is near 2x the size can produce Ω(n) time complexity for both operations repeatedly.
However, if you consider all sequences of push and pop to be equally likely, both operations have O(1) average time complexity, for two reasons:
First, since the sequences are random, I believe the size of the array will mostly be O(1). This is a random walk on the natural numbers.
Second, the array will be near size a power of 2 only rarely.
This shows an example where amortized complexity is strictly greater than average complexity.
They never have different asymptotically different results. average-case means that weird data might not trigger the average case and might be slower. asymptotic analysis means that even weird data will have the same performance. But on average they'll always have the same complexity.
Where they differ is the worst-case analysis. For algorithms where slowdowns come every few items regardless of their values, then the worst-case and the average-case are the same, and we often call this "asymptotic analysis". For algorithms that can have slowdowns based on the data itself, the worst-case and average-case are different, and we do not call either "asymptotic".
In "Pairing Heaps with Costless Meld", the author gives a priority queue with O(0) time per meld. Obviously, the average time per meld is greater than that.
Consider any data structure with worst-case and best-case inserts and removes taking I and R time. Now use the physicist's argument and give the structure a potential of nR, where n is the number of values in the structure. Each insert increases the potential by R, so the total amortized cost of an insert is I+R. However, each remove decreases the potential by R. Thus, each removal has an amortized cost of R-R=0!
The average cost is R; the amortized cost is 0; these are different.
I know, we should drop the non-dominant terms when calculating time complexity of an algorithm. I am wondering if we should drop them when calculating space complexity. For example, if I have a string of N letters, I'd like to:
construct a list of letters from this string -> Space: O(N);
sort this list -> Worst-case space complexity for Timsort (I use Python): O(N).
In this case, would the entire solution take O(N) + O(N) space or just O(N)?
Thank you.
Welcome to SO!
First of all, I think you do misunderstand complexity: Complexity is defined independently of constant factors. It depends only on the large scale behavior of the data set size N. Thus, O(N) + O(N) is the same complexity as O(N).
Thus, your question might have been:
If I construct a list of letters using an algorithm with O(N) space complexity, followed by a sort algorithm with O(N) space complexity, would the entire solution use twice as much space?
But this question cannot be answered, since a complexity does not give you any measure how much space is actually used.
A well-known example: A brute force sorting algorithm, BubbleSort, with time complexity O(N^2) is faster for small data sets than a very good sorting algorithm, QuickSort, with average time complexity O(Nlog(N)).
EDIT:
It is no contradiction, that one can compute a space complexity, and that it does not say how much space is actually used.
A simple example:
Say, for a certain problem algorithm 1 has linear space complexity O(n), and algorithm 2 space complexity O(n^2).
One could thus assume (but this is wrong) that algorithm 1 would always use less space than algorithm 2.
First, it is clear that for large enough n algorithm 2 will use more space than algorithm 1, because n^2 grows faster than n.
However, consider the case where n is small enough, say n = 1, and algorithm 1 is implemented on a computer that uses storage in doubles (64 bits), whereas algorithm 2 is implemented on a computer that uses bytes (8 bits). Then, obviously, the O(n^2) algorithm uses less space than the O(n) algorithm.
This is a constant doubt I'm having. For example, I have a 2-d array of size n^2 (n being the number of rows and columns). Suppose I want to print all the elements of the 2-d array. When I calculate the time complexity of the algorithm with respect to n it's O(n^2 ). But if I calculated the time with respect to the input size (n^2 ) it's linear. Are both these calculations correct? If so, why do people only use O(n^2 ) everywhere regarding 2-d arrays?
That is not how time complexity works. You cannot do "simple math" like that.
A two-dimensional square array of extent x has n = x*x elements. Printing these n elements takes n operations (or n/m if you print m items at a time), which is O(N). The necessary work increases linearly with the number of elements (which is, incidentially, quadratic in respect of the array extent -- but if you arranged the same number of items in a 4-dimensional array, would it be any different? Obviously, no. That doesn't magically make it O(N^4)).
What you use time complexity for is not stuff like that anyway. What you want time complexity to tell you is an approximate idea of how some particular algorithm may change its behavior if you grow the number of inputs beyond some limit.
So, what you want to know is, if you do XYZ on one million items or on two million items, will it take approximately twice as long, or will it take approximately sixteen times as long, for example.
Time complexity analysis is irrespective of "small details" such as how much time an actual operations takes. Which tends to make the whole thing more and more academic and practically useless in modern architectures because constant factors (such as memory latency or bus latency, cache misses, faults, access times, etc.) play an ever-increasing role as they stay mostly the same over decades while the actual cost-per-step (instruction throughput, ALU power, whatever) goes down steadily with every new computer generation.
In practice, it happens quite often that the dumb, linear, brute force approach is faster than a "better" approach with better time complexity simply because the constant factor dominates everything.
To complete task: find gcd(a,b) for integers a>b>0
Consider an algorithm that checks all of the numbers up to b and keeps track of the max number that divides a and b. It would use the % operator twice per check (for a and b). What would the complexity of this algorithm be?
I have not yet taken any formal CS courses in complexity theory (I will soon) so I am just looking for a quick answer.
The modulo operation is implemented in hardware, and it's pseudo O(1). Strictly speaking, it is not constant, but it depends on the number of bits of a and b. However, even then the number of bits is the same for all input sizes, so we usually ignore this factor.
The worst-case complexity of brute force GCD is just O(n) (also O(a), O(b), or O(min(a,b)); they're all the same), and it happens when when the GCD is either 1, a, or b.
Here's a problem.
I am really confused about the c being equal to 0.5 part. Actually overall I am confused how the logn can become n^(0.5). Couldn't I just let c be equal to 100 which would mean 100 < d which results in a different case being used? What am I missing here?
You of course could set c = 100 so that n^c is a (very, veeery) rough asymptotical upper bound to log(n), but this would give you a horrendous and absolutely useless estimate on your runtime T(n).
What it tells you, is that: every polynomial function n^c grows faster than the logarithm, no matter how small c is, as long as it remains positive. You could take c=0.0000000000001, it would seem to grow ridiculously small in the beginning, but at some point it would become larger than log(n) and diverge to infinity much faster than log(n) does. Therefore, in order to get rid of the n^2 log(n) term and being able to apply the polynomial-only version of the Master theorem, you upper bound the logarithmic term by something that grows slowly enough (but still faster than log(n)). In this example, n^c with c=0.5 is sufficient, but you could also take c=10^{-10000} "just to make sure".
Then you apply the Master theorem, and get a reasonable (and sharp) asymptotic upper bound for your T(n).