I am a college student majoring in Artificial Intelligence. I feel stupid because I just can't understand Time Complexity. And my teacher won't help me out. Im failing and clueless. Can anyone help me to understand Time Complexity? For example what is the Big O of this code? or what is the Time Complexity for this binary tree. I got 4/50 on my recent exam so Im really desperate now
Big-O is basically the amount of time/space (time complexity / space complexity) that a specific algorithm runs in.
It depends on the operations of the algorithm. I will exemplify the time complexity.
E.g. the function:
## l is a list of size=n
def linear(l):
for i in l:
print(i)
iterates over a list of size=n, so the algorithm will will need to get all the n values until accomplished its purpose.
So, it's big-o is O(n) as it runs linearly with the amount of n.
The function:
## l is a list of k lists, each of them has size=n
def quadratic(matrix):
for l in matrix:
for i in l:
print(i)
Here, you have to iterate over k lists, and over n elements in each of them.
So, the algorithm runs in O(k*n).
Related
I have a convex quadratic programming problem:
min x^TPx+c^Tx
Ax \leq b
where P is a positive definite matrix. A is a m * n matrix and m is much greater than n, so the number of constraints is much greater than the number of variables.
My question is: 1. how to analyze the time complexity of this problem. 2. How the time complexity of the convex quadratic programming problem relates to the number of constraints.
I have tried to solve my problem using both cvxopt and mosek, the results of both show that the time complexity seems to be linear to the number of constraints.
However, when I tried to find the literature, I found that all the literatures I found only discussed how the time complexity relates to the number of variables, or assume A is a full rank matrix. I will appreciate it if you can recommend some related references. Thank you.
The "Traveling Salesman Problem" is a problem where a person has to travel between "n" cities - but choose the itinerary such that:
Each city is visited only once
The total distance traveled is minimized
I have heard that if a modern computer were the solve this problem using "brute force" (i.e. an exact solution) - if there are more than 15 cities, the time taken by the computer will exceed a hundred years!
I am interested in understanding "how do we estimate the amount of time it will take for a computer to solve the Traveling Salesman Problem (using "brute force") as the number of cities increase". For instance, from the following reference (https://www.sciencedirect.com/topics/earth-and-planetary-sciences/traveling-salesman-problem):
My Question: Is there some formula we can use to estimate the number of time it will take a computer to solve Traveling Salesman using "brute force"? For example:
N cities = N! paths
Each of these N! paths will require "N" calculations
Thus, N * N calculations would be required for the computer to check all paths and then be certain that the shortest path has been found : If we know the time each calculation takes, perhaps we could estimate the total run time as "time per calculation * N*N! "
But I am not sure if this factors in the time to "store and compare" calculations.
Can someone please explain this?
I have heard that if a modern computer were the solve this problem using "brute force" (i.e. an exact solution) - if there are more than 15 cities, the time taken by the computer will exceed a hundred years!
This is not completely true. While the naive brute-force algorithm runs with a n! complexity. A much better algorithm using dynamic programming runs in O(n^2 2^n). Just to give you an idea, with n=25, n! ≃ 2.4e18 while n^2 2^n ≃ 1e12. The former is too huge to be practicable while the second could be OK although it should take a pretty long time on a PC (one should keep in mind that both algorithm complexities contain an hidden constant variable playing an important role to compute a realistic execution time). I used an optimized dynamic programming solution based on the Held–Karp algorithm to compute the TSP of 20 cities on my machine with a relatively reasonable time (ie. no more than few minutes of computation).
Note that in practice heuristics are used to speed up the computation drastically often at the expense of a sub-optimal solution. Some algorithm can provide a good result in a very short time compared to the previous exact algorithms (polynomial algorithms with a relatively small exponent) with a fixed bound on the quality of the result (for example the distance found cannot be bigger than 2 times the optimal solution). In the end, heuristics can often found very good results in a reasonable time. One simple heuristic is to avoid crossing segments assuming an Euclidean distance is used (AFAIK a solution with crossing segments is always sub-optimal).
My Question: Is there some formula we can use to estimate the number of time it will take a computer to solve Travelling Salesman using "brute force"?
Since the naive algorithm is compute bound and quite simple, you can do such an approximation based on the running-time complexity. But to get a relatively precise approximation of the execution time, you need a calibration since not all processors nor implementations behave the same way. You can assume that the running time is C n! and find the value of C experimentally by measuring the computation time taken by a practical brute-force implementation. Another approach is to theoretically find the value of C based on low-level architectural properties (eg. frequency, number of core used, etc.) of the target processor. The former is much more precise assuming the benchmark is properly done and the number of points is big enough. Moreover, the second method requires a pretty good understanding of the way modern processors work.
Numerically, assuming a running time t ≃ C n!, we can say that ln t ≃ ln(C n!) ≃ ln(C) + ln(n!). Based on the Stirling's approximation, we can say that ln t ≃ ln C + n ln n + O(ln n), so ln C ≃ ln t - n ln n - O(ln n). Thus, ln C ≃ ln t - n ln n - O(ln n) and finally, C ≃ exp(ln t - n ln n) (with an O(n) approximation). That being said, the Stirling's approximation may not be precise enough. Using a binary search to numerically compute the inverse gamma function (which is a generalization of the factorial) should give a much better approximation for C.
Each of these N! paths will require "N" calculations
Well, a slightly optimized brute-force algorithm do not need perform N calculation as the partial path length can be precomputed. The last loops just need to read the precomputed sums from a small array that should be stored in the L1 cache (so it take only no more than few cycle of latency to read/store).
I found from various online sources that the time complexity for DTW is quadratic. On the other hand, I also found that standard kNN has linear time complexity. However, when pairing them together, does kNN-DTW have quadratic or cubic time?
In essence, does the time complexity of kNN solely depend on the metric used? I have not found any clear answer for this.
You need to be careful here. Let's say you have n time series in your 'training' set (let's call it this, even though you are not really training with kNN) of length l. Computing the DTW between a pair of time series has a asymptotic complexity of O(l * m) where m is your maximum warping window. As m <= l also O(l^2) holds. (although there might be more efficient implementations, i don't think they are actually faster in practice in most cases, see here). Classifying a time series using kNN requires you to compute the distance between that time series and all time series in the training set which would mean n comparisons, linear with respect to n.
So your final complexity would be in O(l * m * n) or O(l^2 * n). In words: the complexity is quadratic with respect to time series length and linear with respect to the number of training examples.
I have an algorithm and it is mainly composed of k-NN , followed by a computation involving finding permutations, followed by some for loops. Line by line, my computational complexity is :
O(n) - for k-NN
O(2^k) - for a part that computes singlets, pairs, triplets, etc.
O(k!) - for a part that deals with combinatorics.
O(k*k!) - for the final part.
K here is a parameter that can be chosen by the user, in general it is somewhat small (10-100). n is the number of examples in my dataset, and this can get very large.
What is the overall complexity of my algorithm? Is it simply O(n) ?
As k <= 100, f(k) = O(1) for every function f.
In your case, there is a function f such that the overall time is O(n + f(k)), so it is O(n)
Consider the function F: 2^(3*n) + n^2
Can the function A: 2^(3*n) be used as a Big Theta, Omega or O as a characterisation of F? Why?
I'm revising the concepts of Big Omega, Big Theta and Big O and I came across this example but don't know where to start.
No.
2^(3*n) is the leading term, but unless you're doing something very wrong it's not going to take you that long to compute. Wikipedia has a good list of time complexities of various functions. The most complicated operation you're doing is raising to a power, the complexity of which is discussed in other posts.
Since you're looking at a function of the form g(f(x)) where g(x) = 2^x and f(x) = 3x, the time to compute is going to be O(h) + O(k), where h is the time complexity of g, k is the time complexity of f. Think about it: it should never be more than this, if it were you could just break the operation in two and save time by doing the parts separately. Because h is going to dominate this sum you'll typically leave the k term off.
Yes, all three. In general, you only need to pay attention to the fastest growing term, as slower growing terms will be "swamped" by faster growing terms.
In detail:
Obviously F grows faster than A, so F \in \Omega(A) is a no-brainer. There is a positive multiple of A (namely A itself) that is smaller than F, for all sufficiently large n.
Try plotting F against 2*A. You will find that 2*A quickly gets bigger than F and stays bigger. Thus there is a positive multiple of A (namely 2*A) that is bigger than F for sufficiently large arguments. So by the definition of O, F \in O(A).
Finally, since F \in \Omega(A) and F \in O(A), F \in \Theta(A).