Related
Background Information: I solved the N-Queens problem with the C# algorithm below, which returns the total number of solutions given the board of size n x n. It works, but I do not understand why this would be O(n!) time complexity, or if it is a different time complexity. I am also unsure of the space used in the recursion stack (but am aware of the extra space used in the boolean jagged array). I cannot seem to wrap my mind around understanding the time and space complexity of such solutions. Having this understanding would be especially useful during technical interviews, for complexity analysis without the ability to run code.
Preliminary Investigation: I have read several SO posts where the author directly asks the community to provide the time and space complexity of their algorithms. Rather than doing the same and asking for the quick and easy answers, I would like to understand how to calculate the time and space complexity of backtracking algorithms so that I can do so moving forward.
I have also read in numerous locations within and outside of SO that generally, recursive backtracking algorithms are O(n!) time complexity since at each of the n iterations, you look at one less item: n, then n - 1, then n - 2, ... 1. However, I have not found any explanation as to why this is the case. I also have not found any explanation for the space complexity of such algorithms.
Question: Can someone please explain the step-by-step problem-solving approach to identify time and space complexities of recursive backtracking algorithms such as these?
public class Solution {
public int NumWays { get; set; }
public int TotalNQueens(int n) {
if (n <= 0)
{
return 0;
}
NumWays = 0;
bool[][] board = new bool[n][];
for (int i = 0; i < board.Length; i++)
{
board[i] = new bool[n];
}
Solve(n, board, 0);
return NumWays;
}
private void Solve(int n, bool[][] board, int row)
{
if (row == n)
{
// Terminate since we've hit the bottom of the board
NumWays++;
return;
}
for (int col = 0; col < n; col++)
{
if (CanPlaceQueen(board, row, col))
{
board[row][col] = true; // Place queen
Solve(n, board, row + 1);
board[row][col] = false; // Remove queen
}
}
}
private bool CanPlaceQueen(bool[][] board, int row, int col)
{
// We only need to check diagonal-up-left, diagonal-up-right, and straight up.
// this is because we should not have a queen in a later row anywhere, and we should not have a queen in the same row
for (int i = 1; i <= row; i++)
{
if (row - i >= 0 && board[row - i][col]) return false;
if (col - i >= 0 && board[row - i][col - i]) return false;
if (col + i < board[0].Length && board[row - i][col + i]) return false;
}
return true;
}
}
First of all, it's definitely not true that recursive backtracking algorithms are all in O(n!): of course it depends on the algorithm, and it could well be worse. Having said that, the general approach is to write down a recurrence relation for the time complexity T(n), and then try to solve it or at least characterize its asymptotic behaviour.
Step 1: Make the question precise
Are we interested in the worst-case, best-case or average-case? What are the input parameters?
In this example, let us assume we want to analyze the worst-case behaviour, and the relevant input parameter is n in the Solve method.
In recursive algorithms, it is useful (though not always possible) to find a parameter that starts off with the value of the input parameter and then decreases with every recursive call until it reaches the base case.
In this example, we can define k = n - row. So with every recursive call, k is decremented starting from n down to 0.
Step 2: Annotate and strip down the code
No we look at the code, strip it down to just the relevant bits and annotate it with complexities.
We can boil your example down to the following:
private void Solve(int n, bool[][] board, int row)
{
if (row == n) // base case
{
[...] // O(1)
return;
}
for (...) // loop n times
{
if (CanPlaceQueen(board, row, col)) // O(k)
{
[...] // O(1)
Solve(n, board, row + 1); // recurse on k - 1 = n - (row + 1)
[...] // O(1)
}
}
}
Step 3: Write down the recurrence relation
The recurrence relation for this example can be read off directly from the code:
T(0) = 1 // base case
T(k) = k * // loop n times
(O(k) + // if (CanPlaceQueen(...))
T(k-1)) // Solve(n, board, row + 1)
= k T(k-1) + O(k)
Step 4: Solve the recurrence relation
For this step, it is useful to know a few general forms of recurrence relations and their solutions. The relation above is of the general form
T(n) = n T(n-1) + f(n)
which has the exact solution
T(n) = n!(T(0) + Sum { f(i)/i!, for i = 1..n })
which we can easily prove by induction:
T(n) = n T(n-1) + f(n) // by def.
= n((n-1)!(T(0) + Sum { f(i)/i!, for i = 1..n-1 })) + f(n) // by ind. hypo.
= n!(T(0) + Sum { f(i)/i!, for i = 1..n-1 }) + f(n)/n!)
= n!(T(0) + Sum { f(i)/i!, for i = 1..n }) // qed
Now, we don't need the exact solution; we just need the asymptotic behaviour when n approaches infinity.
So let's look at the infinite series
Sum { f(i)/i!, for i = 1..infinity }
In our case, f(n) = O(n), but let's look at the more general case where f(n) is an arbitary polynomial in n (because it will turn out that it really doesn't matter). It is easy to see that the series converges, using the ratio test:
L = lim { | (f(n+1)/(n+1)!) / (f(n)/n!) |, for n -> infinity }
= lim { | f(n+1) / (f(n)(n+1)) |, for n -> infinity }
= 0 // if f is a polynomial
< 1, and hence the series converges
Therefore, for n -> infinity,
T(n) -> n!(T(0) + Sum { f(i)/i!, for i = 1..infinity })
= T(0) n!, if f is a polynomial
Step 5: The result
Since the limit of T(n) is T(0) n!, we can write
T(n) ∈ Θ(n!)
which is a tight bound on the worst-case complexity of your algorithm.
In addition, we've proven that it doesn't matter how much work you do within the for-loop in adddition to the recursive calls, as long as it's polynomial, the complexity stays Θ(n!) (for this form of recurrence relations). (In bold because there are lots of SO answers that get this wrong.)
For a similar analysis with a different form of recurrence relation, see here.
Update
I made a mistake in the annotation of the code (I'll leave it because it is still instructive). Actually, both the loop and the work done within the loop do not depend on k = n - row but on the initial value n (let's call it n0 to make it clear).
So the recurrence relation becomes
T(k) = n0 T(k-1) + n0
for which the exact solution is
T(k) = n0^k (T(0) + Sum { n0^(1-i), for i = 1..k })
But since initially n0 = k, we have
T(k) = k^k (T(0) + Sum { n0^(1-i), for i = 1..k })
∈ Θ(k^k)
which is a bit worse than Θ(k!).
Update: I completely overlooked the complexity added by arr.sort() method. So in Kotlin for array of Int, It compiles to use java.util.DualPivotQuicksort see this which in turn has complexity of O(n^2). see this. Other than that, this is also a valid approach.
I know It can be solved by keeping multiple arrays or using collections (which is what I ended up submitting), I want to know what I missed in the following approach
fun migratoryBirds(arr: Array<Int>): Int {
var maxCount = 0
var maxType = 0
var count = 0
var type = 0
arr.sort()
println(arr.joinToString(" "))
for (value in arr){
if (type != value){
if (count > maxCount){
maxCount = count
maxType = type
}
// new count values
type = value
count = 1
} else {
count++
}
}
return maxType
}
This code passes every scenario except for Test case 2 which has 73966 items for array. On my local machine, that array of 73k+ elements was causing timeout but I did test for array up-to 20k+ randomly generated value 1..5 and every time it succeeded. But I couldn't manage to pass Test case 2 with this approach. So even though I ended up submitting an answer with collection stream approach, I would really like to know what could I be missing in above logic.
I am running array loop only once Complexity should be O(n), So that could not be reason for failing. I am pre-sorting array in ascending order, and I am checking for > not >=, therefore, If two types end up having same count, It will still return the lower of the two types. And this approach is working correctly even for array of 20k+ elements ( I am getting timeout for anything above 25k elements).
The reason it is failing is this line
arr.sort()
Sorting an array takes O(n logn) time. However using something like a hash map this can be solved in O(n) time.
Here is a quick python solution I made to give you the general idea
# Complete the migratoryBirds function below.
def migratoryBirds(arr):
ans = -1
count = -1
dic = {}
for x in arr:
if x in dic:
dic[x] += 1
else:
dic[x] = 1
if dic[x] > count or dic[x] == count and x < ans:
ans = x
count = dic[x]
return ans
I'm trying to find the least common multiple of an array of integers, e.g. if there are 2 numbers given (7, 3) then my task is to find the LCM of the numbers 3 through 7 (3,4,5,6,7 in that case).
My solution would be to add the maximum number to a new variable (var common) until the remainders of all of the numbers in the array (common % numBetween[i]) equal 0. There are more efficient ways of doing this, for example applying the Euclidean Algorithm, but I wanted to solve this my way.
The code:
function smallestCommons(arr) {
var numBetween = [];
var max = Math.max.apply(Math, arr);
var min = Math.min.apply(Math, arr);
while (max - min !== -1) {
numBetween.push(min);
min += 1;
} //this loop creates the array of integers, 1 through 13 in this case
var common = max;
var modulus = [1]; //I start with 1, so that the first loop could begin
var modSum = modulus.reduce(function (a, b) {
return a + b;
}, 0);
while (modSum !== 0) {
modulus = [];
for (var i = 0; i < numBetween.length; i++) {
modulus.push(common % numBetween[i]);
}
if (modSum !== 0) {
common += max;
break; //without this, the loop is infinite
}
}
return common;
}
smallestCommons([1,13]);
Now, the loop is either infinite (without break in the if statement) so I guess the modSum never equals 0, because the modulus variable always contains integers other than 0. I wanted to solve this by "resetting" the modulus to an empty array right after the loop starts, with
modulus = [];
and if I include the break, the loop stops after 1 iteration (common = 26). I can't quite grasp why my code isn't working. All comments are appreciated.
Thanks in advance!
I may be false, but do you actually never change modSum within the while-loop? If so, this is your problem. You wanted to do this by using the function .reduce(), but this does not bind the given function, so you have to call the function each time again in the loop.
I am taking an introduction to Java programing class and I have an array list where I need to exclude the first element from my for loop that finds an average. The first element in the array list is a weight for the average (which is why it needs to be excluded). I also need to drop the lowest value from the remainder of the array list hence my second for loop. I have tried to create a copy of the list and also tried to create a sub list but I cannot get it to work.
public static double Avgerage(ArrayList<Double> inputValues) {
double avg;
double sum = 0;
double weightValue = inputValues.get(0);
double lowest = inputValues.get(0);
for (int i = 1; i > inputValues.size(); i++) {
if (inputValues.get(i) < lowest) {
lowest = inputValues.get(i);
}
}
for (int i = 0; i < inputValues.size(); i++) {
sum = sum + inputValues.get(i);
}
double average = (sum - lowest) / (inputValues.size() - 1);
avg = average * weightValue;
return avg;
}
To start with good programming practice, you should work with interfaces rather than classes, where possible. The appropriate interface here is List<Double>, and when you create it in your class, you should use
List<Double> nameOfList = new ArrayList<Double>();
What we're doing is creating an object which has the behaviour of a List, with the underlying implementation of an ArrayList (more info here.
With regards to the question, you don't appear to be excluding the first element, as you said you wished to - both for loops iterate through all values in the list. Remember to treat the ArrayList like an array - accessing an element does not modify it, like it might in a Queue.
I have edited your code below to demonstrate this, and have also included some other optimisations and corrected the sign error on line 7:
public static double average(List<Double> inputValues) {
double sum = 0;
//Exclude the first element, as it contains the weight
double lowest = inputValues.get(1);
for (int i = 2; i < inputValues.size(); i++) {
lowest = Math.min(inputValues.get(i), lowest);
}
for (int i = 1; i < inputValues.size(); i++) {
sum += inputValues.get(i);
}
double average = (sum - lowest) / (inputValues.size() - 1);
//Scale by the weight
avg *= inputValues.get(0);
return avg;
}
Note: The convention in java is to use camelCase for method names, I have adjusted accordingly.
Also, I don't know your requirements, but optimally, you should be providing logical parameters. If possible do the following before calling the function:
int weight = inputValues.get(0);
inputValues.remove(0);
//And then you would call like this, and update your method signature to match
average(inputValues, weight);
I don't do this inside the method, as the context implies that we would not be modifying values.
I'm looking for a way to generate combinations of objects ordered by a single attribute. I don't think lexicographical order is what I'm looking for... I'll try to give an example. Let's say I have a list of objects A,B,C,D with the attribute values I want to order by being 3,3,2,1. This gives A3, B3, C2, D1 objects. Now I want to generate combinations of 2 objects, but they need to be ordered in a descending way:
A3 B3
A3 C2
B3 C2
A3 D1
B3 D1
C2 D1
Generating all combinations and sorting them is not acceptable because the real world scenario involves large sets and millions of combinations. (set of 40, order of 8), and I need only combinations above the certain threshold.
Actually I need count of combinations above a threshold grouped by a sum of a given attribute, but I think it is far more difficult to do - so I'd settle for developing all combinations above a threshold and counting them. If that's possible at all.
EDIT - My original question wasn't very precise... I don't actually need these combinations ordered, just thought it would help to isolate combinations above a threshold. To be more precise, in the above example, giving a threshold of 5, I'm looking for an information that the given set produces 1 combination with a sum of 6 ( A3 B3 ) and 2 with a sum of 5 ( A3 C2, B3 C2). I don't actually need the combinations themselves.
I was looking into subset-sum problem, but if I understood correctly given dynamic solution it will only give you information is there a given sum or no, not count of the sums.
Thanks
Actually, I think you do want lexicographic order, but descending rather than ascending. In addition:
It's not clear to me from your description that A, B, ... D play any role in your answer (except possibly as the container for the values).
I think your question example is simply "For each integer at least 5, up to the maximum possible total of two values, how many distinct pairs from the set {3, 3, 2, 1} have sums of that integer?"
The interesting part is the early bailout, once no possible solution can be reached (remaining achievable sums are too small).
I'll post sample code later.
Here's the sample code I promised, with a few remarks following:
public class Combos {
/* permanent state for instance */
private int values[];
private int length;
/* transient state during single "count" computation */
private int n;
private int limit;
private Tally<Integer> tally;
private int best[][]; // used for early-bail-out
private void initializeForCount(int n, int limit) {
this.n = n;
this.limit = limit;
best = new int[n+1][length+1];
for (int i = 1; i <= n; ++i) {
for (int j = 0; j <= length - i; ++j) {
best[i][j] = values[j] + best[i-1][j+1];
}
}
}
private void countAt(int left, int start, int sum) {
if (left == 0) {
tally.inc(sum);
} else {
for (
int i = start;
i <= length - left
&& limit <= sum + best[left][i]; // bail-out-check
++i
) {
countAt(left - 1, i + 1, sum + values[i]);
}
}
}
public Tally<Integer> count(int n, int limit) {
tally = new Tally<Integer>();
if (n <= length) {
initializeForCount(n, limit);
countAt(n, 0, 0);
}
return tally;
}
public Combos(int[] values) {
this.values = values;
this.length = values.length;
}
}
Preface remarks:
This uses a little helper class called Tally, that just isolates the tabulation (including initialization for never-before-seen keys). I'll put it at the end.
To keep this concise, I've taken some shortcuts that aren't good practice for "real" code:
This doesn't check for a null value array, etc.
I assume that the value array is already sorted into descending order, required for the early-bail-out technique. (Good production code would include the sorting.)
I put transient data into instance variables instead of passing them as arguments among the private methods that support count. That makes this class non-thread-safe.
Explanation:
An instance of Combos is created with the (descending ordered) array of integers to combine. The value array is set up once per instance, but multiple calls to count can be made with varying population sizes and limits.
The count method triggers a (mostly) standard recursive traversal of unique combinations of n integers from values. The limit argument gives the lower bound on sums of interest.
The countAt method examines combinations of integers from values. The left argument is how many integers remain to make up n integers in a sum, start is the position in values from which to search, and sum is the partial sum.
The early-bail-out mechanism is based on computing best, a two-dimensional array that specifies the "best" sum reachable from a given state. The value in best[n][p] is the largest sum of n values beginning in position p of the original values.
The recursion of countAt bottoms out when the correct population has been accumulated; this adds the current sum (of n values) to the tally. If countAt has not bottomed out, it sweeps the values from the start-ing position to increase the current partial sum, as long as:
enough positions remain in values to achieve the specified population, and
the best (largest) subtotal remaining is big enough to make the limit.
A sample run with your question's data:
int[] values = {3, 3, 2, 1};
Combos mine = new Combos(values);
Tally<Integer> tally = mine.count(2, 5);
for (int i = 5; i < 9; ++i) {
int n = tally.get(i);
if (0 < n) {
System.out.println("found " + tally.get(i) + " sums of " + i);
}
}
produces the results you specified:
found 2 sums of 5
found 1 sums of 6
Here's the Tally code:
public static class Tally<T> {
private Map<T,Integer> tally = new HashMap<T,Integer>();
public Tally() {/* nothing */}
public void inc(T key) {
Integer value = tally.get(key);
if (value == null) {
value = Integer.valueOf(0);
}
tally.put(key, (value + 1));
}
public int get(T key) {
Integer result = tally.get(key);
return result == null ? 0 : result;
}
public Collection<T> keys() {
return tally.keySet();
}
}
I have written a class to handle common functions for working with the binomial coefficient, which is the type of problem that your problem falls under. It performs the following tasks:
Outputs all the K-indexes in a nice format for any N choose K to a file. The K-indexes can be substituted with more descriptive strings or letters. This method makes solving this type of problem quite trivial.
Converts the K-indexes to the proper index of an entry in the sorted binomial coefficient table. This technique is much faster than older published techniques that rely on iteration. It does this by using a mathematical property inherent in Pascal's Triangle. My paper talks about this. I believe I am the first to discover and publish this technique, but I could be wrong.
Converts the index in a sorted binomial coefficient table to the corresponding K-indexes.
Uses Mark Dominus method to calculate the binomial coefficient, which is much less likely to overflow and works with larger numbers.
The class is written in .NET C# and provides a way to manage the objects related to the problem (if any) by using a generic list. The constructor of this class takes a bool value called InitTable that when true will create a generic list to hold the objects to be managed. If this value is false, then it will not create the table. The table does not need to be created in order to perform the 4 above methods. Accessor methods are provided to access the table.
There is an associated test class which shows how to use the class and its methods. It has been extensively tested with 2 cases and there are no known bugs.
To read about this class and download the code, see Tablizing The Binomial Coeffieicent.
Check out this question in stackoverflow: Algorithm to return all combinations
I also just used a the java code below to generate all permutations, but it could easily be used to generate unique combination's given an index.
public static <E> E[] permutation(E[] s, int num) {//s is the input elements array and num is the number which represents the permutation
int factorial = 1;
for(int i = 2; i < s.length; i++)
factorial *= i;//calculates the factorial of (s.length - 1)
if (num/s.length >= factorial)// Optional. if the number is not in the range of [0, s.length! - 1]
return null;
for(int i = 0; i < s.length - 1; i++){//go over the array
int tempi = (num / factorial) % (s.length - i);//calculates the next cell from the cells left (the cells in the range [i, s.length - 1])
E temp = s[i + tempi];//Temporarily saves the value of the cell needed to add to the permutation this time
for(int j = i + tempi; j > i; j--)//shift all elements to "cover" the "missing" cell
s[j] = s[j-1];
s[i] = temp;//put the chosen cell in the correct spot
factorial /= (s.length - (i + 1));//updates the factorial
}
return s;
}
I am extremely sorry (after all those clarifications in the comments) to say that I could not find an efficient solution to this problem. I tried for the past hour with no results.
The reason (I think) is that this problem is very similar to problems like the traveling salesman problem. Until unless you try all the combinations, there is no way to know which attributes will add upto the threshold.
There seems to be no clever trick that can solve this class of problems.
Still there are many optimizations that you can do to the actual code.
Try sorting the data according to the attributes. You may be able to avoid processing some values from the list when you find that a higher value cannot satisfy the threshold (so all lower values can be eliminated).
If you're using C# there is a fairly good generics library here. Note though that the generation of some permutations is not in lexicographic order
Here's a recursive approach to count the number of these subsets: We define a function count(minIndex,numElements,minSum) that returns the number of subsets of size numElements whose sum is at least minSum, containing elements with indices minIndex or greater.
As in the problem statement, we sort our elements in descending order, e.g. [3,3,2,1], and call the first index zero, and the total number of elements N. We assume all elements are nonnegative. To find all 2-subsets whose sum is at least 5, we call count(0,2,5).
Sample Code (Java):
int count(int minIndex, int numElements, int minSum)
{
int total = 0;
if (numElements == 1)
{
// just count number of elements >= minSum
for (int i = minIndex; i <= N-1; i++)
if (a[i] >= minSum) total++; else break;
}
else
{
if (minSum <= 0)
{
// any subset will do (n-choose-k of them)
if (numElements <= (N-minIndex))
total = nchoosek(N-minIndex, numElements);
}
else
{
// add element a[i] to the set, and then consider the count
// for all elements to its right
for (int i = minIndex; i <= (N-numElements); i++)
total += count(i+1, numElements-1, minSum-a[i]);
}
}
return total;
}
Btw, I've run the above with an array of 40 elements, and size-8 subsets and consistently got back results in less than a second.