Problem Adding A Column Vector To A Matrix Using ND4J - kotlin

I'm playing with ND4J basics to come up to speed with its linear algebra capabilities.
I'm running on a Macbook Pro using nd4j-api and nd4j-native dependencies version 1.0.0-M2.1, Open JDK version 17, Kotlin 1.7.20, and IntelliJ 2022.2.2 Ultimate Edition.
I'm writing JUnit 5 tests to perform simple operations: add, subtract, multiply, and divide a 2x2 matrix and a scalar. All are successful and pass just fine.
I was successful at adding a 1x2 row vector to the first and second rows of a 2x2 matrix:
#ParameterizedTest
#ValueSource(longs = [0L, 1L])
fun `add a row vector to each row in a matrix`(rowIndex : Long) {
// setup
val a = Nd4j.create(doubleArrayOf(1.0, 2.0, 3.0, 4.0), intArrayOf(2, 2))
val row = Nd4j.create(doubleArrayOf(11.0, 13.0), intArrayOf(2))
// Adds the row vector to all rows
val expected = arrayOf(
Nd4j.create(doubleArrayOf(12.0, 15.0, 3.0, 4.0), intArrayOf(2, 2)),
Nd4j.create(doubleArrayOf(1.0, 2.0, 14.0, 17.0), intArrayOf(2, 2)))
// exercise
a.getRow(rowIndex).addi(row)
// assert
Assertions.assertEquals(expected[rowIndex.toInt()], a)
}
I try to duplicate the trick by adding a 2x1 column vector to the 2x2 matrix:
#Test
fun `add a column vector to the second column of a matrix`() {
// setup
val a = Nd4j.create(doubleArrayOf(1.0, 2.0, 3.0, 4.0), intArrayOf(2, 2))
val col = Nd4j.create(doubleArrayOf(11.0, 13.0), intArrayOf(2, 1))
// Adds the row vector to all rows
val expected = Nd4j.create(doubleArrayOf(1.0, 2.0, 14.0, 17.0), intArrayOf(2, 2))
// exercise
a.getColumn(1).addi(col)
// assert
Assertions.assertEquals(expected, a)
}
I get an error saying that the array shapes don't match:
java.lang.IllegalStateException: Cannot perform in-place operation "addi": result array shape does not match the broadcast operation output shape: [2].addi([2, 1]) != [2].
In-place operations like x.addi(y) can only be performed when x and y have the same shape, or x and y are broadcastable with x.shape() == broadcastShape(x,y)
I have not been successful in figuring out why. Can anyone see where I've gone wrong and suggest a solution?

We have a function for that already. For matrix + column use addiColumnVector.
For views:
Ensure that you have the exact same shape with the reshape. Do that with some vector:
INDArray vec = Nd4j.zeros(5);
vec.getColumn(0).addi(vec.reshape(5,1));

This solution did the trick. Thanks to Adam Gibson for pointing out the need for reshape:
#ParameterizedTest
#ValueSource(longs = [0L, 1L])
fun `add a column vector to each column in a matrix`(colIndex : Long) {
// setup
val a = Nd4j.create(doubleArrayOf(1.0, 2.0, 3.0, 4.0), intArrayOf(2, 2))
val col = Nd4j.create(doubleArrayOf(11.0, 13.0), intArrayOf(2, 1))
val expected = arrayOf(
Nd4j.create(doubleArrayOf(12.0, 2.0, 16.0, 4.0), intArrayOf(2, 2)),
Nd4j.create(doubleArrayOf(1.0, 13.0, 3.0, 17.0), intArrayOf(2, 2)))
// exercise
// Adds the column vector to successive columns
a.getColumn(colIndex).reshape(intArrayOf(2, 1)).addi(col)
// assert
Assertions.assertEquals(expected[colIndex.toInt()], a)
}

Related

Julia Jump : Getting all feasible solutions to mip

I would like to have instead of only the vector of optimal solution to a mip , all the feasible (suboptimal) vectors.
I found some old questions here, but I am not sure how they work.
First of all, is there any new library tool/way to do that automatically ?
I tried this but, it did nothing:
if termination_status(m) == MOI.FEASIBLE_POINT
println(x)
end
optimize!(m);
If not, what's the easiest way?
I thought of scanning the optimal solution till I find the first non -zero decision variable, then constraint this variable to be zero and solving the model again.
for i in 1:active_variables
if value.(z[i])==1
#constraint(m, x[i] == 0)
break
end
end
optimize!(m);
But I see this problem with this method** :
Ιf I constraint x[i] to be zero, in the next step I will want maybe to drop again this constraint? This comes down to whether there can exist two(or more) different solutions in which x[i]==1
JuMP supports returning multiple solutions.
Documentation: https://jump.dev/JuMP.jl/stable/manual/solutions/#Multiple-solutions
The workflow is something like:
using JuMP
model = Model()
#variable(model, x[1:10] >= 0)
# ... other constraints ...
optimize!(model)
if termination_status(model) != OPTIMAL
error("The model was not solved correctly.")
end
an_optimal_solution = value.(x; result = 1)
optimal_objective = objective_value(model; result = 1)
for i in 2:result_count(model)
#assert has_values(model; result = i)
println("Solution $(i) = ", value.(x; result = i))
obj = objective_value(model; result = i)
println("Objective $(i) = ", obj)
if isapprox(obj, optimal_objective; atol = 1e-8)
print("Solution $(i) is also optimal!")
end
end
But you need a solver that supports returning multiple solutions, and to configure the right solver-specific options.
See this blog post: https://jump.dev/tutorials/2021/11/02/tutorial-multi-jdf/
The following is an example of all-solution finder for a boolean problem. Such problems are easier to handle since the solution space is easily enumerated (even though it can still grow exponentially big).
First, let's get the packages and define the sample problem:
using Random, JuMP, HiGHS, MathOptInterface
function example_knapsack()
profit = [5, 3, 2, 7, 4]
weight = [2, 8, 4, 2, 5]
capacity = 10
minprofit = 10
model = Model(HiGHS.Optimizer)
set_silent(model)
#variable(model, x[1:5], Bin)
#objective(model, FEASIBILITY_SENSE, 0)
#constraint(model, weight' * x <= capacity)
#constraint(model, profit' * x >= minprofit)
return model
end
(it is a knapsack problem from the JuMP docs).
Next, we use recursion to explore the tree of all possible solutions. The tree does not go down branches with no solution (so the running time is not always exponential):
function findallsol(model, x)
perm = shuffle(1:length(x))
res = Vector{Float64}[]
_findallsol!(res, model, x, perm, 0)
return res
end
function _findallsol!(res, model, x, perm, depth)
n = length(x)
depth > n && return
optimize!(model)
if termination_status(model) == MathOptInterface.OPTIMAL
if depth == n
push!(res, value.(x))
return
else
idx = perm[depth+1]
v = value(x[idx])
newcon = #constraint(model, x[idx] == v)
_findallsol!(res, model, x, perm, depth + 1)
delete(model, newcon)
newcon = #constraint(model, x[idx] == 1 - v)
_findallsol!(res, model, x, perm, depth + 1)
delete(model, newcon)
end
end
return
end
Now we can:
julia> m = example_knapsack()
A JuMP Model
Maximization problem with:
Variables: 5
...
Names registered in the model: x
julia> res = findallsol(m, m.obj_dict[:x])
5-element Vector{Vector{Float64}}:
[1.0, 0.0, 0.0, 1.0, 1.0]
[0.0, 0.0, 0.0, 1.0, 1.0]
[1.0, 0.0, 1.0, 1.0, 0.0]
[1.0, 0.0, 0.0, 1.0, 0.0]
[0.0, 1.0, 0.0, 1.0, 0.0]
And we get a vector with all the solutions.
If the problem in question is a boolean problem, this method might be used, as is. In case it has non-boolean variables, the recursion will have to split the feasible space in some even fashion. For example, choosing a variable and cutting its domain in half, and recursing to each half with a smaller domain on this variable (to ensure termination).
P.S. This is not the optimal method. This problem has been well studied. Possible terms to search for are 'model counting' (especially in the boolean domain).
(UPDATE: Changed objective to use FEASIBLE)

Scala: how to get the mean and variance and covariance of a matrix?

I am new to scala and I desperately need some guidance on the following problem:
I have a dataframe like the one below (some elements may be NULL)
val dfDouble = Seq(
(1.0, 1.0, 1.0, 3.0),
(1.0, 2.0, 0.0, 0.0),
(1.0, 3.0, 1.0, 1.0),
(1.0, 4.0, 0.0, 2.0)).toDF("m1", "m2", "m3", "m4")
dfDouble.show
+---+---+---+---+
| m1| m2| m3| m4|
+---+---+---+---+
|1.0|1.0|1.0|3.0|
|1.0|2.0|0.0|0.0|
|1.0|3.0|1.0|1.0|
|1.0|4.0|0.0|2.0|
+---+---+---+---+
I need to get the following statistics out of this dataframe:
a vector that contains the mean of each column (some elements might be NULL and I want to calculate the mean using only the non-NULL elements); I would also like to refer to each element of the vector by name for example, vec_mean["m1_mean"] would return the first element
vec_mean: Vector(m1_mean, m2_mean, m3_mean, m4_mean)
a variance-covariance matrix that is (4 x 4), where the diagonals are var(m1), var(m2),..., and the off-diagonals are cov(m1,m2), cov(m1,m3) ... Here, I would also like to only use the non-NULL elements in the variance-covariance calculation
A vector that contains the number of non-null for each column
vec_n: Vector(m1_n, m2_n, m3_n, m4_n)
A vector that contains the standard deviation of each column
vec_stdev: Vector(m1_stde, m2_stde, m3_stde, m4_stde)
In R I would convert everything to a matrix and then the rest is easy. But in scala, I'm unfamiliar with matrices and there are apparently multiple types of matrices, which are confusing (DenseMatrix, IndexedMatrix, etc.)
Edited: apparently it makes a difference if the content of the dataframe is Double or Int. Revised the elements to be double
Used the following command per suggested answer and it worked!
val rdd = dfDouble0.rdd.map {
case a: Row => (0 until a.length).foldRight(Array[Double]())((b, acc) =>
{ val k = a.getAs[Double](b)
if(k == null)
acc.+:(0.0)
else acc.+:(k)}).map(_.toDouble)
}
Yo can work with Spark RowMatrix. It has these kind of operations like computing the co-variance matrix using each row as an observation, mean, varianze, etc... The only thing that you have to know is how to build it from a Dataframe.
It turns out that a Dataframe in Spark contains a schema that represents the type of information that can be stored in it, and it is not only floating point numbers arrays. So the first thing is to transform this DF to a RDD of vectors(dense vector in this case).
Having this DF:
val df = Seq(
(1, 1, 1, 3),
(1, 2, 0, 0),
(1, 3, 1, 1),
(1, 4, 0, 2),
(1, 5, 0, 1),
(2, 1, 1, 3),
(2, 2, 1, 1),
(2, 3, 0, 0)).toDF("m1", "m2", "m3", "m4")
Convert it to a RDD Row[DenseVector] representation. There must be dozens of ways of doing this. One could be:
val rdd = df.rdd.map {
case a: Row =>
(0 until a.length).foldRight(Array[Int]())((b, acc) => {
val k = a.getAs[Int](b)
if(k == null) acc.+:(0) else acc.+:(k)
}).map(_.toDouble)
}
As you can see in your IDE, the inferred type is RDD[Array[Float]. Now convert this to a RDD[DenseVector]. As simple as doing:
val rowsRdd = rdd.map(Vectors.dense(_))
And now you can build your Matrix:
val mat: RowMatrix = new RowMatrix(rowsRdd)
Once you have the matrix, you can easily compute the different metrix per column:
println("Mean: " + mat.computeColumnSummaryStatistics().mean)
println("Variance: " + mat.computeColumnSummaryStatistics().variance)
It gives:
Mean: [1.375,2.625,0.5,1.375]
Variance:
[0.26785714285714285,1.9821428571428572,0.2857142857142857,1.4107142857142858]
you can read more info about the capabilities of Spark and these distributed types in the doc: https://spark.apache.org/docs/latest/mllib-data-types.html#data-types-rdd-based-api
You can also compute the Covariance matrix, doing the SVD, etc...

Running sum of complicated functions using pandas data frame values

In this simplified example I have three lists of the same length, list a, list b, and list c. I want to find the following running summations
from math import exp
a = [1.3, 4.5, 7.8, 9.2, 4.1]
b = [2.1, 1.1, 1.0, 1.0, -2.0]
c = [3.1, 4.0, 5.0, 6.0, 7.0]
# This simple, but SLOW method
sum1 = 0.0
for i in range(0, 3):
sum1 += a[i] ** b[i] + 3.1 * c[i]
sum2 = 0.0
for i in range(2,4):
sum2 += (b[i] / a[i]) * exp(-c[i])
total = sum1 + sum2
print(total) # yields 52.27644
The above code works just fine; however, for examples with MUCH larger lists it runs very slow. If I were to combine the lists in a pandas data frame, is there some built-in and vectorized capability to conduct this same running summations with the data frame? Something like below.
import pandas as pd
df_dict = {'A': [1.3, 4.5, 7.8, 9.2, 4.1],
'B': [2.1, 1.1, 1.0, 1.0, -2.0],
'C': [3.1, 4.0, 5.0, 6.0, 7.0]}
df = pd.DataFrame(df_dict)
# Some version of a running summation here!
I do not think you need a dataframe here, just use numpy's functions :
step1 = np.power(a[:3], b[:3])
step2 = np.multiply(c[:3], 3.1)
sum1 = np.add(step1, step2).sum()
step3 = np.divide(b[2:4], a[2:4])
step4 = np.exp(np.multiply(c[2:4], -1))
sum2 = np.multiply(step3, step4).sum()
result = sum1 + sum2
result
52.27644589942484
This should be significantly faster as the list size grows; plus you can optimize it further.

Math.Net Multiple Regression Is Wrong After The 4th Independent Variables

I am able to generate correct intercept and coefficients for a multiple regression (Math.Net) adding up to three independent variables. However, once a fourth independent variable is added the returned values are nowheres near close.
Using this code:
Dim i As Integer
Dim g(5)() As Double
g(0) = {1.0, 4.0, 3.2}
g(1) = {2.0, 5.0, 4.1}
g(2) = {3.0, 2.0, 2.5}
g(3) = {4.0, 3.0, 1.6}
g(4) = {4.0, 3.0, 1.6}
Dim d As Double() = {3.5, 5.6, 1.2, 15.2, 3.4, 4.2}
Dim p As Double() = MultipleRegression.QR(Of Double)(g, d, intercept:=True)
For i = 0 To UBound(p)
Debug.WriteLine(p(i))
Next
I get:
-2.45972222222223
1.13194444444445
3.11805555555555
-2.38888888888889
These are correct.
However, if I run the same code, but add a 4th independent variable as such:
Dim i As Integer
Dim g(5)() As Double
g(0) = {1.0, 4.0, 3.2, 5.3}
g(1) = {2.0, 5.0, 4.1, 2.4}
g(2) = {3.0, 2.0, 2.5, 3.6}
g(3) = {4.0, 3.0, 1.6, 2.1}
g(4) = {4.0, 3.0, 1.6, 2.1}
g(5) = {4.0, 3.0, 1.6, 2.1}
Dim d As Double() = {3.5, 5.6, 1.2, 15.2, 3.4, 4.2}
Dim p As Double() = MultipleRegression.QR(Of Double)(g, d, intercept:=True)
For i = 0 To UBound(p)
Debug.WriteLine(p(i))
Next
I get:
6.88018203734109E+17
-9.8476516475107E+16
-3.19472310972754E+16
-4.61094057074081E+16
-5.92835216238101E+16
These number are nowhere close to being correct.
If anyone can provide any direction as to what I am doing wrong, I would be very appreciative. TIA
I have not worked out the math details, but looking intuitively at your problem, of the six observations, three (g(3),g(4),g(5)) have identical independent variables, and the corresponding values of the dependent variable have the highest, median, and lowest values. So these observations don't have any real predictive value. In effect, you are trying to estimate 5 values based on three observations. That's not going to work well, and results in instability in the math.
I've changed your data very slightly, and it returns better values. (I use C#). The problem is with the data, not the program.
double[][] g = new double [6][];
g[0] = new double[4] { 1.0, 4.0, 3.2, 5.3};
g[1] = new double[4] { 2.0, 5.0, 4.1, 2.4};
g[2] = new double[4] { 3.0, 2.0, 2.5, 3.6};
g[3] = new double[4] { 4.0, 3.0, 1.6, 2.12};
g[4] = new double[4] { 4.0, 3.0, 1.6, 2.11};
g[5] = new double[4] { 4.0, 3.0, 1.6, 2.1};
double[] d = new double[6] { 3.5, 5.6, 1.2, 15.2, 3.4, 4.2 };
var p = MultipleRegression.QR(g, d, true);
for (int i = 0; i < p.Length; i++) Console.WriteLine(p[i].ToString());
This returns:
-6386.81388888898
913.902777777791
297.597222222225
428.444444444452
550.000000000007

Transforming mLSTM - Run it on multiple GPUs

I'm running an mLSTM (multiplicative LSTM) transform (based on mLSTM by OpenAi (just the transform, it is already trained) but it takes a really long time to transform more than ~100,000 docs.
I want it to run on multiple GPUs. I saw some examples but I have no idea how to implement it on this mLSTM transform code.
The specific part that I want to run on multiple GPUs is:
def transform(xs):
tstart = time.time()
xs = [preprocess(x) for x in xs]
lens = np.asarray([len(x) for x in xs])
sorted_idxs = np.argsort(lens)
unsort_idxs = np.argsort(sorted_idxs)
sorted_xs = [xs[i] for i in sorted_idxs]
maxlen = np.max(lens)
offset = 0
n = len(xs)
smb = np.zeros((2, n, hps.nhidden), dtype=np.float32)
for step in range(0, ceil_round_step(maxlen, nsteps), nsteps):
start = step
end = step+nsteps
xsubseq = [x[start:end] for x in sorted_xs]
ndone = sum([x == b'' for x in xsubseq])
offset += ndone
xsubseq = xsubseq[ndone:]
sorted_xs = sorted_xs[ndone:]
nsubseq = len(xsubseq)
xmb, mmb = batch_pad(xsubseq, nsubseq, nsteps)
for batch in range(0, nsubseq, nbatch):
start = batch
end = batch+nbatch
batch_smb = seq_rep(
xmb[start:end], mmb[start:end],
smb[:, offset+start:offset+end, :])
smb[:, offset+start:offset+end, :] = batch_smb
features = smb[0, unsort_idxs, :]
print('%0.3f seconds to transform %d examples' %
(time.time() - tstart, n))
return features
This is just a snippet of the full code (I don't think it's OK to copy the entire code here).
The part you're referring to is not the place that splits the computation across GPUs, it only transforms the data (on a CPU!) and runs the session.
The correct place is one that defines the computational graph, e.g. mlstm method. There are many ways to split graph, e.g. place LSTM cells on different GPUs, so that the input sequence can be processed in parallel:
def mlstm(inputs, c, h, M, ndim, scope='lstm', wn=False):
[...]
for idx, x in enumerate(inputs):
with tf.device('/gpu:' + str(i % GPU_COUNT)):
m = tf.matmul(x, wmx) * tf.matmul(h, wmh)
z = tf.matmul(x, wx) + tf.matmul(m, wh) + b
[...]
By the way, there is a useful config option in tensorflow log_device_placement that helps to see the execution details in the output. Here's an example:
import tensorflow as tf
# Creates a graph.
with tf.device('/gpu:0'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='b')
c = tf.add(a, b)
# Creates a session with log_device_placement set to True.
with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:
# Prints the following:
# Device mapping:
# /job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: <GPU name>, pci bus id: 0000:01:00.0, compute capability: 6.1
# Add: (Add): /job:localhost/replica:0/task:0/device:GPU:0
# b: (Const): /job:localhost/replica:0/task:0/device:GPU:0
# a: (Const): /job:localhost/replica:0/task:0/device:GPU:0
print(sess.run(c))