I am trying to make a dynamic number of drop-down menus in a plot, to plot a various number of curves.
I have previously requested help to plot this data, and it worked well.
First thing
Needs["PlotLegends`"]
Here is a example of data (not actual numbers, as they are waaay too long).
data={{year, H, He, Li, C, O, Si, S},
{0, .5, .1, .01, 0.01, 0.01, 0.001, 0.001},
{100, .45, .1, .01, 0.01, 0.01, 0.001, 0.001},
{200, .40, .1, .01, 0.01, 0.01, 0.001, 0.001},
{300, .35, .1, .01, 0.01, 0.01, 0.001, 0.001}}
The compounds variable is the number of compounds+1
compounds=8
For now, my code is this one
Manipulate[
ListLogLogPlot[
{data[[All, {1, i}]],
data[[All, {1, j}]],
data[[All, {1, k}]]},
PlotLegend -> {data[[1, i]],
data[[1, j]],
data[[1, k]]}
],
{{i, 2, "Compound 1"},Thread[Range[2, compounds] -> Drop[data[[1]], 1]]},
{{j, 3, "Compound 2"},Thread[Range[2, compounds] -> Drop[data[[1]], 1]]},
{{k, 4, "Compound 2"},Thread[Range[2, compounds] -> Drop[data[[1]], 1]]},
ContinuousAction -> False
]
As you can see, I can easily add a compound by duplicating each of the 3 lines (data, legend and menu descriptor), but it's lame and inefficient. Plotting a set takes about 20 seconds, so it's about 1 minute here (and I use a pretty efficient cluster).
Is there a solution to add a little menu or field where I can add the number of compounds to plot, so the right number of menus will display? I don't need more than 7 plots, but efficiency...
The numbers 2, 4, 16 are the default values to plot. I can make a list with the default values (2, 14, 16, and some others I may pick), or they could all be set to 2.
Thanks
You could do something like this
Manipulate[
ListLogLogPlot[data[[All, {1, #}]] & /# i],
{{n, 3, "# compounds"}, Range[7],
Dynamic[If[Length[i] != n, i = PadRight[{2, 4, 16}, n, 2]];
PopupMenu[#, Range[7]]] &},
{{i, {2, 4, 16}}, ControlType -> None},
Dynamic[Column[
Labeled[PopupMenu[Dynamic[i[[#]]],
Thread[Range[2, compounds] -> Drop[data[[1]], 1]]],
Row[{"Compound ", #}], Left] & /# Range[n]]
]
]
Without PlotLegend, this runs quite fast for a random data set of about 1000x1000 elements. If I include the PlotLegend option in ListLogLogPlot, it slows down quite a lot so that might be the reason why your code was so slow.
I thought I'd add a DM version. If you're like me you may find that easier than using manipulate. It is essentially a DM version of Heike's answer.
DynamicModule[{data,compounds,n=1,c={2},labels},
data=yourData;
compounds=Length[data[[1]]];
labels=Rule###Transpose[{Range[7],data[[1,2;;]]}];
Column[{
Dynamic[
Grid[
Join[
{{"no. of compounds",PopupMenu[Dynamic[n],Range[7]]}},
Table[
With[{i=i},
c=PadRight[c,n,2];
{"compound"<>ToString[i], PopupMenu[Dynamic[c[[i]]],labels]}
],
{i,n}
]
],
Alignment->{{Right,Left},Center}
],
TrackedSymbols:>{n}
],
Dynamic#ListLogLogPlot[data[[All,{1,#}]]&/#c]
}]
]
I've used Grid because it allows you to easily keep all the controllers and their labels aligned. PadRight[c,n,2] allows you to keep current settings if you change the value of n. I'd avoid plot legends and always make your own.
How about something like:
Manipulate[
Manipulate[ ListLogLogPlot[Table[Subscript[x, n], {n, 1, numCompounds}]],
Evaluate#Apply[Sequence,Table[{{Subscript[x, n], n + 1, "Compound " <> ToString#n},
Thread[Range[2, compounds] -> Drop[data[[1]], 1]]}, {n, 1,
numCompounds}]], ContinuousAction -> False],
{{numCompounds, 3}, 1, compounds - 1, 1}]
Related
I'm trying to figure out how to do shifting on a tensor that has b (batch size), d (depth), h (hight) and w (width) represented as following:
b, d, h, w = tensor.size()
So, I need to find the subtract between the shifted tensor and the tensor itself.
I'm thinking of using torch.narrow or torch.concat to do it for each side (shift the right, left, up then down side) and at each time I subtract from the same tensor side (tensor itself side), then at the end I will add/sum the differences/subtractions of each side (so I will have the final subtraction between the shifted and the tensor itself.
I'm new to PyTorch, it's easy to understand but struggling to implemented and maybe there is a simpler way (directly do the subtraction rather than working on each side and so on .....)
Any help on that please?
Basically, you can split the tensor first, and then cat them in reverse order. I write a function to implement your thoughts. The shift should be a non-negative number and less than or equal to the size of dim.
def tensor_shift(t, dim, shift):
"""
t (tensor): tensor to be shifted.
dim (int): the dimension apply shift.
shift (int): shift distance.
"""
assert 0 <= shift <= t.size(dim), "shift distance should be smaller than or equal to the dim length."
overflow = t.index_select(dim, torch.arange(t.size(dim)-shift, t.size(dim)))
remain = t.index_select(dim, torch.arange(t.size(dim)-shift))
return torch.cat((overflow, remain),dim=dim)
Here are some test results.
a = torch.arange(1,13).view(-1,3)
a
#tensor([[ 1, 2, 3],
# [ 4, 5, 6],
# [ 7, 8, 9],
# [10, 11, 12]])
shift(a, 0, 1) # shift 1 unit along dim=0
#tensor([[10, 11, 12],
# [ 1, 2, 3],
# [ 4, 5, 6],
# [ 7, 8, 9]])
b = torch.arange(1,13).view(-1,2,3)
b
#tensor([[[ 1, 2, 3],
# [ 4, 5, 6]],
#
# [[ 7, 8, 9],
# [10, 11, 12]]])
shift(b, 1, 1) # shift 1 unit along dim=1
#tensor([[[ 4, 5, 6],
# [ 1, 2, 3]],
#
# [[10, 11, 12],
# [ 7, 8, 9]]])
I need to multiply an array (NIR) with a scalar (f) but leaving some values that meet a certain condition intact.
I tried the following:
NIR_f = np.multiply(NIR,f,where=NIR!=-28672.0)
To check I made:
i,j=1119,753
NIR[i][j],NIR_f[i][j]
and I got this:
(-28672.0, 10058.0)
It is assumed that both results should be the same! In that position the condition is not met, therefore the value should remain intact.
Am I using the "where" option wrongly?
Without your array, or a smaller substitute, I can't exactly replicate your problem. But there are potentially 2 issues
float testing is not exact, so it might be matching one -28672.0, and not another.
the remain intact assumption is tricky. leave the value in the output alone, but what was it originally, 0's or NIR values.
Using an integer array to avoid the float issue:
In [20]: arr = np.arange(12).reshape(3,4)
In [21]: arr
Out[21]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
In [22]: np.multiply(arr, 10, where=arr!=10)
Out[22]:
array([[ 0, 10, 20, 30],
[ 40, 50, 60, 70],
[ 80, 90, 481036337249, 110]])
In [24]: np.multiply(arr, 10, where=arr!=10)
Out[24]:
array([[ 0, 10, 20, 30],
[ 40, 50, 60, 70],
[ 80, 90, 0, 110]])
arr[2,2] is random. In effect it started with a np.empty array of the right shape and dtype, and filled all values but that one with the multiplication. To use where correctly we need to specify an out parameter as well.
In [25]: out = np.full(arr.shape,-1)
In [26]: out
Out[26]:
array([[-1, -1, -1, -1],
[-1, -1, -1, -1],
[-1, -1, -1, -1]])
In [27]: np.multiply(arr, 10, where=arr!=10, out=out)
Out[27]:
array([[ 0, 10, 20, 30],
[ 40, 50, 60, 70],
[ 80, 90, -1, 110]])
The issue of inexact floats comes up often enough that I won't try to illustrate that.
I have two piece codes. The first one is:
A = np.arange(3*4*3).reshape(3, 4, 3)
P = np.arange(1, 4)
A[:, 1:, :] = np.einsum('j, ijk->ijk', P, A[:, 1:, :])
and the result A is :
array([[[ 0, 1, 2],
[ 6, 8, 10],
[ 18, 21, 24],
[ 36, 40, 44]],
[[ 12, 13, 14],
[ 30, 32, 34],
[ 54, 57, 60],
[ 84, 88, 92]],
[[ 24, 25, 26],
[ 54, 56, 58],
[ 90, 93, 96],
[132, 136, 140]]])
The second one is:
A = np.arange(3*4*3).reshape(3, 4, 3)
P = np.arange(1, 4)
np.einsum('j, ijk->ijk', P, A[:, 1:, :], out=A[:,1:,:])
and the result A is :
array([[[ 0, 1, 2],
[ 0, 0, 0],
[ 0, 0, 0],
[ 0, 0, 0]],
[[12, 13, 14],
[ 0, 0, 0],
[ 0, 0, 0],
[ 0, 0, 0]],
[[24, 25, 26],
[ 0, 0, 0],
[ 0, 0, 0],
[ 0, 0, 0]]])
So the result is different. Here I want to use out to save memory. Is it a bug in numpy.einsum? Or I missed something?
By the way, my numpy version is 1.13.3.
I haven't used this new out parameter before, but have worked with einsum in the past, and have a general idea of how it works (or at least used to).
It looks to me like it initializes the out array to zero before the start of iteration. That would account for all the 0s in the A[:,1:,:] block. If instead I initial separate out array, the desired values are inserted
In [471]: B = np.ones((3,4,3),int)
In [472]: np.einsum('j, ijk->ijk', P, A[:, 1:, :], out=B[:,1:,:])
Out[472]:
array([[[ 3, 4, 5],
[ 12, 14, 16],
[ 27, 30, 33]],
[[ 15, 16, 17],
[ 36, 38, 40],
[ 63, 66, 69]],
[[ 27, 28, 29],
[ 60, 62, 64],
[ 99, 102, 105]]])
In [473]: B
Out[473]:
array([[[ 1, 1, 1],
[ 3, 4, 5],
[ 12, 14, 16],
[ 27, 30, 33]],
[[ 1, 1, 1],
[ 15, 16, 17],
[ 36, 38, 40],
[ 63, 66, 69]],
[[ 1, 1, 1],
[ 27, 28, 29],
[ 60, 62, 64],
[ 99, 102, 105]]])
The Python portion of einsum doesn't tell me much, except how it decides to pass the out array to the c portion, (as one of the list of tmp_operands):
c_einsum(einsum_str, *tmp_operands, **einsum_kwargs)
I know that it sets up a c-api equivalent of np.nditer, using the str to define the axes and iterations.
It iterates something like this section in the iteration tutorial:
https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.nditer.html#reduction-iteration
Notice in particular the it.reset() step. That sets the out buffer to 0 prior to iterating. It then iterates over the elements of input arrays and the output array, writing the calculation values to the output element. Since it is doing a sum of products (e.g. out[:] += ...), it has to start with a clean slate.
I'm guessing a bit as to what is actually going on, but it seems logical to me that it should zero out the output buffer to start with. If that array is the same as one of the inputs, that will end up messing with the calculation.
So I don't think this approach will work and save you memory. It needs a clean buffer to accumulate the results in. Once that's done it, or you, can write the values back into A. But given the nature of a dot like product, you can't use the same array for input and for output.
In [476]: A[:,1:,:] = np.einsum('j, ijk->ijk', P, A[:, 1:, :])
In [477]: A
Out[477]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 12, 14, 16],
[ 27, 30, 33]],
....)
In the C source code for einsum, there is a section that will take the array specified by out and do some zero-setting.
But in the Python source code for example, there are execution paths that call the tensordot function before ever descending the arguments to call c_einsum.
This means that some operations might be pre-computed (thus modifying your array A on some contraction passes) with tensordot, before any sub-array is ever set to zero by the zero-setter inside the C code for einsum.
Another way to put it is: on each pass at doing the next contraction operations, NumPy has many choices available to it. To use tensordot directly without getting into the C-level einsum code just yet? Or to prepare the arguments and pass to the C level (which will involve over-writing some sub-view of the output array with all zeros)? Or to re-order the operations and repeat the check?
Depending on the order it chooses for these optimizations, you can end up with unexpected all-zeros sub-arrays.
Best bet is to not try to be this clever and use the same array for the output. You say it is because you want to save memory. Yes, in some special cases an einsum operation might be do-able in-place. But it does not currently detect if this is the case and attempt to avoid the zero-setting.
And in a huge number of cases, over-writing into one of the input arrays during the middle of the overall operation would cause many problems, much like trying to append to a list you are directly looping over, etc.
I have 3 states (i.e. state is a vector, vector length is 2 in this example), I have probability distribution (such as 10% state 1, 60% state 2, 30% state 3). I would like to derive a new state that is the sum of probability * state. Of cause, I will need to consider batch too.
1> calculated probability distribution, I have a batch of 4, and 3 possibilities and the distribution is defined below.
dist = tf.constant([[0.1, 0.6, 0.3], [0.2, 0.4, 0.4], [0.3, 0.5, 0.2], [0.3, 0.6, 0.1]])
2> the 3 possible states (and batch of 4). This is a tensor of shape [4, 3, 2] or [batch, 3 possible state, state values]
val = tf.constant([[[10.0, 5.0],[10, 5],[10,5]],[[8, 2],[8, 2],[8, 2]],[[7, 3],[9, 1],[6, 4]],[[1, 2],[3, 4],[5, 6]]])
I would like to get a tensor of [4, 2] or [batch, state values]. In this case, the value should be
[
[10*0.1 + 10*0.6 + 10*0.3, 5*0.1 + 5*0.6 + 5*0.3],
[8*0.2 + 8*0.4 + 8*0.4, 2*0.2 + 2*0.4 + 2*0.4],
[7*0.3 + 9*0.5 + 6*0.2, 3*0.3 + 1*0.5 + 4*0.2],
[1*0.3 + 3*0.6 + 5*0.1, 2*0.3 + 4*0.6 + 6*0.1]
]
or
[
[10, 5],
[8, 2],
[7.8, 2.2],
[2.6, 3.6]
]
How could I do that? Thanks!
Here is what I found. It turns out to be simple element wise mulitiplication (* or tf.multiply)
dist = tf.constant([[0.1, 0.6, 0.3], [0.2, 0.4, 0.4], [0.3, 0.5, 0.2], [0.3, 0.6, 0.1]])
val = tf.constant([[[10.0, 5.0],[10, 5],[10,5]],[[8, 2],[8, 2],[8, 2]],[[7, 3],[9, 1],[6, 4]],[[1, 2],[3, 4],[5, 6]]])
dist.get_shape()
TensorShape([Dimension(4), Dimension(3)])
val.get_shape()
TensorShape([Dimension(4), Dimension(3), Dimension(2)])
val2 = tf.transpose(val, perm=[0, 2, 1])
val2.get_shape()
TensorShape([Dimension(4), Dimension(2), Dimension(3)])
dist2 = tf.expand_dims(dist, 1)
dist2.get_shape()
TensorShape([Dimension(4), Dimension(1), Dimension(3)])
c1 = val2 * dist2
c1.get_shape()
TensorShape([Dimension(4), Dimension(2), Dimension(3)])
c2 = tf.reduce_sum(c1, 2)
print(c2.eval())
[[ 10. 5. ]
[ 8. 2. ]
[ 7.80000019 2.20000005]
[ 2.60000014 3.5999999 ]]
I'm using the tf.unsorted_segment_sum method of TensorFlow and it works fine when the tensor i give as data have only one line. For example:
tf.unsorted_segment_sum(tf.constant([0.2, 0.1, 0.5, 0.7, 0.8]),
tf.constant([0, 0, 1, 2, 2]), 3)
Gives the right result:
array([ 0.3, 0.5 , 1.5 ], dtype=float32)
The question is, if i use a tensor with several lines, how can I get the results for each line? For instance, if I try a tensor with two lines:
tf.unsorted_segment_sum(tf.constant([[0.2, 0.1, 0.5, 0.7, 0.8],
[0.2, 0.2, 0.5, 0.7, 0.8]]),
tf.constant([[0, 0, 1, 2, 2],
[0, 0, 1, 2, 2]]), 3)
The result i would expect is:
array([ [ 0.3, 0.5 , 1.5 ], [ 0.4, 0.5, 1.5 ] ], dtype=float32)
But what I get is:
array([ 0.7, 1. , 3. ], dtype=float32)
I want to know if someone know how to obtain the result for each line without using a for loop?
Thanks in advance
EDIT:
While the solution below may cover some additional strange uses, this problem can be solved much more easily just by transposing the data. It turns out that, even though tf.unsorted_segment_sum does not have an axis parameter, it can work only along one axis, as long as it is the first one. So you can do just as follows:
import tensorflow as tf
with tf.Session() as sess:
data = tf.constant([[0.2, 0.1, 0.5, 0.7, 0.8],
[0.2, 0.2, 0.5, 0.7, 0.8]])
idx = tf.constant([0, 0, 1, 2, 2])
result = tf.transpose(tf.unsorted_segment_sum(tf.transpose(data), idx, 3))
print(sess.run(result))
Output:
[[ 0.30000001 0.5 1.5 ]
[ 0.40000001 0.5 1.5 ]]
ORIGINAL POST:
tf.unsorted_segment_sum does not support working on a single axis. The simplest solution would be to apply the operation to each row and then concatenate them back:
data = tf.constant([[0.2, 0.1, 0.5, 0.7, 0.8],
[0.2, 0.2, 0.5, 0.7, 0.8]])
segment_ids = tf.constant([[0, 0, 1, 2, 2],
[0, 0, 1, 2, 2]])
num_segments = 3
rows = []
for data_i, ids_i in zip(data, segment_ids):
rows.append(tf.unsorted_segment_sum(data_i, ids_i))
result = tf.stack(rows, axis=0)
However, this has drawbacks: 1) it only works for statically-shaped tensors (that is, you need to have a fixed number of rows) and 2) it may not be as efficient. The first one could be circumvented using a tf.while_loop, but, it would be complicated, and also it would require you to concatenate the rows one by one, which is very inefficient. Also, you already stated you want to avoid loops.
A better option is to use different ids for each row. For example, you could add to each value in segment_id something like num_segments * row_index, so you guarantee that each row will have its own set of ids:
num_rows = tf.shape(segment_ids)[0]
rows_idx = tf.range(num_rows)
segment_ids_per_row = segment_ids + num_segments * tf.expand_dims(rows_idx, axis=1)
Then you can apply the operation and the reshape to get the tensor that you want:
seg_sums = tf.unsorted_segment_sum(data, segment_ids_per_row,
num_segments * num_rows)
result = tf.reshape(seg_sums, [-1, num_segments])
Output:
array([[ 0.3, 0.5, 1.5 ],
[ 0.4, 0.5, 1.5 ]], dtype=float32)