Let's suppose i have this array:
import numpy as np
x = np.arange(4)
array([0, 1, 2, 3])
I want to write a very basic formula which will generate from x this array:
array([[0, 1, 2, 3],
[1, 2, 3, 4],
[2, 3, 4, 5],
[3, 4, 5, 6]])
What is the shortest way to do that with python and numpy ?
Thanks
The easiest way I can think of is to use numpy broadcasting.
x[:,None]+x
Out[87]:
array([[0, 1, 2, 3],
[1, 2, 3, 4],
[2, 3, 4, 5],
[3, 4, 5, 6]])
This should do what you want (note that I have introduced a different number of rows (5) than of columns (4) to make a clear distinction):
import numpy as np
A = np.tile(np.arange(4).reshape(1,4),(5,1))+np.tile(np.arange(5).reshape(5,1),(1,4))
print(A)
A break-down of steps:
np.tile(np.arange(4).reshape(1,4),(5,1)) creates a (5,4) matrix with entries 0,1,2,3 in each row:
[[0 1 2 3]
[0 1 2 3]
[0 1 2 3]
[0 1 2 3]
[0 1 2 3]]
np.tile(np.arange(5).reshape(5,1),(1,4)) creates a (5,4) matrix with 0,1,2,3,4 in each column:
[[0 0 0 0]
[1 1 1 1]
[2 2 2 2]
[3 3 3 3]
[4 4 4 4]]
The sum of the two results in what you want:
[[0 1 2 3]
[1 2 3 4]
[2 3 4 5]
[3 4 5 6]
[4 5 6 7]]
Related
Consider numpy array arr , shown below:
arr = ([[1, 5, 6, 3, 3, 7],
[2, 2, 2, 2, 2, 2],
[0, 1, 0, 1, 0, 1],
[4, 8, 4, 8, 4, 8],
[1, 2, 3, 4, 5, 6]])
I want to find all row permutations of arr. NOTE: the order of elements in any given row is unchanged. It is the entire rows that are being permuted.
Because arr has 5 rows, there will be 5! = 120 permutations. I’m hoping these could be ‘stacked’ into a 3d array p, having shape (120, 5, 6):
p = [[[1, 5, 6, 3, 3, 7],
[2, 2, 2, 2, 2, 2],
[0, 1, 0, 1, 0, 1],
[4, 8, 4, 8, 4, 8],
[1, 2, 3, 4, 5, 6]],
[[1, 5, 6, 3, 3, 7],
[2, 2, 2, 2, 2, 2],
[0, 1, 0, 1, 0, 1],
[1, 2, 3, 4, 5, 6]
[4, 8, 4, 8, 4, 8]],
… etc …
[[1, 2, 3, 4, 5, 6],
[4, 8, 4, 8, 4, 8],
[0, 1, 0, 1, 0, 1],
[2, 2, 2, 2, 2, 2],
[1, 5, 6, 3, 3, 7]]]
There is a lot of material online about permitting elements within rows, but I need help in permuting the entire rows themselves.
You can make use of itertools.permutations and np.argsort:
from itertools import permutations
out = np.array([arr[np.argsort(idx)] for idx in permutations(range(5))])
print(out)
[[[1 5 6 3 3 7]
[2 2 2 2 2 2]
[0 1 0 1 0 1]
[4 8 4 8 4 8]
[1 2 3 4 5 6]]
[[1 5 6 3 3 7]
[2 2 2 2 2 2]
[0 1 0 1 0 1]
[1 2 3 4 5 6]
[4 8 4 8 4 8]]
[[1 5 6 3 3 7]
[2 2 2 2 2 2]
[4 8 4 8 4 8]
[0 1 0 1 0 1]
[1 2 3 4 5 6]]
...
[[1 2 3 4 5 6]
[0 1 0 1 0 1]
[4 8 4 8 4 8]
[2 2 2 2 2 2]
[1 5 6 3 3 7]]
[[4 8 4 8 4 8]
[1 2 3 4 5 6]
[0 1 0 1 0 1]
[2 2 2 2 2 2]
[1 5 6 3 3 7]]
[[1 2 3 4 5 6]
[4 8 4 8 4 8]
[0 1 0 1 0 1]
[2 2 2 2 2 2]
[1 5 6 3 3 7]]]
Similar answer, but you do not need to .argsort one more time
from itertools import permutations
import numpy as np
arr = np.array([[1, 5, 6, 3, 3, 7],
[2, 2, 2, 2, 2, 2],
[0, 1, 0, 1, 0, 1],
[4, 8, 4, 8, 4, 8],
[1, 2, 3, 4, 5, 6]])
output = np.array([arr[i, :] for i in permutations(range(5))])
print(output)
[[[1 5 6 3 3 7]
[2 2 2 2 2 2]
[0 1 0 1 0 1]
[4 8 4 8 4 8]
[1 2 3 4 5 6]]
[[1 5 6 3 3 7]
[2 2 2 2 2 2]
[0 1 0 1 0 1]
[1 2 3 4 5 6]
[4 8 4 8 4 8]]
[[1 5 6 3 3 7]
[2 2 2 2 2 2]
[4 8 4 8 4 8]
[0 1 0 1 0 1]
[1 2 3 4 5 6]]
...
[[1 2 3 4 5 6]
[4 8 4 8 4 8]
[2 2 2 2 2 2]
[0 1 0 1 0 1]
[1 5 6 3 3 7]]
[[1 2 3 4 5 6]
[4 8 4 8 4 8]
[0 1 0 1 0 1]
[1 5 6 3 3 7]
[2 2 2 2 2 2]]
[[1 2 3 4 5 6]
[4 8 4 8 4 8]
[0 1 0 1 0 1]
[2 2 2 2 2 2]
[1 5 6 3 3 7]]]
This is a bit faster, here are speed comparisons:
%%timeit
output = np.array([arr[i, :] for i in permutations(range(5))])
381 µs ± 14.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
%%timeit
output = np.array([arr[np.argsort(idx)] for idx in permutations(range(5))])
863 µs ± 97.7 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
Is it possible to change a single level column dataframe to a multi-column dataframe? If we have a dataframe like this,
import pandas as pd
df = pd.DataFrame({
'a': [0, 1, 2, 3],
'b': [4, 5, 6, 7],
'c': [3, 5, 6, 2],
'd': [1, 5, 7, 0],
})
can we change it's column names as below?. So, briefly what I am trying to do is to have 2-levels of column index without changing the values of the dataframe.
A B
a b c d
0 0 4 3 1
1 1 5 5 5
2 2 6 6 7
3 3 7 2 0
Any help?
IIUC, use pd.MultiIndex.from_tuples to create multiindex header and assign to the dataframe.columns:
df = pd.DataFrame({
'a': [0, 1, 2, 3],
'b': [4, 5, 6, 7],
'a2': [3, 5, 6, 2],
'b2': [1, 5, 7, 0],
})
df.columns=pd.MultiIndex.from_tuples([('A','a'),('A','b'),('B','c'),('B','d')])
df
Output:
A B
a b c d
0 0 4 3 1
1 1 5 5 5
2 2 6 6 7
3 3 7 2 0
Best described with an example
import pandas as pd
df = pd.DataFrame({
'a' : ['A','B','C','A','B','C','A','B','C'],
'b': [1,2,3,4,5,6,7,8,9]}
)
And i want to create a column that contains in a list the elements of column b by group of column a
resulting in the following
a b c
0 A 1 [1, 4, 7]
1 A 4 [1, 4, 7]
2 A 7 [1, 4, 7]
3 B 2 [2, 5, 8]
4 B 5 [2, 5, 8]
5 B 8 [2, 5, 8]
6 C 3 [3, 6, 9]
7 C 6 [3, 6, 9]
8 C 9 [3, 6, 9]
I can do this with groupby and apply or agg and then joining the dataframes like so
df_tmp = df.groupby('a')['b'].agg(list).reset_index()
df.merge(df_tmp, on='a')
But i would also be expecting to do the same with transform
df['c'] = df.groupby('a')['b'].transform(list)
but the column c is the same as column b
Also the following
df.groupby('a')['b'].transform(lambda x: len(x))
return a series with the values 3 i.e. the length of the grouped elements is 3 (to be expected)
Also this
df.groupby('a')['b'].transform(lambda x: list(x))
does not provide the expected result.
So to my question, how can i obtain the desired result with groupby and tranform
pandas version is 1.0.5
I come up one fix with below. PS : it should something wrong with transform , when the object type is list tuple or set..
df.groupby('a')['b'].transform(lambda x : [x.tolist()]*len(x))
Out[226]:
0 [1, 4, 7]
1 [1, 4, 7]
2 [1, 4, 7]
3 [2, 5, 8]
4 [2, 5, 8]
5 [2, 5, 8]
6 [3, 6, 9]
7 [3, 6, 9]
8 [3, 6, 9]
Name: b, dtype: object
Interesting problem, not sure what happens with transform in the background. One go-around is to map with groupby().agg():
df['c'] = df['a'].map(df.groupby('a')['b'].agg(list))
Output:
a b c
0 A 1 [1, 4, 7]
1 B 2 [2, 5, 8]
2 C 3 [3, 6, 9]
3 A 4 [1, 4, 7]
4 B 5 [2, 5, 8]
5 C 6 [3, 6, 9]
6 A 7 [1, 4, 7]
7 B 8 [2, 5, 8]
8 C 9 [3, 6, 9]
I would like concatenate two matricies with different size
[[1 1 1]
[2 3 2]
[5 5 3]
[3 2 5]
[4 4 4]]
[[1 3 2 5 4]
[1 2 5 3 4]]
to have this matrix
[[1 1 1 1 1]
[2 3 2 3 2]
[5 5 3 2 5]
[3 2 5 5 3]
[4 4 4 4 4]]
I know size of matricies are differents ((5, 3), 'VS', (2, 5))
but i want concatenate these matricies between them.
It's possible to change axis of second matrix ?
Thanks !
I'm new to numpy, so I might be missing something obviuous here.
The following small argsort() test script gives strange results. Any directions ?
import numpy as np
a = np.array([[3, 5, 6, 4, 1] , [2, 7 ,4 ,1 , 2] , [8, 6, 7, 2, 1]])
print a
print a.argsort(axis=0)
print a.argsort(axis=1)
output:
[[3 5 6 4 1]
[2 7 4 1 2]
[8 6 7 2 1]]
[[1 0 1 1 0] # bad 4th & 5th columns ?
[0 2 0 2 2]
[2 1 2 0 1]]
[[4 0 3 1 2] # what's going on here ?
[3 0 4 2 1]
[4 3 1 2 0]]
As others have indicated this method is working correctly, so in order to provide an answer here is an explanation of how .argsort() works. a.argsort returns the indices (not values) in order that would sort the array along the specified axis.
In your example
a = np.array([[3, 5, 6, 4, 1] , [2, 7 ,4 ,1 , 2] , [8, 6, 7, 2, 1]])
print a
print a.argsort(axis=0)
returns
[[3 5 6 4 1]
[2 7 4 1 2]
[8 6 7 2 1]]
[[1 0 1 1 0]
[0 2 0 2 2]
[2 1 2 0 1]]
because along
[[3 ...
[2 ...
[8 ...
2 is the smallest value. Therefore the current index of 2 (which is 0) takes the first position along this axis in the matrix returned by argsort(). The second smallest value is 3 at index 0, therefore the second position along this axis in the returned matrix will be 0. Finally, the largest element is 2 which occurs at index 2 along the 0 axis, so the final element of the returned matrix will be 2. Thus:
[[1 ...
[0 ...
[2 ...
the same process is repeated along other 4 sequences along axis 0:
[[...5 ...] [[...0 ...]
[...7 ...] becomes ----> [... 2 ...]
[...6 ...]] [... 1 ...]]
[[...6 ...] [[...1 ...]
[...4 ...] becomes ----> [... 0 ...]
[...7 ...]] [... 2 ...]]
[[...4 ...] [[...1 ...]
[...1 ...] becomes ----> [... 2 ...]
[...2 ...]] [... 0 ...]]
[[...1] [[...0]
[...2] becomes ----> [... 2]
[...1]] [... 1]]
changing the axis to from 0 to 1, results in this same process being applied along sequences in the 1st axis:
[[3 5 6 4 1 becomes ----> [[4 0 3 1 2
again because the smallest element is 1 which is at index 4, then 3 at index 0, then 4 at index 3, 5 at index 1 and finally 6 is the largest at index 2.
As before this process is repeated across each of
[2 7 4 1 2] ----> [3 0 4 2 1]
[8 6 7 2 1] ----> [4 3 1 2 0]
giving
[[4 0 3 1 2]
[3 0 4 2 1]
[4 3 1 2 0]]
This actually returns a sorted array, whose elements, rather than the element of the array we want to sort, are the index of that element.
enter image description here
enter image description here
This says the first element in our sorted array would be the element whose index is '1', which in turn is '0'.