(Dyalog) APL learner question
If I have a matrix Y:
Y
4 9 2
3 5 7
8 1 6
I can get two of its members like this:
Y[(1 1) (2 2)]
4 5
I can use the same technique using dfn syntax:
{⍵[(1 1) (2 2)]}Y
4 5
I, however, can't work out how to do the equivalent in a tacit function. In particular it seems that bracket indexing doesn't work in a tacit function, and I can't find a way of using squad indexing with list of indexes.
Is there a way of doing this, or is this a limitation of tacit functions?
Note that in my real example the list of indexes is generated, so I can't simply do (((1 1)⌷⊢),(2 2)⌷⊢)Y or anything similar.
(1 1)(2 2)⌷¨⊂Y
works, also
(1 1)(2 2)⊃⍤0 99⊢Y
The first thing one might try is
Y ← 3 3⍴⍳9
Y
1 2 3
4 5 6
7 8 9
Y[(1 1)(2 2)]
1 5
1 1⌷Y
1
(1 1)(2 2)⌷Y
2 2
2 2
But we see that (1 1)(2 2)⌷Y doesn't work. What is happening is that ⌷ looks at the vectors on its left and builds all combinations of indices, which just builds a 2 by 2 matrix of 2s, as (1 1)(2 2) is interpreted as the indices (1 2), (1, 2), (1, 2), and then (1, 2) again.
It might be easier to see it like this:
(1 2)3⌷Y
3 6
(1 2)3⌷ means "from the first and second rows, give me the element in the 3rd column".
Therefore, if you want to give the indices like that, you are likely to need to use the each operator ¨ with ⌷:
(1 1)(2 2)⌷¨⊂Y
1 5
If you really want that tacitly, then you can use
I ← ⌷¨∘⊂
As other answer(s) have shown, there are more alternatives to indexing. I can also recommend the following webinar on indexing: https://dyalog.tv/Webinar/?v=AgYDvSF2FfU .
Take your time to go through the alternatives in the video, APL isn't like Python: in APL, there's generally more than one obvious way to do it :)
Related
I have a dataframe of few hundreds rows , that can be grouped to ids as follows:
df = Val1 Val2 Val3 Id
2 2 8 b
1 2 3 a
5 7 8 z
5 1 4 a
0 9 0 c
3 1 3 b
2 7 5 z
7 2 8 c
6 5 5 d
...
5 1 8 a
4 9 0 z
1 8 2 z
I want to use GridSearchCV , but with a custom CV that will assure that all the rows from the same ID will always be on the same set.
So either all the rows if a are in the test set , or all of them are in the train set - and so for all the different IDs.
I want to have 5 folds - so 80% of the ids will go to the train and 20% to the test.
I understand that it can't guarentee that all folds will have the exact same amount of rows - since one ID might have more rows than the other.
What is the best way to do so?
As stated, you can provide cv with an iterator. You can use GroupShuffleSplit(). For example, once you use it to split your dataset, you can put the result within GridSearchCV() for the cv parameter.
As mentioned in the sklearn documentation, there's a parameter called "cv" where you can provide "An iterable yielding (train, test) splits as arrays of indices."
Do check out the documentation in future first.
As mentioned previously, GroupShuffleSplit() splits data based on group lables. However, the test sets aren't necessarily disjoint (i.e. doing multiple splits, an ID may appear in multiple test sets). If you want each ID to appear in exactly one test fold, you could use GroupKFold(). This is also available in Sklearn.model_selection, and directly extends KFold to take into account group lables.
I'd like to find the contiguous sequences of equal elements (e.g. of length 2) in a list
my #s = <1 1 0 2 0 2 1 2 2 2 4 4 3 3>;
say grep {$^a eq $^b}, #s;
# ==> ((1 1) (2 2) (4 4) (3 3))
This code looks ok but when one more 2 is added after the sequence of 2 2 2 or when one 2 is removed from it, it says Too few positionals passed; expected 2 arguments but got 1 How to fix it? Please note that I'm trying to find them without using for loop, i.e. I'm trying to find them using a functional code as much as possible.
Optional: In the bold printed section:
<1 1 0 2 0 2 1 2 2 2 4 4 3 3>
multiple sequences of 2 2 are seen. How to print them the number of times they are seen? Like:
((1 1) (2 2) (2 2) (4 4) (3 3))
There are an even number of elements in your input:
say elems <1 1 0 2 0 2 1 2 2 2 4 4 3 3>; # 14
Your grep block consumes two elements each time:
{$^a eq $^b}
So if you add or remove an element you'll get the error you're getting when the block is run on the single element left over at the end.
There are many ways to solve your problem.
But you also asked about the option of allowing for overlapping so, for example, you get two (2 2) sub-lists when the sequence 2 2 2 is encountered. And, in a similar vein, you presumably want to see two matches, not zero, with input like:
<1 2 2 3 3 4>
So I'll focus on solutions that deal with those issues too.
Despite the narrowing of solution space to deal with the extra issues, there are still many ways to express solutions functionally.
One way that just appends a bit more code to the end of yours:
my #s = <1 1 0 2 0 2 1 2 2 2 4 4 3 3>;
say grep {$^a eq $^b}, #s .rotor( 2 => -1 ) .flat
The .rotor method converts a list into a list of sub-lists, each of the same length. For example, say <1 2 3 4> .rotor: 2 displays ((1 2) (3 4)). If the length argument is a pair, then the key is the length and the value is an offset for starting the next pair. If the offset is negative you get sub-list overlap. Thus say <1 2 3 4> .rotor: 2 => -1 displays ((1 2) (2 3) (3 4)).
The .flat method "flattens" its invocant. For example, say ((1,2),(2,3),(3,4)) .flat displays (1 2 2 3 3 4).
A perhaps more readable way to write the above solution would be to omit the flat and use .[0] and .[1] to index into the sub-lists returned by rotor:
say #s .rotor( 2 => -1 ) .grep: { .[0] eq .[1] }
See also Elizabeth Mattijsen's comment for another variation that generalizes for any sub-list size.
If you needed a more general coding pattern you might write something like:
say #s .pairs .map: { .value xx 2 if .key < #s - 1 and [eq] #s[.key,.key+1] }
The .pairs method on a list returns a list of pairs, each pair corresponding to each of the elements in its invocant list. The .key of each pair is the index of the element in the invocant list; the .value is the value of the element.
.value xx 2 could have been written .value, .value. (See xx.)
#s - 1 is the number of elements in #s minus 1.
The [eq] in [eq] list is a reduction.
If you need text pattern matching to decide what constitutes contiguous equal elements you might convert the input list into a string, match against that using one of the match adverbs that generate a list of matches, then map from the resulting list of matches to your desired result. To match with overlaps (eg 2 2 2 results in ((2 2) (2 2)) use :ov:
say #s .Str .match( / (.) ' ' $0 /, :ov ) .map: { .[0].Str xx 2 }
TIMTOWDI!
Here's an iterative approach using gather/take.
say gather for <1 1 0 2 0 2 1 2 2 2 4 4 3 3> {
state $last = '';
take ($last, $_) if $last == $_;
$last = $_;
};
# ((1 1) (2 2) (2 2) (4 4) (3 3))
This question is partially here to help me understand what lex-sorting is in the context of multi-indexes.
Say I have some MultiIndexed DataFrame df, and for the index I want to use:
a = (1, 1, 1)
So to pull the value from the dataframe I write:
df.loc[a, df.columns[i]]
Which works. But the following doesn't:
df.loc[list(a), df.columns[i]]
Giving me the error:
*** KeyError: 'MultiIndex Slicing requires the index to be fully lexsorted tuple len (1), lexsort depth (0)'
Why is this?
Also, another question, what does the following performance warning mean?
PerformanceWarning: indexing past lexsort depth may impact performance.
I'll illustrate the difference between passing a tuple and a list to .loc, using the example with df being
0 1 2
first second
bar one 4 4 7
two 3 4 7
foo one 8 1 8
two 7 5 4
Here df.loc[('foo', 'two')] returns the row indexed by this tuple, namely (7, 5, 4). The parameter specifies both levels of the multiindex.
But df.loc[['foo', 'two']] means you want all rows with the top level of the multiindex being either 'foo' or 'two'. A list means these are the options you want, and since only one level is provided in each option, the selection is based on the first (leftmost) level. The result:
0 1 2
first second
foo one 8 1 8
two 7 5 4
(Since there are no multiindices that begin with 'two', only those with 'foo' are present.)
Without seeing your dataframe, I can't tell where this difference leads to getting KeyError, but I hope the difference itself is clear now.
From my "Id" Column I want to remove the one and zero's from the left.
That is
1000003 becomes 3
1000005 becomes 5
1000011 becomes 11 and so on
Ignore -1, 10 and 1000000, they will be handled as special cases. but from the remaining rows I want to remove the "1" followed by zeros.
Well you can use modulus to get the end of the numbers (they will be the remainder). So just exclude the rows with ids of [-1,10,1000000] and then compute the modulus of 1000000:
print df
Id
0 -1
1 10
2 1000000
3 1000003
4 1000005
5 1000007
6 1000009
7 1000011
keep = df.Id.isin([-1,10,1000000])
df.Id[~keep] = df.Id[~keep] % 1000000
print df
Id
0 -1
1 10
2 1000000
3 3
4 5
5 7
6 9
7 11
Edit: Here is a fully vectorized string slice version as an alternative (Like Alex' method but takes advantage of pandas' vectorized string methods):
keep = df.Id.isin([-1,10,1000000])
df.Id[~keep] = df.Id[~keep].astype(str).str[1:].astype(int)
print df
Id
0 -1
1 10
2 1000000
3 3
4 5
5 7
6 9
7 11
Here is another way you could try to do it:
def f(x):
"""convert the value to a string, then select only the characters
after the first one in the string, which is 1. For example,
100005 would be 00005 and I believe it's returning 00005.0 from
dataframe, which is why the float() is there. Then just convert
it to an int, and you'll have 5, etc.
"""
return int(float(str(x)[1:]))
# apply the function "f" to the dataframe and pass in the column 'Id'
df.apply(lambda row: f(row['Id']), axis=1)
I get that this question is satisfactory answered. But for future visitors, what I like about alex' answer is that it does not depend on there to be exactly four zeros. The accepted answer will fail if you sometimes have 10005, sometimes 1000005 and whatever.
However, to add something more to the way we think about it. If you know it's always going to be 10000, you can do
# backup all values
foo = df.id
#now, some will be negative or zero
df.id = df.id - 10000
#back in those that are negative or zero (here, first three rows)
df.if[df.if <= 0] = foo[df.id <= 0]
It gives you the same as Karl's answer, but I typically prefer these kind of methods for their readability.
I understand the Modulus operator in terms of the following expression:
7 % 5
This would return 2 due to the fact that 5 goes into 7 once and then gives the 2 that is left over, however my confusion comes when you reverse this statement to read:
5 % 7
This gives me the value of 5 which confuses me slightly. Although the whole of 7 doesn't go into 5, part of it does so why is there either no remainder or a remainder of positive or negative 2?
If it is calculating the value of 5 based on the fact that 7 doesn't go into 5 at all why is the remainder then not 7 instead of 5?
I feel like there is something I'm missing here in my understanding of the modulus operator.
(This explanation is only for positive numbers since it depends on the language otherwise)
Definition
The Modulus is the remainder of the euclidean division of one number by another. % is called the modulo operation.
For instance, 9 divided by 4 equals 2 but it remains 1. Here, 9 / 4 = 2 and 9 % 4 = 1.
In your example: 5 divided by 7 gives 0 but it remains 5 (5 % 7 == 5).
Calculation
The modulo operation can be calculated using this equation:
a % b = a - floor(a / b) * b
floor(a / b) represents the number of times you can divide a by b
floor(a / b) * b is the amount that was successfully shared entirely
The total (a) minus what was shared equals the remainder of the division
Applied to the last example, this gives:
5 % 7 = 5 - floor(5 / 7) * 7 = 5
Modular Arithmetic
That said, your intuition was that it could be -2 and not 5. Actually, in modular arithmetic, -2 = 5 (mod 7) because it exists k in Z such that 7k - 2 = 5.
You may not have learned modular arithmetic, but you have probably used angles and know that -90° is the same as 270° because it is modulo 360. It's similar, it wraps! So take a circle, and say that its perimeter is 7. Then you read where is 5. And if you try with 10, it should be at 3 because 10 % 7 is 3.
Two Steps Solution.
Some of the answers here are complicated for me to understand. I will try to add one more answer in an attempt to simplify the way how to look at this.
Short Answer:
Example 1:
7 % 5 = 2
Each person should get one pizza slice.
Divide 7 slices on 5 people and every one of the 5 people will get one pizza slice and we will end up with 2 slices (remaining). 7 % 5 equals 2 is because 7 is larger than 5.
Example 2:
5 % 7 = 5
Each person should get one pizza slice
It gives 5 because 5 is less than 7. So by definition, you cannot divide whole 5items on 7 people. So the division doesn't take place at all and you end up with the same amount you started with which is 5.
Programmatic Answer:
The process is basically to ask two questions:
Example A: (7 % 5)
(Q.1) What number to multiply 5 in order to get 7?
Two Conditions: Multiplier starts from `0`. Output result should not exceed `7`.
Let's try:
Multiplier is zero 0 so, 0 x 5 = 0
Still, we are short so we add one (+1) to multiplier.
1 so, 1 x 5 = 5
We did not get 7 yet, so we add one (+1).
2 so, 2 x 5 = 10
Now we exceeded 7. So 2 is not the correct multiplier.
Let's go back one step (where we used 1) and hold in mind the result which is5. Number 5 is the key here.
(Q.2) How much do we need to add to the 5 (the number we just got from step 1) to get 7?
We deduct the two numbers: 7-5 = 2.
So the answer for: 7 % 5 is 2;
Example B: (5 % 7)
1- What number we use to multiply 7 in order to get 5?
Two Conditions: Multiplier starts from `0`. Output result and should not exceed `5`.
Let's try:
0 so, 0 x 7 = 0
We did not get 5 yet, let's try a higher number.
1 so, 1 x 7 = 7
Oh no, we exceeded 5, let's get back to the previous step where we used 0 and got the result 0.
2- How much we need to add to 0 (the number we just got from step 1) in order to reach the value of the number on the left 5?
It's clear that the number is 5. 5-0 = 5
5 % 7 = 5
Hope that helps.
As others have pointed out modulus is based on remainder system.
I think an easier way to think about modulus is what remains after a dividend (number to be divided) has been fully divided by a divisor. So if we think about 5%7, when you divide 5 by 7, 7 can go into 5 only 0 times and when you subtract 0 (7*0) from 5 (just like we learnt back in elementary school), then the remainder would be 5 ( the mod). See the illustration below.
0
______
7) 5
__-0____
5
With the same logic, -5 mod 7 will be -5 ( only 0 7s can go in -5 and -5-0*7 = -5). With the same token -5 mod -7 will also be -5.
A few more interesting cases:
5 mod (-3) = 2 i.e. 5 - (-3*-1)
(-5) mod (-3) = -2 i.e. -5 - (-3*1) = -5+3
It's just about the remainders. Let me show you how
10 % 5=0
9 % 5=4 (because the remainder of 9 when divided by 5 is 4)
8 % 5=3
7 % 5=2
6 % 5=1
5 % 5=0 (because it is fully divisible by 5)
Now we should remember one thing, mod means remainder so
4 % 5=4
but why 4?
because 5 X 0 = 0
so 0 is the nearest multiple which is less than 4
hence 4-0=4
modulus is remainders system.
So 7 % 5 = 2.
5 % 7 = 5
3 % 7 = 3
2 % 7 = 2
1 % 7 = 1
When used inside a function to determine the array index. Is it safe programming ? That is a different question. I guess.
Step 1 : 5/7 = 0.71
Step 2 : Take the left side of the decimal , so we take 0 from 0.71 and multiply by 7
0*7 = 0;
Step # : 5-0 = 5 ; Therefore , 5%7 =5
Modulus operator gives you the result in 'reduced residue system'. For example for mod 5 there are 5 integers counted: 0,1,2,3,4. In fact 19=12=5=-2=-9 (mod 7). The main difference that the answer is given by programming languages by 'reduced residue system'.
lets put it in this way:
actually Modulus operator does the same division but it does not care about the answer , it DOES CARE ABOUT reminder for example if you divide 7 to 5 ,
so , lets me take you through a simple example:
think 5 is a block, then for example we going to have 3 blocks in 15 (WITH Nothing Left) , but when that loginc comes to this kinda numbers {1,3,5,7,9,11,...} , here is where the Modulus comes out , so take that logic that i said before and apply it for 7 , so the answer gonna be that we have 1 block of 5 in 7 => with 2 reminds in our hand! that is the modulus!!!
but you were asking about 5 % 7 , right ?
so take the logic that i said , how many 7 blocks do we have in 5 ???? 0
so the modulus returns 0...
that's it ...
A novel way to find out the remainder is given below
Statement : Remainder is always constant
ex : 26 divided by 7 gives R : 5
This can be found out easily by finding the number that completely divides 26 which is closer to the
divisor and taking the difference of the both
13 is the next number after 7 that completely divides 26 because after 7 comes 8, 9, 10, 11, 12 where none of them divides 26 completely and give remainder 0.
So 13 is the closest number to 7 which divides to give remainder 0.
Now take the difference (13 ~ 7) = 5 which is the temainder.
Note: for this to work divisor should be reduced to its simplest form ex: if 14 is the divisor, 7 has to be chosen to find the closest number dividing the dividend.
As you say, the % sign is used to take the modulus (division remainder).
In w3schools' JavaScript Arithmetic page we can read in the Remainder section what I think to be a great explanation
In arithmetic, the division of two integers produces a quotient and a
remainder.
In mathematics, the result of a modulo operation is the
remainder of an arithmetic division.
So, in your specific case, when you try to divide 7 bananas into a group of 5 bananas, you're able to create 1 group of 5 (quotient) and you'll be left with 2 bananas (remainder).
If 5 bananas into a group of 7, you won't be able to and so you're left with again the 5 bananas (remainder).