macro for pandas column setting in hy - pandas

I feel I'm writing functions needlessly for the following operation of setting several derived columns sequentially:
(defn add-cols[d]
(do
(setv (get d "col0") "0")
(setv (get d "col1") (np.where (> 0 (get d "existing-col")) -1 1))
(setv (get d "col2") (* (get d "col1") (get d "existing-col")))
d
))
The above is neither succinct nor easy to follow. I'd appreciate any help with converting this pattern to a macro. I'm a beginner with macros but am thinking of creating something like so :
(pandas-addcols d
`col0 : "0",
`col1 : (np.where ( > 0 `existing-col) -1 1),
`col2 : (* `col1 `existing-col))
Would appreciate any help or guidance on the above. The final form of the macro can obviously be different too. Ultimately the most repetitive bit is the multiple "setv" and "get" calls and maybe there are more elegant a generic ways to remove those calls.

A little syntactic sugar that can help is to use a shorter name for get and remove the need to quote the string literal. Here's a simple version of $ from this library. Also, Hy's setv already lets you provide more than one target–value pair.
(import
[numpy :as np]
[pandas :as pd])
(defmacro $ [obj key]
(import [hy [HyString]])
`(get (. ~obj loc) (, (slice None) ~(HyString key))))
(setv
d (pd.DataFrame (dict :a [-3 1 3] :b [4 5 6]))
($ d col0) 0
($ d col1) (np.where (> 0 ($ d a)) -1 1))

Related

Sum of numeric elements in a nested list LISP

Return the SUM of numeric elements in a nested list using LISP. If there are no numeric elements return an empty list/NIL
Examples:
(6 3 -2 5 (4 2 -3) 4) should return 19
(1 2 3 (-4 5) a b c) should return 7
Asking other people to do your homework for you is almost never a good way of learning anything.
But here's an answer, which is written in a Lisp (Racket) and which does show how you ought to go about solving this problem, and also (I think) demonstrates some nice ways of thinking about problems like this ... but which you almost certainly can't cut and paste.
Note that this does not quite agree with the requirements given: which is supposed to return a non-numeric value for a list with no numbers in it. That breaks a recursive algorithm like this, since the empty list is a list with no numbers in it. So this does something more sensible. Making this answer implement the requirements is left as an exercise to the student.
(define (sum-nested-list l)
(let sum-nested-list-loop ([thing l]
[running-total 0])
(match thing
['()
;; We're done if we've run out of list
running-total]
[(list* (? number? x) r)
;; A list whose first element is a number: add it to the running total
;; and carry on on the rest of the list
(sum-nested-list-loop r (+ running-total x))]
[(list* (? list? x) r)
;; A list whose first element is a nested list: recurse into the
;; nested list
(sum-nested-list-loop r (sum-nested-list-loop x running-total))]
[(list* _ r)
;; A list whose first element is not a number or a nested list:
;; carry on on the rest of the list
(sum-nested-list-loop r running-total)]
[_
;; Not a list at all: explode
(error 'sum-numeric-list "what?")])))
(defun flat-sum (tree)
(let ((count 0))
(tree-equal tree tree :test (lambda (left right)
(if (numberp left)
(incf count left) t)))
count))
1> (flat-sum '(1 2 3 (-4 5) a b c))
7

array_eq_sub behavior for zero length

I have the following lemma in why3:
lemma trivial:
forall a : array 'a, b : array 'a.
array_eq_sub a b 0 0
This seems like it would be the base case behavior, but apparently isn't. Any ideas on why this isn't working?
UPDATE
I was able to reduce the issue to a single missing lemma:
lemma array_eq_2:
forall a : array 'a, b : array 'a.
map_eq_sub a.elts b.elts 0 0 -> array_eq_sub a b 0 0
This seems trivial as well, given the definition of array_eq_sub as specified in the documentation. Why can't my prover find a solution?
After struggling with this issue, I decided to take a look at the why3 source code. I found a definition which was different from what was documented:
predicate array_eq_sub (a1 a2: array 'a) (l u: int) =
a1.length = a2.length /\ 0 <= l <= a1.length /\ 0 <= u <= a1.length /\
map_eq_sub a1.elts a2.elts l u
In short, the lengths of the arrays have to be equal in order for a portion of them to be equal. This was different than what was documented, and I suspect may be causing many theorems to be unsound.

Clojure - sequence of objects

I have a Clojure function:
(def obseq
(fn []
(let [a 0
b 1
c 2]
(println (seq '(a b c))))))
It outputs :
(a b c)
I want it to output a sequence containing the values of a, b, and c, not their names.
Desired output:
(1 2 3)
How do I implement this?
Short answer:
clj.core=> (defn obseq []
(let [a 0
b 1
c 2]
(println [a b c])))
#'clj.core/obseq
clj.core=> (obseq)
[0 1 2]
nil
Long answer:
Quoting a form like '(a b c) recursively prevents any evaluation inside the quoted form. So, the values for your 3 symbols a, b, and c aren't substituted. It is much easier to use a vector (square brackets), which never needs to be quoted (unlike a list). Since the vector is unquoted, the 3 symbols are evaluated and replaced by their values.
If you wanted it to stay a list for some reason, the easiest way is to type:
clj.core=> (defn obseq [] (let [a 0 b 1 c 2] (println (list a b c))))
#'clj.core/obseq
clj.core=> (obseq)
(0 1 2)
nil
This version also has no quoting, so the 3 symbols are replaced with their values. Then the function (list ...) puts them into a list data structure.
Note that I also converted your (def obseq (fn [] ...)) into the preferred form (defn obseq [] ...), which has the identical result but is shorter and (usually) clearer.

How can I use a functor name like a variable in Prolog?

We have this assignment for our Prolog course. After two months of one hour per week of Prolog, it is still an enigma to me, my thinking seems unable to adapt from procedural languages - yet.
There is a knowledge base containing predicates/functors with the same name and arities 1, 2 and 3.
The call form should be
search(functor_name, argument, S).
The answers should find all occurrences with this functor name and argument, regardless of arity.
The answers should be of the form:
S = functor_name(argument);
S = functor_name(argument,_);
S = functor_name(_,argument);
S = functor_name(argument,_,_);
S = functor_name(_,argument,_);
S = functor_name(_,_,argument);
false.
I have found out that I could use call to test if the entry in the knowledge base exists.
But call does not seem to work with a variable for the functor name. I am totally baffled, no idea how to use a variable for a functor name.
UPDATE:
My question has been partly answered.
My new code gives me true and false for arities 1, 2 and 3 (see below).
search(Person,Predicate) :-
ID = Person, Key = Predicate, current_functor(Key,1),
call(Key,ID)
; ID = Person, Key = Predicate, current_functor(Key,2),
(call(Key,ID,_);call(Key,_,ID))
; ID = Person, Key = Predicate, current_functor(Key,3),
(call(Key,ID,_,_);call(Key,_,ID,_);call(Key,_,_,ID)).
UPDATE2:
Another partial answer has come in. That one gives me S as a list of terms, but the "other" arguments are placeholders:
search2(Predicate, Arg, S) :-
( Arity = 2 ; Arity = 3 ; Arity = 4 ),
functor(S, Predicate, Arity),
S =.. [_,Predicate|Args],
member(Arg, Args).
The result is quite nice. Still missing: the Predicate should not be inside the brackets and the other arguments should be taken literally from the knowledge base, not written as placeholders. The current result looks like this:
?- search2(parent,lars,S).
S = parent(parent, lars) ;
S = parent(parent, lars, _G1575) ;
S = parent(parent, _G1574, lars) ;
S = parent(parent, lars, _G1575, _G1576) ;
S = parent(parent, _G1574, lars, _G1576) ;
S = parent(parent, _G1574, _G1575, lars).
I am giving up with this question, because the question was posed in the wrong way from the beginning. I should have asked more specifically - which I could not, because I am still no good in Prolog.
#false helped me most. I am accepting his answer.
There are two approaches here, one "traditional" (1970s) that implements literally what you want:
search(F, Arg, S) :-
( N = 1 ; N = 2 ; N = 3 ), % more compactly: between(1,3, N)
functor(S, F, N),
S =.. [_|Args], % more compactly: between(1,N, I), arg(I,S,Arg)
member(Arg, Args).
The other reconsiders the explicit construction of the goal. Actually, if you have a functor F, and arguments A1, A2, A3 you can immediately write the goal call(F, A1, A2, A3) without any use of functor/3 or (=..)/2.
There are many advantages of using call(F, A1, A2, A3) in place of Goal =.. [F, A1, A2, A3], call(Goal): In many situations it is cleaner, faster, and much easier to typecheck. Further, when using a module system, the handling of potential module qualifications for F will work seamlessly. Whereas (=..)/2 will have to handle all ugly details explicitly, that is more code, more errors.
search(F,A,call(F,A)).
search(F,A,call(F,A,_)).
search(F,A,call(F,_,A)).
search(F,A,call(F,A,_,_)).
search(F,A,call(F,_,A,_)).
search(F,A,call(F,_,_,A)).
If you want to shorten this, then rather construct call/N dynamically:
search(F, Arg, S) :-
( N = 2 ; N = 3 ; N = 4 ),
functor(S, call, N),
S =.. [_,F|Args],
member(Arg, Args).
Note that call needs an extra argument for the functor F!
You can use the "univ" operator, =.., to construct a goal dynamically:
?- F=member, X=1, L=[1,2,3], Goal =.. [F, X, L], call(Goal).
F = member,
X = 1,
L = [1, 2, 3],
Goal = member(1, [1, 2, 3]) .

foldl is tail recursive, so how come foldr runs faster than foldl?

I wanted to test foldl vs foldr. From what I've seen you should use foldl over foldr when ever you can due to tail reccursion optimization.
This makes sense. However, after running this test I am confused:
foldr (takes 0.057s when using time command):
a::a -> [a] -> [a]
a x = ([x] ++ )
main = putStrLn(show ( sum (foldr a [] [0.. 100000])))
foldl (takes 0.089s when using time command):
b::[b] -> b -> [b]
b xs = ( ++ xs). (\y->[y])
main = putStrLn(show ( sum (foldl b [] [0.. 100000])))
It's clear that this example is trivial, but I am confused as to why foldr is beating foldl. Shouldn't this be a clear case where foldl wins?
Welcome to the world of lazy evaluation.
When you think about it in terms of strict evaluation, foldl looks "good" and foldr looks "bad" because foldl is tail recursive, but foldr would have to build a tower in the stack so it can process the last item first.
However, lazy evaluation turns the tables. Take, for example, the definition of the map function:
map :: (a -> b) -> [a] -> [b]
map _ [] = []
map f (x:xs) = f x : map f xs
This wouldn't be too good if Haskell used strict evaluation, since it would have to compute the tail first, then prepend the item (for all items in the list). The only way to do it efficiently would be to build the elements in reverse, it seems.
However, thanks to Haskell's lazy evaluation, this map function is actually efficient. Lists in Haskell can be thought of as generators, and this map function generates its first item by applying f to the first item of the input list. When it needs a second item, it just does the same thing again (without using extra space).
It turns out that map can be described in terms of foldr:
map f xs = foldr (\x ys -> f x : ys) [] xs
It's hard to tell by looking at it, but lazy evaluation kicks in because foldr can give f its first argument right away:
foldr f z [] = z
foldr f z (x:xs) = f x (foldr f z xs)
Because the f defined by map can return the first item of the result list using solely the first parameter, the fold can operate lazily in constant space.
Now, lazy evaluation does bite back. For instance, try running sum [1..1000000]. It yields a stack overflow. Why should it? It should just evaluate from left to right, right?
Let's look at how Haskell evaluates it:
foldl f z [] = z
foldl f z (x:xs) = foldl f (f z x) xs
sum = foldl (+) 0
sum [1..1000000] = foldl (+) 0 [1..1000000]
= foldl (+) ((+) 0 1) [2..1000000]
= foldl (+) ((+) ((+) 0 1) 2) [3..1000000]
= foldl (+) ((+) ((+) ((+) 0 1) 2) 3) [4..1000000]
...
= (+) ((+) ((+) (...) 999999) 1000000)
Haskell is too lazy to perform the additions as it goes. Instead, it ends up with a tower of unevaluated thunks that have to be forced to get a number. The stack overflow occurs during this evaluation, since it has to recurse deeply to evaluate all the thunks.
Fortunately, there is a special function in Data.List called foldl' that operates strictly. foldl' (+) 0 [1..1000000] will not stack overflow. (Note: I tried replacing foldl with foldl' in your test, but it actually made it run slower.)
Upon looking at this problem again, I think all current explanations are somewhat insufficient so I've written a longer explanation.
The difference is in how foldl and foldr apply their reduction function. Looking at the foldr case, we can expand it as
foldr (\x -> [x] ++ ) [] [0..10000]
[0] ++ foldr a [] [1..10000]
[0] ++ ([1] ++ foldr a [] [2..10000])
...
This list is processed by sum, which consumes it as follows:
sum = foldl' (+) 0
foldl' (+) 0 ([0] ++ ([1] ++ ... ++ [10000]))
foldl' (+) 0 (0 : [1] ++ ... ++ [10000]) -- get head of list from '++' definition
foldl' (+) 0 ([1] ++ [2] ++ ... ++ [10000]) -- add accumulator and head of list
foldl' (+) 0 (1 : [2] ++ ... ++ [10000])
foldl' (+) 1 ([2] ++ ... ++ [10000])
...
I've left out the details of the list concatenation, but this is how the reduction proceeds. The important part is that everything gets processed in order to minimize list traversals. The foldr only traverses the list once, the concatenations don't require continuous list traversals, and sum finally consumes the list in one pass. Critically, the head of the list is available from foldr immediately to sum, so sum can begin working immediately and values can be gc'd as they are generated. With fusion frameworks such as vector, even the intermediate lists will likely be fused away.
Contrast this to the foldl function:
b xs = ( ++xs) . (\y->[y])
foldl b [] [0..10000]
foldl b ( [0] ++ [] ) [1..10000]
foldl b ( [1] ++ ([0] ++ []) ) [2..10000]
foldl b ( [2] ++ ([1] ++ ([0] ++ [])) ) [3..10000]
...
Note that now the head of the list isn't available until foldl has finished. This means that the entire list must be constructed in memory before sum can begin to work. This is much less efficient overall. Running the two versions with +RTS -s shows miserable garbage collection performance from the foldl version.
This is also a case where foldl' will not help. The added strictness of foldl' doesn't change the way the intermediate list is created. The head of the list remains unavailable until foldl' has finished, so the result will still be slower than with foldr.
I use the following rule to determine the best choice of fold
For folds that are a reduction, use foldl' (e.g. this will be the only/final traversal)
Otherwise use foldr.
Don't use foldl.
In most cases foldr is the best fold function because the traversal direction is optimal for lazy evaluation of lists. It's also the only one capable of processing infinite lists. The extra strictness of foldl' can make it faster in some cases, but this is dependent on how you'll use that structure and how lazy it is.
I don't think anyone's actually said the real answer on this one yet, unless I'm missing something (which may well be true and welcomed with downvotes).
I think the biggest different in this case is that foldr builds the list like this:
[0] ++ ([1] ++ ([2] ++ (... ++ [1000000])))
Whereas foldl builds the list like this:
((([0] ++ [1]) ++ [2]) ++ ... ) ++ [999888]) ++ [999999]) ++ [1000000]
The difference in subtle, but notice that in the foldr version ++ always has only one list element as its left argument. With the foldl version, there are up to 999999 elements in ++'s left argument (on average around 500000), but only one element in the right argument.
However, ++ takes time proportional to the size of the left argument, as it has to look though the entire left argument list to the end and then repoint that last element to the first element of the right argument (at best, perhaps it actually needs to do a copy). The right argument list is unchanged, so it doesn't matter how big it is.
That's why the foldl version is much slower. It's got nothing to do with laziness in my opinion.
The problem is that tail recursion optimization is a memory optimization, not a execution time optimization!
Tail recursion optimization avoids the need to remember values for each recursive call.
So, foldl is in fact "good" and foldr is "bad".
For example, considering the definitions of foldr and foldl:
foldl f z [] = z
foldl f z (x:xs) = foldl f (z `f` x) xs
foldr f z [] = z
foldr f z (x:xs) = x `f` (foldr f z xs)
That's how the expression "foldl (+) 0 [1,2,3]" is evaluated:
foldl (+) 0 [1, 2, 3]
foldl (+) (0+1) [2, 3]
foldl (+) ((0+1)+2) [3]
foldl (+) (((0+1)+2)+3) [ ]
(((0+1)+2)+3)
((1+2)+3)
(3+3)
6
Note that foldl doesn't remember the values 0, 1, 2..., but pass the whole expression (((0+1)+2)+3) as argument lazily and don't evaluates it until the last evaluation of foldl, where it reaches the base case and returns the value passed as the second parameter (z) wich isn't evaluated yet.
On the other hand, that's how foldr works:
foldr (+) 0 [1, 2, 3]
1 + (foldr (+) 0 [2, 3])
1 + (2 + (foldr (+) 0 [3]))
1 + (2 + (3 + (foldr (+) 0 [])))
1 + (2 + (3 + 0)))
1 + (2 + 3)
1 + 5
6
The important difference here is that where foldl evaluates the whole expression in the last call, avoiding the need to come back to reach remembered values, foldr no. foldr remember one integer for each call and performs a addition in each call.
Is important to bear in mind that foldr and foldl are not always equivalents. For instance, try to compute this expressions in hugs:
foldr (&&) True (False:(repeat True))
foldl (&&) True (False:(repeat True))
foldr and foldl are equivalent only under certain conditions described here
(sorry for my bad english)
For a, the [0.. 100000] list needs to be expanded right away so that foldr can start with the last element. Then as it folds things together, the intermediate results are
[100000]
[99999, 100000]
[99998, 99999, 100000]
...
[0.. 100000] -- i.e., the original list
Because nobody is allowed to change this list value (Haskell is a pure functional language), the compiler is free to reuse the value. The intermediate values, like [99999, 100000] can even be simply pointers into the expanded [0.. 100000] list instead of separate lists.
For b, look at the intermediate values:
[0]
[0, 1]
[0, 1, 2]
...
[0, 1, ..., 99999]
[0.. 100000]
Each of those intermediate lists can't be reused, because if you change the end of the list then you've changed any other values that point to it. So you're creating a bunch of extra lists that take time to build in memory. So in this case you spend a lot more time allocating and filling in these lists that are intermediate values.
Since you're just making a copy of the list, a runs faster because it starts by expanding the full list and then just keeps moving a pointer from the back of the list to the front.
Neither foldl nor foldr is tail optimized. It is only foldl'.
But in your case using ++ with foldl' is not good idea because successive evaluation of ++ will cause traversing growing accumulator again and again.
Well, let me rewrite your functions in a way that difference should be obvious -
a :: a -> [a] -> [a]
a = (:)
b :: [b] -> b -> [b]
b = flip (:)
You see that b is more complex than a. If you want to be precise a needs one reduction step for value to be calculated, but b needs two. That makes the time difference you are measuring, in second example twice as much reductions must be performed.
//edit: But time complexity is the same, so I wouldn't bother about it much.