Is there a way (with loop or iterate, doesn't matter) to iterate over sequence backwards?
Apart from (loop for i downfrom 10 to 1 by 1 do (print i)) which works with indexes, and requires length, or (loop for elt in (reverse seq)) which requires reversing sequence (even worse then the first option).
For lists the easiest is (dolist (x (reverse list)) ..) or using the more efficient nreverse if the list can be modified.
For vectors an alternative is dotimes with index calculation, something like:
(let* ((vec #(1 2 3))
(len (length vec)))
(dotimes (i len)
(print (aref vec (- len i 1)))))
Typically lists are iterated over from the start as each cons points to the next. Doing it from the back is inherently inefficient.
If you nevertheless have a list and wish fast reverse or random access, an option is to coerce it to a vector using e.g (coerce my-list 'array) and then access the elements using aref (or coerce to simple-vector and use svref).
If you are the one building the list, consider creating an adjustable vector with fill-pointer (see make-array documentation) and then use vector-push-extend to add items. That gives fast random access from the beginning.
Iterate can do it:
(iterate (for x :in-sequence #(1 2 3) :downto 0)
(princ x))
; => 321
As others have noted, this will be very inefficient if used on lists.
Related
I'm currently reading through Seibel's "Practical common lisp" and found this example macro:
(defmacro check (&rest forms)
`(progn
,#(loop for f in forms collect `(do-stuff ,f ',f))
(defun test ()
(check ( (= (+ 1 2 ) 3) (= (+ 1 2 ) 4 )))
)
do-stuff simply then format the two args in a certain way allowing for 'testing' the truth of a form, but that's not important for my question.
What I was interested in was to translate the loop into a DO, unfortunately, I'm totally lacking in the necessary skill to do so:
(defmacro check (&rest forms)
`(progn
,#(do ((index 0 (list-length forms))
(accumulator nil))
((= index (list-length forms)) accumulator)
(push `(do-stuff ,(nth index forms) ',(nth index forms)) accumulator)
))
This does the job, I can also do this (put every form into a variable inside the do):
(defmacro check (&rest forms)
`(progn
,#(do* ((index 0 (list-length forms))
(accumulator nil)
(f (nth index forms) (nth index forms)))
((= index (list-length forms)) accumulator)
(push `(do-stuff ,f ',f) accumulator)
))
My problem is the following :
Is there a more efficient way to write this do loop ? Is this a good way to implement it ?
Something in the LOOP version is making me wonder if there is not a simple way to extract an element of a list without the need to define an index variable, or to regroup the COLLECTED elements without the need to define an accumulator list...
If you use do you shouldn't use nth. Just iterate over the list, not the indexes.
(do ((l forms (cdr l))
(accumulator nil))
((null l) (nreverse accumulator))
(let ((f (car l)))
(push `(do-stuff ,f ',f) accumulator)))
You can also use the built-in dolist:
(let ((accumulator nil))
(dolist (f forms (nreverse accumulator))
(push `(do-stuff ,f ',f) accumulator)))
Finally there's mapcar:
(mapcar (lambda (f) `(do-stuff ,f ',f)) forms)
Is there a more efficient way to write this do loop ? Is this a good way to implement it ?
The complexity of your code is quadratic to the size N of the list, since for each item you call nth to access an element inside, resulting in a O(N*N) execution time. There is a more efficient way to do it (the original LOOP version is an example of a linear algorithm).
Here is a different version where instead of calling push followed by nreverse, the items are queued at the end of the list during traversal. I added comments to explain what each part does.
By the way I don't claim that this is more efficient that using nreverse, I think we can't know without testing. Note however that there are as many operations in both cases (cons a new item, and eventually mutate the cdr slot), they are just done either in two passes or one pass.
In fact the code below is very not far from being an implementation of MAPCAR where there is only one list to traverse (not the variadic version in the standard).
First, define a helper function that transforms one form:
(defun expand-check (form)
`(do-stuff ,form ',form))
Recall that you could just (mapcar #'expand-check checks) to have the desired result.
Anyway, here is a DO version:
(defun expand-checks (checks)
;; LIST-HOLDER is just a temporary cons-cell that allows us to treat
;; the rest of the queue operations without having to worry about
;; the corner case of the first item (the invariant is that LAST is
;; always a cons-cell, never NIL). Here LIST-HOLDER is initially
;; (:HANDLE), the first value being discarded later.
(let ((list-holder (list :handle)))
;; DO is sufficient because the iterator values are independant
;; from each other (no need to use DO*).
(do (;; descend the input list
(list checks (cdr list))
;; update LAST so that it is always the last cons cell, this
;; is how we can QUEUE items at the end of the list without
;; traversing it. This queue implementation was first
;; described by Peter Norvig as far as I known.
(last list-holder (cdr last)))
;; End iteration when LIST is empty
((null list)
;; In which case, return the rest of the LIST-HOLDER, which
;; is the start of the list that was built.
(rest list-holder))
;; BODY of the DO, create a new cons-cell at the end of the
;; queue by mutating the LAST const cell.
(setf (cdr last)
(list (expand-check
(first list)))))))
Firstly, anything of the form
(loop for v in <list> collect (f v ...))
Can be easily expressed as mapcar:
(mapcar (lambda (v)
(f v ...))
<list>)
The interesting case is when the loop only collects a value sometimes, or when the iteration is over some more complicated thing.
In that case one nice approach is to factor out the iteration bit and the 'collecting values' bit, using do or whatever to perform the iteration and some other mechanism to collect values.
One such is collecting. So, for instance, you could use dolist to iterate over the list and collecting to collect values. And perhaps we might only want to collect non-nil values or something to make it more interesting:
(collecting
(dolist (v <list>)
(when v
(collect (f v ...)))))
All of these are more verbose than the simple loop case, but for instance collecting can do things which are painful to express with loop.
I've been reading through https://lispcast.com/when-to-use-a-macro, and it states (about clojure's macros)
Another example is performing expensive calculations at compile time as an optimization
I looked up, and it seems clojure has unhygienic macros. Can this also be applied to hygienic ones? Particularly talking about Scheme. As far as I understand hygienic macros, they only transform syntax, but the actual execution of code is deferred until the runtime no matter what.
Yes. Macro hygiene just refers to whether or not macro expansion can accidentally capture identifiers. Whether or not a macro is hygienic, regular macro expansion (as opposed to reader macro expansion) occurs at compile-time. Macro expansion replaces the macro's code with the results of it being executed. Two major use cases for them are to transform syntax (i.e. DSLs), to enhance performance by eliminating computations at run time or both.
A few examples come to mind:
You prefer to write your code with angles in degrees but all of the calculations are actually in radians. You could have macros eliminate these trivial, but unnecessary (at run time) conversions, at compile time.
Memoization is a broad example of compute optimization that macros can be used for.
You have a string representing a SQL statement or complex textual math expression which you want to parse and possibly even execute at compile time.
You could also combine the examples and have a memoizing SQL parser. Pretty much any scenario where you have all the necessary inputs at compile time and can therefore compute the result is a candidate.
Yes, hygienic macros can do this sort of thing. As an example here is a macro called plus in Racket which is like + except that, at macroexpansion-time, it sums sequences of adjacent literal numbers. So it does some of the work you might expect to be done at run-time at macroexpansion-time (so, effectively, at compile-time). So, for instance
(plus a b 1 2 3 c 4 5)
expands to
(+ a b 6 c 9)
Some notes on this macro.
It's probably not very idiomatic Racket, because I'm a mostly-unreformed CL hacker, which means I live in a cave and wear animal skins and say 'ug' a lot. In particular I am sure I should use syntax-parse but I can't understand it.
It might not even be right.
There are subtleties with arithmetic which means that this macro can return different results than +. In particular + is defined to add pairwise from left to right, while plus does not in general: all the literals get added firsto in particular (assuming you have done (require racket/flonum, and +max.0 &c have the same values as they do on my machine), then (+ -max.0 1.7976931348623157e+308 1.7976931348623157e+308) has a value of 1.7976931348623157e+308, while (plus -max.0 1.7976931348623157e+308 1.7976931348623157e+308) has a value of +inf.0, because the two literals get added first and this overflows.
In general this is a useless thing: it's safe to assume, I think, that any reasonable compiler will do these kind of optimisations for you. The only purpose of it is to show that it's possible to do the detect-and-compile-away compile-time constants.
Remarkably, at least from the point of view of caveman-lisp users like me, you can treat this just like + because of the last in the syntax-case: it works to say (apply plus ...) for instance (although no clever optimisation happens in that case of course).
Here it is:
(require (for-syntax racket/list))
(define-syntax (plus stx)
(define +/stx (datum->syntax stx +))
(syntax-case stx ()
[(_)
;; return additive identity
#'0]
[(_ a)
;; identity with one argument
#'a]
[(_ a ...)
;; the interesting case: there's more than one argument, so walk over them
;; looking for literal numbers. This is probably overcomplicated and
;; unidiomatic
(let* ([syntaxes (syntax->list #'(a ...))]
[reduced (let rloop ([current (first syntaxes)]
[tail (rest syntaxes)]
[accum '()])
(cond
[(null? tail)
(reverse (cons current accum))]
[(and (number? (syntax-e current))
(number? (syntax-e (first tail))))
(rloop (datum->syntax stx
(+ (syntax-e current)
(syntax-e (first tail))))
(rest tail)
accum)]
[else
(rloop (first tail)
(rest tail)
(cons current accum))]))])
(if (= (length reduced) 1)
(first reduced)
;; make sure the operation is our +
#`(#,+/stx #,#reduced)))]
[_
;; plus on its own is +, but we want our one. I am not sure this is right
+/stx]))
It is possible to do this even more aggressively in fact, so that (plus a b 1 2 c 3) is turned into (+ a b c 6). This has probably even more exciting might-get-different answers implications. It's worth noting what the CL spec says about this:
For functions that are mathematically associative (and possibly commutative), a conforming implementation may process the arguments in any manner consistent with associative (and possibly commutative) rearrangement. This does not affect the order in which the argument forms are evaluated [...]. What is unspecified is only the order in which the parameter values are processed. This implies that implementations may differ in which automatic coercions are applied [...].
So an optimisation like this is clearly legal in CL: I'm not clear that it's legal in Racket (although I think it should be).
(require (for-syntax racket/list))
(define-for-syntax (split-literals syntaxes)
;; split a list into literal numbers and the rest
(let sloop ([tail syntaxes]
[accum/lit '()]
[accum/nonlit '()])
(if (null? tail)
(values (reverse accum/lit) (reverse accum/nonlit))
(let ([current (first tail)])
(if (number? (syntax-e current))
(sloop (rest tail)
(cons (syntax-e current) accum/lit)
accum/nonlit)
(sloop (rest tail)
accum/lit
(cons current accum/nonlit)))))))
(define-syntax (plus stx)
(define +/stx (datum->syntax stx +))
(syntax-case stx ()
[(_)
;; return additive identity
#'0]
[(_ a)
;; identity with one argument
#'a]
[(_ a ...)
;; the interesting case: there's more than one argument: split the
;; arguments into literals and nonliterals and handle approprately
(let-values ([(literals nonliterals)
(split-literals (syntax->list #'(a ...)))])
(if (null? literals)
(if (null? nonliterals)
#'0
#`(#,+/stx #,#nonliterals))
(let ([sum/stx (datum->syntax stx (apply + literals))])
(if (null? nonliterals)
sum/stx
#`(#,+/stx #,#nonliterals #,sum/stx)))))]
[_
;; plus on its own is +, but we want our one. I am not sure this is right
+/stx]))
I am having trouble understanding how to compare numbers by value vs by address.
I have tried the following:
(setf number1 5)
(setf number2 number1)
(setf number3 5)
(setf list1 '(a b c d) )
(setf list2 list1)
(setf list3 '(a b c d) )
I then used the following predicate functions:
>(eq list1 list2) T
>(eq list1 list3) Nil
>(eq number1 number2) T
>(eq number1 number3) T
Why is it that with lists eq acts like it should (both pointers for list1 and list3 are different) yet for numbers it does not act like I think it should as number1 and number3 should have different addresses. Thus my question is why this doesn't act like I think it should and if there is a way to compare addresses of variables containing numbers vs values.
Equality Predicates in Common Lisp
how to compare numbers by value vs by address.
While there's a sense in which can be applied, that's not really the model that Common Lisp provides. Reading about the built-in equality predicates can help clarify the way in which objects are stored in memory (implicitly)..
EQ is generally what checks the "same address", but that's not how it's specified, and that's not exactly what it does, either. It "returns true if its arguments are the same, identical object; otherwise, returns false."
What does it mean to be the same identical object? For things like cons-cells (from which lists are built), there's an object in memory somewhere, and eq checks whether two values are the same object. Note that eq could return true or false on primitives like numbers, since the implementation is free to make copies of them.
EQL is like eq, but it adds a few extra conditions for numbers and characters. Numbers of the same type and value are guaranteed to be eql, as are characters that represent the same character.
EQUAL and EQUALP are where things start to get more complex and you actually get something like element-wise comparison for lists, etc.
This specific case
Why is it that with lists eq acts like it should (both pointers for
list1 and list3 are different) yet for numbers it does not act like I
think it should as number1 and number3 should have different
addresses. Thus my question is why this doesn't act like I think it
should and if there is a way to compare addresses of variables
containing numbers vs values.
The examples in the documentation for eq show that (eq 3 3) (and thus, (let ((x 3) (y 3)) (eq x y)) can return true or false. The behavior you're observing now isn't the only possible one.
Also, note that in compiled code, constant values can be coalesced into one. That means that the compiler has the option of making the following return true:
(let ((x '(1 2 3))
(y '(1 2 3)))
(eq x y))
One of the problems is that testing it in one implementation in a specific setting does not tell you much. Implementations may behave differently when the ANSI Common Lisp specification allows it.
do not assume that two numbers of the same value are EQ or not EQ. This is unspecified in Common Lisp. Use EQL or = to compare numbers.
do not assume that two literal lists, looking similar in a printed representation, are EQ or not EQ. This is unspecified in Common Lisp for the general case.
For example:
A file with the following content:
(defvar *a* '(1 2 3))
(defvar *b* '(1 2 3))
If one now compiles and loads the file it is unspecified if (eq *a* *b*) is T or NIL. Common Lisp allows an optimizing compiler to detect that the lists have the similar content and then will allocate only one list and both variables will be bound to the same list.
An implementation might even save space when not the whole lists are having similar content. For example in (a 1 2 3 4) and (b 1 2 3 4) a sublist (1 2 3 4) could be shared.
For code with a lot of list data, this could help saving space both in code and memory. Other implementations might not be that sophisticated. In interactive use, it is unlikely that an implementation will try to save space like that.
In the Common Lisp standard quite a bit behavior is unspecified. It was expected that implementations with different goals might benefit from different approaches.
It's a Project Euler problem .
I learned from Fastest way to list all primes below N
and implemented a clojure :
(defn get-primes [n]
(loop [numbers (set (range 2 n))
primes []]
(let [item (first numbers)]
(cond
(empty? numbers)
primes
:else
(recur (clojure.set/difference numbers (set (range item n item)))
(conj primes item))))))
used like follows:
(reduce + (get-primes 2000000))
but It is so slow..
I am wondering why, Can someone enlighten me?
This algorithm is not even correct: at each iteration except the final one it adds the value of (first numbers) at that point to primes, but there is no guarantee that it will in fact be a prime, since the set data structure in use is unordered. (This is also true of the Python original, as mentioned by its author in an edit to the question you link to.) So, you'd first need to fix it by changing (set (range ...)) to (into (sorted-set) (range ...)).
Even then, this is simply not a great algorithm for finding primes. To do better, you may want to write an imperative implementation of the Sieve of Eratosthenes using a Java array and loop / recur, or maybe a functional SoE-like algorithm such as those described in Melissa E. O'Neill's beautiful paper The Genuine Sieve of Eratosthenes.
I need to count the number of times that particular pairs of integers occur in some process (a heuristic detecting similar images using locality sensitive hashing - integers represent images and the "neighbours" below are images that have the same hash value, so the count indicates how many different hashes connect the given pair of images).
The counts are stored as a map from (ordered) pair to count (matches below).
The input (nbrs-list below) is a list of a list of integers that are considered neighbours, where every distinct (ordered) pair in the inner list ("the neighbours") should be counted. So, for example, if nbrs-list is [[1,2,3],[10,8,9]] then the pairs are [1,2],[1,3],[2,3],[8,9],[8,10],[9,10].
The routine collect is called multiple times; the matches parameter accumulates results and the nbrs-list is new on each call. The smallest number of neighbours (size of the inner list) is 1 and the largest ~1000. Each integer in nbrs-list occurs just once in any call to collect (this implies that, on each call, no pair occurs more than once) and the integers cover the range 0 - 1,000,000 (so the size of nbrs-list is less than 1,000,000 - since each value occurs just once and sometimes they occur in groups - but typically larger than 100,000 - as most images have no neighbours).
I have pulled the routines out of a larger chunk of code, so they may contain small edit errors, sorry.
(defn- flatten-1
[list]
(apply concat list))
(defn- pairs
([nbrs]
(let [nbrs (seq nbrs)]
(if nbrs (pairs (rest nbrs) (first nbrs) (rest nbrs)))))
([nbrs a bs]
(lazy-seq
(let [bs (seq bs)]
(if bs
(let [b (first bs)]
(cons (if (> a b) [[b a] 1] [[a b] 1]) (pairs nbrs a (rest bs))))
(pairs nbrs))))))
(defn- pairs-map
[nbrs]
(println (count nbrs))
(let [pairs-list (flatten-1 (pairs nbrs))]
(apply hash-map pairs-list)))
(defn- collect
[matches nbrs-list]
(let [to-add (map pairs-map nbrs-list)]
(merge-with + matches (apply (partial merge-with +) to-add))))
So the above code expands each set of neighbours to ordered pairs; creates a map from pairs to 1; then combines maps using addition of values.
I'd like this to run faster. I don't see how to avoid the O(n^2) expansion of pairs, but I imagine I can at least reduce the constant overhead by avoiding so many intermediate structures. At the same time, I'd like the code to be fairly compact and readable...
Oh, and now I am exceeding the "GC overhead limit". So reducing memory use/churn is also a priority :o)
[Maybe this is too specific? I am hoping the lessons are general and haven't seem many posts about optimising "real" clojure code. Also, I can profile etc, but my code seems so ugly I am hoping there's an obvious, cleaner approach - particularly for the pairs expansion.]
I guess you want the frequency with which each pair occurs ?
Try function frequencies. It uses transients under the hood, which should avoid GC overheads.
(I hope I haven't misunderstood your question)
If you just want to count the pairs in lists as this, then [[1,2,3],[8,9,10]] is
(defn count-nbr-pairs [n]
(/ (* n (dec n))
2))
(defn count-pairs [nbrs-list]
(apply + (map #(count-nbr-pairs (count %)) nbrs-list)))
(count-pairs [[1 2 3 4] [8 9 10]])
; => 9
This of course assumes, that you don't need to remove duplicate pairs.
=>(require '[clojure.math.combinatorics :as c])
=>(map #(c/combinations % 2) [[1 2 3] [8 9 10]])
(((1 2) (1 3) (2 3)) ((8 9) (8 10) (9 10)))
It's a pretty small library, take a look at the source
Performance wise you're looking at around the following number for your usecase of 1K unique values under 1M
=> (time (doseq
[i (c/combinations (take 1000 (shuffle (range 1000000))) 2)]))
"Elapsed time: 270.99675 msecs"
That's including generating the target set, which takes about 100 ms on my machine.
the above suggestions seemed to help, but were insufficient. i finally got decent performance with:
packing pairs into a long value. this works because MAX_LONG > 1e12 and long instances are comparable (so work well as hash keys, unlike long[2]). this had a significant effect on lowering memory use compared to [n1 n2].
using a TLongByteHashMap primitive type hash map and mutating it.
handling the pair code with nested doseq loops (or nested for loops when using immutable data structures).
improving my locality sensitive hash. a big part of the problem was that it was too weak, so finding too many neighbours - if the neighbours of a million images are ill-constrained then you get a million million pairs, which consumes a little too much memory...
the inner loop now looks like:
(doseq [i (range 1 (count nbrs))]
(doseq [j (range i)]
(let [pair (pack size (nth nbrs i) (nth nbrs j))]
(if-let [value (.get matches pair)]
(.put matches pair (byte (inc value)))
(.put matches pair (byte 0))))))
where
(defn- pack
[size n1 n2]
(+ (* size n1) n2))
(defn- unpack
[size pair]
[(int (/ pair size)) (mod pair size)])