I've been reading through https://lispcast.com/when-to-use-a-macro, and it states (about clojure's macros)
Another example is performing expensive calculations at compile time as an optimization
I looked up, and it seems clojure has unhygienic macros. Can this also be applied to hygienic ones? Particularly talking about Scheme. As far as I understand hygienic macros, they only transform syntax, but the actual execution of code is deferred until the runtime no matter what.
Yes. Macro hygiene just refers to whether or not macro expansion can accidentally capture identifiers. Whether or not a macro is hygienic, regular macro expansion (as opposed to reader macro expansion) occurs at compile-time. Macro expansion replaces the macro's code with the results of it being executed. Two major use cases for them are to transform syntax (i.e. DSLs), to enhance performance by eliminating computations at run time or both.
A few examples come to mind:
You prefer to write your code with angles in degrees but all of the calculations are actually in radians. You could have macros eliminate these trivial, but unnecessary (at run time) conversions, at compile time.
Memoization is a broad example of compute optimization that macros can be used for.
You have a string representing a SQL statement or complex textual math expression which you want to parse and possibly even execute at compile time.
You could also combine the examples and have a memoizing SQL parser. Pretty much any scenario where you have all the necessary inputs at compile time and can therefore compute the result is a candidate.
Yes, hygienic macros can do this sort of thing. As an example here is a macro called plus in Racket which is like + except that, at macroexpansion-time, it sums sequences of adjacent literal numbers. So it does some of the work you might expect to be done at run-time at macroexpansion-time (so, effectively, at compile-time). So, for instance
(plus a b 1 2 3 c 4 5)
expands to
(+ a b 6 c 9)
Some notes on this macro.
It's probably not very idiomatic Racket, because I'm a mostly-unreformed CL hacker, which means I live in a cave and wear animal skins and say 'ug' a lot. In particular I am sure I should use syntax-parse but I can't understand it.
It might not even be right.
There are subtleties with arithmetic which means that this macro can return different results than +. In particular + is defined to add pairwise from left to right, while plus does not in general: all the literals get added firsto in particular (assuming you have done (require racket/flonum, and +max.0 &c have the same values as they do on my machine), then (+ -max.0 1.7976931348623157e+308 1.7976931348623157e+308) has a value of 1.7976931348623157e+308, while (plus -max.0 1.7976931348623157e+308 1.7976931348623157e+308) has a value of +inf.0, because the two literals get added first and this overflows.
In general this is a useless thing: it's safe to assume, I think, that any reasonable compiler will do these kind of optimisations for you. The only purpose of it is to show that it's possible to do the detect-and-compile-away compile-time constants.
Remarkably, at least from the point of view of caveman-lisp users like me, you can treat this just like + because of the last in the syntax-case: it works to say (apply plus ...) for instance (although no clever optimisation happens in that case of course).
Here it is:
(require (for-syntax racket/list))
(define-syntax (plus stx)
(define +/stx (datum->syntax stx +))
(syntax-case stx ()
[(_)
;; return additive identity
#'0]
[(_ a)
;; identity with one argument
#'a]
[(_ a ...)
;; the interesting case: there's more than one argument, so walk over them
;; looking for literal numbers. This is probably overcomplicated and
;; unidiomatic
(let* ([syntaxes (syntax->list #'(a ...))]
[reduced (let rloop ([current (first syntaxes)]
[tail (rest syntaxes)]
[accum '()])
(cond
[(null? tail)
(reverse (cons current accum))]
[(and (number? (syntax-e current))
(number? (syntax-e (first tail))))
(rloop (datum->syntax stx
(+ (syntax-e current)
(syntax-e (first tail))))
(rest tail)
accum)]
[else
(rloop (first tail)
(rest tail)
(cons current accum))]))])
(if (= (length reduced) 1)
(first reduced)
;; make sure the operation is our +
#`(#,+/stx #,#reduced)))]
[_
;; plus on its own is +, but we want our one. I am not sure this is right
+/stx]))
It is possible to do this even more aggressively in fact, so that (plus a b 1 2 c 3) is turned into (+ a b c 6). This has probably even more exciting might-get-different answers implications. It's worth noting what the CL spec says about this:
For functions that are mathematically associative (and possibly commutative), a conforming implementation may process the arguments in any manner consistent with associative (and possibly commutative) rearrangement. This does not affect the order in which the argument forms are evaluated [...]. What is unspecified is only the order in which the parameter values are processed. This implies that implementations may differ in which automatic coercions are applied [...].
So an optimisation like this is clearly legal in CL: I'm not clear that it's legal in Racket (although I think it should be).
(require (for-syntax racket/list))
(define-for-syntax (split-literals syntaxes)
;; split a list into literal numbers and the rest
(let sloop ([tail syntaxes]
[accum/lit '()]
[accum/nonlit '()])
(if (null? tail)
(values (reverse accum/lit) (reverse accum/nonlit))
(let ([current (first tail)])
(if (number? (syntax-e current))
(sloop (rest tail)
(cons (syntax-e current) accum/lit)
accum/nonlit)
(sloop (rest tail)
accum/lit
(cons current accum/nonlit)))))))
(define-syntax (plus stx)
(define +/stx (datum->syntax stx +))
(syntax-case stx ()
[(_)
;; return additive identity
#'0]
[(_ a)
;; identity with one argument
#'a]
[(_ a ...)
;; the interesting case: there's more than one argument: split the
;; arguments into literals and nonliterals and handle approprately
(let-values ([(literals nonliterals)
(split-literals (syntax->list #'(a ...)))])
(if (null? literals)
(if (null? nonliterals)
#'0
#`(#,+/stx #,#nonliterals))
(let ([sum/stx (datum->syntax stx (apply + literals))])
(if (null? nonliterals)
sum/stx
#`(#,+/stx #,#nonliterals #,sum/stx)))))]
[_
;; plus on its own is +, but we want our one. I am not sure this is right
+/stx]))
Related
I'm currently reading through Seibel's "Practical common lisp" and found this example macro:
(defmacro check (&rest forms)
`(progn
,#(loop for f in forms collect `(do-stuff ,f ',f))
(defun test ()
(check ( (= (+ 1 2 ) 3) (= (+ 1 2 ) 4 )))
)
do-stuff simply then format the two args in a certain way allowing for 'testing' the truth of a form, but that's not important for my question.
What I was interested in was to translate the loop into a DO, unfortunately, I'm totally lacking in the necessary skill to do so:
(defmacro check (&rest forms)
`(progn
,#(do ((index 0 (list-length forms))
(accumulator nil))
((= index (list-length forms)) accumulator)
(push `(do-stuff ,(nth index forms) ',(nth index forms)) accumulator)
))
This does the job, I can also do this (put every form into a variable inside the do):
(defmacro check (&rest forms)
`(progn
,#(do* ((index 0 (list-length forms))
(accumulator nil)
(f (nth index forms) (nth index forms)))
((= index (list-length forms)) accumulator)
(push `(do-stuff ,f ',f) accumulator)
))
My problem is the following :
Is there a more efficient way to write this do loop ? Is this a good way to implement it ?
Something in the LOOP version is making me wonder if there is not a simple way to extract an element of a list without the need to define an index variable, or to regroup the COLLECTED elements without the need to define an accumulator list...
If you use do you shouldn't use nth. Just iterate over the list, not the indexes.
(do ((l forms (cdr l))
(accumulator nil))
((null l) (nreverse accumulator))
(let ((f (car l)))
(push `(do-stuff ,f ',f) accumulator)))
You can also use the built-in dolist:
(let ((accumulator nil))
(dolist (f forms (nreverse accumulator))
(push `(do-stuff ,f ',f) accumulator)))
Finally there's mapcar:
(mapcar (lambda (f) `(do-stuff ,f ',f)) forms)
Is there a more efficient way to write this do loop ? Is this a good way to implement it ?
The complexity of your code is quadratic to the size N of the list, since for each item you call nth to access an element inside, resulting in a O(N*N) execution time. There is a more efficient way to do it (the original LOOP version is an example of a linear algorithm).
Here is a different version where instead of calling push followed by nreverse, the items are queued at the end of the list during traversal. I added comments to explain what each part does.
By the way I don't claim that this is more efficient that using nreverse, I think we can't know without testing. Note however that there are as many operations in both cases (cons a new item, and eventually mutate the cdr slot), they are just done either in two passes or one pass.
In fact the code below is very not far from being an implementation of MAPCAR where there is only one list to traverse (not the variadic version in the standard).
First, define a helper function that transforms one form:
(defun expand-check (form)
`(do-stuff ,form ',form))
Recall that you could just (mapcar #'expand-check checks) to have the desired result.
Anyway, here is a DO version:
(defun expand-checks (checks)
;; LIST-HOLDER is just a temporary cons-cell that allows us to treat
;; the rest of the queue operations without having to worry about
;; the corner case of the first item (the invariant is that LAST is
;; always a cons-cell, never NIL). Here LIST-HOLDER is initially
;; (:HANDLE), the first value being discarded later.
(let ((list-holder (list :handle)))
;; DO is sufficient because the iterator values are independant
;; from each other (no need to use DO*).
(do (;; descend the input list
(list checks (cdr list))
;; update LAST so that it is always the last cons cell, this
;; is how we can QUEUE items at the end of the list without
;; traversing it. This queue implementation was first
;; described by Peter Norvig as far as I known.
(last list-holder (cdr last)))
;; End iteration when LIST is empty
((null list)
;; In which case, return the rest of the LIST-HOLDER, which
;; is the start of the list that was built.
(rest list-holder))
;; BODY of the DO, create a new cons-cell at the end of the
;; queue by mutating the LAST const cell.
(setf (cdr last)
(list (expand-check
(first list)))))))
Firstly, anything of the form
(loop for v in <list> collect (f v ...))
Can be easily expressed as mapcar:
(mapcar (lambda (v)
(f v ...))
<list>)
The interesting case is when the loop only collects a value sometimes, or when the iteration is over some more complicated thing.
In that case one nice approach is to factor out the iteration bit and the 'collecting values' bit, using do or whatever to perform the iteration and some other mechanism to collect values.
One such is collecting. So, for instance, you could use dolist to iterate over the list and collecting to collect values. And perhaps we might only want to collect non-nil values or something to make it more interesting:
(collecting
(dolist (v <list>)
(when v
(collect (f v ...)))))
All of these are more verbose than the simple loop case, but for instance collecting can do things which are painful to express with loop.
I'm just building up a function at the REPL and ran into this.
I define a symbol S and give it a value:
(def S '(FRUIT COLORS (YELLOW GREEN) SKIN (EDIBLE INEDIBLE)))
I want, eventually, a function that takes the first entry in the parameter list and any and all subsequent parameter pairs and applies them to the first entry. I never got that far in my coding. I want to use a loop / recur construct (should I?), and here's how far I got in the REPL:
(loop [KV# (rest S)]
(if (empty? KV#)
nil
(
(pprint S, (first KV#), (second KV#))
(recur (rest (rest KV#)))
)
)
)
I get a "can only recur from tail position" compiler error.
After looking everywhere about this including 7 or 8 articles in Stack Overflow, I can only ask: Huh?!
I'm new at this. If recur isn't in the tail position, could someone please explain to me why?Something to do with 'if' statement syntax? GAHH! Clojure's not for the weak! Thank you.
You've made one of my favorite mistakes in Clojure - you've tried to use parentheses to group code. You need to use a (do ...) form to group forms together, as in:
(loop [KV# (rest S)]
(if (empty? KV#)
nil ; then
(do ; else
(pprint S, (first KV#), (second KV#))
(recur (rest (rest KV#)))
)
)
)
This gets rid of the "recur not in tail position" problem, but still fails - an arity exception on pprint - but I'll leave that for you to solve.
How did I spot this? My rule is that any time I find two left-parens together I immediately assume I've made a mistake and I need to figure out what I did wrong. In this case it was a little harder to spot because the left-parens were separated by intervening white space - but still, from the view of the lexical scanner they're adjacent to one another. So you just have to learn to think like a lexical scanner. :-)
SBCL compiler optimizations are based on the idea that if a type is declared, then "open coding" allows generic operations to be replaced with specific ones.
For example
(defun add (a b)
(declare (type fixnum a b))
(+ a b))
Will allow the generic + to be replaced with a single instruction for fixnum.
However, I have found that in practice, this seems to rarely be possible because:
In order for a function to be specialized/optimized it must be inlinable. The declaration must be marked explicitly with a (declaim (inline ...)), so the author of a function must anticipate that others might want to inline it. (In theory the compiler could generate multiple versions, but this doesn't seem to be the case.)
Most standard functions do not appear inlineable.
For example, one would expect that the following declaration is sufficient for open coding to take place:
(defun max-integers (array)
(declare (optimize (speed 3) (space 0) (safety 0)))
(declare (inline reduce))
(declare (type (simple-array fixnum (*)) array))
(reduce (lambda (a b) (if (> b a) b a)) array))
However, the assembly shows it's making a function call to the generic reduce:
; Size: 22 bytes. Origin: #x1001BC8109
; 09: 488B15B0FFFFFF MOV RDX, [RIP-80] ; no-arg-parsing entry point
; #<FUNCTION (LAMBDA
; # ..)>
; 10: B904000000 MOV ECX, 4
; 15: FF7508 PUSH QWORD PTR [RBP+8]
; 18: B8781C3220 MOV EAX, #x20321C78 ; #<FDEFN REDUCE>
; 1D: FFE0 JMP RAX
The conclusion seems to be that the compiler cannot actually do much type optimization, as each usage of reduce, map, etc is a barrier to type propagation, and they are building blocks of everything else.
How can I overcome this and take advantage of optimizations by declaring types?
I really want to avoid writing type specific versions of each function or "macroifying" what should be a function.
I think one answer is that if you want to write FORTRAN-style array-bashing code, write FORTRAN-style array-bashing code. In particular using things like reduce is probably not the way to do this.
For instance if you change your function to the perfectly readable
(defun max-integers/loop (array)
(declare (optimize (speed 3) (space 0) (safety 0))
(type (simple-array fixnum (*)) array))
(loop for i of-type fixnum across array
maximizing i))
Then SBCL does a far, far better job of optimising it.
It's worth pointing out another confusion in your question: You say that for something like
(defun add (a b)
(declare (type fixnum a b))
(+ a b))
SBCL will optimize + to the machine instruction. No, it won't. The reason it won't is because the fixnum type is not closed under addition: consider what (add most-positive-fixnum 1) should do. If you want to generate very fast code for integers you need to make sure that your integer types are small enough that the compiler can be sure that the operations you're doing on them remain machine integers (or, if you want to live dangerously, cover your code with (the fixnum ...) and set safety to 0 when compiling, which seems to allow the compiler to just return the wrong answer for addition in the way people usually expect computers to do).
You can't force the implementation to open-code functions that weren't declared INLINE when they were defind -- it simply hasn't saved the information needed.
However, the overhead of calling REDUCE is probably negligible compared to the actual processing. So what you can do is declare the types of a and b, to optimize the callback function.
(reduce (lambda (a b) (declare (type fixnum a b)) (if (> b a) b a)) array)
I guess you were hoping that if it open-coded reduce it would automatically propagate this type from the declaration of array, so you wouldn't need to do this.
How you can have a different behaviour if a variable is defined or not in racket language?
There are several ways to do this. But I suspect that none of these is what you want, so I'll only provide pointers to the functions (and explain the problems with each one):
namespace-variable-value is a function that retrieves the value of a toplevel variable from some namespace. This is useful only with REPL interaction and REPL code though, since code that is defined in a module is not going to use these things anyway. In other words, you can use this function (and the corresponding namespace-set-variable-value!) to get values (if any) and set them, but the only use of these values is in code that is not itself in a module. To put this differently, using this facility is as good as keeping a hash table that maps symbols to values, only it's slightly more convenient at the REPL since you just type names...
More likely, these kind of things are done in macros. The first way to do this is to use the special #%top macro. This macro gets inserted automatically for all names in a module that are not known to be bound. The usual thing that this macro does is throw an error, but you can redefine it in your code (or make up your own language that redefines it) that does something else with these unknown names.
A slightly more sophisticated way to do this is to use the identifier-binding function -- again, in a macro, not at runtime -- and use it to get information about some name that is given to the macro and decide what to expand to based on that name.
The last two options are the more useful ones, but they're not the newbie-level kind of macros, which is why I suspect that you're asking the wrong question. To clarify, you can use them to write a kind of a defined? special form that checks whether some name is defined, but that question is one that would be answered by a macro, based on the rest of the code, so it's not really useful to ask it. If you want something like that that can enable the kind of code in other dynamic languages where you use such a predicate, then the best way to go about this is to redefine #%top to do some kind of a lookup (hashtable or global namespace) instead of throwing a compilation error -- but again, the difference between that and using a hash table explicitly is mostly cosmetic (and again, this is not a newbie thing).
First, read Eli's answer. Then, based on Eli's answer, you can implement the defined? macro this way:
#lang racket
; The macro
(define-syntax (defined? stx)
(syntax-case stx ()
[(_ id)
(with-syntax ([v (identifier-binding #'id)])
#''v)]))
; Tests
(define x 3)
(if (defined? x) 'defined 'not-defined) ; -> defined
(let ([y 4])
(if (defined? y) 'defined 'not-defined)) ; -> defined
(if (defined? z) 'defined 'not-defined) ; -> not-defined
It works for this basic case, but it has a problem: if z is undefined, the branch of the if that considers that it is defined and uses its value will raise a compile-time error, because the normal if checks its condition value at run-time (dynamically):
; This doesn't work because z in `(list z)' is undefined:
(if (defined? z) (list z) 'not-defined)
So what you probably want is a if-defined macro, that tells at compile-time (instead of at run-time) what branch of the if to take:
#lang racket
; The macro
(define-syntax (if-defined stx)
(syntax-case stx ()
[(_ id iftrue iffalse)
(let ([where (identifier-binding #'id)])
(if where #'iftrue #'iffalse))]))
; Tests
(if-defined z (list z) 'not-defined) ; -> not-defined
(if-defined t (void) (define t 5))
t ; -> 5
(define x 3)
(if-defined x (void) (define x 6))
x ; -> 3
It's a Project Euler problem .
I learned from Fastest way to list all primes below N
and implemented a clojure :
(defn get-primes [n]
(loop [numbers (set (range 2 n))
primes []]
(let [item (first numbers)]
(cond
(empty? numbers)
primes
:else
(recur (clojure.set/difference numbers (set (range item n item)))
(conj primes item))))))
used like follows:
(reduce + (get-primes 2000000))
but It is so slow..
I am wondering why, Can someone enlighten me?
This algorithm is not even correct: at each iteration except the final one it adds the value of (first numbers) at that point to primes, but there is no guarantee that it will in fact be a prime, since the set data structure in use is unordered. (This is also true of the Python original, as mentioned by its author in an edit to the question you link to.) So, you'd first need to fix it by changing (set (range ...)) to (into (sorted-set) (range ...)).
Even then, this is simply not a great algorithm for finding primes. To do better, you may want to write an imperative implementation of the Sieve of Eratosthenes using a Java array and loop / recur, or maybe a functional SoE-like algorithm such as those described in Melissa E. O'Neill's beautiful paper The Genuine Sieve of Eratosthenes.