I am trying to learn how to make type declarations in lisp. I figured out that aref causes problems:
(defun getref (seq k)
(declare (optimize (speed 3) (safety 0)))
(declare (type (vector fixnum *) seq) (type fixnum k))
(aref seq k))
Compiled, it says:
; in: DEFUN GETREF
; (AREF MORE-LISP::SEQ MORE-LISP::K)
; ==>
; (SB-KERNEL:HAIRY-DATA-VECTOR-REF ARRAY SB-INT:INDEX)
;
; note: unable to
; avoid runtime dispatch on array element type
; due to type uncertainty:
; The first argument is a (VECTOR FIXNUM), not a SIMPLE-ARRAY.
;
; compilation unit finished
; printed 1 note
And so in every other function, where I want to use aref (and I do, since I need adjustable vectors), this happens too. How do I fix it?
It's not a problem and not an error. It just an information (a note) from the SBCL compiler that it can't optimize the code better. The code will work just fine. You can safely ignore it.
If you can't use a simple vector (a one-dimensional simple array), then this is the price to pay for it: aref might be slightly slower.
The optimization hint you get comes from the docstring of a deftransform defined in sbcl/src/compiler/generic/vm-tran.lisp:
(deftransform hairy-data-vector-ref ((array index) (simple-array t) *)
"avoid runtime dispatch on array element type"
...)
It has a comment which says:
This and the corresponding -SET transform work equally well on non-simple
arrays, but after benchmarking (on x86), Nikodemus didn't find any cases
where it actually helped with non-simple arrays -- to the contrary, it
only made for bigger and up to 100% slower code.
The code for arrays is quite complex and it is hard to say why and how things are designed as they are. You should probably ask SBCL developers on sbcl-help. See the mailing lists section on
Sourceforge.
Currently it seems preferable to favor simple arrays if possible.
Related
SBCL compiler optimizations are based on the idea that if a type is declared, then "open coding" allows generic operations to be replaced with specific ones.
For example
(defun add (a b)
(declare (type fixnum a b))
(+ a b))
Will allow the generic + to be replaced with a single instruction for fixnum.
However, I have found that in practice, this seems to rarely be possible because:
In order for a function to be specialized/optimized it must be inlinable. The declaration must be marked explicitly with a (declaim (inline ...)), so the author of a function must anticipate that others might want to inline it. (In theory the compiler could generate multiple versions, but this doesn't seem to be the case.)
Most standard functions do not appear inlineable.
For example, one would expect that the following declaration is sufficient for open coding to take place:
(defun max-integers (array)
(declare (optimize (speed 3) (space 0) (safety 0)))
(declare (inline reduce))
(declare (type (simple-array fixnum (*)) array))
(reduce (lambda (a b) (if (> b a) b a)) array))
However, the assembly shows it's making a function call to the generic reduce:
; Size: 22 bytes. Origin: #x1001BC8109
; 09: 488B15B0FFFFFF MOV RDX, [RIP-80] ; no-arg-parsing entry point
; #<FUNCTION (LAMBDA
; # ..)>
; 10: B904000000 MOV ECX, 4
; 15: FF7508 PUSH QWORD PTR [RBP+8]
; 18: B8781C3220 MOV EAX, #x20321C78 ; #<FDEFN REDUCE>
; 1D: FFE0 JMP RAX
The conclusion seems to be that the compiler cannot actually do much type optimization, as each usage of reduce, map, etc is a barrier to type propagation, and they are building blocks of everything else.
How can I overcome this and take advantage of optimizations by declaring types?
I really want to avoid writing type specific versions of each function or "macroifying" what should be a function.
I think one answer is that if you want to write FORTRAN-style array-bashing code, write FORTRAN-style array-bashing code. In particular using things like reduce is probably not the way to do this.
For instance if you change your function to the perfectly readable
(defun max-integers/loop (array)
(declare (optimize (speed 3) (space 0) (safety 0))
(type (simple-array fixnum (*)) array))
(loop for i of-type fixnum across array
maximizing i))
Then SBCL does a far, far better job of optimising it.
It's worth pointing out another confusion in your question: You say that for something like
(defun add (a b)
(declare (type fixnum a b))
(+ a b))
SBCL will optimize + to the machine instruction. No, it won't. The reason it won't is because the fixnum type is not closed under addition: consider what (add most-positive-fixnum 1) should do. If you want to generate very fast code for integers you need to make sure that your integer types are small enough that the compiler can be sure that the operations you're doing on them remain machine integers (or, if you want to live dangerously, cover your code with (the fixnum ...) and set safety to 0 when compiling, which seems to allow the compiler to just return the wrong answer for addition in the way people usually expect computers to do).
You can't force the implementation to open-code functions that weren't declared INLINE when they were defind -- it simply hasn't saved the information needed.
However, the overhead of calling REDUCE is probably negligible compared to the actual processing. So what you can do is declare the types of a and b, to optimize the callback function.
(reduce (lambda (a b) (declare (type fixnum a b)) (if (> b a) b a)) array)
I guess you were hoping that if it open-coded reduce it would automatically propagate this type from the declaration of array, so you wouldn't need to do this.
I've been reading through https://lispcast.com/when-to-use-a-macro, and it states (about clojure's macros)
Another example is performing expensive calculations at compile time as an optimization
I looked up, and it seems clojure has unhygienic macros. Can this also be applied to hygienic ones? Particularly talking about Scheme. As far as I understand hygienic macros, they only transform syntax, but the actual execution of code is deferred until the runtime no matter what.
Yes. Macro hygiene just refers to whether or not macro expansion can accidentally capture identifiers. Whether or not a macro is hygienic, regular macro expansion (as opposed to reader macro expansion) occurs at compile-time. Macro expansion replaces the macro's code with the results of it being executed. Two major use cases for them are to transform syntax (i.e. DSLs), to enhance performance by eliminating computations at run time or both.
A few examples come to mind:
You prefer to write your code with angles in degrees but all of the calculations are actually in radians. You could have macros eliminate these trivial, but unnecessary (at run time) conversions, at compile time.
Memoization is a broad example of compute optimization that macros can be used for.
You have a string representing a SQL statement or complex textual math expression which you want to parse and possibly even execute at compile time.
You could also combine the examples and have a memoizing SQL parser. Pretty much any scenario where you have all the necessary inputs at compile time and can therefore compute the result is a candidate.
Yes, hygienic macros can do this sort of thing. As an example here is a macro called plus in Racket which is like + except that, at macroexpansion-time, it sums sequences of adjacent literal numbers. So it does some of the work you might expect to be done at run-time at macroexpansion-time (so, effectively, at compile-time). So, for instance
(plus a b 1 2 3 c 4 5)
expands to
(+ a b 6 c 9)
Some notes on this macro.
It's probably not very idiomatic Racket, because I'm a mostly-unreformed CL hacker, which means I live in a cave and wear animal skins and say 'ug' a lot. In particular I am sure I should use syntax-parse but I can't understand it.
It might not even be right.
There are subtleties with arithmetic which means that this macro can return different results than +. In particular + is defined to add pairwise from left to right, while plus does not in general: all the literals get added firsto in particular (assuming you have done (require racket/flonum, and +max.0 &c have the same values as they do on my machine), then (+ -max.0 1.7976931348623157e+308 1.7976931348623157e+308) has a value of 1.7976931348623157e+308, while (plus -max.0 1.7976931348623157e+308 1.7976931348623157e+308) has a value of +inf.0, because the two literals get added first and this overflows.
In general this is a useless thing: it's safe to assume, I think, that any reasonable compiler will do these kind of optimisations for you. The only purpose of it is to show that it's possible to do the detect-and-compile-away compile-time constants.
Remarkably, at least from the point of view of caveman-lisp users like me, you can treat this just like + because of the last in the syntax-case: it works to say (apply plus ...) for instance (although no clever optimisation happens in that case of course).
Here it is:
(require (for-syntax racket/list))
(define-syntax (plus stx)
(define +/stx (datum->syntax stx +))
(syntax-case stx ()
[(_)
;; return additive identity
#'0]
[(_ a)
;; identity with one argument
#'a]
[(_ a ...)
;; the interesting case: there's more than one argument, so walk over them
;; looking for literal numbers. This is probably overcomplicated and
;; unidiomatic
(let* ([syntaxes (syntax->list #'(a ...))]
[reduced (let rloop ([current (first syntaxes)]
[tail (rest syntaxes)]
[accum '()])
(cond
[(null? tail)
(reverse (cons current accum))]
[(and (number? (syntax-e current))
(number? (syntax-e (first tail))))
(rloop (datum->syntax stx
(+ (syntax-e current)
(syntax-e (first tail))))
(rest tail)
accum)]
[else
(rloop (first tail)
(rest tail)
(cons current accum))]))])
(if (= (length reduced) 1)
(first reduced)
;; make sure the operation is our +
#`(#,+/stx #,#reduced)))]
[_
;; plus on its own is +, but we want our one. I am not sure this is right
+/stx]))
It is possible to do this even more aggressively in fact, so that (plus a b 1 2 c 3) is turned into (+ a b c 6). This has probably even more exciting might-get-different answers implications. It's worth noting what the CL spec says about this:
For functions that are mathematically associative (and possibly commutative), a conforming implementation may process the arguments in any manner consistent with associative (and possibly commutative) rearrangement. This does not affect the order in which the argument forms are evaluated [...]. What is unspecified is only the order in which the parameter values are processed. This implies that implementations may differ in which automatic coercions are applied [...].
So an optimisation like this is clearly legal in CL: I'm not clear that it's legal in Racket (although I think it should be).
(require (for-syntax racket/list))
(define-for-syntax (split-literals syntaxes)
;; split a list into literal numbers and the rest
(let sloop ([tail syntaxes]
[accum/lit '()]
[accum/nonlit '()])
(if (null? tail)
(values (reverse accum/lit) (reverse accum/nonlit))
(let ([current (first tail)])
(if (number? (syntax-e current))
(sloop (rest tail)
(cons (syntax-e current) accum/lit)
accum/nonlit)
(sloop (rest tail)
accum/lit
(cons current accum/nonlit)))))))
(define-syntax (plus stx)
(define +/stx (datum->syntax stx +))
(syntax-case stx ()
[(_)
;; return additive identity
#'0]
[(_ a)
;; identity with one argument
#'a]
[(_ a ...)
;; the interesting case: there's more than one argument: split the
;; arguments into literals and nonliterals and handle approprately
(let-values ([(literals nonliterals)
(split-literals (syntax->list #'(a ...)))])
(if (null? literals)
(if (null? nonliterals)
#'0
#`(#,+/stx #,#nonliterals))
(let ([sum/stx (datum->syntax stx (apply + literals))])
(if (null? nonliterals)
sum/stx
#`(#,+/stx #,#nonliterals #,sum/stx)))))]
[_
;; plus on its own is +, but we want our one. I am not sure this is right
+/stx]))
I use SBCL (64-bit v1.4.0) for numerical calculation.
After enabling optimization, following compiler note appears:
note: doing float to pointer coercion (cost 13) to "<return value>"
The code I use is as follows:
(defun add (a b)
(declare (optimize (speed 3) (safety 0)))
(declare (double-float a b))
(the double-float (+ a b)))
I've also tried ftype and got the same note.
On the other hand, following code doesn't show the note:
(defun add-fixnum (a b)
(declare (optimize (speed 3) (safety 0)))
(declare (fixnum a b))
(the fixnum (+ a b)))
I think double-float and fixnum are both 64 bits wide.
Why can not SBCL return a double-float value via a register like C language? And are there any way to avoid float to pointer coercion without inline expansion?
The problem is that Lisp data is dynamically typed, and the return value of a function has to include the type information. The type tag in most implementations is stored in the low-order bits of a value.
This allows a special optimization for fixnums. Their type tag is all zeroes, and the value is the integer shifted left by the number of bits in the type tag. When you add these values, the result still has zeroes in the tag bits, so you can perform arithmetic on the values using normal CPU operations.
But this doesn't work for floating point values. After performing the CPU operations, it has to add the type tag to the value. This is what it means by "float to pointer coercion" (a more common word for it in many languages is "boxing").
Declaring the return type doesn't avoid this, because the callers don't necessarily have access to the declarations -- Lisp allows you to compile the callers in a separate compilation unit than the functions they call.
If you declare the function INLINE, then this doesn't need to be done, because the callers know the type it's returning, and the hardware value can be returned directly to them without adding the tag.
A more detailed explanation can be found in this ancient comp.lang.lisp thread. It's referring to CMUCL, which is what SBCL is derived from (notice that the wording of the warning is exactly the same).
How you can have a different behaviour if a variable is defined or not in racket language?
There are several ways to do this. But I suspect that none of these is what you want, so I'll only provide pointers to the functions (and explain the problems with each one):
namespace-variable-value is a function that retrieves the value of a toplevel variable from some namespace. This is useful only with REPL interaction and REPL code though, since code that is defined in a module is not going to use these things anyway. In other words, you can use this function (and the corresponding namespace-set-variable-value!) to get values (if any) and set them, but the only use of these values is in code that is not itself in a module. To put this differently, using this facility is as good as keeping a hash table that maps symbols to values, only it's slightly more convenient at the REPL since you just type names...
More likely, these kind of things are done in macros. The first way to do this is to use the special #%top macro. This macro gets inserted automatically for all names in a module that are not known to be bound. The usual thing that this macro does is throw an error, but you can redefine it in your code (or make up your own language that redefines it) that does something else with these unknown names.
A slightly more sophisticated way to do this is to use the identifier-binding function -- again, in a macro, not at runtime -- and use it to get information about some name that is given to the macro and decide what to expand to based on that name.
The last two options are the more useful ones, but they're not the newbie-level kind of macros, which is why I suspect that you're asking the wrong question. To clarify, you can use them to write a kind of a defined? special form that checks whether some name is defined, but that question is one that would be answered by a macro, based on the rest of the code, so it's not really useful to ask it. If you want something like that that can enable the kind of code in other dynamic languages where you use such a predicate, then the best way to go about this is to redefine #%top to do some kind of a lookup (hashtable or global namespace) instead of throwing a compilation error -- but again, the difference between that and using a hash table explicitly is mostly cosmetic (and again, this is not a newbie thing).
First, read Eli's answer. Then, based on Eli's answer, you can implement the defined? macro this way:
#lang racket
; The macro
(define-syntax (defined? stx)
(syntax-case stx ()
[(_ id)
(with-syntax ([v (identifier-binding #'id)])
#''v)]))
; Tests
(define x 3)
(if (defined? x) 'defined 'not-defined) ; -> defined
(let ([y 4])
(if (defined? y) 'defined 'not-defined)) ; -> defined
(if (defined? z) 'defined 'not-defined) ; -> not-defined
It works for this basic case, but it has a problem: if z is undefined, the branch of the if that considers that it is defined and uses its value will raise a compile-time error, because the normal if checks its condition value at run-time (dynamically):
; This doesn't work because z in `(list z)' is undefined:
(if (defined? z) (list z) 'not-defined)
So what you probably want is a if-defined macro, that tells at compile-time (instead of at run-time) what branch of the if to take:
#lang racket
; The macro
(define-syntax (if-defined stx)
(syntax-case stx ()
[(_ id iftrue iffalse)
(let ([where (identifier-binding #'id)])
(if where #'iftrue #'iffalse))]))
; Tests
(if-defined z (list z) 'not-defined) ; -> not-defined
(if-defined t (void) (define t 5))
t ; -> 5
(define x 3)
(if-defined x (void) (define x 6))
x ; -> 3
I started learning Clojure recently. Generally it looks interesting, but I can't get used to some syntactic inconveniences (comparing to previous Ruby/C# experience).
Prefix notation for nested expressions. In Ruby I get used to write complex expressions with chaining/piping them left-to-right: some_object.map { some_expression }.select { another_expression }. It's really convenient as you move from input value to result step-by-step, you can focus on a single transformation and you don't need to move cursor as you type. Contrary to that when I writing nested expressions in Clojure, I write the code from inner expression to outer and I have to move cursor constantly. It slows down and distracts. I know about -> and ->> macros but I noticed that it's not an idiomatic. Did you have the same problem when you started coding in Clojure/Haskell etc? How did you solve it?
I felt the same about Lisps initially so I feel your pain :-)
However the good news is that you'll find that with a bit of time and regular usage you will probably start to like prefix notation. In fact with the exception of mathematical expressions I now prefer it to infix style.
Reasons to like prefix notation:
Consistency with functions - most languages use a mix of infix (mathematical operators) and prefix (functional call) notation . In Lisps it is all consistent which has a certain elegance if you consider mathematical operators to be functions
Macros - become much more sane if the function call is always in the first position.
Varargs - it's nice to be able to have a variable number of parameters for pretty much all of your operators. (+ 1 2 3 4 5) is nicer IMHO than 1 + 2 + 3 + 4 + 5
A trick then is to use -> and ->> librerally when it makes logical sense to structure your code this way. This is typically useful when dealing with subsequent operations on objects or collections, e.g.
(->>
"Hello World"
distinct
sort
(take 3))
==> (\space \H \W)
The final trick I found very useful when working in prefix style is to make good use of indentation when building more complex expressions. If you indent properly, then you'll find that prefix notation is actually quite clear to read:
(defn add-foobars [x y]
(+
(bar x y)
(foo y)
(foo x)))
To my knowledge -> and ->> are idiomatic in Clojure. I use them all the time, and in my opinion they usually lead to much more readable code.
Here are some examples of these macros being used in popular projects from around the Clojure "ecosystem":
Ring cookie parsing
Leiningen internals
ClojureScript compiler
Proof by example :)
If you have a long expression chain, use let. Long runaway expressions or deeply nested expressions are not especially readable in any language. This is bad:
(do-something (map :id (filter #(> (:age %) 19) (fetch-data :people))))
This is marginally better:
(do-something (map :id
(filter #(> (:age %) 19)
(fetch-data :people))))
But this is also bad:
fetch_data(:people).select{|x| x.age > 19}.map{|x| x.id}.do_something
If we're reading this, what do we need to know? We're calling do_something on some attributes of some subset of people. This code is hard to read because there's so much distance between first and last, that we forget what we're looking at by the time we travel between them.
In the case of Ruby, do_something (or whatever is producing our final result) is lost way at the end of the line, so it's hard to tell what we're doing to our people. In the case of Clojure, it's immediately obvious that do-something is what we're doing, but it's hard to tell what we're doing it to without reading through the whole thing to the inside.
Any code more complex than this simple example is going to become pretty painful. If all of your code looks like this, your neck is going to get tired scanning back and forth across all of these lines of spaghetti.
I'd prefer something like this:
(let [people (fetch-data :people)
adults (filter #(> (:age %) 19) people)
ids (map :id adults)]
(do-something ids))
Now it's obvious: I start with people, I goof around, and then I do-something to them.
And you might get away with this:
fetch_data(:people).select{|x|
x.age > 19
}.map{|x|
x.id
}.do_something
But I'd probably rather do this, at the very least:
adults = fetch_data(:people).select{|x| x.age > 19}
do_something( adults.map{|x| x.id} )
It's also not unheard of to use let even when your intermediary expressions don't have good names. (This style is occasionally used in Clojure's own source code, e.g. the source code for defmacro)
(let [x (complex-expr-1 x)
x (complex-expr-2 x)
x (complex-expr-3 x)
...
x (complex-expr-n x)]
(do-something x))
This can be a big help in debugging, because you can inspect things at any point by doing:
(let [x (complex-expr-1 x)
x (complex-expr-2 x)
_ (prn x)
x (complex-expr-3 x)
...
x (complex-expr-n x)]
(do-something x))
I did indeed see the same hurdle when I first started with a lisp and it was really annoying until I saw the ways it makes code simpler and more clear, once I understood the upside the annoyance faded
initial + scale + offset
became
(+ initial scale offset)
and then try (+) prefix notation allows functions to specify their own identity values
user> (*)
1
user> (+)
0
There are lots more examples and my point is NOT to defend prefix notation. I just hope to convey that the learning curve flattens (emotionally) as the positive sides become apparent.
of course when you start writing macros then prefix notation becomes a must-have instead of a convenience.
to address the second part of your question, the thread first and thread last macros are idiomatic anytime they make the code more clear :) they are more often used in functions calls than pure arithmetic though nobody will fault you for using them when they make the equation more palatable.
ps: (.. object object2 object3) -> object().object2().object3();
(doto my-object
(setX 4)
(sety 5)`