EDIT: I changed the example code after the first answer because I came up with a simple version that begs the same questions.
I am currently learning Common Lisp's scoping properties. After I thought I had a solid understanding I decided to code up some examples that I could predict the outcome of, but apparently I was wrong. I have three question, each one relating to an example below:
Example 1:
(defmethod fun1 (x)
(print x)
(fun2))
(defmethod fun2 ()
(print x))
(fun1 5)
Output:
5
*** - EVAL: variable X has no value
Question: This makes sense. x is statically scoped and fun2 has no way of finding the value of x without having it passed explicitly.
Example 2:
(defvar x 100)
(defmethod fun1 (x)
(print x)
(fun2))
(defmethod fun2 ()
(print x))
(fun1 5)
Output:
5
5
Question: I don't understand why x is suddenly visible to fun2 with the value that fun1 gave it, instead of having a value of 100...
Example 3:
(setf x 100)
(defmethod fun1 (x)
(print x)
(fun2))
(defmethod fun2 ()
(print x))
(fun1 5)
Output:
5
100
Question: Should I ignore these results since calling setf on an undeclared variable is apparently undefined? This happens to be what I would expect in my second example...
Any insight would be greatly appreciated...
The effects of setting an undefined variable using setf is undefined in ANSI Common Lisp.
defvar will define a special variable. This declaration is global and also has effect on let bindings. That's the reason that by convention these variables are written as *foo*. If you have ever defined x with defvar, it is declared special and there is no way to declare it lexical later.
let by default provides local lexical variables. If the variable was already declared special (for example because of a defvar), then it just creates a new local dynamic binding.
Update
Example 1 .
Nothing to see.
Example 2
x has been declared special. All uses of the variable x now use dynamic binding.
When calling the function, you bind x to 5. Dynamically. Other functions can now access this dynamic binding and get that value.
Example 3
This is undefined behavior in Common Lisp. You are setting an undeclared variable. What happens then is implementation dependent. Your implementation (most do something similar) sets the symbol value of x to 100. In fun1, x is lexically bound. In fun2 evaluating x retrieves the symbol value (or possibly to the dynamically bound value) of x.
As an example for an implementation that did (does?) something else: the CMUCL implementation would also have declare x in example 3 by default to be special. Setting an undefined variable also declares it special.
NOTE
In portable standard compliant Common Lisp code the global variables are defined with defvar and defparameter. Both declare these variables to be special. ALL uses of these variables now involve dynamic binding.
Remember:
((lambda (x)
(sin x))
10)
is basically the same as
(let ((x 10))
(sin x))
Which means that variable bindings in let bindings and variable bindings in function calls are working the same way. If x would have been declared special in some place earlier, both would involve dynamic binding.
This is specified in the Common Lisp standard. See for example the explanation to the SPECIAL declaration.
Related
I am reading the Wikipedia article about CLOS.
It says that:
This dispatch mechanism works at runtime. Adding or removing methods thus may lead to changed effective methods (even when the generic function is called with the same arguments) at runtime. Changing the method combination also may lead to different effective methods.
Then, I inserted:
; declare the common argument structure prototype
(defgeneric f (x y))
; define an implementation for (f integer t), where t matches all types
(defmethod f ((x integer) y) 1)
Using SBCL and SLIME, I compiled the regions with code and had the following result:
CL-USER> (f 1 2)
1
Then, I added to the definition:
; define an implementation for (f integer real)
(defmethod f ((x integer) (y real)) 2)
Again, I repeated the process compiling the new region and using the REPL to eval:
CL-USER> (f 1 2.0)
2
First question, if CLOS had the opposite behavior of run-time dispatch (compile-time dispatch, I suppose), what would the result be?
Second question, I decided to comment out the second method, leaving just the generic function and the first written method. Then, I re-compiled the region with Emacs.
When calling the fuction fin the REPL with (f 1 2) I thought I would get 1 since the second method is out. Instead, I got 2.
CL-USER> (f 1 2.0)
2
Why did this happen?
The only way I can get back to (f 1 2) returning 1 is re-starting the Slime REPL and compiling the region (with the second method being commented out). Third question, is there a better way to have this result without having to re-start the REPL?
Thanks!
These two are actually the same question. The answer is: you are modifying a system while it is running.
If CLOS objects weren't re-definable at run-time, this would simply not work, or you'd not be allowed to do that. Try such re-definitions with basic structs (i. e. the things you get when using defstruct), and you will often run into pretty severe warnings or even errors when the change is not compatible. Of course, structs have other limitations, too, e. g. only single dispatch, so that it's not so easy to make an exactly analogous example. But try to remove a slot from a defstruct.
Just commenting out some source code doesn't change the fact that you evaluated (compiled and loaded) it before. You are manipulating a running system, and the source code is just that: source. If you want to remove a method from the running system, you can use remove-method (see also How to remove a defmethod for a struct). Most Lisp IDEs have ways to do that interactively, e. g. in SLIME using the SLIME inspector.
I want to define a function (not a macro) that can set a variable in the scope its called.
I have tried:
(defun var-set (var-str val)
(let ((var-interned
(intern (string-upcase var-str))))
(set var-interned val)
))
(let ((year "1400"))
(var-set "year" 1388)
(labeled identity year))
Which doesn't work because of the scoping rules. Any "hacks" to accomplish this?
In python, I can use
previous_frame = sys._getframe(1)
previous_frame_locals = previous_frame.f_locals
previous_frame_locals['my-var'] = some_value
Any equivalent API for lisp?
You cannot do that because after compilation the variable might not even exist in any meaningful sense.
E.g., try to figure out by looking at the output of (disassemble (lambda (x) (+ x 4))) where you
would write the new values of x.
You have to tell both the caller and the callee (at compile time!) that the variable is special:
(defun set-x (v)
(declare (special x))
(setq x v))
(defun test-set (a)
(let ((x a))
(declare (special x))
(set-x 10)
x))
(test-set 3)
==> 10
See Dynamic and Lexical variables in Common Lisp for further details on lexical vs dynamic bindings.
You can't. This is why it is called lexical scope: you have access to variable bindings if and only if you can see them. The only way to get at such a binding is to create some object for which it is visible and use that. For instance:
(defun foo (x)
(bar (lambda (&optional (v nil vp)
(if vp (setf x vp) x))))
(defun bar (a)
...
(funcall a ...))
Some languages, such as Python have both rather rudimentary implementations of variable bindings and a single implementation (or a mandated implementation) which allow you to essentially poke around inside the system to subvert lexical scoping. Some CL implementations may have rudimentary implementation of variable bindings (probably none do) but Common Lisp the language does not mandate such implementations and nor should it.
As an example of the terrible consequences of mandating that some kind of access to lexical variables must be allowed consider this:
(defun outer (n f)
(if (> n 0)
(outer (g n) f)
(funcall f)))
If f could somehow poke at the lexical bindings of outer this would mean that all those bindings would need to exist at the point f was called: tail-call elimination would thus be impossible. If the language mandated that such bindings should be accessible then the language is mandating that tail-call elimination is not possible. That would be bad.
(It is quite possible that implementations, possibly with some debugging declarations, allow such access in some circumstances. That's very different than the language mandating such a thing.)
What are you trying to achieve?
What about a closure? Write the defun inside the let, and create an accessor and a setter function if needed.
(let ((year "1400"))
(defun current-year ()
year)
(defun set-year (val)
(setf year val)))
and
CL-USER> (current-year)
"1400"
CL-USER> (set-year 1200)
1200
CL-USER> (current-year)
1200
That Python mechanism violates the encapsulation which motivates the existence of lexical scope.
Because a lexical scope is inaccessible by any external means other than invocations of function bodies which are in that scope, a compiler is free to translate a lexical scope into any representation which performs the same semantics. Variables named in the source code of the lexical scope can disappear entirely. For instance, in your example, all references to year can be replaced by copies of the pointer to the "1400" string literal object.
Separately from the encapsulation issue there is also the consideration that a function does not have any access at all to a parent lexical scope, regardless of that scope's representation. It does not exist. Functions do not implicitly pass their lexical scope to children. Your caller may not have a lexical environment at all, and there is no way to know. The essence of the lexical environment is that no aspect of it is passed down to children, other than via the explicit passage of lexical closures.
Python's feature is poorly considered because it makes programs dependent on the representation of scopes. If a compiler like PyPy is to make that code work, it has to constrain its treatment of lexical scopes to mimic the byte code interpreted version of Python.
Firstly, each function has to know who called it, so it has to receive some parameter(s) about that, including a link to the caller's environment. That's going to be a source of inefficiency even in code that doesn't take advantage of it.
The concept of a well-defined "previous frame" means that the compiler cannot merge together frames. Code which expects some variable to be in the third frame up from here will break if those frames are all inlined together due to a nested lexical scope being flattened, or due to function inlining.
As soon as you provide an interface to the parent lexical environment, and applications start using it, you no longer have lexical scoping. You have a form of dynamic scoping with lexical-like visibility rules.
The application logic can implement de facto dynamic scope on top of this API, because you can write a loop which searches for a variable across the chain of lexical scopes. Does my parent have an x variable? If not, does the grandparent, if there is one? You can search the dynamic chain of invocations for the most recent one which binds x, and that is dynamic scope.
There is nothing wrong with dynamic scope, if it is a separate discipline that is not entangled in the implementation of lexical scope.
That said, an API for tracing frames and getting at local variables is is the sort of introspection that is very useful in developing a debugger. Another angle on this is that if you work that API into an application, you're using debugging features in production.
(defvar *lexical-variables* '())
(defun get-var (name)
(let ((var (cdr (assoc name *lexical-variables*))))
(unless var (error "No lexical variable named ~S" name))
var))
(defun deref (var)
(funcall (if (symbolp var)
(or (cdr (assoc var *lexical-variables*))
(error "No lexical variable named ~S" var))
var)))
(defun (setf deref) (new-value var)
(funcall (if (symbolp var)
(or (cdr (assoc var *lexical-variables*))
(error "No lexical variable named ~S" var))
var)
new-value))
(defmacro with-named-lexical-variable ((&rest vars) &body body)
(let ((vvar (gensym))
(vnew-value (gensym))
(vsetp (gensym)))
`(let ((*lexical-variables* (list* ,#(mapcar (lambda (var)
`(cons ',var
(lambda (&optional (,vnew-value nil ,vsetp))
(if ,vsetp
(setf ,var ,vnew-value)
,var))))
vars)
*lexical-variables*)))
,#body)))
(defun var-set (var-str val)
(let ((var-interned (intern (string-upcase var-str))))
(setf (deref var-interned) val)))
(let ((x 1)
(y 2))
(with-named-lexical-variable (x y)
(var-set "x" 3)
(setf (deref 'y) 4)
(mapcar (function deref) '(x y))))
;; -> (3 4)
(let ((year "1400"))
(with-named-lexical-variable (year)
(var-set "year" 1388))
year)
;; --> 1388
If I have a function that accepts function argument, for optimization purposes I can declare it to be a function, let's say
(defun foo (f)
(declare (type function f))
...)
However, I can be even more specific:
(defun foo (f)
(declare (type (function (double-float) double-float) f))
...)
i.e. telling that f will accept one double-float argument and return one double-float value. SBCL, however, seem to be able to perform a better optimization on the former and for the latter it says that it doesn't know if f is fdefinition (try to compile with (optimize (speed 3)) declaration to reproduce).
So, my questions are:
Am I doing something wrong? Especially If SBCL would do exactly the same thing for just function and (function ...) I would be OK with it, but it actually does worse. Or should it be considered a bug in SBCL?
Is function type declaration in general useless in CL in terms of optimization for some reason?
SysInfo: SBCL 1.3.18
From the SBCL Manual (4.2.3 Getting Existing Programs to Run):
Some incorrect declarations can only be detected by run-time type
checking [...] because the SBCL compiler does much more
type inference than other Common Lisp compilers, so an incorrect
declaration can do more damage.
It's possible, that's why your function does worse with the variable type declarations included.
Further:
The most common problem is with variables whose constant initial value
doesn't match the type declaration. Incorrect constant initial values
will always be flagged by a compile-time type error, and they are
simple to fix once located. Consider this code fragment:
(prog (foo)
(declare (fixnum foo))
(setq foo ...)
...)
Here foo is given an initial value of nil, but is declared to be a fixnum. Even if it is never read, the initial value of a
variable must match the declared type. There are two ways to fix this
problem. Change the declaration
(prog (foo)
(declare (type (or fixnum null) foo))
(setq foo ...)
...)
or change the initial value
(prog ((foo 0))
(declare (fixnum foo))
(setq foo ...)
...)
This is from the manual for the current version of SBCL (1.4), so it may or may not apply to your situation.
I know the language exists, but I can't put my finger on it.
dynamic scope
and
static typing?
We can try to reason about what such a language might look like. Obviously something like this (using a C-like syntax for demonstration purposes) cannot be allowed, or at least not with the obvious meaning:
int x_plus_(int y) {
return x + y; // requires that x have type int
}
int three_plus_(int y) {
double x = 3.0;
return x_plus_(y); // calls x_plus_ when x has type double
}
So, how to avoid this?
I can think of a few approaches offhand:
Commenters above mention that Fortran pre-'77 had this behavior. That worked because a variable's name determined its type; a function like x_plus_ above would be illegal, because x could never have an integer type. (And likewise one like three_plus_, for that matter, because y would have the same restriction.) Integer variables had to have names beginning with i, j, k, l, m, or n.
Perl uses syntax to distinguish a few broad categories of variables, namely scalars vs. arrays (regular arrays) vs. hashes (associative arrays). Variables belonging to the different categories can have the exact same name, because the syntax distinguishes which one is meant. For example, the expression foo $foo, $foo[0], $foo{'foo'} involves the function foo, the scalar $foo, the array #foo ($foo[0] being the first element of #foo), and the hash %foo ($foo{'foo'} being the value in %foo corresponding to the key 'foo'). Now, to be quite clear, Perl is not statically typed, because there are many different scalar types, and these are types not distinguished syntactically. (In particular: all references are scalars, even references to functions or arrays or hashes. So if you use the syntax to dereference a reference to an array, Perl has to check at runtime to see if the value really is an array-reference.) But this same approach could be used for a bona fide type system, especially if the type system were a very simple one. With that approach, the x_plus_ method would be using an x of type int, and would completely ignore the x declared by three_plus_. (Instead, it would use an x of type int that had to be provided from whatever scope called three_plus_.) This could either require some type annotations not included above, or it could use some form of type inference.
A function's signature could indicate the non-local variables it uses, and their expected types. In the above example, x_plus_ would have the signature "takes one argument of type int; uses a calling-scope x of type int; returns a value of type int". Then, just like how a function that calls x_plus_ would have to pass in an argument of type int, it would also have to provide a variable named x of type int — either by declaring it itself, or by inheriting that part of the type-signature (since calling x_plus_ is equivalent to using an x of type int) and propagating this requirement up to its callers. With this approach, the three_plus_ function above would be illegal, because it would violate the signature of the x_plus_ method it invokes — just the same as if it tried to pass a double as its argument.
The above could just have "undefined behavior"; the compiler wouldn't have to explicitly detect and reject it, but the spec wouldn't impose any particular requirements on how it had to handle it. It would be the responsibility of programmers to ensure that they never invoke a function with incorrectly-typed non-local variables.
Your professor was presumably thinking of #1, since pre-'77 Fortran was an actual real-world language with this property. But the other approaches are interesting to think about. :-)
I haven't found elsewhere has it written down, but AXIOM CAS (and various forks, including FriCAS which is still been actively developed) uses a script language called SPAD with both a very novel strong static dependent type system and dynamic scoping (although it is possibly an unintended implementation bug).
Most of the time the user won't realize that, but when they start trying to build closures like other functional languages it reveals its dynamic scoping nature:
FriCAS Computer Algebra System
Version: FriCAS 2021-03-06
Timestamp: Mon May 17 10:43:08 CST 2021
-----------------------------------------------------------------------------
Issue )copyright to view copyright notices.
Issue )summary for a summary of useful system commands.
Issue )quit to leave FriCAS and return to shell.
-----------------------------------------------------------------------------
(1) -> foo (x,y) == x + y
Type: Void
(2) -> foo (1,2)
Compiling function foo with type (PositiveInteger, PositiveInteger)
-> PositiveInteger
(2) 3
Type: PositiveInteger
(3) -> foo
(3) foo (x, y) == x + y
Type: FunctionCalled(foo)
(4) -> bar x y == x + y
Type: Void
(5) -> bar
(5) bar x == y +-> x + y
Type: FunctionCalled(bar)
(6) -> (bar 1)
Compiling function bar with type PositiveInteger ->
AnonymousFunction
(6) y +-> #1 + y
Type: AnonymousFunction
(7) -> ((bar 1) 2)
(7) #1 + 2
Type: Polynomial(Integer)
Such a behavior is similar to what will happen when trying to build a closure by using (lambda (x) (lambda (y) (+ x y))) in a dynamically scoped Lisp, such as Emacs Lisp. Actually the underlying representation of functions is essentially the same as Lisp in the early days since AXIOM has been first developed on top of an early Lisp implementation on IBM mainframe.
I believe it is however a defect (like what JMC happened did when implementing the first version of LISP language) because the implementor made the parser to do uncurrying as in the function definition of bar, but it is unlikely to be useful without the ability to build the closure in the language.
It is also worth notice that SPAD automatically renames the variables in anomalous functions to avoid captures so its dynamic scoping could be used as a feature as in other Lisps.
Dynamic scope means, that the variable and its type in a specific line of your code depends on the functions, called before. This means, you can not know the type in a specific line of your code, because you can not know, which code has been executed before.
Static typing means, that you have to know the type in every line of your code, before the code starts to run.
This is irreconcilable.
I see that the Practical Common Lisp uses (defvar *db* nil) for setting up a global variable. Isn't it OK to use setq for the same purpose?
What are the advantages/disadvantages of using defvar vs. setq?
There are several ways to introduce variables.
DEFVAR and DEFPARAMETER introduce global dynamic variables. DEFVAR optionally sets it to some value, unless it is already defined. DEFPARAMETER sets it always to the provided value.
SETQ does not introduce a variable.
(defparameter *number-of-processes* 10)
(defvar *world* (make-world)) ; the world is made only once.
Notice that you likely never want to DEFVAR variables with names like x, y, stream, limit, ... Why? Because these variables then would be declared special and its difficult to undo that. The special declaration is global and all further uses of the variable would use dynamic binding.
BAD:
(defvar x 10) ; global special variable X, naming convention violated
(defvar y 20) ; global special variable Y, naming convention violated
(defun foo ()
(+ x y)) ; refers to special variables X and y
(defun bar (x y) ; OOPS!! X and Y are special variables
; even though they are parameters of a function!
(+ (foo) x y))
(bar 5 7) ; -> 24
BETTER: Always mark special variables with * in their names!
(defvar *x* 10) ; global special variable *X*
(defvar *y* 20) ; global special variable *Y*
(defun foo ()
(+ *x* *y*)) ; refers to special variables X and y
(defun bar (x y) ; Yep! X and Y are lexical variables
(+ (foo) x y))
(bar 5 7) ; -> 42
Local variables are introduced with DEFUN, LAMBDA, LET, MULTIPLE-VALUE-BIND and many others.
(defun foo (i-am-a-local-variable)
(print i-am-a-local-variable))
(let ((i-am-also-a-local-variable 'hehe))
(print i-am-also-a-local-variable))
Now, by default the local variables in above two forms are lexical, unless they are declared SPECIAL. Then they would be dynamic variables.
Next, there are also several forms to set a variable to new values. SET, SETQ, SETF and others. SETQ and SETF can set both lexical and special (dynamic) variables.
It is required for portable code that one sets variables that are already declared. The exact effect of setting a not declared variable is undefined by the standard.
So, if you know what your Common Lisp implementation does, you can use
(setq world (make-new-world))
in the Read-Eval-Print-Loop at the toplevel. But don't use it in your code, since the effect is not portable. Typically SETQ will set the variable. But some implementation might also declare the variable SPECIAL when it doesn't know it (CMU Common Lisp does that by default). That's almost always not what one would want. Use it for casual use if you know what you do, but not for code.
Same here:
(defun make-shiny-new-world ()
(setq world (make-world 'shiny)))
First, such variables should be written as *world* (with the surrounding * characters), to make clear that it is a global special variable. Second, it should have been declared with DEFVAR or DEFPARAMETER before.
A typical Lisp compiler will complain that above variable is undeclared. Since global lexical variables don't exist in Common Lisp, the compiler has to generate code for a dynamic lookup. Some compiler then say, okay, we assume that this is a dynamic lookup, let's declare it to be special - since that is what we assume anyway.
defvar introduces a dynamic variable while setq is used to assign a value to a dynamic or lexical variable. The value of a dynamic variable is looked up in the environment that calls the function, while the value of a lexical variable is looked up in the environment where the function was defined. The following example will make the difference clear:
;; dynamic variable sample
> (defvar *x* 100)
*X*
> (defun fx () *x*)
FX
> (fx)
100
> (let ((*x* 500)) (fx)) ;; gets the value of *x* from the dynamic scope.
500
> (fx) ;; *x* now refers to the global binding.
100
;; example of using a lexical variable
> (let ((y 200))
(let ((fy (lambda () (format t "~a~%" y))))
(funcall fy) ;; => 200
(let ((y 500))
(funcall fy) ;; => 200, the value of lexically bound y
(setq y 500) ;; => y in the current environment is modified
(funcall fy)) ;; => 200, the value of lexically bound y, which was
;; unaffected by setq
(setq y 500) => ;; value of the original y is modified.
(funcall fy))) ;; => 500, the new value of y in fy's defining environment.
Dynamic variables are useful for passing around a default value. For instance, we can bind the dynamic variable *out* to the standard output, so that it becomes the default output of all io functions. To override this behavior, we just introduce a local binding:
> (defun my-print (s)
(format *out* "~a~%" s))
MY-PRINT
> (my-print "hello")
hello
> (let ((*out* some-stream))
(my-print " cruel ")) ;; goes to some-stream
> (my-print " world.")
world
A common use of lexical variables is in defining closures, to emulate objects with state. In the first example, the variable y in the binding environment of fy effectively became the private state of that function.
defvar will assign a value to a variable only if it is not already assigned. So the following re-definition of *x* will not change the original binding:
> (defvar *x* 400)
*X*
> *x*
100
We can assign a new value to *x* by using setq:
> (setq *x* 400)
400
> *x*
400
> (fx)
400
> (let ((*x* 500)) (fx)) ;; setq changed the binding of *x*, but
;; its dynamic property still remains.
500
> (fx)
400
DEFVAR establishes a new variable. SETQ assigns to a variable.
Most Lisp implementations I've used will issue a warning if you SETQ to a variable that doesn't yet exist.
defvar and defparameter both introduce global variables. As Ken notes, setq assigns to a variable.
In addition, defvar will not clobber something previously defvar-ed. Seibel says later in the book (Chapter 6): "Practically speaking, you should use DEFVAR to define variables that will contain data you'd want to keep even if you made a change to the source code that uses the variable."
http://www.gigamonkeys.com/book/variables.html
For instance, if you have a global *db* for the database in the Simple Database chapter:
(defvar *db* nil)
...and you start playing with it at the REPL - adding, deleting things, etc - but then you make a change to the source file which contains that defvar form, reloading that file will not wipe out *db* and all the changes you might have made... I believe that setq will, as will defparameter. A more experienced Lisper please correct me if I'm wrong though.