Crew,
I'm one of those types that insists on defining my variables with SETF. I've upgraded to a new machine (and a new version of SBCL) and it's not letting me get away with doing that (naturally, I get the appropriate "==> undefined variable..." error)
My problem here is, I've already written 20,000 lines of code (incorrectly) defining my variables with SETF and i don't like the prospect of re-writing all of my code to get the interpreter to digest it all.
Is there a way to shut down that error such that interpretation can continue?
Any help is appreciated.
Sincerely,
-Todd
One option is to set up your package environment so that a bare symbol setf refers to my-gross-hack::setf instead of cl:setf. For example, you could set things up like:
(defpackage #:my-gross-hack
(:use #:cl)
(:shadow #:setf))
;;; define your own setf here
(defpackage #:my-project
(use #:cl)
(shadowing-import-from #:my-gross-hack #:setf))
;;; proceed to use setf willy-nilly
You can handle warnings, so that compilation terminates; while doing so, you can collect enough information to fix your code.
Tested with SBCL 1.3.13.
Handle warnings
I have warnings when using setf with undefined variables. The following invokes the debugger, from which I can invoke muffle-warning:
(handler-bind ((warning #'invoke-debugger))
(compile nil '(lambda () (setf *shame* :on-you))))
The warning is of type SIMPLE-WARNING, which has the following accessors: SIMPLE-CONDITION-FORMAT-CONTROL and SIMPLE-CONDITION-FORMAT-ARGUMENTS.
(defparameter *setf-declarations* nil)
(defun handle-undefined-variables (condition)
(when (and (typep condition 'simple-warning)
(string= (simple-condition-format-control condition)
"undefined ~(~A~): ~S"))
(let* ((arguments (simple-condition-format-arguments condition))
(variable (and (eq (first arguments) :variable)
(second arguments))))
(when variable
(proclaim `(special ,variable))
(push variable *setf-declarations*)
(invoke-restart 'muffle-warning)))))
Use that as a handler:
(handler-bind ((warning #'handle-undefined-variables))
;; compilation, quickload, asdf ...
)
The above handler is not robust: the error message might change in future versions, the code assumes the arguments follow a given pattern, ... But this needs only work once, since from now on you are going to declare all your variables.
Fix your code
Now that your code compiles, get rid of the ugly. Or at least, add proper declarations.
(with-open-file (out #P"declarations.lisp" :direction :output)
(let ((*package* (find-package :cl-user)))
(format out
"(in-package :cl-user)~%~%~{(defvar ~(~S~))~%~}"
*setf-declarations*)))
This iterates over all the symbols you collected and write declarations inside a single file.
In my example, it would contain:
(in-package :cl-user)
(defvar *shame*)
Try to cleanly recompile without handling errors, by loading this file early in your compilation process, but after packages are defined. Eventually, you may want to find the time to move those declarations in place of the setf expressions that triggered a warning.
Related
I am getting an error only when the code is entered line by line in the repl. It works when the whole program is pasted at once, or from the command line.
class A {
method a () {
return 1;
}
}
class B {
method b () {
return 2;
}
}
This is the error statement:
===SORRY!=== Error while compiling:
Package 'B' already has a method 'b' (did you mean to declare a multi method?)
This screen shot might make it clearer. On the left I just pasted the code, and on the right I entered it line by line. The code is still working but what is causing the error?
For some reason, I could not reproduce this using just one class.
I can reproduce that error, and looks like a REPL bug, or simply something the REPL is not prepared to do. This, for instance, will also raise an exception:
class A {
method a() {
return 1;
}
};
class foo {
has $.bar = 3;
};
In either form, either pasting it directly or in pieces. It's always the second class. It's probably related to the way EVAL works, but I really don't know. At the end of the day, the REPL can only take you so far and I'm not totally sure this is within the use case. You might want to use Comma or any other IDE, like emacs, for anything that's more complicated than a line; Comma also provides help for evaluating expressions, and even grammars.
I think Comma is the bees knees. And I almost never use the repl. But I enjoy:
Golfing Your example is a more than adequate MRE. But I love minimizing bug examples.
Speculating I think I can see what's going on.
Searching issue queues Rakudo has two issue queues on GH: old and new.
Spelunking compiler code Rakudo is mostly written in Raku; maybe we can work out what this problem is in the REPL code (which is part of the compiler)?
Golfing
First, the bug:
Welcome to 𝐑𝐚𝐤𝐮𝐝𝐨™ v2021.03.
Implementing the 𝐑𝐚𝐤𝐮™ programming language v6.d.
Built on MoarVM version 2021.03.
To exit type 'exit' or '^D'
> # 42
Nil
> { subset a
*
===SORRY!=== Error while compiling:
Redeclaration of symbol 'a'.
at line 3
------> <BOL>⏏<EOL>
Commentary:
To get on the fairway, enter any line that's not just whitespace, and press Enter.
Pick the right iron; open a block with {, declare some named type, and press Enter. The REPL indicates you're on the green by displaying the * multi-line prompt.
To sink the ball, just hit Enter.
Second, golfing in aid of speculation:
> # 42
Nil
> { BEGIN say 99
99
* }
99
>
(BEGIN marks code that is to be run during compilation as soon as the compiler encounters it.)
Speculating
Why does the initial # 42 evaluation matter? Presumably the REPL tries to maintain declarations / state (of variables and types etc) during a REPL session.
And as part of that it's presumably remembering all previous code in a session.
And presumably it's seeing anything but blank lines as counting as previous code.
And the mere existence of some/any previous code somehow influences what state the REPL is maintaining and/or what it's asking the compiler to do.
Maybe.
Why does a type declaration matter when, say, a variable declaration doesn't?
Presumably the REPL and/or compiler is distinguishing between these two kinds of declaration.
Ignoring the REPL, when compiling code, a repeated my declaration only raises a warning, whereas a repeated type declaration is an error. Quite plausibly that's why?
Why does a type declaration have this effect?
Presumably the type successfully compiles and only after that an exception is thrown (because the code is incomplete due to the missing closing brace).
Then the REPL asks the compiler to again try to compile the multi-line code thus far entered, with whatever additional code the user has typed (in my golf version I type nothing and just hit Enter, adding no more code).
This repeated compile attempt includes the type declaration again, which type declaration, because the compilation state from the prior attempt to compile the multi-line code is somehow being retained, is seen by the compiler as a repeat declaration, causing it to throw an exception that causes the REPL to exit multi-line mode and report the error.
In other words, the REPL loop is presumably something like:
As each line is entered, pass it to the compiler, which compiles the code and throws an exception if anything goes wrong.
If an exception is thrown:
2.1 If in multi-line mode (with * prompt), then exit multi-line mode (go back to > prompt) and display exception message.
2.2 Else (so not in multi-line mode), if analysis (plausibly very basic) of the exception and/or entered code suggests multi-line mode would be useful, then enter that mode (with * prompt). In multi-line mode, the entire multi-line of code so far is recompiled each time the user presses Enter.
2.3 Else, display exception message.
(Obviously there's something else going on related to initialization given the need to start with some evaluation to manifest this bug, but that may well be a completely distinct issue.)
Searching
I've browsed through all open Rakudo issues in its old and new queues on GH that match 'repl'. I've selected four that illustrate the range of difficulties the REPL has with maintaining the state of a session:
REPL loses custom operators. "Interestingly, if a postfix operator like this is exported by a module which is loaded by the REPL, the REPL can successfully parse that operator just once, after which it will fail with an error similar to the above." Is this related to the way the bug this SO is focused on doesn't manifest until it's a second or later evaluation?
Perl6 REPL forgets the definition of infix sub. Looks like a dupe of the above issue, but includes extra debugging goodies from Brian Duggan. ❤️
REPL messes up namespaces when Foo is used after Foo::Bar.
In REPL cannot bind to scalars declared on earlier lines.
One thing I haven't done is checked whether these bugs all still exist. My guess is they do. And there are many others like them. Perhaps they have a somewhat common cause? I've no idea. Perhaps we need to look at the code...
Spelunking
A search of the Rakudo sources for 'repl' quickly led to a REPL module. Less than 500 lines of high level Raku code! \o/ (For now, let's just pretend we can pretty much ignore digging into the code it calls...)
From my initial browser, I'll draw attention to:
A sub repl:
sub repl(*%_) {
my $repl := REPL.new(nqp::getcomp("Raku"), %_, True);
nqp::bindattr($repl,REPL,'$!save_ctx',nqp::ctxcaller(nqp::ctx));
$repl.repl-loop(:no-exit);
}
Blame shows that Liz added this a couple months ago. It's very tangential to this bug, but I'm guessing methods and variables with ctx in their name are pretty central to things so this is hopefully a nice way to start pondering that.
method repl-eval. 30 lines or so.
REPL: loop { ... }. 60 lines or so.
That'll do for tonight. I'll post this then return to it all another day.
sub make-regex {
my $what's-in-the-box = rand > .5 ?? 'x' !! 'y';
/$what's-in-the-box/
}
my $lol-who-knows = make-regex;
$lol-who-knows.gist.say;
How do you see the innards of the regex (i.e. x or y)? Forcing a match is not a solution.
How do you see the innards of the regex (i.e. x or y)?
You:
Statically compile the regex. Doing so will involve use of a raku compiler, i.e. Rakudo.
Dynamically evaluate the regex so that the $what's-in-the-box variable in your regex gets interpolated and turns into x or y. Doing so will involve running the regex as code. That in turn means both using Rakudo and running the regex with a Match object invocant (or sub-class instance or, conceivably, a mock object equivalent).
View the resulting regex. Doing so will involve using compiler (Rakudo) toolchain specific introspection or debug functionality.
Forcing a match is not a solution.
You can run a regex with no input if your concern is just to avoid the need for a successful match:
say Match.new.&$lol-who-knows; # #<failed match>
But it must be run, otherwise the $what's-in-the-box variable won't turn into an x or y. You might think you could cheat on this by writing something that mimics this part of raku regex construction/use but there's good reason to think that's not going to work out[1].
And then you must view its innards, using Rakudo toolchain features, after you've started running it and before it finishes running.
A regex is a method
In raku, a Regex is code, a sub-class of Method.
Until you run it, by using it in a match, that code is just /$what's-in-the-box/ (aka regex { $what's-in-the-box }). It's the equivalent of something like:
method {
...self should be a Match or a sub-class of it
...if self is not an instance, create a new one
...do matching -- compiler evaluates $what's-in-the-box during this
...return updated self / new instance
}
(To see a bit more detail, see Moritz's answer to the SO Can I change the slang inside a method?.)
A routine is a closure
You create your regex inside the closure named make-regex. The compiler spots that you've used $what's-in-the-box and therefore hangs on to the variable even after the make-regex closure returns. Later on, if/when your regex is run, the $what's-in-the-box variable will be replaced by its value at the moment the regex is run.
Footnotes
[1] There are plenty of huge complications you'll encounter if you try to do an end-run around using the compiler. Even something simple like interpolation is non-trivial. To quote Jonathan Worthington, in 2020:
I thought oh I'm only building [a tool that's] not the real compiler. I can ... have a simpler model of the symbol resolution. In the end it turned out that no I couldn't really. That started to create us some problems. When we aligned the way that we resolved symbols with the same algorithms and lookup structures that was being used in the compiler then suddenly it all became a lot simpler.
Apologies: I don't have sufficient knowledge to rework this as an easy to understand code snippet.
I've been using the SBCL compiler notes as signs to what might be improved but I'm well out of my depth with this —
; compiling (DEFUN EXECUTE-PARALLEL ...)
; file: /home/dunham/8000-benchmarksgame/bench/spectralnorm/spectralnorm.sbcl-8.sbcl
; in: DEFUN EXECUTE-PARALLEL
; (FUNCALL FUNCTION START END)
; --> SB-C::%FUNCALL THE
; ==>
; (SB-KERNEL:%COERCE-CALLABLE-FOR-CALL FUNCTION)
;
; note: unable to
; optimize away possible call to FDEFINITION at runtime
; because:
; FUNCTION is not known to be a function
—
#+sb-thread
(defun execute-parallel (start end function)
(declare (type int31 start end))
(let* ((num-threads 4))
(loop with step = (truncate (- end start) num-threads)
for index from start below end by step
collecting (let ((start index)
(end (min end (+ index step))))
(sb-thread:make-thread
(lambda () (funcall function start end))))
into threads
finally (mapcar #'sb-thread:join-thread threads))))
#-sb-thread
(defun execute-parallel (start end function )
(funcall function start end))
(The program is here. Measurements for similar programs are here.)
Is it practical to make SBCL "optimize away possible call to FDEFINITION" or is that compiler note an explanation rather than an opportunity?
The reason for the possible call to fdefinition is that it doesn't know that function is a function: it might be the name of one: in general it may be a function designator rather than a function. To keep the compiler quiet, explain to it that it is a function with a suitable type declaration, which is (declare (type function function)): you just need to declare that its type is function).
Rainer is right: there is ε chance that this is ever going to be a performance problem, given you're starting a new thread. In particular it is fairly likely that adding a declaration will make no difference at all:
without a declaration the call to funcall will get compiled as something like 'check the type of the object: if it is a function, call it; if it is not, call fdefinition on it and call the result;';
with a declaration then the overall function looks like 'check the object is a function, signalling an error if not ... call the function'.
In both cases, if the object is a function, there is one type check and one call: the type check is just in a different place. In the first case, the code will still work if the object is merely the name of a function, while with the type check it won't.
And in both of these cases this is code where you care calling make-thread: if this is anything like as fast as a function call, even via fdefinition I would be really impressed by the threading system! Almost certainly the performance of this function is entirely dominated by the overhead of making threads.
In real code, avoid optimizations like that - unless really needed
Is it practical to make SBCL "optimize away possible call to FDEFINITION" or is that compiler note an explanation rather than an opportunity?
Generally it does not matter, especially since most Lisp code should not be compiled with optimization qualities (speed 3) (safety 0) (space 0), since it may open up the software to runtime errors and crashes depending on the implementation and program used. Calling things unchecked (without safety), other than functions or symbols naming functions, via funcall might be dangerous enough to cause a program crash.
For a specific benchmark one might check via timings if a type declaration and a specialized fdefinition compilation brings any advantage.
a type declaration
A type declaration to make clear that a variable named fn is referencing an object of type function would be:
(declare (type function fn))
in the specific benchmark program FDEFINITION won't be called anyway
In the example you have provided, fdefinition will not be called anyway.
(setf foo (lambda (x) x)) ; foo references a function object
(funcall foo 3)
funcall is probably implemented by something like this:
(etypecase f
((or cons symbol) (funcall (fdefinition f) ...))
(function ...))
Since your code passes a function object, there is never the need to call fdefinition.
The optimization benefit then will be that the runtime type dispatch can be removed and the dead code for the cons or symbol case...
You ask a question about removing an fdefinition but actually your question relies on a premise that the sbcl notes are a good way to drive optimisations and improvements. The notes are a good way to spot obvious issues and places where type declarations can help. They do not tell you what actually makes your program slow. The correct way to improve the performance of a program is to 1. Think if there is a faster algorithm, and 2. Measure it’s performance and work out what is slow.
A single fdefinition call will only matter if it happens in a tight loop (ie it is not single but very plural)
In this case it happens to start a thread. If you are starting threads in a tight loop then your performance problem comes from starting threads in a tight loop. Don’t do that.
If you aren’t starting threads in a tight loop (looking at your code, it appears you are not), there are bigger fish to fry. Why waste time on an fdefinition that maybe gets called 4 times per call to execute-parallel when you can optimise the inner function instead.
What is the conventional, well-behaved way for a macro to indicate that it's been passed invalid arguments?
(defmacro defthisthing [name & definitions] . . .)
I'm writing a macro right now to accept a whole bunch of definitions. If the same thing gets defined twice, the macro should complain. Similarly, if one of the definitions uses a term that's not defined elsewhere in the same macro invocation, the macro should complain, hopefully with line and column numbers so the programmer can see exactly where the error is.
I'm currently thinking that throwing an exception makes the most sense, because invalid macro arguments are in fact a compilation error. Everything should shut down no differently than if the compiler found unbalanced parentheses.
If that's correct, what is the conventional exception to throw? And how do you include the filename and line number of the offending snippet of code?
And if that's not correct, what is the more Clojurely approach?
Throwing exceptions sounds fine. I just checked the Clojure source and this is how it is done there:
(defmacro let ...) calls (defmacro assert-args ...), which throws exceptions if the arguments don’t meet their requirements.
Using SWI-Prolog (Multi-threaded, 64 bits, Version 7.3.5),
we proceed step by step:
Define dcg nonterminal a//1 in module dcgAux (pronounced: "di-SEE-goh"):
:- module(dcgAux,[a//1]).
a(0) --> [].
a(s(N)) --> [a], a(N).
Run the following queries—using phrase/2 and apply:foldl/4:
?- use_module([library(apply),dcgAux]).
true.
?- phrase( foldl( a,[s(0),s(s(0))]),[a,a,a]).
true.
?- phrase( foldl(dcgAux:a,[s(0),s(s(0))]),[a,a,a]).
true.
?- phrase(apply:foldl(dcgAux:a,[s(0),s(s(0))]),[a,a,a]).
true.
?- phrase(apply:foldl( a,[s(0),s(s(0))]),[a,a,a]).
ERROR: apply:foldl_/4: Undefined procedure: apply:a/3
нет! Quite a surprise—and not a good one. Have we been missing some unknown unknowns?
To get rid of above irritating behavior, we must first find out the reason(s) causing it:
?- import_module(apply,M), M=user.
false.
?- phrase(apply:foldl(a,[s(0),s(s(0))]),[a,a,a]).
ERROR: apply:foldl_/4: Undefined procedure: apply:a/3
?- add_import_module(apply,user,end).
true.
?- import_module(apply,M), M=user. % sic!
M = user. % `?- import_module(apply,user).` fails!
?- phrase(apply:foldl(a,[s(0),s(s(0))]),[a,a,a]).
true.
What's going on? The way I see it is this:
Module expansion of the goal passed to foldl/4 is limited.
Quoting from the SWI-Prolog manual page on import_module/2:
All normal modules only import from user, which imports from system.
SWI's library(apply) only "inherits" from system, but not user.
If we clone module apply to applY (and propagate the new module name), we observe:
?- use_module(applY).
true.
?- phrase(applY:foldl(a,[s(0),s(s(0))]),[a,a,a]). % was: ERROR
true. % now: OK!
Please share your ideas on how I could/should proceed!
(I have not yet run a similar experiment with other Prolog processors.)
This is an inherent feature/bug of predicate based module systems in the Quintus tradition. That is, this module system was first developed for Quintus Prolog. It was adopted subsequently by SICStus (after 0.71), then (more or less) by 13211-2, then by YAP, and (with some modifications) by SWI.
The problem here is what exactly an explicit qualification means. As long as the goal is no meta-predicate, things are trivially resolvable: Take the module of the innermost qualification. However, once you have meta-predicates, the meta-arguments need to be informed of that module ; or not. If the meta-arguments are informed, we say that the colon sets the calling context, if not, then some other means is needed for that purpose.
In Quintus tradition, the meta-arguments are taken into account. With the result you see. As a consequence you cannot compare two implementations of the same meta-predicate in the same module directly. There are other approaches most notably IF and ECLiPSe that do not change the calling context via the colon. This has advantages and disadvantages. The best is to compare them case by case.
Here is a recent case. Take lambdas and how they are put into a module
in SICStus, in SWI, and in ECLiPSe.
As for the Quintus/SICStus/YAP/SWI module system, I'd rather use it in the most conservative manner possible. That is:
no explicit qualification, consider the infix : as something internal
clean, checkable meta-declarations - insert on purpose an undefined predicate just to see whether or not cross-referencing is able to detect the problem (in SWI that's check or make).
use the common subset, avoid the many bells and whistles - there are many well meant extensions...
do more versatile things the pedestrian way: reexport by adding an appropriate module and add a dummy definition. Similarly, instead of renaming, import things from an interface module.
be always aware that module systems have inherently some limits. No matter how you twist or turn it. There is no completely seamless module system, for the very purpose of modules is to separate code and concerns.
1: To be precise, SICStus' adaptation of Quintus modules did only include : for module sensitive arguments in meta_predicate declarations. The integers 0..9, which are so important for higher-order programming based on call/N were only introduced about 20 years later in
4.2.0 released 2011-03-08.