I've been playing around with redis to keep track of the ratelimit of an external api in a distributed system. I've decided to create a key for each route where a limit is present. The value of the key is how many request I can still make until the limit resets. And the reset is made by setting the TTL of the key to when the limit will reset.
For that I wrote the following lua script:
if redis.call("EXISTS", KEYS[1]) == 1 then
local remaining = redis.call("DECR", KEYS[1])
if remaining < 0 then
local pttl = redis.call("PTTL", KEYS[1])
if pttl > 0 then
--[[
-- We would exceed the limit if we were to do a call now, so let's send back that a limit exists (1)
-- Also let's send back how much we would have exceeded the ratelimit if we were to ignore it (ramaning)
-- and how long we need to wait in ms untill we can try again (pttl)
]]
return {1, remaining, pttl}
elseif pttl == -1 then
-- The key expired the instant after we checked that it existed, so delete it and say there is no ratelimit
redis.call("DEL", KEYS[1])
return {0}
elseif pttl == -2 then
-- The key expired the instant after we decreased it by one. So let's just send back that there is no limit
return {0}
end
else
-- Great we have a ratelimit, but we did not exceed it yet.
return {1, remaining}
end
else
return {0}
end
Since a watched key can expire in the middle of a multi transaction without aborting it. I assume the same is the case for lua scripts. Therefore I put in the cases for when the ttl is -1 or -2.
After I wrote that script I looked a bit more in depth at the eval command page and found out that a lua script has to be a pure function.
In there it says
The script must always evaluates the same Redis write commands with
the same arguments given the same input data set. Operations performed
by the script cannot depend on any hidden (non-explicit) information
or state that may change as script execution proceeds or between
different executions of the script, nor can it depend on any external
input from I/O devices.
With this description I'm not sure if my function is a pure function or not.
After Itamar's answer I wanted to confirm that for myself so I wrote a little lua script to test that. The scripts creates a key with a 10ms TTL and checks the ttl untill it's less then 0:
redis.call("SET", KEYS[1], "someVal","PX", 10)
local tmp = redis.call("PTTL", KEYS[1])
while tmp >= 0
do
tmp = redis.call("PTTL", KEYS[1])
redis.log(redis.LOG_WARNING, "PTTL:" .. tmp)
end
return 0
When I ran this script it never terminated. It just went on to spam my logs until I killed the redis server. However time dosen't stand still while the script runs, instead it just stops once the TTL is 0.
So the key ages, it just never expires.
Since a watched key can expire in the middle of a multi transaction without aborting it. I assume the same is the case for lua scripts. Therefore I put in the cases for when the ttl is -1 or -2.
AFAIR that isn't the case w/ Lua scripts - time kinda stops (in terms of TTL at least) when the script's running.
With this description I'm not sure if my function is a pure function or not.
Your script's great (without actually trying to understand what it does), don't worry :)
Is it possible to have one Redis Lua script hit more than one database? I currently have information of one type in DB 0 and information of another type in DB 1. My normal workflow is doing updates on DB 1 based on an API call along with meta information from DB 0. I'd love to do everything in one Lua script, but can't figure out how to hit multiple dbs. I'm doing this in Python using redis-py:
lua_script(keys=some_keys,
args=some_args,
client=some_client)
Since the client implies a specific db, I'm stuck. Ideas?
It is usually a wrong idea to put related data in different Redis databases. There is almost no benefit compared to defining namespaces by key naming conventions (no extra granularity regarding security, persistence, expiration management, etc ...). And a major drawback is the clients have to manually handle the selection of the correct database, which is error prone for clients targeting multiple databases at the same time.
Now, if you still want to use multiple databases, there is a way to make it work with redis-py and Lua scripting.
redis-py does not define a wrapper for the SELECT command (normally used to switch the current database), because of the underlying thread-safe connection pool implementation. But nothing prevents you to call SELECT from a Lua script.
Consider the following example:
$ redis-cli
SELECT 0
SET mykey db0
SELECT 1
SET mykey db1
The following script displays the value of mykey in the 2 databases from the same client connection.
import redis
pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
r = redis.Redis(connection_pool=pool)
lua1 = """
redis.call("select", ARGV[1])
return redis.call("get",KEYS[1])
"""
script1 = r.register_script(lua1)
lua2 = """
redis.call("select", ARGV[1])
local ret = redis.call("get",KEYS[1])
redis.call("select", ARGV[2])
return ret
"""
script2 = r.register_script(lua2)
print r.get("mykey")
print script2( keys=["mykey"], args = [1,0] )
print r.get("mykey"), "ok"
print
print r.get("mykey")
print script1( keys=["mykey"], args = [1] )
print r.get("mykey"), "misleading !!!"
Script lua1 is naive: it just selects a given database before returning the value. Its usage is misleading, because after its execution, the current database associated to the connection has changed. Don't do this.
Script lua2 is much better. It takes the target database and the current database as parameters. It makes sure that the current database is reactivated before the end of the script, so that next command applied on the connection still run in the correct database.
Unfortunately, there is no command to guess the current database in the Lua script, so the client has to provide it systematically. Please note the Lua script must reset the current database at the end whatever happens (even in case of previous error), so it makes complex scripts cumbersome and awkward.
I have a bug I've been trying to track down for a few weeks now, and have pretty much isolated exactly what's going wrong. However, why this happens is beyond me.
I have a very large Oracle package (around 4,100 lines of code) and I'm calling several procedures. However, data seems to be getting lost between procedure calls.
The data that is being lost is:
dpmethodstate varchar_state_local_type;
First, I call this procedure:
PROCEDURE delivery_plan_set_state (
messages OUT ReferenceCursor,
state IN varchar_state_local_type
) AS
BEGIN
logMessage('state COUNT is: ' || state.COUNT);
dpmethodstate := state;
FOR I IN 1..dpmethodstate.COUNT LOOP
logMessage(dpmethodstate(I));
END LOOP;
logMessage('delivery_plan_set_state end - dpmethodstate count is now ' || dpmethodstate.COUNT);
OPEN messages FOR SELECT * FROM TABLE(messageQueue);
messageQueue := NULL;
END delivery_plan_set_state;
I pass in state, which is a valid array of a single string. I can verify in the logs that dpmethodstate has a COUNT of 1 when the procedure ends.
Next, I call the execute_filter procedure which looks like this:
PROCEDURE execute_filter (
--Whole bunch of OUT parameters
) AS
--About 50 different local variables being set here
BEGIN
SELECT TO_CHAR(SYSTIMESTAMP, 'HH24:MI:SS.ff') INTO TIMING FROM DUAL;
logMessage('[' || TIMING || '] execute_filter begin');
logMessage('a) dpmethodstate Count is: ' || dpmethodstate.COUNT);
--Rest of procedure
However, this time dpmethodstate.COUNT is 0. The value I set from delivery_plan_set_state has vanished!
When I look at my logs, it looks something like this:
proposed
delivery_plan_set_state end - dpmethodstate count is now 1
[21:39:48.719017] execute_filter begin
a) dpmethodstate Count is: 0
As you can see, dpmethodstate got lost between procedure calls. There's a few things to note:
Nothing else in this package is capable of setting the value for dpmethodstate besides delivery_plan_set_state. And I can see nothing else has called it.
My client side code is written in C#, and not much happens between the two procedure calls.
This happens perhaps 1 out of every 100 times, so it's very difficult to track down or debug.
First off, what's the best way to debug a problem like this? Is there any more logging I can do? Also, how does Oracle maintain state between procedure calls and is there anything that can intermittently reset this state? Any pointers would be much appreciated!
Is dpmethodstate a package global variable? I'm assuming it is, but I don't see that explicitly mentioned.
Since package global variables have session scope, are you certain that the two procedure calls are always using the same physical database connection and that nothing is using this connection in the interim? If you're using some sort of connection pool where you get a connection from the pool before each call and return the connection to the pool after the first call, it wouldn't be terribly unusual in a development environment (or low usage environment) to get the same connection 99% of the time for the second call but get a different session 1% of the time.
Can you log the SID and SERIAL# of the session where you are setting the value and where you are retrieving the value?
SELECT sid, serial#
FROM v$session
WHERE sid = sys_context( 'USERENV', 'SID' );
If those are different, you wouldn't expect the value to persist.
Beyond that, there are other ways to clear session state but they require someone to take explicit action. Calling DBMS_SESSION.RESET_PACKAGE or DBMS_SESSION.MODIFY_PACKAGE_STATE(DBMS_SESSION.REINITIALIZE) will clear out any session state set in your session. Compiling the package will do the same thing but that should throw an error warning you that your session state was discarded when you try to read it.
I'm currently trying to code an add-on the popular game World Of Warcraft for a friend. I don't understand too much about the game myself and debugging it within the game is difficult as he's having to do all the testing.
I'm pretty new to Lua, so this may be a very easy question to answer. But when a Lua error occurs in WoW it throws it on screen and gets in the way, this is very bad to a game player as it will stop their gameplay if it throws the exception at the wrong time. I'm looking for a way to cleanly handle the error being thrown. Here's my code so far for the function.
function GuildShoppingList:gslSlashProc()
-- Actions to be taken when command /gsl is procced.
BankTab = GetCurrentGuildBankTab()
BankInfo = GetGuildBankText(BankTab)
local Tabname, Tabicon, TabisViewable, TabcanDeposit, TabnumWithdrawals, remainingWithdrawals = GetGuildBankTabInfo(BankTab)
p1 = BankInfo:match('%-%- GSL %-%-%s+(.*)%s+%-%- ENDGSL %-%-')
if p1 == nil then
self:Print("GSL could not retrieve information, please open the guild bank and select the info tab allow data collection to be made")
else
self:Print("Returning info for: "..Tabname)
for id,qty in p1:gmatch('(%d+):(%d+)') do
--do something with those keys:
local sName, sLink, iRarity, iLevel, iMinLevel, sType, sSubType, iStackCount = GetItemInfo(id);
local iSum = qty/iStackCount
self:Print("We need "..sLink.." x"..qty.."("..iSum.." stacks of "..iStackCount..")")
end
end
end
The problem being when checking to see if p1 is nil, it still throws a Lua error about trying to call p1 as nil. It will be nil at times and this needs to be handled correctly.
What would be the correct most efficient way to go about this?
You might want to wrap your function in a pcall or xpcall which enables you to intercept any error thrown by Lua.
Aside of that, I personally find this construct easier to read:
p1=string.match(str,pat)
if p1 then
-- p1 is valid, eg not nil or false
else
-- handle the problems
end
Whenever I create a large Mathematica project I run into this problem: Preventing avalanche of runtime errors in Mathematica, i.e., Mathematica's error message are opaque, archaic, and legion.
The idea then is to disable all of Mathematica's own error messages and implement type checking and error messages of your own in every Function and Module. However I have not found a simple and efficient way of doing this and end up with, e.g., some function generating an error 20 function calls deep and then get a whole cascade of error messages all the way back up to the main routine.
How would you set up a simple mechanism for this that only generates one error message at the function that experiences the error and a simple list of the chain of function calls?
EDIT: Since it has come up in a couple of answers; I am specifically looking for something lightweight regarding the output it produces (otherwise I could just stick with Mathematica's error messages) and obviously also lightweight in computational overhead. So while Stack and Trace are definitely light on the overhead, their output in complex projects is not quick to parse and some work needs to be done simplifying it.
YAsI - Yet Another (silly?) Idea ...
Re-reading your question ...
The idea then is to disable all of Mathematica's own error messages and implement type checking and error messages of your own in every Function and Module.
Found this:
$MessagePrePrint = ( #; Print[Stack[_][[;; -5]]]; Abort[]) &
v[x_, y_] := w[x, y];
w[x_, y_] := x/y;
StackComplete#v[1, 0];
During evaluation of In[267]:= {StackComplete[v[1,0]];,
StackComplete[v[1,0]], v[1,0], w[1,0], 1/0, 1/0, Message[Power::infy,1/0]}
Out[267]= $Aborted
conclusion ... Aborts at first message and leaves a "reasonable" stack trace. "Reasonable" means "Should be improved".
But it is completely non-intrusive!
To get the ball rolling here is one idea that I've been toying with; the creation of a pseudo stack.
First make a global variable theStack={} and then in every Function or Module start with AppendTo[theStack,"thisFuncName"] and end with theStack=Most#theStack. Assuming moderate (~a few tens) depth of function calls, this should not add any significant overhead.
Then implement your own typing/error checking and use Print#theStack;Abort[]; on errors.
Refinements of this method could include:
Figuring out a way to dynamically get "thisFuncionName" so that the AppendTo[] can be made into an identical function call for all Functions and Module.
Using Message[] Instead of Print[].
Pushing other important variables / stateful information on theStack.
One attempt to implement #Timo's idea (theStack)
Incomplete and perhaps flawed, but just to keep thinking about it:
Clear["Global`*"];
funcDef = t_[args___] \[CircleMinus] a_ :>
{t["nude", args] := a,
ReleaseHold[Hold[t[args] :=
(If[! ValueQ[theStack], theStack = {}];
AppendTo[theStack, ToString[t]];
Check[ss = a, Print[{"-TheStack->", Evaluate#theStack}];
Print#Hold[a]; Abort[]];
theStack = Most#theStack;
Return[ss])
]]};
v[x_, y_]\[CircleMinus] (Sin# g[x, y]) /. funcDef;
g[x_, y_]\[CircleMinus] x/y /. funcDef;
v[2, 3]
v[2, 0]
Output:
Out[299]= Sin[2/3]
During evaluation of In[295]:= Power::infy: Infinite expression 1/0 encountered. >>
During evaluation of In[295]:= {-TheStack->,{v,g}}
During evaluation of In[295]:= Hold[2/0]
Out[300]= $Aborted
A suggestion for extracting stack, maybe something that relies on Trace?
An example of using Trace below, from Chris Chiasson. This code saves evaluation tree of 1 + Sin[x + y] + Tan[x + y] into ~/temp/msgStream.m
Developer`ClearCache[];
SetAttributes[recordSteps, HoldAll];
recordSteps[expr_] :=
Block[{$Output = List#OpenWrite["~/temp/msgStream.m"]},
TracePrint[Unevaluated[expr], _?(FreeQ[#, Off] &),
TraceInternal -> True];
Close /# $Output;
Thread[
Union#Cases[
ReadList["~/temp/msgStream.m", HoldComplete[Expression]],
symb_Symbol /;
AtomQ#Unevaluated#symb &&
Context#Unevaluated#symb === "System`" :>
HoldComplete#symb, {0, Infinity}, Heads -> True],
HoldComplete]
];
recordSteps[1 + Tan[x + y] + Sin[x + y]]
To answer Samsdram's question, the code below (also from Chris) gives evaluation tree of a Mathematica expression. Here is the post from MathGroup with source code and examples.
(Attributes## = {HoldAllComplete}) & /# {traceToTreeAux, toVertex,
HoldFormComplete, getAtoms, getAtomsAux}
MakeBoxes[HoldFormComplete[args___], form_] :=
MakeBoxes[HoldForm[args], form]
edge[{head1_, pos1_, xpr1_}, {head2_, pos2_, xpr2_}] :=
Quiet[Rule[{head1, vertexNumberFunction#pos1, xpr1}, {head2,
vertexNumberFunction#pos2, xpr2}], {Rule::"rhs"}]
getAtomsAux[atom_ /; AtomQ#Unevaluated#atom] :=
Sow[HoldFormComplete#atom, getAtomsAux]
getAtomsAux[xpr_] := Map[getAtomsAux, Unevaluated#xpr, Heads -> True]
getAtoms[xpr_] := Flatten#Reap[getAtomsAux#xpr][[2]]
toVertex[traceToTreeAux[HoldForm[heldXpr_], pos_]] := toVertex[heldXpr]
toVertex[traceToTreeAux[HoldForm[heldXprs___], pos_]] :=
toVertex#traceToTreeAux[Sequence[], pos]
(*this code is strong enough to not need the ToString commands,but \
some of the resulting graph vertices give trouble to the graphing \
routines*)
toVertex[
traceToTreeAux[xpr_, pos_]] := {ToString[
Short#Extract[Unevaluated#xpr, 0, HoldFormComplete], StandardForm],
pos, ToString[Short#First#originalTraceExtract#{pos}, StandardForm]}
traceToTreeAux[xpr_ /; AtomQ#Unevaluated#xpr, ___] := Sequence[]
traceToTreeAux[_HoldForm, ___] := Sequence[]
traceToTreeAux[xpr_, pos_] :=
With[{lhs = toVertex#traceToTreeAux[xpr, pos],
args = HoldComplete ## Unevaluated#xpr},
Identity[Sequence][
ReleaseHold[
Function[Null, edge[lhs, toVertex##], HoldAllComplete] /# args],
ReleaseHold#args]]
traceToTree[xpr_] :=
Block[{vertexNumber = -1, vertexNumberFunction,
originalTraceExtract},
vertexNumberFunction[arg_] :=
vertexNumberFunction[arg] = ++vertexNumber;
originalTraceExtract[pos_] :=
Extract[Unevaluated#xpr, pos, HoldFormComplete]; {MapIndexed[
traceToTreeAux, Unevaluated#xpr, {0, Infinity}]}]
TraceTreeFormPlot[trace_, opts___] :=
Block[{$traceExpressionToTree = True},
Through#{Unprotect, Update}#SparseArray`ExpressionToTree;
SparseArray`ExpressionToTree[trace, Infinity] = traceToTree#trace;
With[{result = ToExpression#ToBoxes#TreeForm[trace, opts]},
Through#{Unprotect, Update}#SparseArray`ExpressionToTree;
SparseArray`ExpressionToTree[trace, Infinity] =.;
Through#{Update, Protect, Update}#SparseArray`ExpressionToTree;
result]];
TraceTreeFormPlot[Trace[Tan[x] + Sin[x] - 2*3 - 55]]
Perhaps we have been over thinking this. What if we just tweaked the pattern matching on the arguments a little. For instance, if we modified the function to check for a numeric quantity and added some code to print an error if it fails. For instance,
TypeNumeric[x_] := If[! NumericQ[Evaluate[x]],
Print["error at "]; Print[Stack[]]; Print["Expression "]; Print[x]; Print["Did
not return a numeric value"];Return[False],
(*Else*)
Return[True];]
SetAttributes[TypeNumeric, HoldAll];
Step 2: If you have a function, f[x_] that requires a numeric quantity, just write it with the standard pattern test and all should be well
Input:
f[x_?TypeNumeric] := Sqrt[x]
f[Log[y]]
f[Log[5]]
Output:
error at
{f}
Expression
Log[y]
Did not return a numeric value
f[Log[y]]
Sqrt[Log[5]]
I believe this will work and, it makes robust type checking as simple as a writing a function or two. The problem is that this could be hugely inefficient because this code evaluates the expression x twice, once for the type checking and once for real. This could be bad if an expensive function call is involved.
I haven't figured out the way around this second problem and would welcome suggestions on that front. Are continuations the way out of this problem?
Hope this helps.