Redis StackExchange LuaScripts with parameters - redis

I'm trying to use the following Lua script using C# StackExchange library:
private const string LuaScriptToExecute = #"
local current
current = redis.call(""incr"", KEYS[1])
if current == 1 then
redis.call(""expire"", KEYS[1], KEYS[2])
return 1
else
return current
end
Whenever i'm evaluating the script "as a string", it works properly:
var incrementValue = await Database.ScriptEvaluateAsync(LuaScriptToExecute,
new RedisKey[] { key, ttlInSeconds });
If I understand correctly, each time I invoke the ScriptEvaluateAsync method, the script is transmitted to the redis server which is not very effective.
To overcome this, I tried using the "prepared script" approach, by running:
_setCounterWithExpiryScript = LuaScript.Prepare(LuaScriptToExecute);
...
...
var incrementValue = await Database.ScriptEvaluateAsync(_setCounterWithExpiryScript,
new[] { key, ttlInSeconds });
Whenever I try to use this approach, I receive the following error:
ERR Error running script (call to f_7c891a96328dfc3aca83aa6fb9340674b54c4442): #user_script:3: #user_script: 3: Lua redis() command arguments must be strings or integers
What am I doing wrong?
What is the right approach in using "prepared" LuaScripts that receive dynamic parameters?

If I look in the documentation: no idea.
If I look in the unit test on github it looks really easy.
(by the way, is your ttlInSeconds really RedisKey and not RedisValue? You are accessing it thru KEYS[2] - shouldnt that be ARGV[1]? Anyway...)
It looks like you should rewrite your script to use named parameters and not arguments:
private const string LuaScriptToExecute = #"
local current
current = redis.call(""incr"", #myKey)
if current == 1 then
redis.call(""expire"", #myKey, #ttl)
return 1
else
return current
end";
// We should load scripts to whole redis cluster. Even when we dont have any.
// In that case, there will be only one EndPoint, one iteration etc...
_myScripts = _redisMultiplexer.GetEndPoints()
.Select(endpoint => _redisMultiplexer.GetServer(endpoint))
.Where(server => server != null)
.Select(server => lua.Load(server))
.ToArray();
Then just execute it with anonymous class as parameter:
for(var setCounterWithExpiryScript in _myScripts)
{
var incrementValue = await Database.ScriptEvaluateAsync(
setCounterWithExpiryScript,
new {
myKey: (RedisKey)key, // or new RedisKey(key) or idk
ttl: (RedisKey)ttlInSeconds
}
)// .ConfigureAwait(false); // ? ;-)
// when ttlInSeconds is value and not key, just dont cast it to RedisKey
/*
var incrementValue = await
Database.ScriptEvaluateAsync(
setCounterWithExpiryScript,
new {
myKey: (RedisKey)key,
ttl: ttlInSeconds
}
).ConfigureAwait(false);*/
}
Warning:
Please note that Redis is in full-stop mode when executing scripts. Your script looks super-easy (you sometimes save one trip to redis (when current != 1) so i have a feeling that this script will be counter productive in greater-then-trivial scale. Just do one or two calls from c# and dont bother with this script.

First of all, Jan's comment above is correct.
The script line that updated the key's TTL should be redis.call(""expire"", KEYS[1], ARGV[1]).
Regarding the issue itself, after searching for similar issues in RedisStackExchange's Github, I found that Lua scripts do not work really well in cluster mode.
Fortunately, it seems that "loading the scripts" isn't really necessary.
The ScriptEvaluateAsync method works properly in cluster mode and is sufficient (caching-wise).
More details can be found in the following Github issue.
So at the end, using ScriptEvaluateAsync without "preparing the script" did the job.
As a side note about Jan's comment above that this script isn't needed and can be replaced with two C# calls, it is actually quite important since this operation should be atomic as it is a "Rate limiter" pattern.

Related

Readding all key batch wise using stack exchange Redis client

i am using StackExchange.Redis (2.1.58) , i write below code by using cursor .
Dictionary<string, string> keyResult = new Dictionary<string, string>();
var Server = Connection.GetServer(Connection.GetEndPoints()[0]);
long current_cursor = 0;
int next_cursor =-1;
long page_size = 10;
while(next_cursor!=0)
{
var allkeys = Server.Keys(RedisCache.Database, argKeyPattern, 10, 0, next_cursor==-1 ? 0 : next_cursor);
var cursor = ((StackExchange.Redis.IScanningCursor)allkeys);
foreach (var key in allkeys)
{
if (current_cursor == cursor.Cursor)
{
keyResult.Add(key, RedisCache.StringGet(key));
}
else
{
next_cursor = Convert.ToInt32(cursor.Cursor);
current_cursor = next_cursor;
break;
}
}
}
this code works fine , my question is there any other approach to read keys from Redis batch wise in more efficient way ?
Thanks !!
From the documentation
The Keys(...) method deserves special mention: it is unusual in that it does not have an *Async counterpart. The reason for this is that behind the scenes, the system will determine the most appropriate method to use (KEYS vs SCAN, based on the server version), and if possible will use the SCAN approach to hand you back an IEnumerable that does all the paging internally - so you never need to see the implementation details of the cursor operations. If SCAN is not available, it will use KEYS, which can cause blockages at the server.
From first looks, KEYS command should be avoid. However, the library already fix that for you by using SCAN if the command is available. So I think you're good here.
if SCAN command is not available so Keys() will fallback to using KEYS command
SCAN command is available in version 2.8++

Mojolicious template cache is stale

I'm currently developing a small single-page Web app using Mojolicious. The app has a Javascript frontend (using Backbone) that talks to a REST-ish API; the layout of the source is roughly:
use Mojolicious::Lite;
# ... setup code ...
get '/' => sub {
my $c = shift;
# fetch+stash data for bootstrapped collections...
$c->render('app_template');
};
get '/api_endpoint' => sub {
my $c = shift;
# fetch appropriate API data...
$c->render(json => $response);
};
# ... more API endpoints ...
app->start;
The app template uses EP, but very minimally; the only server-side template directives just insert JSON for bootstrapped collections. It's deployed via Apache as a plain CGI script. (This isn't optimal, but it's for low-traffic internal use, and more intricate server configuration is problematic in context.) Perl CGI is configured via mod_perl.
This works most of the time, but occasionally the renderer somehow gets the idea that it should cache the template and ignore changes to it. The debug records in error_log show "Rendering cached template" rather than the normal "Rendering template", and my new changes to the template stop appearing in the browser. I can't find a reliable way to stop this, though it will eventually stop on its own according to conditions I can't discern.
How can I make the app notice template changes reliably? Alternatively, how can I disable template caching completely?
How can I make the app notice template changes reliably?
This is what the morbo development server is for. Morbo wouldn't be used for your live code deployment, but for a development environment where you are continually changing your code and templates. Generally changes to live code and templates are meant to be handled by restarting the application server, or Apache in your case. (Hypnotoad has a hot-restart capability for this purpose)
Alternatively, how can I disable template caching completely?
To do this, add the following setup code (outside of routes, after use Mojolicious::Lite):
app->renderer->cache->max_keys(0);
For old answer see below.
I turned the findings of this answer into a plugin and released it on CPAN as Mojolicious::Plugin::Renderer::WithoutCache after discussing wit Grinnz on IRC, where they encouraged a release.
You can use it like this:
use Mojolicious::Lite;
plugin 'Renderer::WithoutCache';
It will create a new Cache object that does nothing, and install that globally into the renderer. That way, it doesn't need to be created every time like my initial answer below did.
In theory, this should be faster than Grinnz' approach (which is more sensible), and since you explicitly don't want to cache, you obviously want things to be as fast as possible, right? It's supposedly faster because the real Mojo::Cache would still need to go and try to set the cache, but then abort because there are no more free keys, and it also would try to look up the values from the cache every time.
I benchmarked this with both Dumbbench and Benchmark. Both of them showed negligible results. I ran them each a couple of times, but they fluctuated a lot, and it's not clear which one is faster. I included output of a run where my implementation was faster, but it still shows how minuscule the difference is.
Benchmark with Dumbbench:
use Dumbbench;
use Mojolicious::Renderer;
use Mojolicious::Controller;
use Mojolicious::Plugin::Renderer::WithoutCache::Cache;
my $controller = Mojolicious::Controller->new;
my $renderer_zero_keys = Mojolicious::Renderer->new;
$renderer_zero_keys->cache->max_keys(0);
my $renderer_nocache = Mojolicious::Renderer->new;
$renderer_nocache->cache( Mojolicious::Plugin::Renderer::WithoutCache::Cache->new );
my $bench = Dumbbench->new(
target_rel_precision => 0.005,
initial_runs => 5000,
);
$bench->add_instances(
Dumbbench::Instance::PerlSub->new(
name => 'max_keys',
code => sub {
$renderer_zero_keys->render( $controller, { text => 'foobar' } );
}
),
Dumbbench::Instance::PerlSub->new(
name => 'WithoutCache',
code => sub {
$renderer_nocache->render( $controller, { text => 'foobar' } );
}
),
);
$bench->run;
$bench->report;
__END__
max_keys: Ran 8544 iterations (3335 outliers).
max_keys: Rounded run time per iteration: 5.19018e-06 +/- 4.1e-10 (0.0%)
WithoutCache: Ran 5512 iterations (341 outliers).
WithoutCache: Rounded run time per iteration: 5.0802e-06 +/- 5.6e-09 (0.1%)
Benchmark with Benchmark:
use Benchmark 'cmpthese';
use Mojolicious::Renderer;
use Mojolicious::Controller;
use Mojolicious::Plugin::Renderer::WithoutCache::Cache;
my $controller = Mojolicious::Controller->new;
my $renderer_zero_keys = Mojolicious::Renderer->new;
$renderer_zero_keys->cache->max_keys(0);
my $renderer_nocache = Mojolicious::Renderer->new;
$renderer_nocache->cache( Mojolicious::Plugin::Renderer::WithoutCache::Cache->new );
cmpthese(
-5,
{
'max_keys' => sub {
$renderer_zero_keys->render( $controller, { text => 'foobar' } );
},
'WithoutCache' => sub {
$renderer_nocache->render( $controller, { text => 'foobar' } );
},
}
);
__END__
Rate max_keys WithoutCache
max_keys 190934/s -- -2%
WithoutCache 193846/s 2% --
I recon in a heavy load environment with lots of calls it would eventually make a difference, but that is very hard to prove. So if you don't like to think about the internals of the cache, this plugin might be useful.
Old answer:
Looking at Mojolicious::Plugin::EPRenderer I found out that there is a cache. It's a Mojo::Cache instance, which has the methods get, set and max_keys, and inherits from Mojo::Base (like probably everything in Mojolicious).
The ::EPRenderer gets a $renderer, which is a Mojolicious::Renderer. It holds the Mojo::Cache instance. I looked at $c with Data::Printer, and found out that there is a $c->app that holds all of those.
Knowing this, you can easily make your own cache class that does nothing.
package Renderer::NoCache;
use Mojo::Base -base;
sub get {}
sub set {}
sub max_keys {}
Now you stick it into $c.
package Foo;
use Mojolicious::Lite;
get '/' => sub {
my $c = shift;
$c->app->renderer->cache( Renderer::NoCache->new );
$c->render(template => 'foo', name => 'World');
};
app->start;
__DATA__
## foo.html.ep
Hello <%= $name =%>.
Now every attempt to get or set the cache simply does nothing. It will try caching, but it will never find anything.
Of course it's not great to make a new object every time. It would be better to make that object once at startup and get it into the internal permanent version of app. You have CGI, so it might not make a difference.
You could also just monkey-patch the get out of Mojo::Cache. This more hacky approach will do the same thing:
package Foo;
use Mojolicious::Lite;
*Mojo::Cache::get = sub { };
get '/' => sub {
my $c = shift;
$c->render(template => 'foo', name => 'World');
};
app->start;
But beware: we just disabled fetching from every cache in your application that uses Mojo::Cache. This might not be what you want.

RavenDB fails with ConcurrencyException when using new transaction

This code always fails with a ConcurrencyException:
[Test]
public void EventOrderingCode_Fails_WithConcurrencyException()
{
Guid id = Guid.NewGuid();
using (var scope1 = new TransactionScope())
using (var session = DataAccess.NewOpenSession)
{
session.Advanced.UseOptimisticConcurrency = true;
session.Advanced.AllowNonAuthoritativeInformation = false;
var ent1 = new CTEntity
{
Id = id,
Name = "George"
};
using (var scope2 = new TransactionScope(TransactionScopeOption.RequiresNew))
{
session.Store(ent1);
session.SaveChanges();
scope2.Complete();
}
var ent2 = session.Load<CTEntity>(id);
ent2.Name = "Gina";
session.SaveChanges();
scope1.Complete();
}
}
It fails at the last session.SaveChanges. Stating that it is using a NonCurrent etag. If I use Required instead of RequiresNew for scope2 - i.e. using the same Transaction. It works.
Now, since I load the entity (ent2) it should be using the newest Etag unless this is some cached value attached to scope1 that I am using (but I have disabled Caching). So I do not understand why this fails.
I really need this setup. In the production code the outer TransactionScope is created by NServiceBus, and the inner is for controlling an aspect of event ordering. It cannot be the same Transaction.
And I need the optimistic concurrency too - if other threads uses the entity at the same time.
BTW: This is using Raven 2.0.3.0
Since no one else have answered, I had better give it a go myself.
It turns out this was a human error. Due to a bad configuration of our IOC container the DataAccess.NewOpenSession gave me the same Session all the time (across other tests). In other words Raven works as expected :)
Before I found out about this I also experimented with using TransactionScopeOption.Suppress instead of RequiresNew. That also worked. Then I just had to make sure that whatever I did in the suppressed scope could not fail. Which was a valid option in my case.

Error when trying add data to RavenDb

I'm using autofac and the interfaces are correctly resolved but this code fails with "No connection could be made because the target machine actively refused it 127.0.0.1:8081"
using (var store = GetService<IDocumentStore>())
{
using (var session = store.OpenSession())
{
session.Store(new Entry { Author = "bob", Comment = "My son says this", EntryId = Guid.NewGuid(), EntryTime = DateTime.Now, Quote = "I hate you dad." });
session.SaveChanges();
}
}
Here is the registration
builder.Register<IDocumentStore>(c =>
{
var store = new DocumentStore { Url = "http://localhost:8081" };
store.Initialize();
return store;
}).SingleInstance();
When I navigate to http://localhost:8081 I do get the silverlight management UI. Although I'm running a Windows VM and vmware and Silverlight5 don't play together. That's another issue entirely. Anyways does anyone see what I'm doing wrong here or what I should be doing differently? Thanks for any code, tips, or tricks.
On a side note, can I enter some dummy records from a command line interface? Any docs or examples of how I can do that?
Thanks All.
Just curious, are you switching RavenDB to listen on 8081? The default is 8080. If you're getting the management studio to come up, I suspect you are.
I'm not too familiar with autofac but, it looks like you're wrapping your singleton DocumentStore in a using statement.
Try:
using (var session = GetService<IDocumentStore>().OpenSession())
{
}
As far as dummy records go, the management studio will ask you if you want to generate some dummy data if your DB is empty. If you can't get silverlight to work in the VM, I'm not sure if there's another automated way to do it.
Perhaps using smuggler:
http://ravendb.net/docs/server/administration/export-import
But you'd have to find something to import.

Efficient way to run multiple scripts using javax.script

I am developing a game where I'd like to have multiple scripts that all implement the same structure. Each script would need to be run in its own scope so that code doesn't overlap other scripts. For example:
structure.js
function OnInit() {
// Define resources to load, collision vars, etc.
}
function OnLoop() {
// Every loop
}
function ClickEvent() {
// Someone clicked me
}
// Other fun functions
Now, lets say I have: "BadGuy.js", "ReallyReallyBadGuy.js", "OtherBadGuy.js" - They all look like the above in terms of structure. Within the game whenever an event takes place, I'd like to invoke the appropriate function.
The problem comes down to efficiency and speed. I found a working solution by creating an engine for each script instance (using getEngineByName), but that just doesn't seem ideal to me.
If there isn't a better solution, I'll probably resort to each script having its own unique class / function names. I.e.
BadGuy.js
var BadGuy = new Object();
BadGuy.ClickEvent = function() {
}
I don't think you need to create a new ScriptEngine for every "Guy". You can manage them all in one engine. So with advance apologies for butchering you game scenario.....
Get one instance of the Rhino engine.
Issue eval(script) statements to add new JS Objects to the engine, along with the different behaviours (or functions) that you want these Objects to support.
You have a couple of different choices for invoking against each one, but as long as each "guy" has a unique name, you can always reference them by name and invoke a named method against it.
For more performance sensitive operations (perhaps some sort of round based event loop) you can precompile a script in the same engine which can then be executed without having to re-evaluate the source.
Here's a sample I wrote in Groovy.
import javax.script.*;
sem = new ScriptEngineManager();
engine = sem.getEngineByExtension("js");
engine.getBindings(ScriptContext.ENGINE_SCOPE).put("out", System.out);
eventLoop = "for(guy in allGuys) { out.println(allGuys[guy].Action(action)); }; "
engine.eval("var allGuys = []");
engine.eval("var BadGuy = new Object(); allGuys.push(BadGuy); BadGuy.ClickEvent = function() { return 'I am a BadGuy' }; BadGuy.Action = function(activity) { return 'I am doing ' + activity + ' in a BAD way' }");
engine.eval("var GoodGuy = new Object(); allGuys.push(GoodGuy); GoodGuy.ClickEvent = function() { return 'I am a GoodGuy' }; GoodGuy.Action = function(activity) { return 'I am doing ' + activity + ' in a GOOD way' }");
CompiledScript executeEvents = engine.compile(eventLoop);
println engine.invokeMethod(engine.get("BadGuy"), "ClickEvent");
println engine.invokeMethod(engine.get("GoodGuy"), "ClickEvent");
engine.getBindings(ScriptContext.ENGINE_SCOPE).put("action", "knitting");
executeEvents.eval();