Redis (PHP-Redis) SCAN and KEYS show different results with same pattern - redis

I am using PHP-Redis with Redis Version 3.1.6
$result = $redis->keys('source_1234_[a-zA-Z]*_[0-9]*');
produces
{array} [6]
0 = "source_1234_test_1"
1 = "source_1234_test_2"
2 = "source_1234_test_3"
3 = "source_1234_test_4"
4 = "source_1234_test_5"
5 = "source_1234_test_6"
However
$iterator = 0;
$result = $redis->scan($iterator, 'source_1234_[a-zA-Z]*_[0-9]*');
returns
FALSE
I am reading the docs for KEYS and SCAN but all it says it that supports glob-style patterns.
So checking http://www.globtester.com/ I can confirm that the pattern is valid and should return the correct results. Why is there a difference and why does SCAN return FALSE in this case?

Two problems with your code:
(a) You need to set the iterator to NULL, not 0.
0 is returned from the call to SCAN to indicate that all keys have been scanned. It will therefore stop and return false.
(b) SCAN iterates over sets of all keys, returning the matches from each set for each call. You're only calling scan once. It will scan the first COUNT keys and return false if none of those happen to match.
See https://redis.io/commands/scan#number-of-elements-returned-at-every-scan-call:
SCAN family functions do not guarantee that the number of elements returned per call are in a given range. The commands are also allowed to return zero elements, and the client should not consider the iteration complete as long as the returned cursor is not zero.[...]
To get identical results to KEYS you need to iterate over all sets of keys:
$iterator = NULL
while($iterator != 0) {
$arr_keys = $redis->scan($iterator, 'source_1234_[a-zA-Z]*_[0-9]*')
foreach($arr_keys as $str_key) {
echo $str_key;
}
}

Try something like that:
$redisClient = new Redis();
$redisClient->connect($config['devices']['redis_ip']);
$redisClient->setOption(Redis::OPT_SCAN, Redis::SCAN_RETRY);
$start_time = microtime(TRUE);
$it = NULL;
while ($keys = $redisClient->scan($it, "useragent:*")) {
foreach ($keys as $key){
// Do something with the key
usleep(1000);
}
}
$end_time = microtime(TRUE);
$time = $end_time - $start_time;
echo "\t[done]: Total time: {$time} seconds\n";
$redisClient->close();

Since I was looking for this, I'm posting what worked for me
$iterator = null;
while(false !== ($keys = $redis->scan($iterator, $pattern))) {
foreach($keys as $key) {
echo $key . PHP_EOL;
}
}
I took this from the documentation of scan of php-redis.
This should print every key matching the $pattern.

Related

Why am I getting a missing hash-tags error when I try to run JedisCluster.scan() using a match pattern?

I'm trying to run scan on my redis cluster using Jedis. I tried using the .scan(...) method as follows for a match pattern but I get the following error:
"JedisCluster only supports SCAN commands with MATCH patterns containing hash-tags"
my code is as follows (excerpted):
private final JedisCluster redis;
...
String keyPrefix = "helloWorld:*";
ScanParams params = new ScanParams()
.match(keyPrefix)
.count(100);
String cur = SCAN_POINTER_START;
boolean done = false;
while (!done) {
ScanResult<String> resp = redis.scan(cur, params);
...
cur = resp.getStringCursor();
if (resp.getStringCursor().equals(SCAN_POINTER_START)) {
done = true;
}
}
When I run my code, it gives a weird error talking about hashtags:
"JedisCluster only supports SCAN commands with MATCH patterns containing hash-tags"
In the redis-cli I could just use match patterns like that what I wrote for the keyPrefix variable. Why am I getting an error?
How do I get Jedis to show me all the the keys that match a given substring?
The problem was that the redis variable is a RedisCluster object and not a Redis object.
A redis cluster object has a collection of nodes and scanning that is different than scanning an individual node.
To solve the issue, you can scan through each individual node as such:
String keyPrefix = "helloWorld:*";
ScanParams params = new ScanParams()
.match(keyPrefix)
.count(100);
redis.getClusterNodes().values().stream().forEach(pool -> {
boolean done = false;
String cur = SCAN_POINTER_START;
try (Jedis jedisNode = pool.getResource()) {
while (!done) {
ScanResult<String> resp = jedisNode.scan(cur, params);
...
cur = resp.getStringCursor();
if (cur.equals(SCAN_POINTER_START)) {
done = true;
}
}
}
});

Kotlin: function to stop iteration when predicate satisfied

I'm looking for a function in Kotlin which stops an iteration once a predicate is fulfilled:
val services = listOf(service1, service2)
...
var res: Result = null
services.stopIfPredicateFulFilled { service ->
res = service.doSomething()
res != null
}
While this example is not really nice since res is overwritten in each iteration I hope the intention is clear.
forEach doesn't do the job the way I expect it to be done. So, I was wondering if there isn't anything else.
You can use the functions find { ... } and firstOrNull { ... } (they are equivalent, just named differently). They find the first element satisfying the predicate and return that element, ignoring all the remaining elements.
services.find { service ->
res = service.doSomething()
res != null
}

Filter cached sqlJdbs query in Pentaho CE

I use sqlJdbs query as a data provider for my CCC controls. I use geospatial request in my query that's why I cache my results(Cache=True). Otherwise the request made long.
It works fine. However I have to use parameters in my query to filter resulting rows:
SELECT ...
FROM ...
WHERE someField IN (${aoi_param})
Is there some way to cache full set of rows and then apply WHERE to cached results without rebuilding new cache for each set of values in the ${aoi_param}?
What is the best practice?
So, I am not really sure that it is the best practice, but I solved my problem this way:
I included aoi_param to the Listeners and Parameters of my chart control
Then I filtered data set in Post Fetch:
function f(data){
var _aoi_param = this.dashboard.getParameterValue('${p:aoi_param}');
function isInArray(myValue, myArray) {
var arrayLength = myArray.length;
for (var i = 0; i < arrayLength; i++) {
if (myValue == myArray[i]) return true;
}
return false;
}
function getFiltered(cdaData, filterArray) {
var allCdaData = cdaData;
cdaData = {
metadata: allCdaData.metadata,
resultset: allCdaData.resultset.filter(function(row){
// 2nd column is an AOI id in my dataset
return isInArray(row[2], filterArray);
})
};
return cdaData;
}
var dataFiltered = getFiltered(data, _aoi_param);
return dataFiltered;
}
excluded WHERE someField IN (${aoi_param}) from the query of my sql over sqlJdbc component

SCAN command performance with phpredis

I'm replacing KEYS with SCAN using phpredis.
$redis = new Redis();
$redis->connect('127.0.0.1', 6379);
$redis->setOption(Redis::OPT_SCAN, Redis::SCAN_RETRY);
$it = NULL;
while($arr_keys = $redis->scan($it, "mykey:*", 10000)) {
foreach($arr_keys as $str_key) {
echo "Here is a key: $str_key\n";
}
}
According to redis documentation, I use SCAN to paginate searches to avoid disadvantage of using KEYS.
But in practice, using above code costs me 3 times lower than using just a single $redis->keys()
So I'm wondering if I've done something wrong, or I have to pay speed to avoid KEYS's threat?
Note that I totally have 400K+ keys in my db, and 4 mykey:* keys
A word of caution of using the example:
$it = NULL;
while($arr_keys = $redis->scan($it, "mykey:*", 10000)) {
foreach($arr_keys as $str_key) {
echo "Here is a key: $str_key\n";
}
}
That can return empty array's if none of the 10000 keys scanned matches and then it will give up, and you didn't get all the keys you wanted! I would recommend doing more like this:
$it = null;
do
{
$arr_keys = $redis->scan($it, $key, 10000);
if (is_array($arr_keys) && !empty($arr_keys))
{
foreach ($arr_keys as $str_key)
{
echo "Here is a key: $str_key\n";
}
}
} while ($arr_keys !== false);
And why it takes so long, 400k+ keys, 10000, that's 40 scan requests to redis, if it's not on the local machine, add latency for every 40 redis query to your speed.
Since using keys in production environments is just forbidden because it blocks the entire server while iterating global space keys, then, there's no discussion here about use or not to use keys.
In the other hand, if you want to speed up things, you should go further with Redis: you should index your data.
I doubt that these 400K keys couldn't be categorized in sets or sorted sets, or even hashes, so when you need a particular subset of your 400K-keys-database you could run any scan-equivalent command against a set of 1K items, instead of 400K.
Redis is about indexing data. If not, you're using it like just a simple key-value store.

Why can I not use Continuation when using a proxy class to access MS CRM 2013?

So I have a standard service reference proxy calss for MS CRM 2013 (i.e. right-click add reference etc...) I then found the limitation that CRM data calls limit to 50 results and I wanted to get the full list of results. I found two methods, one looks more correct, but doesn't seem to work. I was wondering why it didn't and/or if there was something I'm doing incorrectly.
Basic setup and process
crmService = new CrmServiceReference.MyContext(new Uri(crmWebServicesUrl));
crmService.Credentials = System.Net.CredentialCache.DefaultCredentials;
var accountAnnotations = crmService.AccountSet.Where(a => a.AccountNumber = accountNumber).Select(a => a.Account_Annotation).FirstOrDefault();
Using Continuation (something I want to work, but looks like it doesn't)
while (accountAnnotations.Continuation != null)
{
accountAnnotations.Load(crmService.Execute<Annotation>(accountAnnotations.Continuation.NextLinkUri));
}
using that method .Continuation is always null and accountAnnotations.Count is always 50 (but there are more than 50 records)
After struggling with .Continutation for a while I've come up with the following alternative method (but it seems "not good")
var accountAnnotationData = accountAnnotations.ToList();
var accountAnnotationFinal = accountAnnotations.ToList();
var index = 1;
while (accountAnnotationData.Count == 50)
{
accountAnnotationData = (from a in crmService.AnnotationSet
where a.ObjectId.Id == accountAnnotationData.First().ObjectId.Id
select a).Skip(50 * index).ToList();
accountAnnotationFinal = accountAnnotationFinal.Union(accountAnnotationData).ToList();
index++;
}
So the second method seems to work, but for any number of reasons it doesn't seem like the best. Is there a reason .Continuation is always null? Is there some setup step I'm missing or some nice way to do this?
The way to get the records from CRM is to use paging here is an example with a query expression but you can also use fetchXML if you want
// Query using the paging cookie.
// Define the paging attributes.
// The number of records per page to retrieve.
int fetchCount = 3;
// Initialize the page number.
int pageNumber = 1;
// Initialize the number of records.
int recordCount = 0;
// Define the condition expression for retrieving records.
ConditionExpression pagecondition = new ConditionExpression();
pagecondition.AttributeName = "address1_stateorprovince";
pagecondition.Operator = ConditionOperator.Equal;
pagecondition.Values.Add("WA");
// Define the order expression to retrieve the records.
OrderExpression order = new OrderExpression();
order.AttributeName = "name";
order.OrderType = OrderType.Ascending;
// Create the query expression and add condition.
QueryExpression pagequery = new QueryExpression();
pagequery.EntityName = "account";
pagequery.Criteria.AddCondition(pagecondition);
pagequery.Orders.Add(order);
pagequery.ColumnSet.AddColumns("name", "address1_stateorprovince", "emailaddress1", "accountid");
// Assign the pageinfo properties to the query expression.
pagequery.PageInfo = new PagingInfo();
pagequery.PageInfo.Count = fetchCount;
pagequery.PageInfo.PageNumber = pageNumber;
// The current paging cookie. When retrieving the first page,
// pagingCookie should be null.
pagequery.PageInfo.PagingCookie = null;
Console.WriteLine("#\tAccount Name\t\t\tEmail Address");while (true)
{
// Retrieve the page.
EntityCollection results = _serviceProxy.RetrieveMultiple(pagequery);
if (results.Entities != null)
{
// Retrieve all records from the result set.
foreach (Account acct in results.Entities)
{
Console.WriteLine("{0}.\t{1}\t\t{2}",
++recordCount,
acct.EMailAddress1,
acct.Name);
}
}
// Check for more records, if it returns true.
if (results.MoreRecords)
{
// Increment the page number to retrieve the next page.
pagequery.PageInfo.PageNumber++;
// Set the paging cookie to the paging cookie returned from current results.
pagequery.PageInfo.PagingCookie = results.PagingCookie;
}
else
{
// If no more records are in the result nodes, exit the loop.
break;
}
}