According to Redis documentation on PFADD command:
Return value
Integer reply, specifically:
1 if at least 1 HyperLogLog internal register was altered. 0 otherwise.
Can anyone explain the following two points?
Does this mean PFADD will return "1" if the counter was really incremented by 1? is it guaranteed that after running PFADD, the new PFCOUNT will be PFCOUNT(before) + output of PFADD? In other words, can a single-threaded client keep track of the count using only the output of PFADD?
When PFADD returns "0" or "1", do they translate to a "cache hit" and a "cache miss" respectively?
Does this mean PFADD will return "1" if the counter was really incremented by 1?
No.
The return value is purely boolean, i.e it only indicates whether or not the
underlying HyperLogLog was modified.
Is it guaranteed that after running PFADD, the new PFCOUNT will be PFCOUNT(before) + output of PFADD?
No, since the output of PFADD does not represent a count (see above).
That being said, you may want to use the output of PFADD as a trigger to call
PFCOUNT again, as explained by antirez in the original blog post:
This is interesting for the user since as we add elements the
probability of an element actually modifying some register decreases.
The fact that the API is able to provide hints about the fact that a
new cardinality is available allows for programs that continuously add
elements and retrieve the approximated cardinality only when a new one
is available.
At last:
When PFADD returns "0" or "1", do they translate to a "cache hit" and a "cache miss" respectively?
No. As detailed above it only indicates that a new cardinality is available.
Related
I'm using Redis Key Space Notification to get my application notified when a specified key gets expired.
But when the key gets expired, Redis deletes the key, i need to remove this behavior because my application can use this expired information in another moment.
Is there a way to remove this behavior?
As #sazzad and #nitrin0 said, there's no way to change that.
As another option to get a similar result, I'd suggest you use a sorted set to track these "psuedo-expirations", and when they "expire", a background process does whatever else you need the key for: move it, transform it, reset the expiration, etc.
Use the command zadd to both create a new sorted set and to add members to it. The key for the set can be anything, but I'd use the members as the keys from the data that expires so you can easily work with both the real data, and the member in the sorted set.
ZADD name-of-sorted-set NX timestamp-when-data-expires key-of-real-data
Let's break this down:
name-of-sorted-set is what you'd use in the other Z* commands to work with this specific sorted set.
NX means "Only add new elements. Don't update already existing elements.". The other option is XX which is "Only update elements that already exist. Don't add new elements." For this, the only options are NX or nothing.
timestamp-when-data-expires is the score for this member, and we'll use it as the exact timestamp when the data would otherwise "expire", so you'll have to do some calculations in your application to provide the timestamp instead of just the seconds until it expires.
key-of-real-data is the exact key used for the real data this represents. Using the exact key here will help easily combine the two when you're working with this sorted set to find which members have "expired", since the members are the keys you'd use to move, delete, transform, the data.
Next I'd have a background process run zrangebyscore to see if there are any members whose scores (timestamps) are within some range:
ZRANGEBYSCORE name-of-sorted-set min-timestamp max-timestamp WITHSCORES LIMIT 0 10
Let's break this down too:
name-of-sorted-set is the key for the set we chose in ZADD above
min-timestamp is the lower end of the range to find members that have "expired"
max-timestamp is the higher end of the range
WITHSCORES tells Redis to return the name of the members AND their scores
LIMIT allows us to set an offset (the 0) and a count of items to return (the 10). This is just an example, but for very large data sets you'll likely have to make use of both the offset and count limits.
ZRANGEBYSCORE will return something like this if using redis-cli:
1) "first-member"
2) "1631648102"
3) "second-member"
4) "1631649154"
5) "third-member"
6) "1631650374"
7) "fourth-member"
8) "1631659171"
9) "fifth-member"
10) "1631659244"
Some Redis clients will change that, so you'll have to test it in your application. In redis-cli the member-score pair is returned over two lines.
Now that you have the members (keys of the actual data) that have "expired" you can do whatever it is you need to do with them, then probably either remove them from the set entirely, or remove them and replace them. Since in this example we created the sorted set with the NX example, we can't update existing records, only insert new ones.
I like to schedule observations of variable duration (planning entity) into hourly time slots over several nights. I would need to impose that there are no gaps in particular groups, and need collectors for minimum, maximum and sum. Is there a workaround to have a groupBy with one groupKeyMapping and three collectors?
constraintFactory.from(OB.class)
.groupBy(OB::getGroupID, min(OB::getStart), max(OB::getEnd), sum(OB::getDuration))
I tried to workaround this using toList() and computing values myself but strangely it doesn't pass down a List<OB> but single OBs. The code below prints class my.package.OB
constraintFactory.from(OB.class)
.groupBy(OB::getGroupID, toList())
.filter((groupID, entries) -> {
println "=> ${entries.class} "
return true
})
This was a gap in our API (see PLANNER-2330). It is being addressed for OptaPlanner 8.3.0. We encourage you to upgrade - not only to get this particular feature, but also some nice performance improvements, assuming you're using higher cardinality joins.
As per Java Card v3.1 new package is defined javacardx.security.derivation
https://docs.oracle.com/en/java/javacard/3.1/jc_api_srvc/api_classic/javacardx/security/derivation/package-summary.html
KDF X9.63 works on three inputs: input secret, counter and shared info.
Depends on length of generated key material, multiple rounds on hash is carried out to generated final output.
I am using this KDF via JC API to generated 64 bytes of output (which is carried out by 2 rounds of SHA-256) for a 16 bytes-Encryption Key, a 16 bytes-IV, and a 32 bytes-MAC Key.
Note: This is just pseudo code to put my question with necessary details.
DerivationFunction df = DerivationFunction.getInstance(DerivationFunction.ALG_KDF_ANSI_X9_63, false);
df.init(KDFAnsiX963Spec(MessageDigest.ALG_SHA_256, input, sharedInfo, (short) 64);
SecretKey encKey = KeyBuilder.buildKey(KeyBuilder.TYPE_AES, (short)16, false);
SecretKey macKey = KeyBuilder.buildKey(KeyBuilder.TYPE_HMAC, (short)32, false);
df.nextBytes(encKey);
df.nextBytes(IVBuffer, (short)0, (short)16);
df.lastBytes(macKey);
I have the following questions:
When rounds of KDF are performed? Are these performed during df.init() or during df.nextBytes() & df.lastBytes()?
One KDF round will generate 32 bytes output (considering SHA-256 algorithm) then how API's df.nextBytes() & df.lastBytes() will work with any output expected length < 32 bytes?
In this KDF counter is incremented in every next round then how counter will be managed between df.nextBytes() & df.lastBytes() API's?
When rounds of KDF are performed? Are these performed during df.init() or during df.nextBytes() & df.lastBytes()?
That seems to be implementation specific to me. It will probably be faster to perform all the calculations at one time, but in that case it still makes sense to wait for the first request of the bytes. On the other hand RAM is also often an issue, so on demand generation also makes some sense. That requires a somewhat trickier implementation though.
The fact that the output size is pre-specified probably indicates that the simpler method of generating all the key material at once is at least foreseen by the API designers (they probably created an implementation before subjecting it to peer review in the JCF).
One KDF round will generate 32 bytes output (considering SHA-256 algorithm) then how API's df.nextBytes() & df.lastBytes() will work with any output expected length < 32 bytes?
It will commonly return the leftmost bytes (of the hash output) and likely leave the rest of the bytes in a buffer. This buffer will likely be destroyed together with the rest of the state when lastBytes is called (so don't forget to do so).
Note that the API clearly states that you have to re-initialize the DerivationFunction instance if you want to use it again. So that is a very strong indication that they though of destruction of key material (something that is required by FIPS and Common Criteria certification, not just common sense).
Other KDF's could have a different way of returning bytes, but using the leftmost bytes and then add rounds to the right is so common you can call it universal. For the ANSI X9.63 KDF this is certainly the case and it is clearly specified in the standard that way.
In this KDF counter is incremented in every next round then how counter will be managed between df.nextBytes() & df.lastBytes() API's?
These are methods of the same class and cannot be viewed separately, so they are not separate API's. Class instances can keep state in anyway they want. It might simply hold the counter as class variable, but if it decided to generate the bytes during init or the first nextBytes / lastBytes call then the counter is not even required anymore.
The timestamp minimum is 0, and the sequence part starts at 0. Why is Redis Streams minimum message ID '0-1' and not '0-0'?
Is '0-0' used internally? Is this why you can have 'empty' streams?
It appears to be a bug - there is an open pull request to fix it at https://github.com/antirez/redis/pull/6574
This makes perfect sense. If id 0-0 was allowed, there'd be no way to start fetching stream from the very beginning if using an explicit index (as opposed to -). Sometimes it is convenient to use an explicit index and not -.
In other words, it would be confusing and undesirable if simply passing 0 would result in skipping first element.
Is there good way to support pop members from the Redis Sorted Set just like the api LPOP of the List ?
What I figured out for poping message from the Redis Sorted Set is using ZRANGE +ZREM , however it is not thread security and need the distributed lock when multi threads accessing them at the same time from the different host.
Please kind suggesting if there is better way to pop the members from the Sorted Set?
In Redis 5.0 or above, you can use [B]ZPOP{MIN|MAX} key [count] for this scenario.
The MIN version takes the item(s) with the lowest scores; MAX takes the item(s) with the highest scores. count defaults to 1, and the B prefix blocks until the data is available.
ZPOPMIN
ZPOPMAX
BZPOPMIN
BZPOPMAX
You can write a Lua script to do the job: wrap these two commands in a single Lua script. Redis ensures that the Lua script runs in an atomic way.
local key = KEYS[1]
local result = redis.call('ZRANGE', key, 0, 0)
local member = result[1]
if member then
redis.call('ZREM', key, member)
return member
else
return nil
end