I have a Redis module with a custom command that issues commands to two different keys at once.
The custom commands receives a key (e.g. myKey), and some values, and issues HSET myKey_sd1 ... and ZADD myKey_sd2 ... commands.
I wonder what would happen in a cluster configuration, where each key could exist in a separate node.
Right now, inside the module's custom command I enclosure the key that is given to me in { and }, so I end up issuing HSET {myKey}_sd1 and ZADD {myKey}_sd2
Which works but I wonder if it's necessary.
I also wonder what happens if I receive a key that is already enclosed in curly brackets (because the user of the module wants to control the hash slot by himself) - e.g. 123_{myKey}
Right now, I would still enclose this key in my own curly brackets when I issue the HSET and ZADD - HSET {123_{myKey}}_sd1 ... and ZSET {123_{myKey}}_sd2
In this case, as far as I know, Redis would compute the hash for 123_{myKey which undermines the input of the user (since they didn't wish to enclose the 123_ part)
To fix it, I could have my custom command look for a { in the key that it received, if it didn't find a {, it would enclose the key in { and }, like I described above.
However, if it did find a {, it would leave the key as is, resulting in (in the case of the example above) HSET 123_{myKey}_sd1 ... and ZADD 123_{myKey}_sd2 - which works both for the user and for the inner workings of the module.
So I wonder
Is enclosing the keys that I receive from the user in { and } the right way to support clustering?
If so, am I handling the case where the user provides a key which is already enclosed in { and } in the right way?
Is enclosing the keys that I receive from the user in { and } the right way to support clustering?
The Redis Module cluster documentation is "missing", but I believe it's the case that your module must make sure to use the same node as the key that was passed as an argument.
So, yes, you must use hash tags to ensure that your keys are using the same hash slot as the passed key.
If so, am I handling the case where the user provides a key which is already enclosed in { and } in the right way?
Your proposed fix (not your current implementation) is right. You must hash to the same slot as the key that is passed, which means respecting any specified hash slot.
Just make sure you're following the exact algorithm for interpreting hash tags. You don't want any edge cases here.
Related
As we may know, good naming convention for Redis keys is "object-type:id"
How to get all key namespaces (symbols before :)?
E.g.
127.0.0.1:6379> KEYS *
1) "bar:1"
2) "baz:1"
3) "baz:3"
4) "foo:1"
5) "foo:2"
6) "baz:2"
I want a command like
127.0.0.1:6379> SCAN 0 MATCH "(^\w+):" COUNT 10
1) "6"
2) "foo"
3) "bar"
4) "baz"
But seems like MATCH option of SCAN command does not support PCRE pattern syntax.
In real, I got
127.0.0.1:6379> SCAN 0 MATCH "(^\w+):" COUNT 10
1) "6"
2) (empty list or set)
Any ideas ?
You could create an index key and add each namespace to it every time you add a new key:
SADD "namespaces" "foo". But there would be problem with deleting them, as you'd have to check if the deleted object was the last of it's type.
The second option would be creating a module (if you're using Redis >= 4.0) with a command that you could use to add and remove said keys, that would have it's own type in which it would store object-type counters (so it wouldn't have to call KEYS * which is quite slow). The problem here would be if you had key expiration, or oom key deletion configured. The counter would be incorrect once a key gets deleted by Redis itself.
The third option is to create a module with one command that would call KEYS *, then get the first part of the key and add it to a hash map/dictionary (using RedisModuleDict) and then return it using RedisModule_ReplyWithString(ctx, str) in a loop
I think that you could also write it in Lua instead of creating a module, but a module written in C is probably going to be faster... and I don't know Lua so I won't be able to help you there.
You could always write a simple script in bash that would get the object-type from each key and SADD it to a temporary key then read it, print and delete afterwards, but that's the same as the third option but slower and has to use a temporary key which might be used by some other script or something.
I am investigating the CyberSource REST API and want to test the JSON Web Token Authentication method as documented here: https://developer.cybersource.com/api/developer-guides/dita-gettingstarted/authentication/GenerateHeader/jwtTokenAuthentication.html
I am unable to replicate the sha256 hash of the JSON payload described in the JWT Payload/Claim Set section.
{
"clientReferenceInformation" : {
"code" : "TC50171_3"
},
"orderInformation" : {
"amountDetails" : {
"totalAmount" : "102.21",
"currency" : "USD"
}
}
}
I've attempted to use the sha256sum command in binary and text format on a file containing the payload example. I've also attempted running this command on different permutations of this payload, such as without whitespace or newlines.
I expect to get the example hash of
2b4fee10da8c5e1feaad32b014021e079fe4afcf06af223004af944011a7cb65c
but instead get
f710ef58876f83e36b80a83c8ec7da75c8c1640d77d598c470a3dd85ae1458d3 and other dissimilar hashes.
What am I doing wrong?
Since the alleged "example" hash contains 33 hex characters one can see that it is not a possible valid output of SHA256. So there is nothing you can do to make your example match theirs.
There is also a base64 example in that discussion, but it is also not valid base64. By adding an extra padding character '=' to the base64 it can be made valid, and decoding it reveals that it mostly matches the alleged SHA256 hash.
My guess is that the values on that page are just examples of what values look like to the human eye rather than test vectors you are supposed to match exactly.
Probably you are not doing anything wrong. Hash functions have a avalanche effect, wherein any different bit in the input changes a lot the output hash. If the site's original example used a different encoding, or had a different order for the JSON elements, or even had more or less tabs, spaces, line breaks, or any other "trash" character, you'll have a hard time to find a fitting message for the hash showed in the site.
Usually, cryptographic solutions use canonicalizations to avoid this kind of problem (different hash values for semantically equal messages). However, the JWT specification doesn't specify any type of canonicalization for JSON.
In short, I think you don't have to worry about this. Your JWT implementation will be correct as long you use a valid (correctly implemented) hash function.
Also, I noticed that the JWT specification doesn't specify a "Digest" field for the JWT payload. So, you may not even need to use this field. Unless CyberSource REST API makes it mandatory.
I'm new to Aerospike and am probably missing something fundamental, but I'm trying to see an enumeration of the Keys in a Set (I'm purposefully avoiding the word "list" because it's a datatype).
For example,
To see all the Namespaces, the docs say to use SHOW NAMESPACES
To see all the Sets, we can use SHOW SETS
If I want to see all the unique Keys in a Set ... what command can I use?
It seems like one can use client.scan() ... but that seems like a super heavy way to get just the key (since it fetches all the bin data as well).
Any recommendations are appreciated! As of right now, I'm thinking of inserting (deleting) into (from) a meta-record.
Thank you #pgupta for pointing me in the right direction.
This actually has two parts:
In order to retrieve original keys from the server, one must -- during put() calls -- set policy to save the key value server-side (otherwise, it seems only a digest/hash is stored?).
Here's an example in Python:
aerospike_client.put(key, {'bin': 'value'}, policy={'key': aerospike.POLICY_KEY_SEND})
Then (modified Aerospike's own documentation), you perform a scan and set the policy to not return the bin data. From this, you can extract the keys:
Example:
keys = []
scan = client.scan('namespace', 'set')
scan_opts = { 'concurrent': True, 'nobins': True, 'priority': aerospike.SCAN_PRIORITY_MEDIUM }
for x in (scan.results(policy=scan_opts)): keys.append(x[0][2])
The need to iterate over the result still seems a little clunky to me; I still think that using a 'master-key' Record to store a list of all the other keys will be more performant, in my case -- in this way, I can simply make one get() call to the Aerospike server to retrieve the list.
You can choose not bring the data back by setting includeBinData in ScanPolicy to false.
In Redis, I have hash keys in the following format
keys {
'img::opt': 'nameX',
'img:*:opt': 'nameY',
'img:#:opt': 'nameZ',
'img:A:opt': 'nameN'
}
It is actually in the format of 'extension:owner:spec'
I want to perform hscan based on owner which returns,
1. Everything except blank(will return 2nd, 3rd and 4th keys)
2. Only blank and star(*)(will return 1st and 2nd keys)
For first, I can use pattern as 'img:?*:opt'. How to write a pattern which can be used for 2nd case.
I tried,
img:[^][*]:opt
img:[^|*]:opt
img:[*^]:opt
But none of them are working. Is it possible to pattern match empty string?
Redis' glob-style pattern matching syntax doesn't support the not ('^') operator.
However, as you're looking for two specific keys, why not access them directly by simply doing:
HGET img:*:opt img::opt
Note: in Redis versions prior to 4, HGET needs to be replaced with HMGET
I'm wondering if there is a way to determine if two MediaStreams are equal.
What do you mean by "equal"?
I'd like to determine if the two streams are using the same hardware sources (Same microphone and camera are being used).
Acquiring streamB with the exact same constraints as streamA would mean they are equal.
Here is what I've tried so far:
comparing via the MediaStream id e.g.: streamA.id == streamB.id
This falls away since according to the spec:
When a MediaStream object is created, the User Agent must generate an identifier string, and must initialize the object's id attribute to that string. A good practice is to use a UUID [rfc4122], which is 36 characters long in its canonical form. To avoid fingerprinting, implementations should use the forms in section 4.4 or 4.5 of RFC 4122 when generating UUIDs.
Compare the id's of the MediaStreamTracks - same story, a UUID is generated per track.
Compare the tracks labels, which in the current Chrome contain names/identifiers of the hardware. This is very close to what I'm looking for, however (emphasis mine):
User Agents may label audio and video sources (e.g., "Internal microphone" or "External USB Webcam"). The label attribute must return the label of the object's corresponding source, if any. If the corresponding source has or had no label, the attribute must instead return the empty string
Is there a different approach I could take? Should I never end up in a situation where I compare two media streams? Would you say I can trust the label attribute?
Thanks for your time.
groupId together with kind is probably the closest thing you will get. Until you get multiple mics/cams on the same device...