The timestamp minimum is 0, and the sequence part starts at 0. Why is Redis Streams minimum message ID '0-1' and not '0-0'?
Is '0-0' used internally? Is this why you can have 'empty' streams?
It appears to be a bug - there is an open pull request to fix it at https://github.com/antirez/redis/pull/6574
This makes perfect sense. If id 0-0 was allowed, there'd be no way to start fetching stream from the very beginning if using an explicit index (as opposed to -). Sometimes it is convenient to use an explicit index and not -.
In other words, it would be confusing and undesirable if simply passing 0 would result in skipping first element.
Related
This script shows that the timestamp that redis returns seems to jump forward a lot and then go backwards from time to time. This happens regularly throughout the run, on every run. What's going on? Is this documented anywhere? I stumbled on this as it messes up my sliding window rate limiter.
res_0, res_1 = 0, 0
for _ in range(200):
script = f"""
local time = redis.call("TIME")
local current_time = time[1] .. "." .. time[2]
return current_time
"""
res_0 = res_1
res_1 = float(await redis.eval(script, numkeys=0))
print(res_1 - res_0)
time.sleep(0.01)
1667745169.747809
0.011765003204345703
0.01197195053100586
0.011564016342163086
0.011634111404418945
0.012428998947143555
0.011847972869873047
0.011600971221923828
0.011788129806518555
0.012033939361572266
0.012130022048950195
0.01160883903503418
0.011954069137573242
0.012022972106933594
0.011958122253417969
0.011713981628417969
0.011844873428344727
0.012138128280639648
0.011618852615356445
0.011570215225219727
0.011890888214111328
0.011478900909423828
0.7928261756896973
-0.5926899909973145
0.11812996864318848
0.11584997177124023
0.12353992462158203
0.1199800968170166
0.11719989776611328
0.12331008911132812
-0.8117339611053467
0.011723995208740234
0.01131582260131836
The most likely reason for this behavior is floating point arithmetic on the calling code side: parsing floats would inevitably round (not sure about the extent, since you didn't tell what platform you are coding against) the input value and the original result precision is lost. So, I would suggest to review your logic so that you process the two components of the result returned by TIME independently using a couple of integers / longs instead.
In addition to that, apart from the obvious possibility of an issue with the clock of the Redis server, there may also be the chance you are contacting different Redis hosts along with each iteration - this may hold true in the event you are using a multi-node Redis topology (replication or cluster).
I want to stream real-time sensor data(webcam, laser point cloud, etc.) from one robot to multiple observers.
In this use case, only the newest data is useful. For example, when a new frame of point cloud arrives, the older ones will be useless.
Redis has nice publisher/consumer support, but it has buffers according to (Redis Pubsub and Message Queueing).
So are there better alternatives? Something like ROS's publishers/subscribers. They have a message queue size parameter.
/**
* The subscribe() call is how you tell ROS that you want to receive messages
* on a given topic.
*
* The second parameter to the subscribe() function is the size of the message
* queue. If messages are arriving faster than they are being processed, this
* is the number of messages that will be buffered up before beginning to throw
* away the oldest ones.
*/
ros::Subscriber sub = n.subscribe("chatter", 1000, chatterCallback);
Maybe you can use redis list data structure for your purpose, like a queue. The list data structure in redis is made with linked list and adding a new item is O(1). Whenever your robot produces data it can put it in a list with LPUSH command, and when you want to get the latest item from the list use LRANGE "key-name" 0 0. This command will retrive the latest pushed item. Also if you want to not accumulate the data in queue, you may try to use LTRIM before LRANGE to maintain the latest records. For example LTRIM "key-name" 0 10 will keep the records of last 10 elements. This trim interval should be set according to your observer processing speeds. ref: https://redis.io/docs/data-types/lists/
I have a requirement like this.
Flux<Integer> s1 = .....;
s1.flatMap(value -> anotherSource.find(value));
I need a way to stop this s1 when anotherSource.find gives me first empty. how to do that?
Note:
One possible solution is to throw error then capture it to stop.
anotherSource.find(value).switchIfempty(Mono.error(..))
I am looking for better solution than this.
You won't find a specific operator for this, you'll have to combine operators to achieve it. (Note that doesn't make it a "hack" per-se, reactive frameworks are generally intended to be used in a way where you combine basic operators together to achieve your use-case.)
I would agree that using an error to achieve is far from ideal though as it potentially disrupts the flow of real errors in the reactive chain - so that should really be a last resort.
The approach I've generally taken in cases where I want the stream to stop based on an inner publisher is to materialise the inner stream, filter out the onComplete() signals and then re-add the onComplete() wherever appropriate (in this case, if it's empty.) You can then dematerialise the outer stream and it'll respond to the completed signal wherever you've injected it, stopping the stream:
s1.flatMap(
value ->
anotherSource
.find(value)
.materialize()
.filter(s -> !s.isOnComplete())
.defaultIfEmpty(Signal.complete()))
.dematerialize()
This has the advantage of preserving any error signals, while also not requiring another object or special value.
I'm currently using Erlang for a big project but i have a question regarding a proper proceeding.
I receive bytes over a tcp socket. The bytes are according to a fixed protocol, the sender is a pyton client. The python client uses class inheritance to create bytes from the objects.
Now i would like to (in Erlang) take the bytes and convert these to their equivelant messages, they all have a common message header.
How can i do this as generic as possible in Erlang?
Kind Regards,
Me
Pattern matching/binary header consumption using Erlang's binary syntax. But you will need to know either exactly what bytes or bits your are expecting to receive, or the field sizes in bytes or bits.
For example, let's say that you are expecting a string of bytes that will either begin with the equivalent of the ASCII strings "PUSH" or "PULL", followed by some other data you will place somewhere. You can create a function head that matches those, and captures the rest to pass on to a function that does "push()" or "pull()" based on the byte header:
operation_type(<<"PUSH", Rest/binary>>) -> push(Rest);
operation_type(<<"PULL", Rest/binary>>) -> pull(Rest).
The bytes after the first four will now be in Rest, leaving you free to interpret whatever subsequent headers or data remain in turn. You could also match on the whole binary:
operation_type(Bin = <<"PUSH", _/binary>>) -> push(Bin);
operation_type(Bin = <<"PULL", _/binary>>) -> pull(Bin).
In this case the "_" variable works like it always does -- you're just checking for the lead, essentially peeking the buffer and passing the whole thing on based on the initial contents.
You could also skip around in it. Say you knew you were going to receive a binary with 4 bytes of fluff at the front, 6 bytes of type data, and then the rest you want to pass on:
filter_thingy(<<_:4/binary, Type:6/binary, Rest/binary>>) ->
% Do stuff with Rest based on Type...
It becomes very natural to split binaries in function headers (whether the data equates to character strings or not), letting the "Rest" fall through to appropriate functions as you go along. If you are receiving Python pickle data or something similar, you would want to write the parsing routine in a recursive way, so that the conclusion of each data type returns you to the top to determine the next type, with an accumulated tree that represents the data read so far.
I only covered 8-bit bytes above, but there is also a pure bitstring syntax, which lets you go as far into the weeds with bits and bytes as you need with the same ease of syntax. Matching is a real lifesaver here.
Hopefully this informed more than confused. Binary syntax in Erlang makes this the most pleasant binary parsing environment in a general programming language I've yet encountered.
http://www.erlang.org/doc/programming_examples/bit_syntax.html
I can just speak a little English so I hope you can understand what I said.
I fork a child process , then I do ADD in child process. EX: 56+48=104
If the value lower than 255 , I can use "wexitstatus(status)" to get the answer.
But if the value higher than 256, it would be wrong !
How can I do?
If the program returns an exit code > 255, the program is simply wrong and needs to be fixed. That's against Unix standard. If you're not using standard Unix, you're probably going to need specialist help from within your organisation contacts.
From manpage for wait():
WEXITSTATUS(stat_val)
If the value of WIFEXITED(stat_val) is non-zero, this macro evaluates to the low-order 8 bits of the status argument that the child process passed to _exit() or exit(), or the value the child process returned from main().
It's limited to 8-bits, which means 1 byte, which means the int from WEXITSTATUS can only range from 0-255. In fact, any Unix program will only ever return a max of 255.
Additionally, many OS's/programs reserve > 127 for system designated codes, so you shouldn't even use anything above that.
If you need more return codes than 126 (since 0 is no error), consider writing it to STDOUT and hooking that.