We are given an InputStream which represents the response body of a text/event-stream HTTP response. The stream emits events periodically, every few seconds.
Now we need to find a way to convert this InputStream into Flux<MyClass>. The goal is to use a StepVerifier on that stream, to check that each object has the expected structure, and to close the stream in the end.
How could this be done?
Related
The following code only prints 10000 i.e. only the last element
val channel = BroadcastChannel<Int>(Channel.CONFLATED)
val flowJob = channel.asFlow().buffer(Channel.UNLIMITED).onEach {
println(it)
}.launchIn(GlobalScope)
for (i in 0..100) {
channel.offer(i*i)
}
flowJob.join()
Code can be ran in the playground.
But since the Flow is launched in separate dispatching thread, and value is sent to the Channel and since Flow has an unlimited buffer, it should receive each element till onEach is invoked. But why only the last element is able to get received?
Is this the expected behavior or some bug? If its expected behavior how would somebody try to push only the newest elements to the flow, but all the flow that has certain buffer can receive the element.
Actually, this is about the "Conflate" way of buffering. For buffering a flow you have a couple of ways such as using buffer() method or collectLatest() or conflate(). Each of them has their own way to buffer. So conflate() method's way is that when the flow emits values, it tries to collect but when the collector is too slow, then conflate() skips the intermediate values for the sake of the flow. And it's doing it even tho every time it's emitted in a separate coroutine. So in a channel, a similar thing is happening basically.
Here is the official doc explanation:
When a flow represents partial results of the operation or operation
status updates, it may not be necessary to process each value, but
instead, only most recent ones. In this case, the conflate operator
can be used to skip intermediate values when a collector is too slow
to process them.
Check out this link.
The explanation is for flow but you need to focus on the feature that you are using. And in this case, conflation is same for channel and flow.
The problem here is the Channel.CONFLATED. Taken from the docs:
Channel that buffers at most one element and conflates all subsequent `send` and `offer` invocations,
so that the receiver always gets the most recently sent element.
Back-to-send sent elements are _conflated_ -- only the the most recently sent element is received,
while previously sent elements **are lost**.
Sender to this channel never suspends and [offer] always returns `true`.
This channel is created by `Channel(Channel.CONFLATED)` factory function invocation.
This implementation is fully lock-free.
so this is why you only get the most recent (last) element. I'd use an UNLIMITED Channel instead:
val channel = Channel<Int>(Channel.UNLIMITED)
val flowJob = channel.consumeAsFlow().onEach {
println(it)
}.launchIn(GlobalScope)
for (i in 0..100) {
channel.offer(i*i)
}
flowJob.join()
As some of the comments stated, using Channel.CONFLATED will store only the last value, and you are offering to the channel, even if your flow has a buffer.
Also join() will suspend until the Job is not complete, in your case infinitely, that's why you needed the timeout.
val channel = Channel<Int>(Channel.RENDEZVOUS)
val flowJob = channel.consumeAsFlow().onEach {
println(it)
}.launchIn(GlobalScope)
GlobalScope.launch{
for (i in 0..100) {
channel.send(i * i)
}
channel.close()
}
flowJob.join()
Check out this solution (playground link), with the Channel.RENDEZVOUS your channel will accept new elements only if the others are already consumed.
This is why we have to use send instead of offer, send suspends until it can send elements, while offer returns a boolean indicating if send was succesfull.
At last, we have to close the channel, in order for join() not to suspend until eternity.
Proto3 release notes states:
The
deterministic serialization is, however, NOT canonical across languages; it
is also unstable across different builds with schema changes due to unknown
fields. Users who need canonical serialization, e.g. persistent storage in
a canonical form, fingerprinting, etc, should define their own
canonicalization specification and implement the serializer using reflection
APIs rather than relying on this API.
What I would like to achieve is to have a deterministic serialisation of protobuf message to carry a crypto signature along with it. As I understand due to differences in serialisers binary data could differ and signature would become invalid.
package Something
message Request {
Payload payload = 1;
// signature of serialised payload
bytes signature = 2;
message Payload {
string user_id_from = 1;
uint64 amount = 2;
string user_id_to = 3;
}
}
What is the way to do this using protobuf reflection?
This doesn't answer the question directly, but may solve your issue: don't store the payload as a message, but store the serialized bytes alongside with the signature.
message Request {
// Serialized Payload message.
bytes payload = 1;
// signature of serialised payload
bytes signature = 2;
}
message Payload {
string user_id_from = 1;
uint64 amount = 2;
string user_id_to = 3;
}
This may be a little less convenient to work with in code, but has the advantage of preserving all the forwards and backwards-compatibility guarantees of protobuf.
It also frees you from serializing the message twice when writing it (once as a subfield, once to get the signature).
I need to be able to create an event processing rule that when you add a new device, you take a string value from one fragment (e.g.: c8y_Hardware.imei) and use that string to populate another fragment (e.g: c8y_Mobile.imei). So the new device would then have the same value in both c8y_Hardware.imei and c8y_Mobile.imei.
We have attempted setting up the appropriate CEP rules, but they are not working (they do compile and save).
insert into UpdateManagedObject
select
m.id as id,
{
"c8y_Mobile.imei", getString(m,"c8y_Hardware.imei")
} as fragments
from
ManagedObjectCreated as m
where
getString(m,"c8y_Hardware.imei") != "";
Any guidance on where we are messing up our syntax would be greatly appreciated.
It should be: m.managedObject.id as id.
Usually you would also get an error on compile but it can be that the streams also have an id so that it technically works in CEP. You should be able to check if it triggers on the debug stream and see the id that has been set.
Same applies for all other Cumulocity streams. The streams itself e.g. ManagedObjectCreated or AlarmUpdated etc. are not the objects directly. They have always a property like in this case managedObject or for AlarmUpdated it is alarm. This property is the actual payload.
The helper methods like getString are written in a way that you can pass either the payload or the full stream object so there it does not matter.
I send the following message with content type application/json:
However whene i get messages from the same RabbitMQ Web console, it shows the payload as String.
What am I doing wrong? Or am I fundamentally misunderstanding and the Payload is always of type String?
From the official docs:
AMQP messages also have a payload (the data that they carry), which AMQP brokers treat as an opaque byte array. The broker will not inspect or modify the payload. It is possible for messages to contain only attributes and no payload. It is common to use serialisation formats like JSON, Thrift, Protocol Buffers and MessagePack to serialize structured data in order to publish it as the message payload. AMQP peers typically use the "content-type" and "content-encoding" fields to communicate this information, but this is by convention only.
So basically, RabbitMQ has no knowledge on JSON, messages all are just byte arrays to it
From NodeJS Context:
If we want to send JSON object as message, we may get the following error:
The first argument must be of type string or an instance of Buffer,
ArrayBuffer, or Array or an Array-like Object. Received an instance of
Object
So, we can convert the JSON payload as string and parse it in the worker. We stringify the JSON object before sending the data the Queue-
let payloadAsString = JSON.stringify(payload);
And from worker's end, we can then JSON.parse
let payload = JSON.parse(msg.content.toString());
//then access the object as we normally do, i.e. :
let id = payload.id;
For anyone using .Net to send objects via RabbitMQ.
You have to serialise your JSON object to byte array, send via RabbitMQ then de-serialise after receiving. You can do this like this:
Install the Newtonsoft JSON library
using Newtonsoft.Json;
Create a model for your JSON object message (in this case AccountMessage)
Serialise your object into byte array like this:
byte[] messagebuffer = Encoding.Default.GetBytes(JsonConvert.SerializeObject(accountMessage) );
After receiving the message data, you can de-serialise like this:
AccountMessage receivedMessage = JsonConvert.DeserializeObject<AccountMessage>(Encoding.UTF8.GetString(body));
from here
Content Type and Encoding
The content (MIME media) type and content encoding fields allow publishers communicate how message payload should be deserialized and decoded by consumers.
RabbitMQ does not validate or use these fields, it exists for applications and plugins to use and interpret.
by the way, using the rabbitMQ web gui, you use the words content_type, however in code (javascript confirmed), you use the key name contentType. it's a subtle difference, but enough to drive you crazy.
I have a very strange situation.
Basically I have code that uses a decryptor created by:
Dim des3 As New TripleDESCryptoServiceProvider
des3.Mode = CipherMode.CBC
Return des3.CreateDecryptor(_encKey, _initVec)
The _encKey and _initVec are hardcoded.
I use it by calling:
Dim res() As Byte = decrypt(Convert.FromBase64String(_data))
m_transformDec.TransformFinalBlock(res, 0, res.Length)
Here _data is a string containing the encrypted value. m_transformDec is the Decryptor created previously.
Usually this works. Occasionally, I get a "bad data" error. I print out the value of _data, and it is always the same.
The code is multithreaded, which I suspect is the reason for both the problem, and it being hard to reproduce. The decryptor is created in the creation of the class, and the decryption is done in a Shared function, but I don't see anything there which is not thread-safe.
Any ideas?
You should not assume anything is safe for concurrent calls unless you have reason to believe it is. In the docs, you have the boilerplate text that instance members are not guaranteed to be thread-safe, so you should defensively lock the des3 object when you're using it.
You should not be hard coding the initialization vector; it should be randomly chosen when encrypting data, then stored in some way with the encrypted data (many people choose to tack it onto the beginning of the data, then remove it and use it for decryption; use whatever storage scheme you prefer, though). Using the same IV defeats the purpose of the IV, which serves to make plaintext attacks more difficult.