Looking to understand whether there is a a bulletproof event from the namagedObject side of c8y where we know the device has just connected.
I have a microservice that listens for events in real time and I want to trigger a process once we know a device has connected to send its payload.
We have used:
"c8y_Connection": {"status":"CONNECTED"}
We have had the microservice log to Slack all events from managedObjects where we saw for three days the "status":"CONNECTED" value in the payload of our demo devices at reporting times.
But after three days, we see no more this "CONNECTED" state (all payloads showing "DISCONNECTED").
What I am trying to achieve from the inventoryObject event is to understand when a device had connected and sent payload to know when data had arrived. I then go get the data and process it externally. This is post registration and as part of the daily data send cycle for my type of device.
What would be the best way to understand when a device has sent payload in a microservice? I want to notify an external application with either “data is arriving for id 35213” or even better, “data has arrived for device 35213, and here’s the $payload”.
Just as a general information ahead:
The c8y_Connection fragment showing connected shows an active MQTT connection or an active long polling connection and it is only evaluated once every minute.
So if the client is just sending data and immediately disconnecting afterwards this might not picked up.
If you want to see the device having send something to Cumulocity maybe the c8y_Availability fragment is a better as it holds the timestamp when the device last send something.
{ "lastMessage": "2022-10-11T14:49:50.201+09:00", "status": "UNAVAILABLE"}
Also here the evaluation (or better the update to database) only happens every minute.
Both c8y_Availability and c8y_Connection however are only generated if the availability monitoring has been activated for the device (by defining a required interval for the device).
So if you have activated the availability monitoring and you see a "lastMessage" you can reliably say that the device has already send something to Cumulocity.
Related
RFC 5277 defines notification replay support. Just wondering what customer problems this notification replay can solve? What could be the need to scan through list of past notifications? At any point of time, controllers can always fire "get" RPC and determine current state.
Any inputs on this are appreciated.
Notification replay is mostly required to sync NETCONF clients with servers without the need to do a full RPC.
In some systems, the configuration + operational data becomes quite large, so the notification replay allows for a delta sync.
The main problem with this is that the server will only keep a limited time window of notifications; so if a client asks for a replay of the last week, the server may be only able to provide a subset of that.
RESTCONF has a different solution for this same issue, the last-modified header field(https://www.rfc-editor.org/rfc/rfc8040#section-3.4.1.1), which allows for doing a get of 'data that has changed since timestamp x'.
I am using IoTHub device client SDK on an embedded device. The application will send telemetry message to iot hub periodically. The iot device connect to a wireless router and wireless connect to internet via WAN port.
When the wireless router lost internet connection, iot device will not get notified immediately about the disconnection. It takes about 60s to get notified, before that iot device will continue to send telemetry message with IoTHubDeviceClient_LL_SendEventAsync(), all those message get queued in SDK layer and eat memory. Since it's on embedded device with limited resource, memory get eaten up and cause program been killed by a lower memory killer app.
Is there way to specified total size of iot message can be queued in sdk layer? If exceed this quota, IoTHubDeviceClient_LL_SendEventAsync() will failed immediately.
Actually this is also needed for normal scenario too. When iot device send message, seems message been queued in low layer and get flushed out at certain time. I don't see any API that can control the flush. That create another problem, even when there is internet connection, from application level, there is no control of how many message been queued and how long it been queued, in turn, app has no control of how much memory been used by process. On my device, there is system monitor that will kill process use too much memory.
The question is what do you do even in that case if the message failure occurs in the case that the queue is full? Do you lose the information then because of lack of storage capacity? From the IoT perspective, I would recommend in this case to consider if your device is reliable IoT device to handle these edge cases as well. And also knowing the limits of the devices, and knowing how long it can be without the internet connection helps to mitigate these risks from your application, not SDK.
From the GitHub, default sendMessageAsync method throws timeout exception in case your message sending fails, unless you have some kind of retry policies implemented(according to the documentation C SDK does not allow custom retry policies
https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-reliability-features-in-sdks).
According to the documentation in case of connection failure based on the retry policy(if you have set it), SDK will try to initiate connection this way or that way and queue the messages created in the meantime:
https://github.com/Azure/azure-iot-sdk-c/blob/master/doc/connection_and_messaging_reliability.md
So, an expectation here is that SDK does not take responsibility for the memory limits. This is up to the application to deal. Since your device has some limitations, I would recommend implementing your own queuing mechanism(maybe set no-retry as a policy and that way avoid queuing). That way you have under the control what will happen in the case that there is no internet connection and have under the control memory limitations. Maybe your business case accepts that you calculate an average value and instead of 50 you store 1 message over the time etc..
If this something you do not like, the documentation says also that you set the timeout for the queue - maybe not the memory limit but timeout yes, so maybe you can try to investigate this a bit deeper:
"There are two timeout controls in this system. An original one in the iothub_client_ll layer - which controls the "waiting to send" queue - and a modern one in the protocol transport layer - that applies to the "in progress" list. However, since IoTHubClient_LL_DoWork causes the Telemetry messages to be immediately* processed, sent and moved to the "in progress" list, the first timeout control is virtually non-applicable.
Both can be fine-tuned by users through IoTHubClient_LL_SetOption, and because of that removing the original control could cause a break for existing customers. For that reason it has been kept as is, but it will be re-designed when we move to the next major version of the product."
When subscribing to real-time notifications, I go through the normal handshake, subscribe, connect flow.
Once the connection returns with events, I reconnect and wait for the next response to return. My question is:
If events are generated the first response and the next reconnect, could they be lost?
As an example: A synchronous application which processes returned response data after it returns and only reconnects once the data processing has finished could cause a significant delay between the response and the next reconnect. Are the cumulocity events generated during that delay buffered in the real-time queue for that particular client id or are they just lost?
Another possible example is when the client ID is no longer valid (this seems to happen every day at midnight), I have to resubscribe, causing a period of time during which no one is subscribed.
The client ID that you receive when handshaking is connected to a queue on the server side. That queue keeps all notifications that you are not able to receive until the next connect. It delivers them when you reconnect. (Try it out with Postman: After a connect returns, send a couple of events, then connect again. You will notice that you will get all events at once.)
However, as you noticed, the queue is not kept forever. If you are not able to reconnect within two hours (I believe), the queue is thrown away in order to not block server resources. This is what you noticed. In that case, you need to query the database to determine any missed events (e.g., poll any operations in pending state from devices).
As far as I can tell there are 7 events dispatched by a NoFlo port:
attach,
connect,
begingroup,
data,
endgroup,
disconnect,
detach
To me some of these events sound very similar such as attach + connect, and disconnect + detach. What is the difference?
What does begingroup and endgroup mean?
When do these events get emitted and when are they generally used?
I've seen the documentation at: http://noflojs.org/documentation/components/#portevents
Would my assumption be correct to assume that attach and detach are for handling NoFlo UI cases eg changing the state of the components look?
Another assumption would be that connect gets fired every time before data is sent? Then data gets fired. Then disconnect? Seems a bit odd to me...
I'm completely in the dark when it comes to groups.
attach and detach happen when the NoFlo Network attaches (or removes) a socket to the port. So usually they happen at network start-up time, before IIPs get sent.
The exception to this is when you're live-editing the graph with a tool like Flowhub. In that situation attach/detach can happen whenever you connect or remove wires.
Most components don't need to care about the attachment events.
connect happens before the upstream connection sends data, and disconnect when the upstream connection says that it has sent everything it is intending to send. So in effect they're beginning of transmission and end of transmission events. An upstream component may choose to connect again after a disconnect if it has a new batch of data to send.
data is the event for actual payload-containing packets.
begingroup and endgroup are the "bracket IPs" containing metadata about the data being sent. They can be used for creating tree structures with packet data.
For example, filesystem/ReadFile will send the file contents as a data packet, but the filename is sent via a bracket IP using a begingroup/endgroup packets around the actual file contents.
The noflo-groups library provides lots of components for utilizing group information for synchronization, routing, etc.
I have a project coming up where I need to send and receive messages through a specific mobile operator, which only provides an SMPP interface. The whole project will be a hosted website. I have already read quite a lot, but I do not yet quite understand what is actually needed from my side to use the protocol.
Should my application try to maintain a constant connection to the smpp?
Can I simply connect, send a message and then disconnect?
Are receiving messages based on push or pull?
Thanks for the help.
SMPP is a peer-to-peer protocol. That should mean that SMS Gateway (your side) and SMSC (your mobile operator) need to have a proper bind/connection established. Even when there are no SMS or DLRs to send/receive, there is a continous exchange of smpp PDU (enquire_link/enquire-link_resp) that ensure that the bind is established.
In detail, if you send an enquire_link PDU and you get no response (enquire_link_resp) the bind is broken. Your sms won't be delivered (will remain enqueued in your gateway store), and you won't receive MOs (incoming sms) or DLRs (delivery report). To re-establish the connection you should re-initiate the connection.
So, my answer would be that you need a constant connection to SMSC.
You are stating you want to receive messages, as a result at least a bind_receiver is needed. Because you don't know when messages are going to come in, you will have to be constantly connected, rather than disconnecting after each event.
With regards to your question about "push or pull" this depends on how you solve the first problem. If you can build a solution that is constantly connected, the result will be a push (the carrier will push it to you as soon as they receive the message). If (for some reason) you cannot maintain a constant connection, you'll end up building a pull mechanism. You'll connect to the carrier ever X seconds to see if they have a message waiting for you.
I do need to highlight 2 pitfalls though:
A number of carriers in the world, do not store or even accept messages if you are not connected, therefore, depending on which carrier you interact with, you might be forced to use a continuous connection.
Most carriers do not allow you to open and close connections in quick succession. Once you disconnect, you can not reconnect for a time frame of X seconds.
Therefore a constant connection is really the way to go. Alternatively, you can look into a company like Nexmo, which will provide you with a HTTP Call every time a message arrives.
I'm not sure which language your developing your application in, but if you use any of the popular languages (Java, PHP, Perl) there are modules out there that handle basic SMPP Connectivity for you. A quick google search for your language and "SMPP Client" will give you a list of references.