I am using QuestDb with UDP Influx Line Protocol publisher and want to see if there are gaps in the metrics due to package loss or any other reason.
Is there anything I can do on a publisher side or questdb to monitor the percentage of data loss if any?
One thing you can do on the producer side is tag & sequence each publisher
The tag has to be unique to a publisher
The sequence can be a long value that is monotonically incremented and (ideally) the sequence number should not be reset when a publisher restarts
On the server side you can execute the following query, assuming you have the column names tag and seq for tag and sequence respectively
(SELECT a.tag, a.seq high, b.seq low
FROM tab a
ASOF JOIN tab b ON (tag))
WHERE high > low + 1;
Related
I am running a dataflow job, which readed from BigQuery and scans around 8 GB of data and result in more than 50,000,000 records. Now at group by step I want to group based on a key and one column need to be concatenated . But After concatenated size of concatenated column becomes more than 100 MB that why I have to do that group by in dataflow job because that group by can not be done in Bigquery level due to row size limit of 100 MB.
Now the dataflow job scales well when reading from BigQuery but stuck at Group by Step , I have 2 version of dataflow code, but both are stucking at group by step. When I checked the stack driver logs, it says, processing stuck at lull for more than 1010 sec time(similar kind of message) and Refusing to split GroupedShuffleReader <dataflow_worker.shuffle.GroupedShuffleReader object at 0x7f618b406358> kind of message
I expect the group by state to be completed within 20 mins but is stuck for more than 1 hours and never gets finished
I figured out the thing myself.
Below are the 2 changes that I did in my pipeline:
1. I added a Combine function just after the Group by Key, see screenshot
since the Group by key when running on multiple worker, does a lot of network traffic exchange, and by default the network we use, does not allow the inter network communication, so I have to create a firewall rule to allow traffic from one worker to another worker i.e. ip range to network traffic.
I'm developing a Connector with some bank, and we're using the ISO8583 protocol, right now, i'm setting the STAN(field 11) with some random number generated with a random generator but sometimes I have some number collisions, the question is, could I safely use this generator or do I need to make the STAN a sequential number?
Thanks in advance.
The System Trace Audit Number (STAN) ISO-8583 number has different values and is maintained basically between relationships within the transaction. That is it can stay the same or the same transaction will have many STANs over its transaction path but it SHOULD be the same between two end point and it is usually controlled in settings whos STAN to use.
For Example:
Terminal -> Terminal Driver -> Switch 1->Switch 2->Issuer
The STAN is say assign by the terminal driver and then remains constant at minimum for the following relationships... though may change for each relationship.
Terminal Driver - Switch 1
Switch 1 -> Switch 2
Switch 2 -> Issuer
Note that internally within each system to the STAN may be unique as well but it needs to keep a unique STAN for each relationship.. and it shouldn't change between the request and response as it is needed for multi-part transactions (Single PA, Multiple Completions & Multi-PA, Single Completion) as well as for reversals and such in Data Element 90.
Depends on your remote endpoint, but I've seen many requiring sequential numbers, and detecting duplicates.
Usually STAN is the number increased for each request.
Random STAN generation is not the best case for network messages sequences.
The duplication of STANs can be due to different sources, i.e. Host clients or Terminals.
STAN itself cannot be the only field to detect unique transaction requests. It must be mixed together with other fields like RRN, Terminal ID, Merchant ID.
See also "In ISO message, what's the use of stan and rrn ?"
Suppose I have stream mystream, and a group mygroup, how do I get the length of unconsumed messages?
No, there is no way to do that afaik.
It is possible to get the last message ID delivered in a group and in a stream with the XINFO GROUPS and XINFO STREAM commands, respectively.
However, there is no command that returns the length of a stream subrange. Such command, was it to exist, would probably require linear time complexity and in that case, it will probably not be implemented.
Use XINFO GROUPS
The command XINFO GROUPS mygroup will provide a field in the response for lag.
According to the documentation:
lag: the number of entries in the stream that are still waiting to be delivered to the group's consumers, or a NULL when that number can't be determined.
If you are wondering why lag can be null:
There are two special cases in which this mechanism is unable to report the lag:
A consumer group is created or set with an arbitrary last delivered ID (the XGROUP CREATE and XGROUP SETID commands, respectively). An arbitrary ID is any ID that isn't the ID of the stream's first entry, its last entry or the zero ("0-0") ID.
One or more entries between the group's last-delivered-id and the stream's last-generated-id were deleted (with XDEL or a trimming operation).
In both cases, the group's read counter is considered invalid, and the returned value is set to NULL to signal that the lag isn't currently available.
More details can be found at https://redis.io/commands/xinfo-groups/
I have application that accepts data for updating products prices and I wondering how I can optimize it.
Data is received in some kind of queue ( rabbitMQ )
Few key notes:
I can't change incoming data format ( data is received from third party )
Updates must be performed in order from product perspective ( due attributes )
Each of product CAN have additional attributes by which system can behave differently when updating prices internally
I was thinking about using some messaging system too to distribute processing something like that:
where :
Q1 is queue for handling only p1 product updates.
Q2 is queue for handling only p2 product updates.
and so on..
However I have found it is likely to be more anti-pattern: Dynamic queue creation with RabbitMQ
For example seems with RabbitMQ it would be even quite hard to achieve since we need to have predefined queues in order to listen them.
The question is:
1) Should I use another pattern in case this is not valid and which pattern I should use
2) In case this pattern is valid is there some kind different messaging system that would allow distribute data by this pattern
I am considering implementing a Lambda Architecture in order to process events transmitted by multiple devices.
In most cases (averages etc.) its seems to fit my requirements. However, I am stuck trying to model a specific use case. In short...
Each device has a device_id. Every device emits 1 event per second. Each event has an event_id ranging from {0-->10}.
An event_id of 0 indicates START & an event_id of 10 indicates END
All the events between START & END should be grouped into one single group (event_group).
This will produce tuples of event_groups i.e. {0,2,2,2,5,10}, (0,4,2,7,...5,10), (0,10)
This (event_group) might be small i.e. 10 minutes or very large say 3hours.
According to Lambda Architecture these events transmitted by every device are my "Master Data Set".
Currently, the events are sent to HDFS & Storm using Kafka (Camus, Kafka Spout).
In the Streaming process I group by device_id, and use Redis to maintain a set of incoming events in memory, based on a key which is generated each time an event_id=0 arrives.
The problem lies in HDFS. Say I save a file with all incoming events every hour. Is there a way to distinguish these (group_events)?
Using Hive I can group tuples in the same manner. However, each file will also contain "broken" event_groups
(0,2,2,3) previous computation (file)
(4,3,) previous computation (file)
(5,6,7,8,10) current computation (file)
so that I need to merge them based on device_id into (0,2,2,3,4,3,5,6,7,8,10) (multiple files)
Is a Lambda Architecture a fit for this scenario? Or should the streaming process be the only source of truth? I.e. write to hbase, hdfs itself won't this affect the overall latency.
As far as I understand your process, I don't see any issue, as the principle of Lambda Architecure is to re-process regularly all your data on a batch mode.
(by the way, not all your data, but a time frame, usually larger than the speed layer window)
If you choose a large enough time window for your batch mode (let's say your aggregation window + 3 hours, in order to include even the longest event groups), your map reduce program will be able to compute all your event groups for the desired aggregation window, whatever file the distincts events are stored (Hadoop shuffle magic !)
The underlying files are not part of the problem, but the time windows used to select data to process are.