Merge two message threads into one - splunk

have two message threads, each thread consists of ten messages. I need to request to display these two chains in one.
The new thread must consist of ten different messages: five messages from one system, five messages from another (backup) system. Messages from the system use the same SrcMsgId value. Each system has a unique SrcMsgId within the same chain. The message chain from the backup system enters the splunk immediately after the messages from the main system. Messages from the standby system also have a Mainsys_srcMsgId value - this value is identical to the main system's SrcMsgId value. Tell me how can I display a chain of all ten messages? Perhaps first messages from the first system (main), then from the second (backup) with the display of the time of arrival at the server.
Specifically, we want to see all ten messages one after the other, in the order in which they arrived at the server. Five messages from the primary, for example: ("srcMsgId": "rwfsdfsfqwe121432gsgsfgd71") and five from the backup: ("srcMsgId": "rwfsdfsfqwe121432gsgsfgd72"). The problem is that messages from other systems also come to the server, all messages are mixed (chaotically), which is why we want to organize all messages from one system and its relative in the search. Messages from the backup system are associated with the main system only by this parameter: "Mainsys_srcMsgId" - using this key, we understand that messages come from the backup system (secondary to the main one).
Examples of messages from the primary and secondary system:
Main system:
{
"event": "Sourcetype test please",
"sourcetype": "testsystem-2",
"host": "some-host-123",
"fields":
{
"messageId": "ED280816-E404-444A-A2D9-FFD2D171F32",
"srcMsgId": "rwfsdfsfqwe121432gsgsfgd71",
"Mainsys_srcMsgId": "",
"baseSystemId": "abc1",
"routeInstanceId": "abc2",
"routepointID": "abc3",
"eventTime": "1985-04-12T23:20:50Z",
"messageType": "abc4",
.....................................
Message from backup system:
{
"event": "Sourcetype test please",
"sourcetype": "testsystem-2",
"host": "some-host-123",
"fields":
{
"messageId": "ED280816-E404-444A-A2D9-FFD2D171F23",
"srcMsgId": "rwfsdfsfqwe121432gsgsfgd72",
"Mainsys_srcMsgId": "rwfsdfsfqwe121432gsgsfgd71",
"baseSystemId": "abc1",
"routeInstanceId": "abc2",
"routepointID": "abc3",
"eventTime": "1985-04-12T23:20:50Z",
"messageType": "abc4",
"GISGMPRequestID": "PS000BA780816-E404-444A-A2D9-FFD2D1712345",
"GISGMPResponseID": "PS000BA780816-E404-444B-A2D9-FFD2D1712345",
"resultcode": "abc7",
"resultdesc": "abc8"
}
}
When we want to combine in a query only five messages from one chain, related: "srcMsgId".
We make the following request:
index="bl_logging" sourcetype="testsystem-2"
| транзакция maxpause=5m srcMsgId Mainsys_srcMsgId messageId
| таблица _time srcMsgId Mainsys_srcMsgId messageId продолжительность eventcount
| сортировать srcMsgId_time
| streamstats current=f window=1 значения (_time) as prevTime по теме
| eval timeDiff=_time-prevTime
| delta _time как timediff

Related

How frequently are the Azure Storage Queue metrics updated?

I observed that it took about 6 hours from the time of setting up Diagnostics (the newer offering still in preview) for the Queue Message Count metric to move from 0 to the actual total number of messages in queue. The other capacity metrics Queue Capacity and Queue Count took about 1 hour to reflect actual values.
Can anyone shed light on how these metrics are updated? It would be good to know how to predict the accuracy of the graphs.
I am concerned because if the latency of these metrics is typically this large then an alert based on queue metrics could take too long to raise.
Update:
Platform metrics are created by Azure resources and give you visibility into their health and performance. Each type of resource creates a distinct set of metrics without any configuration required. Platform metrics are collected from Azure resources at one-minute frequency unless specified otherwise in the metric's definition.
And 'Queue Message Count' is platform metrics.
So it should update the data every 1 minute.
But it didn't. And this is not a problem that only occur on portal. Even you use rest api to get the QueueMessageCount, it still not update after 1 minute:
https://management.azure.com/subscriptions/xxx-xxx-xxx-xxx-xxx/resourceGroups/0730BowmanWindow/providers/Microsoft.Storage/storageAccounts/0730bowmanwindow/queueServices/default/providers/microsoft.insights/metrics?interval=PT1H&metricnames=QueueMessageCount&aggregation=Average&top=100&orderby=Average&api-version=2018-01-01&metricnamespace=Microsoft.Storage/storageAccounts/queueServices
{
"cost": 59,
"timespan": "2021-05-17T08:57:56Z/2021-05-17T09:57:56Z",
"interval": "PT1H",
"value": [
{
"id": "/subscriptions/xxx-xxx-xxx-xxx-xxx/resourceGroups/0730BowmanWindow/providers/Microsoft.Storage/storageAccounts/0730bowmanwindow/queueServices/default/providers/Microsoft.Insights/metrics/QueueMessageCount",
"type": "Microsoft.Insights/metrics",
"name": {
"value": "QueueMessageCount",
"localizedValue": "Queue Message Count"
},
"displayDescription": "The number of unexpired queue messages in the storage account.",
"unit": "Count",
"timeseries": [
{
"metadatavalues": [],
"data": [
{
"timeStamp": "2021-05-17T08:57:00Z",
"average": 1.0
}
]
}
],
"errorCode": "Success"
}
],
"namespace": "Microsoft.Storage/storageAccounts/queueServices",
"resourceregion": "centralus"
}
This may be an issue that needs to be reported to the azure team. It is so slow, it even loses its practicality. I think send an alert based on this is a bad thing(it’s too slow).
Maybe you can design you own logic by code to check the QueueMessageCount.
Just a sample(C#):
1, Get Queues
Then get all of the queue names.
2, Get Properties
Then get the number of the message in each queue.
3, sum the obtained numbers.
4, send custom alert.
Original Answer:
At first, after I send message to one queue in queue storage, the 'Queue Message Count' also remains stubbornly at zero on my side, but a few hours later it can get the 'Queue Message Count':
I thought it would be a bug, but it seems to work well now.

RabbitMQ keep processing order of jobs in a same group

I have a list of jobs in my application queue(RabbitMQ).
Some of these jobs are group together and must do in the order.(not continuous, but by order of dispatch time)
For example, consider this 4 jobs in the queue:
[
{ "group": "x", "dispatched_timestamp": 10001, "field1": "some data", "field2": "some other data"},
{ "group": "g", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"},
{ "group": "x", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"},
{ "group": "t", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"}
]
I must sure the first job in group "x" execute successfully before the thirth job(same group).
But i don't care if the fourth job execute sooner than the first(or whatever).
Because sometimes it may happen which all three job deliver to 3 consumer, but the first job fail for some reason(but the second and thirth job has done successful).
I know with this conditions there will be some situations which all jobs in the queue are belongs to same group, so multiple consumers can't work on them and they must deliver one by one. that's ok.
There's no such thing in AMQ protocol that can lead to this exact solution, there're some ways to solve this problem.
Define queue for each message group
Set concurrency as 1
Let me quote the message ordering from the doc
Section 4.7 of the AMQP 0-9-1 core specification explains the
conditions under which ordering is guaranteed: messages published in
one channel, passing through one exchange and one queue and one
outgoing channel will be received in the same order that they were
sent. RabbitMQ offers stronger guarantees since release 2.7.0.
Ref: https://www.rabbitmq.com/semantics.html
First of the foremost things for you is to preserve the message ordering, once we have ordered messages we can utilize the concurrency to handle the messages in order.
Let's say your queue has 5 messages as shown
Queue: Queue1
+--------------+
Head-->|m1|m2|m3|m4|m5| <---- Tail
+--------------+
There's the concept of competing consumers, competing consumers means there're more than consumers/subscribers for the same queue. If there is more than one consumer than each of them will run autonomously, which means ordering on the consumer side won't be preserved. To preserve the ordering on consumer side, we should not use competing consumers.
Even though now consumers are not competing, we can still lose message ordering, if we have more than one executor. More than one executor simply means we can poll the queue, send a polled message to any of the executors. Based on the CPU execution policy etc we will still lose the ordering, so now we need to restrict the number of executors to 1.
As we have only one executor each of the polled messages will be executed in orders, so it will become a serial execution.
For Queue1
The executor will consume the message in the following order
-> m1
-> m2
-> m3
-> m4
-> m5
Still, there's one missing piece, what happens if the execution of m1 is failing?
You can retry for N number of times before consuming the next message, to achieve this don't acknowledge unless you have successfully executed any polled message.
From design points of view, this does not look good, since you're processing messages in serial instead of parallel, though you don't have any other alternatives.

Azure Function Apps - maintain max batch size with maxDequeueCount

I have following host file:
{
"version": "2.0",
"extensions": {
"queues": {
"maxPollingInterval": "00:00:02",
"visibilityTimeout": "00:00:30",
"batchSize": 16,
"maxDequeueCount": 3,
"newBatchThreshold": 8
}
}
}
I would expect with setup there could never be more than batchSize+newBatchThreshold number of instances running. But I realized when messages are dequed they are run instantly and not just added to the back of the queue. This means you can end up with a very high amount of instances causing a lot of 429 (to many requests). Is there anyway to configure the function app to just add the dequeded messages to the back of the queue?
It was not related to dequeueCount. The problem was because it was a consumption plan, and then you cant control the amount of instances. After chaning to a Standard plan it worked as expected.

How to fetch the current run status (i.e. success or failure) of all metronome jobs?

We are using metronome, and we want to create a dashboard for our jobs scheduled by it against its rest API.
Alas, the job endpoint
/v1/jobs
does not contain the last state, i.e. success or failure, but only its configuration.
Googling on how to get the history of a job, I found out that I can query the job history through embed=history GET parameter for each jobId.
I could now combine fetching the id list so that I could then fetch each job's history through:
/v1/jobs/{job_id}?embed=history
Yet this includes all the runs and also requires us to fetch each job individually.
Is there a way to get the metronome job status without querying all the jobs one by one?
You can click on each GET or POST endpoint on the official docs to see if it supports additional query params.
The endpoint for jobs indeed supports historic data
As you can see you can use embed=history or embed=historySummary, for your use-case embed=historySummary is better suited as it only contains the timestamps of the last run in this form and is less expensive and time-consuming:
[
{
"id": "your_job_id",
"historySummary": {
"failureCount": 6,
"lastFailureAt": "2018-01-26T12:18:46.406+0000",
"lastSuccessAt": "2018-04-19T13:50:14.132+0000",
"successCount": 226
},
...
},
...
]
You can compare those dates to figure out if the last run was successful. Yet keep in mind that lastFailureAt and lastSuccessAt might be null as a job might have been never run in the first place:
{
"id": "job-that-never-ran",
"labels": {},
"run": {
...
}
"historySummary": {
"successCount": 0,
"failureCount": 0,
"lastSuccessAt": null,
"lastFailureAt": null
}
},

bitcoin-cli: how to create a wallet and utxo address

I am a relative newbie in bitcoin and blockchain and hope you can help me with some of the questions.
So I launched a "regtest" network and generated 101 blocks using
bitcoin-cli -regtest generate 101
Now, if I launch 'bitcoin-cli -regtest getaddressesbyaccount ""', I get the public address of my default account:
[
"mwpKJNJ4UZL7yFyj53RSVcwauGAK84UvV2"
]
And of course, I should not have any other accounts as for now.
When I launch 'bitcoin-cli -regtest listunspent':
[
{
"txid": "694030f8638318c8c54054515ec716159edc494b14234885deb48f294b75a2fe",
"vout": 0,
"address": "n1queZpweTHjrMLvwSmcfrrJSQjsrYG3nG",
"scriptPubKey": "21038cadb266ed1ae6c474f5c1b74fc5f6790eacde843a673a16cfc924a100f2a679ac",
"amount": 50.00000000,
"confirmations": 101,
"spendable": true,
"solvable": true,
"safe": true
}
]
First question:
I understand that the only transaction listed by "listunspent" is UTXO,
meaning this is a transaction what I received to my address "n1queZpweTHjrMLvwSmcfrrJSQjsrYG3nG" with 50 BTC as amount.
Where this address comes from? By what bitcoin-cli command I can see/find it in my wallet?
Second question:
How can I create a new wallet with some balances and switch between them ( using bitcoin-cli )?
Basically, I would like to be able to test my app using bitcoin-cli - I need to be able to create wallets, switch between them and send btc between the addresses.
Coinbase coins can't be transferred until 100 blocks after they were created.
(Why did you generate "101" blocks specifically?)
So, the amount in your wallet you see is from the first block you mined. You can verify that by bitcoin-cli -regtest getblock "<hash of first block>" which you had got in return to the generate 101 command you ran earlier (an array of 101 block hashes).
Try the following
generate one more block bitcoin-cli -regtest generate 1
now listunspent and you should see 2 utxos instead of 1.
Depending on what you want to test, maybe simply creating a new address and sending money to it is enough for you?
[Edit]
Shut down core properly.
Rename your wallet.dat file
When you restart, a new wallet(wallet.dat) will be created. You can use them by supplying -wallet arg to bitcoin-qt
For example, if you are on linux:
Create 4 wallets by starting bitcoin core, stopping bitcoin core and then renaming the wallet.dat in your ~/.bitcoin folder (then repeating the process). For example, run this process 4 times to generate :
mywallet.dat
wifeswallet.dat
kidswallet.dat
businesswallet.dat
Then, in linux, in your .bashrc :
alias mywallet="bitcoin-qt -wallet=~/.bitcoin/mywallet.dat"
alias wifeswallet="bitcoin-qt -wallet=~/.bitcoin/wifeswallet.dat"
alias kidswallet="bitcoin-qt -wallet=~/.bitcoin/kidswallet.dat"
alias businesswallet="bitcoin-qt -wallet=~/.bitcoin/businesswallet.dat"