I want to trigger an alert when there are no logs coming from our services in Splunk but not sure how to do that.
I can search our logs using this [| inputlookup app | search app=app_name env=prod service=app_name] where app is the csv lookup table with app, env, and service properties that provide lookup values for our search.
One other thing to note is I have access to the sourcetype or the source where
sourcetype=kube:container:app_name_env
source=*k8s_app_name_env_*
But again, not sure what search query I should create the alert based on. I know how to create alerts in splunk but not sure how to trigger it if there are no logs coming from the source above. Any suggestions? Thanks!
In the Alert actions, have it send a message when there are no results:
Related
If I execute a BigQuery job using the REST API (i.e. bigquery.googleapis.com) in the response I get back a selfLink that looks something like this:
https://bigquery.googleapis.com/bigquery/v2/projects/my-project/jobs/job_0123456789ABCDEF?location=EU
In the UI (ie.. console.cloud.google.com) I can see the very same job in the project's query history:
Is it possible to use the information within that API response and construct a URL that will allow a person to visit that URL in the browser and be taken directly to the information about that query in the UI? This would be really useful because we could log a message containing that URL so that anyone viewing the logs can see a user-friendly UI regarding that job.
I suspect the answer is "no" but just thought I'd ask.
I believe you can share this link:
https://console.cloud.google.com/bigquery?project=<my-project>j=<bq:<location>:<job_id>>&page=queryresults
For example: https://console.cloud.google.com/bigquery?project=my-project&j=bq:US:2846160a-9a13-4192-9bff-e691ff2adab6&page=queryresults
If a user has BQ Job List permission in that project, then when they open up the link they will be be able to see the query that was run in the UI, along with the job information.
But they can't see the query results, which is intended behavior. Instead they will get a warning:
Access Denied: User does not have permission to access results of another user's job.
On AWS Cloudwatch Log Insights, often after you filter your logs and find an interesting log message you want to see what was happening right before or after that message.
What is the best way to find log messages right next to another message?
I imagine a good way is to filter based on #tiemstamp and they conveniently provide a date_floor function, but I cannot figure out the syntax that works for equality to timestamps.
In Logs Insights, if you query for:
fields #timestamp, #message, #logStream
| filter #message like /<Your Log Message>/
you'll get a link where you can access the log stream. Clicking there will bring you right to the context before/after the log you're interested in.
It's a fair bit of work for something that should be a single click but it's the only work-around I'm aware of. Feel free to go bug the AWS team to build this as a 1-click feature right from the logs themselves.
Edit:
Something I didn't know when I wrote this answer: this trick only works if you're querying a single log group. If you're querying multiple, it still shows the logStream but it's not clickable.
I've run into a few different issues with the PagerDuty integration in Splunk Cloud.
The documentation on PagerDuty's site is either outdated, not applicable to Splunk Cloud or else there's something wrong with the way my Splunk Cloud account is configured (could be a permissions issue): https://www.pagerduty.com/docs/guides/splunk-integration-guide/. I don't see an Alert Actions page in Splunk Cloud, I have a Searches, Reports and Alerts page though.
I've configured PD alerts in Splunk using the alert_logevent app but it's not clear if I should instead be using some other app. These alerts do fire when there are search hits but I'm seeing another issue (below). The alert_webhook app type seems like it might be appropriate but I was unable to get it to work correctly. I cannot create an alert type using the pagerduty_incident app. . . although I can set it as a Trigger Action (I guess this is how it's supposed to work, I don't find the UI to intuitive here).
When my alerts fire and create incidents in PagerDuty, I do not see a way to set the PagerDuty incident severity.
Also, the PD incidents include a link back to Splunk, which I believe should open the query with the search hits which generated the alert. However, the link brings me to a page with a Page Not Found! error. It contains a link to "more information about my request" which brings up a Splunk query with no hits. This query looks like "index=_internal, host=SOME_HOST_ON_SPLUNK_CLOUD, source=*web_service.log, log_level=ERROR, requestid=A_REQUEST_ID". It it not clear to me if this is a config issue, bug in Splunk Cloud or possibly even a permissions issue for my account.
Any help is appreciated.
I'm also a Splunk Cloud + PagerDuty customer and ran into the same issue. The PagerDuty App for Splunk seems to create all incidents as Critical but you can set different severities with event rules.
One way to do this is dynamically is to rename your Splunk alerts with the desired severity level and then create a PagerDuty event rule for each level that looks for the keyword in the Summary. For example...
If the following condition is met:
Summary contains "TEST"
Then perform the following actions:
Set severity = Info
screenshot of the example in the event rule edit screen
It's a bit of pain to rename your existing alerts in Splunk but it works.
If your severity levels in Splunk are programmatically set like in Enterprise Security, then another method would be to modify the PagerDuty App for Splunk to send the $alert.severity$ custom alert action token as a custom detail in the webhook payload and use that as event rule condition instead of Summary... but that seems harder.
I need to get a list of users for a specific Push adapter/event source, and, I'm trying to use the API console requests, which says the format is:
http://{hostname}:{port}/{context-root}/console/api/{api-context}/{action}/{parameters}
and I'm using:
http://192.168.1.106:10080/Module_07_04_nativeAPIForiOSPush/console/api/Push/get/PushAdapter/PushEventSource
to search the demo project, which has one subscriber. However, I get 404 return from a browser request.
The first column of the docs is the "api-context", but, it lists "Push" and "Event Sources", which, obviously seems invalid.
What is the correct format to find users subscribed to a push for a specific adapter/event source?
WL server does not provide API for listing subscribed users. By design you should maintain your own DB of subscribed users, this is why you have onSubscribe onUnsubscribe callbacks in event source. As an alternative - you can look into WL's DB tables to find this info.
I wanna capture the error message from the web based application during the run-time of QTP if server couldn't able to process the request which was hardcoded.
Example: If i'm trying run for sign-up page were i hardcoded all the fields but in the website they asking for minimum of 8 character for the field password but i hardcoded only 6 character.
I'm facing an error while running QTP and that web application shows additional "Pop-up" page. I wanna capture that message in that pop-up window and need to store string for future purpose.
Pls help me
You can typically use an Output Value to capture strings from elements in the application. How to do this exactly depends on the way your application displays the error message, you will have to give some more details if you want a more specific answer.
Use the QTP Recovery Scenario feature and setup trigger for a popup having a specific window title.