I want to view AmazonS3 JMX metrics. Specifically the number of retries performed by the client. The only problem is that according to this article it seems that it supports only Cloudwatch, and I want to see it momentarily using jconsole.
Enabling without credential file, doesn't seem to work.
Is there some kind of workaround?
Related
I'm pretty new to S3. I'm trying to create a Bucket and receive notifications on Object Created events using code only (not with the AWS Management UI).
I'm writing in dotnet so I'm using the AWSSDK.Core nuget package.
Until now I've managed to create a bucket using the sdk.
It seems like a trivial task though I couldn't find references around the web to accomplish it.
Also, the object storage is S3 compatible, not AWS S3.
I tried configuring a SNS Topic, but it seems that in order to enable notifications, the API requires SQS as a Queueing service, not RabbitMQ.
I did see another approach - configuration of a lambda function that transfers messages to RabbitMQ, but couldn't find references and documentation as well.
Any help is appreciated :)
I am trying to set up pods logs shipping from EKS to ElasticSearch Cloud.
According to Fluent Bit for Amazon EKS on AWS Fargate is here, ElasticSearch should be supported:
You can choose between CloudWatch, Elasticsearch, Kinesis Firehose and Kinesis Streams as outputs.
According to FluentBit Configuration Parameters for ElasticSearch having Cloud_ID and Cloud_Auth parameters should be enough to ship logs to Elasticsearch Cloud.
An example here shows how to configure ES output for FluentBit, so my config looks like:
[OUTPUT]
Name es
Match *
Logstash_Format On
Logstash_Prefix ${logstash_prefix}
tls On
tls.verify Off
Pipeline date_to_timestamp
Cloud_ID ${es_cloud_id}
Cloud_Auth ${es_cloud_auth}
Trace_Output On
I am running a simple ngnix container to generate some logs (as in one of the linked examples), but they don't seem to appear in my ElasticSearch / Kibana.
Am I missing anything? How do I ship logs to ElasticSearch Cloud?
Also, Trace_Output On is supposed to log FluentBits' attempts to ship logs, but where can I see these logs on EKS?
I also ran into this. It seems to me only AWS ElasticSearch is supported when using the AWS managed FluentBit (from what I can tell).
https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-eks-adds-built-in-logging-support-for-aws-fargate/
You can work around this by using a sidecar fluentbit container (which can send to ElasticSearch) if that's an option for you. You will need to modify the application to have logs written to the filesystem.
Or you can use the managed FluentBit with the cloudwatch output, subscribe with to the log group with a lambda function and send it to ES.
My understanding on aws xray is, xray is similar to dynatrace and I am trying to use xray for monitoring apache performance. I do not see any document related to xray with apache except below.
https://mvnrepository.com/artifact/com.amazonaws/aws-xray-recorder-sdk-apache-http
Can anyone please suggest if it is possible to use aws xray with apache and if yes can you also point some document related to it. Thanks.
I assume that by "apache" you mean the Apache Tomcat servlet container, since you are referring to a maven artifact which is a Java build tool.
Disclamer: I don't know what "dynatrace" is and I don't know which logging you specifically want.
But as far as the Apache Tomcat servlet container and X-Ray goes - here is the link to get started:
http://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java.html
Start by adding AWSXRayServletFilter as a servlet filter to trace incoming requests. A servlet filter creates a segment While the segment is open you can use the SDK client's methods to add information to the segment and create subsegments to trace downstream calls. The SDK also automatically records exceptions that your application throws while the segment is open.
As for the mentioned maven artifact:
aws-xray-recorder-sdk-apache-http – Instruments outbound HTTP calls made with Apache HTTP clients
So, you'll need this if, let's say, a client makes a request to your Tomcat server and your Tomcat server makes a request to another server thus acting as a client in this case.
The current app (with a competitor) is using Spring-Boot and Spring-Sessions to save session data independent from the instance in Redis.
How can we replicate this in Bluemix?
Is there a way to add on Spring-Sessions to the Redis service?
The other option would be the IBM Session Cache. Does that work with Spring-Boot and Spring-Session? Does the Session Cache service work without Tomcat? (Jetty for example?)
I haven't played with Spring-Sessions yet, but I might be able to point you in the right direction.
There is a general Redis service available on Bluemix. The open source Java buildpack (Tomcat) promises auto configuration of Redis. I would try this first. To use this buildpack:
cf push <appname> -b java_buildpack
If that doesn't work, you could read the credentials from VCAP_SERVICES and configure JedisConnectionFactory programmatically connect to it.
If you use the IBM Session Cache service with a Liberty application on Bluemix, the configuration is auto generated such that standard JEE HTTPSession objects are persisted to the service. It might work with other java runtimes, but configuration would be more manual.
Hope this helps.
Does the session cache work without Tomcat (Jetty for example)?
In general, the session cache should work with any webserver/servletcontainer, when Bluemix supports Jetty runtime for example, the session cache should support.
Session cache is based on IBM WebSphere eXtreme Scale caching technology, and the base product had been tested against WebSphere, Liberty and Tomcat runtime for the HTTP session use cases, but not tested under Jetty.
We have a large extended network of users that we track using badges. The total traffic is in the neighborhood of 60 Million impressions a month. We are currently considering switching from a fairly slow, database-based logging solution (custom-built on PHP—messy...) to a simple log-based alternative that relies on Amazon S3 logs and Splunk.
After using Splunk for some other analyisis tasks, I really like it. But it's not clear how to set up a source like S3 with the system. It seems that remote sources require the Universal Forwarder installed, which is not an option there.
Any ideas on this?
Very late answer but I was looking for the same thing and found a Splunk app that does what you want, http://apps.splunk.com/app/1137/. I have yet not tried it though.
I would suggest logging j-son preprocessed data to a documentdb database. For example, using azure queues or simmilar service bus messaging technologies that fit your scenario in combination with azure documentdb.
So I'll keep your database based approach and modify it to be a schemaless easy to scale document based DB.
I use http://www.insight4storage.com/ from AWS Marketplace to track my AWS S3 storage usage totals by prefix, bucket or storage class over time; plus it shows me the previous versions storage by prefix and per bucket. It has a setting to save the S3 data as splunk format logs that might work for your use case, in addition to its UI and webservice API.
You use Splunk Add-On for AWS.
This is what I understand,
Create a Splunk instance. Use the website version or the on-premise
AMI of splunk to create an EC2 where splunk is running.
Install Splunk Add-On for AWS application on the EC2.
Based on the input logs type (e.g. Cloudtrail logs, Config logs, generic logs, etc) configure the Add-On and supply AWS account id or IAM Role, etc parameters.
The Add-On will automatically ping AWS S3 source and fetch the latest logs after specified amount of time (default to 30 seconds).
For generic use case (like ours), you can try and configure Generic S3 input for Splunk