I'm new to Google Compute Engine and I just setup a new instance that will eventually become a web server. All the firewall rules were left at default when the instance was created, which leaves open a few ports (RDP, SSH, etc) for administration. No software was installed or DNS records pointing to the server were created and I left the instance running.
After a couple weeks, I looked at the billing and there were over 300 mebibytes of data billed due to egress traffic to China and America. I'm wondering if this is a normal.
Is there any particular reason hundreds of megs of traffic went out on a brand new, firewalled instance? Does the Google cloud service offer some kind of network analysis tool to breakdown traffic by type/destination?
Thanks for any advice.
You can use Google Cloud Stackdriver monitoring to see the history of your ingress and egress traffic of your VM instances. However to get more information of traffic (like IPs, packets, requests, etc..) you will need to install a third party monitoring tool or to use a packet capturing tool (Wireshark, tcpdump,...) to analyze your traffics.
Related
I got a mail from amazon s3 webservices stating below details
"We are writing to you today to let you know about changes which impact your use of the Amazon Simple Storage Service (S3). In efforts to best serve our customers, we have improved the systems powering the Amazon S3 API and are in the process of shutting down legacy application server capacity. We have detected access on the legacy capacity for Amazon S3 buckets that you own. The legacy capacity is no longer in service, as the DNS entry for the S3 endpoint no longer includes the IP addresses associated with it. We will be shutting down the legacy capacity and retiring the set of IP addresses fronting this capacity after April 1, 2020."
I want to find out which legacy system I am using, and how to prevent from affecting my services.
Imagine you had a web site, www.example.com.
In DNS, that name was pointed to your web server at 203.0.113.100.
You decide to buy a new web server, and you give it a new IP address, let's say 203.0.113.222.
You update the DNS for example.com to point to 203.0.113.222. Within seconds, traffic starts arriving at the new server. Over the coming minutes, more and more traffic arrives at the new server, and less and less arrives at the old server.
Yet, for some strange reason, a few of your site's prior visitors are still hitting that old server. You check the DNS and it's correct. Days go by, then weeks, and somehow a few visitors who used your old server before the cutover are still hitting it.
How is that possible?
That's the gist of the communication here from AWS. They see your traffic arriving on unexpected S3 server IP addresses, for no reason that they can explain.
You're trying to connect to the right endpoint -- that's not the issue -- the problem is that for some reason you have somehow "cached" (using the term in a very imprecise sense) an old DNS lookup and are accessing a bucket by hitting a wrong, old S3 IP address.
If you have a Java backend service accessing S3, those can notorious for holding on to DNS lookups forever. You might need to restart that service, and look into how to resolve that issue and enable correct behavior which is -- as I understand it -- not how Java behaves by default. (Not claiming to be a Java expert but I've encountered this sort of DNS behavior many times.)
If you have an HAProxy or Nginx server that's front-ending for an S3 bucket and has been up for a while, those might need a restart and you should look into how to correctly configure them not to resolve DNS only at startup. I ran into exactly this issue once, years ago, except my HAProxy was forwarding requests to Amazon CloudFront on only 1 of the several IP addresses it could have been using. They took that CloudFront edge server offline, or it failed, or whatever, and the DNS was updated... but my proxy was not able to re-query DNS so it just kept trying and failing until I restarted it. Then I fixed it so that it periodically repeated the DNS lookup so it always had a current address.
If you have your own DNS resolver servers, you might want to verify that they aren't somehow misbehaving, and you might want to ensure that you don't for some reason have any /etc/hosts (or equivalent) static host entried for anything related to S3.
There could be any number of causes but I'm confident at least in my interpretation of what they say is happening.
AWS
I am using network load balancer infront of private VPC in the API gateway. Basically for APIs in the gateway the endpoint is network load balancer's DNS name.
The issue is, performance sucks (+5 seconds).. If I use the IP address of the EC2 instead of NLB DNS the response is very good (less than 100ms).
Can somebody point me what is the issue? Any configuration screw up I did while creating NLB?
I have been researching for the past 2 days and couldn't find any solution.
Appreciate your response.
I had a similar issue that was due to failing health checks. When all health checks fails, the targets are tried randomly (typically target in each AZ), however, at that stage I had only configured an EC2 in one of the AZs. The solution was to fix the health checks. They require the SecurityGroup (on the EC2 instances) to allow the entire VPC CIDR range (or at least the port the health checks are using).
Is there a way to make physical F5 BigIP LB to route traffic to both EC2 instances(Autoscaling) and physical machines? I came across this article https://devcentral.f5.com/articles/using-big-ip-gtm-to-integrate-with-amazon-web-services but it seems it is routing traffic to an entire AWS zone, not to a couple of EC2 instances behind a ELB.
yes, you can route traffic to any resources from BIG-IP, whether they are locally defined on the same L3 network or remote, you just need to make sure you have routes defined on BIG-IP pointing in the right direction. If you are trying to cloudburst, you can define priority level in the pool so that your physical servers get the traffic unless the minimum threshold is crossed, at which point the remote servers (other datacenter or cloud servers, doesn't matter) will be automatically engaged.
You can also add orchestration to where your cloud servers aren't up and active unless you are getting close to a threshold, at which point the BIG-IP can trigger an action to spin up those servers, then add them to the pool dynamically. There are many options available to you with BIG-IP
I have service running on Google Compute Engine. I've got few instances in Europe in a target pool and few instances in US in a target pool. At the moment I have a domain name, which is hooked up to the Europe target pool IP, and can load balance between those two instances very nicely.
Now, can I configure the Compute Engine Load Balancer so that the one domain name is connected to both regions? All load balancing rules seem to be related to a single region, and I don't know how I could get all the instances involved.
Thanks!
You can point one domain name (A record) at multiple IP addresses, i.e. mydomain.com -> 196.240.7.22 204.80.5.130, but this setup will send half the users to the U.S., and the other half to Europe.
What you probably want to look for is a service that provides geo-aware or geo-located DNS. A few examples include loaddns.com, Dyn, or geoipdns.com, and it also looks like there are patches to do the same thing with BIND.
You should configure your DNS server. Google does not have a DNS service, as one part of their offering, at the moment. You can use Amazon's Route 53 to route your requests. It has a nice feature called latency based routing which allows you to route clients to different IP addresses (in your case - target pools) based on latency. You can find more information here - http://aws.amazon.com/about-aws/whats-new/2012/03/21/amazon-route-53-adds-latency-based-routing/
With Google's HTTP load balancing, you can load balance traffic over these VMs in different regions by exposing via one IP. Google eliminates the need for GEO DNS. Have a look at the doc:
https://developers.google.com/compute/docs/load-balancing/
Hope it helps.
For a cloud instance that runs Apache, I'm guessing the cloud has an IP address.
One of the benefits of using a cloud is scaling, but I'm not sure how that scaling happens. I thought that new instances are created automatically to accommodate rise in traffic. IF that's correct (correct me if I'm wrong), then does that mean that each new instance would have its own IP or what? because if that's the case, it would complicate matters a lot when pointing a domain to a cloud.
The cloud sits behind a load balancer which is able to redirect traffic to different spawned instances of Apache servers. In that since you can grow and shrink to any number of servers based on how much traffic you are receiving.