I'm writing an aspnet core app which can be deployed to either azure or aws. The Microsoft libraries for accessing azure logging / configuration sources are well-behaved and fail silently if they're not in an appropriate environment. However the AWS SDK blows up with and exception Unable to get IAM security credentials from EC2 Instance Metadata Service. if the providers are configured outside of AWS. Are there some environment variables I can look at to determine if my application is running in AWS so I can skip those?
All EC2 instances (and therefore all AWS hosting methods) have access to an instance meta-data http service running on a local-link address at http://169.254.169.254/latest/meta-data/. The best process I can come up with is to make a call to this service. If the call fails -- the process is not hosted on an EC2 instance.
Related
I have developed a set of Lagom microservices. The development environment provides with default Service Gateway and Service Locator.
In a production environment I would like my services to:
register to a service registry
be available to a web app through a service locator that uses this registry
What should I use as Service Registry / Service Locator / Service Gateway ?
A simple NGINX would be a reasonable service gateway but it implies a very static configuration based on redirect rules (no actual registration).
I cannot find any code sample on this subject and the documentation is very poor (it describes well development tools but doesn't help when it comes to actual production).
The documentation on that area is vague on purpose because the ecosystem is very vast and changes fast.
You could, for example, use Consul or ZooKeeper to keep track of the instances that are runnning for each service and where they are running (where means IP:PORT). Then you would need to use a Consul-based or a ZooKeeper-based Service Locator instance. The preferred target deployment environment these days is Kubernetes (in any of its flavors) so the service location is based on DNS-SRV lookups on the DNS server provided by k8s. The registration step happens automatically in a k8s setup for each pod so you won't need to care for that.
Then, the reverse proxy on the edge capable of directing each request to the appropriate process is a plain-old HTTP proxy that can check your service location (or cache the service location information). These days the recommendation is configuring an Ingress/Route (for k8s or OpenShift) edge proxy for each of your lagom services.
See the guide on Deploying a Lagom application to Openshift for a thorough explanation.
I'm trying to deploy a node.js application into aws using Elastic BeansTalk. However, while creating a web server environment, it fails because my account does not havve the permission to use auto scaling due to cost. Is there a way to disable autoscaling completely with elastic beanstalk?
Yes, just create a single instance environment.
As per the AWS docs, single instance environments do not have load balancers or auto-scaling. You can read more about Elastic Beanstalk environment types here:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-types.html?ref_=pe_395030_31184250_9
My Airflow application is running in AWS EC2 instance which has IAM role as well. Currently I am creating Airflow S3 connection using hardcoded access and secret key. But I want my application to pickup this AWS credentials from this instance itself.
How to achieve this?
We have a similar setup, our Airflow instance run inside containers deployed inside an EC2 machine. We set up the policies to access S3 on the EC2 machine instance profile. You don't need to pick up the credentials in the EC2 machine, because the machine has an instance profile that should have all the permissions that you need. From the Airflow side, we only use aws_default connection, in the extra parameter we only setup the default region, but there aren't any credentials.
Here a details article about Intance Profiles: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
The question is answered but for future reference, it is possible to do it without relying on aws_default and just doing it via Environment Variables. Here is an example to write logs to s3 using an AWS connection to benefit form IAM:
AIRFLOW_CONN_AWS_LOG="aws://"
AIRFLOW__CORE__REMOTE_LOG_CONN_ID=aws_log
AIRFLOW__CORE__REMOTE_LOGGING=true
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER="s3://path to bucket"
I have an Express API server running on localhost on my own machine. How do I make it accessible from the Internet and not just my own machine?
Preferably, it would be deployed on AWS.
In AWS there are multiple ways of hosting your express application based on flexibility vs convenience.
AWS Elastic Beanstalk:
This will provide you more convenience by creating an autoscaling and loadbalancing environment with version management and roll back support from one place in AWS web console. Also provide you IDE support for deployments and CLI commands for CI/CD support.
AWS ECS:
If you plans to dockerize your application(Which I highly recommend) you can use AWS ECS to manage your docker cluster with container level Autoscaling and loadbalancing support for more convenience. This also provides CLI for CI/CD.
AWS EC2:
If you need more flexibility, you can get a virtual server in AWS and also manually configure autoscaling and loadbalancing which I prefer as the least option simply for a web app since you have to do most of the things manually.
All this services will provide you with publicly accessible URL if you configure them properly to grant access from outside. You need to configure networking and security groups properly either exposing the loadbalancer or instance IP/DNS URL to the outside.
This has to be a pretty simple task. I have an Azure Cloud Service that works just fine with HTTP and HTTPS. Now I want to create a net.tcp connection to the cloud service. The first thing I discover is that net.tcp is not supported by IIS Express. That's fine. So I try switching to regular IIS and all I get when I try to run the application is a 404 error. If I was building an WCF Web Application, I would get a form that allows me to select (and build) the virtual directory for the application, but there's no analog for this in the Cloud Service.
I can take a fresh 'Cloud Service' project build using visual studio, make no changes to it except change IIS Express to IIS. When I launch it, I get a 404 error in the web browser.
Does anyone have a working example of tcp.net (with IIS Express or standard IIS) in a Cloud Service?
This video by the Azure team is a great overview and cleared up a lot of conceptual issues I had. An Azure Cloud Service is a container around one or more virtual machines. The virtual machines basically come in two flavors: Web Role and Worker Role. What threw me is the description that the Worker Role was intended for background processes: not true. A Web Role is basically a VM with IIS, a Worker Role is basically a VM without IIS. Since the off-the-shelf IIS doesn't have net.tcp installed, you have to jump through hoops in order to get the protocol installed. Instead, the Worker Role can be used as a self-hosted web service. If you want the performance and throughput of TCP, this appears to be the direction to go.