AWS CLI via 2 proxies - apache

Question

I have a scenario where i need to execute AWS CLI commands via 2 proxies against the AWS cloudwatch.
Server A(AWS CLI) -----> Server B (Apache proxy Web server) -----> Corporate Proxy IP (X.X.X.X) -----> Internet
My Challenge here is that the AWS CLI commands do not have a context (/something) based on which a rewrite rule (to be written on Server B) can be applied to forward the request from Server A to Corporate Proxy IP and finally to internet (AWS).
Connectivity from Corporate Proxy IP is already there to Internet.
My main motive is to fetch cloudwatch metrics on Server A via the 2 proxies. According to me this is not achievable but need inputs if this can be achieved and if YES, what Rewrite rule
should be written on Server B to proxy the AWS CLI commands to Corporate Proxy.
AWS CLI commands Eg. would be as below:
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilization --dimensions Name=InstanceId,Value=i-xxx --statistics Average --start-time date -u '+%FT%TZ' -d '10 mins ago' --end-time date -u '+%FT%TZ' --period 60
I'm aware that we can use HTTP_PROXY to forward requests via proxy, however that would only forward my request from Server A to Server B (Apache proxy Web Server).
Thanks in advance & appreciate a quick response.

Answer 1

Ok, so I actually recently built out a reverse proxy server (nginx) specifically to forward AWS CLI requests to help with corporate firewalls. Unfortunately, I cannot publish the code used to make that work, but can give you some insights into the issues with setting up a system like this.
This ones the most obvious. You'll need to have a redirection rule that understands the request being pushed through it and rewrites it to a syntax that the upstream AWS server can understand. In default AWS commands, that context is part of the URL (e.g. https://ec2.us-west-2.amazonaws.com). If you're passing through an upstream reverse proxy, you'll need to pass that context up somehow. You can either have a star DNS record to capture all requests to your proxy the same way amazon does it (e.g. \*.\*.{proxy-address} => {proxy-ip} then aws ec2 --endpoint-url https://ec2.us-west-2.proxy describe-instances) or you can manually inject the information into the path (e.g. aws ec2 --endpoint-url https://proxy/ec2/us-west-2 describe-instances). Then, on your proxy server, you parse out the information and set your upstream based on it. My final solution was to place the full default AWS endpoint url into the path of my proxy https://proxy/ec2.us-west-2.amazonaws.com then have regex upstream to parse out the endpoint URL in case there is information in the path placed by amazon, then set the upstream server to the endpoint URL resolved by the regex.
After you complete #1, you'll now run into the second issue of generated signatures. If you're using the --endpoint-url flag to the CLI, it will sign the request with the Host header set to the proxy server URL. Now, when this is rewritten upstream, that Host header will no longer match the signature. So, you'll need to re-sign any request passing through the proxy. There's a couple sneaky ways around this. What I ended up doing was creating an AWS CLI wrapper which overloaded the signing mechanism to sign the request as if it was sending it to the default AWS endpoint, then overwriting the Host header to point towards my reverse proxy. Re-signing this way is advantageous because it removes proxy latency due to not having to translate the request, but is quite difficult to implement in a way that will dynamically ingest any new signature methods AWS may release.
It is also worth noting that if you dig deep enough into the botocore source code, you'll find some reverse proxy support that is build in, but appears to be defunct/not used (it is not exposed to the client). Hopefully, they flush out that functionality in the near future and this will no longer be an issue.