MinIO Signature Mismatch - reverse-proxy

I have set up MinIO behind a reverse proxy in EKS. Everything worked well until MinIO was updated to RELEASE.2021-11-03T03-36-36Z. Now I am getting the following error when trying to access my MinIO bucket using the mc command-line utility: mc: <ERROR> Unable to list folder. The request signature we calculated does not match the signature you provided. Check your key and signing method.
mc version is RELEASE.2021-11-16T20-37-36Z. When I port-forward the MinIO container to localhost and access it in a browser at http://localhost:9001 I can get to it, but I can't log in anymore. I get the error:
Invalid Login, 403 Forbidden`. This is seen in my MinIO container
It also logs the following:
API: SYSTEM()
Time: 03:19:57 UTC 11/23/2021
DeploymentID: 60a8ed7a-7448-4a3d-9220-ff823facd54e
Error: The request signature we calculated does not match the signature you provided. Check your key and signing method. (*errors.errorString)
requestHeaders={"method":"POST","reqURI":"/minio/admin/v3/update?updateURL=","header":{"Authorization":["AWS4-HMAC-SHA256 Credential=<credential-scrubbed>/20211123//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=37850012ca8d27498793c514aa826f1c29b19ceae96057b9d46e24599cc8081b"],"Connection":["keep-alive"],"Content-Length":["0"],"Host":["<host-info-scrubbed>"],"User-Agent":["MinIO (darwin; amd64) madmin-go/0.0.1 mc/RELEASE.2021-11-16T20-37-36Z"],"X-Amz-Content-Sha256":["<scrubbed>"],"X-Amz-Date":["20211123T031957Z"],"X-Forwarded-For":["10.192.57.142"],"X-Forwarded-Host":["<host-info-scurbbed>"],"X-Forwarded-Path":["/minio/admin/v3/update"],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Real-Ip":["10.192.57.142"]}}
5: cmd/auth-handler.go:154:cmd.validateAdminSignature()
4: cmd/auth-handler.go:165:cmd.checkAdminRequestAuth()
3: cmd/admin-handler-utils.go:41:cmd.validateAdminReq()
2: cmd/admin-handlers.go:87:cmd.adminAPIHandlers.ServerUpdateHandler()
1: net/http/server.go:2046:http.HandlerFunc.ServeHTTP()
When checking the proxy logs (NGINX), I see:
10.192.57.142 - - [24/Nov/2021:21:18:17 +0000] "GET / HTTP/1.1" 403 334 "-" "MinIO (darwin; amd64) minio-go/v7.0.16 mc/RELEASE.2021-11-16T20-37-36Z"
Any suggestions or advice on what I can do to resolve this would be great! I'm using the mc client on OSX.

Related

Minio uploads through the web interface and API receives "Unauthorized request."

I can successfully upload files to my Minio server using mc command line client (logged in as root):
./mc cp roobina.jpg minio/mag
roobina.jpg: 63.50 KiB / 63.50 KiB
But when I try to upload a file to a bucket using minio's own web interface I receive this error:
Unauthorized request.
When using api (in a php application using AmazonS3 libraries), I receive this error:
Error:Error executing "PutObject" on "https://s3.***.net/clbu/public/4d/4b/d1ad580690058a636ad58e5af931541336ec.jpg"; AWS HTTP error: Client error: `PUT https://s3.***.net/clbu/public/4d/4b/d1ad580690058a636ad58e5af931541336ec.jpg` resulted in a `403 Forbidden` response:
Forbidden (truncated...) Unable to parse error information from response - Error parsing XML: String could not be parsed as XML
Could someone please help?
After looking at different possible causes, I found that mod_security of apache (used as reverse proxy of minio:9000) was interfering with uploads causing the problem.
I disabled mod_security on the reverse proxy account and the problem is now solved.

400 bad request when attempting connection to AWS Neptune with IAM enabled

I am unable to connect to neptune instance that has IAM enabled. I have followed the AWS documentation (corrected a few of my silly errors on the way) but without luck.
When I connect via my Java application using the SigV4Signer and when I use the gremlin console, I get a 400 bad request websocket error.
o.a.t.g.d.Handler$GremlinResponseHandler : Could not process the response
io.netty.handler.codec.http.websocketx.WebSocketHandshakeException: Invalid handshake response getStatus: 400 Bad Request
at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:267)
at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker.finishHandshake(WebSocketClientHandshaker.java:302)
at org.apache.tinkerpop.gremlin.driver.handler.WebSocketClientHandler.channelRead0(WebSocketClientHandler.java:69)
When I run com.amazon.neptune.gremlin.driver.example.NeptuneGremlinSigV4Example (from my machine over port-forwarding AND from the EC2 jumphost) I get:
java.util.concurrent.TimeoutException: Timed out while waiting for an available host - check the client configuration and connectivity to the server if this message persists
I am able to connect to my neptune instance using the older deprecated certificate mechanism. I am using a jumphost ec2 instance and port-forwarding.
I believe that the SigV4 aspect is working as in the neptune audit logs I can see attempts to connect with the aws_access_key:
1584098990319, <jumphost_ip>:47390, <db_instance_ip>:8182, HTTP_GET, [unknown], [unknown], "HttpObjectAggregator$AggregatedFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: CompositeByteBuf(ridx: 0, widx: 0, cap: 0, components=0)) GET /gremlin HTTP/1.1 upgrade: websocket connection: upgrade sec-websocket-key: g44zxck9hTI9cZrq05V19Q== sec-websocket-origin: http://localhost:8182 sec-websocket-version: 13 Host: localhost:8182 X-Amz-Date: 20200313T112950Z Authorization: AWS4-HMAC-SHA256 Credential=<my_access_key>/20200313/eu-west-2/neptune-db/aws4_request, SignedHeaders=host;sec-websocket-key;sec-websocket-origin;sec-websocket-version;upgrade;x-amz-date, Signature=<the_signature> content-length: 0", /gremlin
But when I look
This is the policy that I created:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"neptune-db:*"
],
"Resource": [
"arn:aws:neptune-db:eu-west-2:<my_aws_account>:*/*"
]
}
]
}
I have previously tried with a policy that references my cluster resource id.
I created a new api user with this policy attached as its only permission. (I've tried this twice).
IAM is showing my that the graph-user I created has not successfully logged in (duh).
Seems that the issue is with the IAM set-up somewhere along the line. Is it possible to get more information out of AWS with regards to why the connection attempt is failing?
I am using the most recent release of Neptune and the 3.4.3 Gremlin Driver and console. I am using Java 8 when running the NeptuneGremlinSigV4Example and building the libraries to deploy to the console.
thanks
It appears from the audit log output that the SigV4 Signature that is being created is using localhost as the Host header. This is most likely due to the fact that you're using a proxy to connect to Neptune. By default, the NeptuneGremlinSigV4Example assumes that you're connecting directly to a Neptune endpoint and reuses the endpoint as the Host header in creating the Signature.
To get around this, you can use the following example code that overrides this process and allows you to use a proxy and still sign the request properly.
https://github.com/aws-samples/amazon-neptune-samples/tree/master/gremlin/gremlin-java-client-demo
I was able to get this to work using the following.
Create an SSL tunnel from you local workstation to your EC2 jumphost:
ssh -i <key-pem-file> -L 8182:<neptune-endpoint>:8182 ec2-user#<ec2-jumphost-hostname>
Set the following environment variables:
export AWS_ACCESS_KEY_ID=<access_key>
export AWS_SECRET_ACCESS_KEY=<secret_key>
export SERVICE_REGION=<region_id> (i.e. us-west-2)
Once the tunnel is up and your environment variables are set, use the following format with the Gremlin-Java-Client-Demo:
java -jar target/gremlin-java-client-demo.jar --nlb-endpoint localhost --lb-port 8182 --neptune-endpoint <neptune-endpoint> --port 8182 --enable-ssl --enable-iam-auth

Puppet agent is not running successfully after updating ssl certs

I am running puppet 3.7. The certs are expiring for me so I have updated the certs (after creating a backup so I am able to get back to the original state and that's fine). After updating the certs on puppetmaster using this, updating certs on the agent using this and updating certs on puppetdb using this, I am unable to run puppet agent successfully on a client box. It gives me the following error:
root#ip-10-181-36:/var/lib/puppet# sudo puppet agent -t
Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
(at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in 'issue_deprecation_warning')
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: newer-generic-host(127.0.0.1) access to /node/ip-10-181-36 [find] authenticated at :39
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: newer-generic-host(127.0.0.1) access to /catalog/ip-10-181-36 [find] authenticated at :1
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: newer-generic-host(127.0.0.1) access to /report/ip-10-181-36 [save] authenticated at :91
I am stuck at this point and no googling or reading docs or seeing the logs is helping. Does anyone have any ideas?

AWS Boto S3 API read KMS encrypted keys

I tried to read keys which are encrypted by using AWS KMS, I first hit the following error.
S3ResponseError: 400 Bad Request
InvalidArgumentRequests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.Authorizationnull1D584F77747CBB01LQIxPHmUGGDMnnI45xqWHtrB1+96tc7pDIEi6bVEE5i425SRypqeBXzvsH0CqPzwJe4xVv1UjhQ=
After setting os.environ['S3_USE_SIGV4'] = 'True', the above 400 error is gone, but now I hit the 403 error.
S3ResponseError: 403 Forbidden
May I ask if anyone hit the same issue before?
This error is caused by a wrong S3 hostname which was s3-ap-southeast-1.s3.amazonaws.com but should be s3-ap-southeast-1.amazonaws.com

Failure to login to an Orbeon Forms application when using HTTPS

I have an Orbeon Forms v3.7.1 based application that fails when I try to login using HTTPS instead of HTTP.
I am using Apache as a proxy server connecting requests to a an application running on WebLogic 12.1.3. For various reasons, we recently had to migrate the Apache plugin module from mod_proxy to mod_weblogic.
For both modules, we have configured the module to map the incoming HTTPS requests to HTTP going to the WebLogic server. The main difference we noticed that seems to be causing the error is that the Referer header changed. When using the mod_proxy module, the Referer is listed as the Apache server using HTTP. When using the mod_weblogic module, the Referer is listed as the incoming Apache URL using HTTPS.
When using the mod_weblogic module, we get the following error in the Orbeon log when we try to login to the application. It seems to be failing while parsing the URL. Any ideas how to fix this problem?
Here is the exception listed in the orbeon log file:
2015-03-19 22:28:40,340 ERROR ProcessorService - Exception at line 20, column 46 of https://baseqa20151.delphi-tech.com:443/wl1213-test/baseqa20151/oasis2Portal/owsPortal/phs/get-navigation
org.orbeon.oxf.common.ValidationException: line 20, column 46 of https://baseqa20151.delphi-tech.com:443/wl1213-test/baseqa20151/oasis2Portal/owsPortal/phs/get-navigation: Fatal error: The entity name must immediately follow the '&' in the entity reference.
https://baseqa20151.delphi-tech.com:443/wl1213-test/baseqa20151/oasis2Portal/owsPortal/phs/get-navigation, line 20, column 46: Fatal error: The entity name must immediately follow the '&' in the entity reference.
at org.orbeon.oxf.xml.XMLUtils$ErrorHandler.fatalError(XMLUtils.java:306)
at orbeon.apache.xerces.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178)
at orbeon.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:351)
at orbeon.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:281)
at orbeon.apache.xerces.impl.XMLScanner.reportFatalError(XMLScanner.java:1459)
at orbeon.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEntityReference(XMLDocumentFragmentScannerImpl.java:1252)
at orbeon.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1717)
at orbeon.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:324)
at orbeon.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:845)
at orbeon.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:768)
at orbeon.apache.xerces.parsers.XMLParser.parse(XMLParser.java:108)
at orbeon.apache.xerces.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1201)
at org.orbeon.oxf.xml.XMLUtils.inputSourceToSAX(XMLUtils.java:350)
at org.orbeon.oxf.xml.XMLUtils.inputStreamToSAX(XMLUtils.java:335)
at org.orbeon.oxf.processor.URIProcessorOutputImpl.readURLToStateIfNeeded(URIProcessorOutputImpl.java:394)
at org.orbeon.oxf.xforms.processor.XFormsURIResolver.resolve(XFormsURIResolver.java:86)
at org.orbeon.oxf.xforms.processor.XFormsURIResolver.readURLAsDocument(XFormsURIResolver.java:117)
at org.orbeon.oxf.xforms.XFormsModel.performDefaultAction(XFormsModel.java:660)
at org.orbeon.oxf.xforms.XFormsContainingDocument.dispatchEvent(XFormsContainingDocument.java:1283)
at org.orbeon.oxf.xforms.XFormsContainer.initializeModels(XFormsContainer.java:173)
at org.orbeon.oxf.xforms.XFormsContainingDocument.initialize(XFormsContainingDocument.java:1525)
at org.orbeon.oxf.xforms.XFormsContainingDocument.<init>(XFormsContainingDocument.java:181)
at org.orbeon.oxf.xforms.processor.XFormsToXHTML.createCacheContainingDocument(XFormsToXHTML.java:326)
at org.orbeon.oxf.xforms.processor.XFormsToXHTML.access$200(XFormsToXHTML.java:50)
at org.orbeon.oxf.xforms.processor.XFormsToXHTML$2.read(XFormsToXHTML.java:152)
at org.orbeon.oxf.processor.ProcessorImpl.readCacheInputAsObject(ProcessorImpl.java:453)
at org.orbeon.oxf.xforms.processor.XFormsToXHTML.doIt(XFormsToXHTML.java:121)
at org.orbeon.oxf.xforms.processor.XFormsToXHTML.access$000(XFormsToXHTML.java:50)
at org.orbeon.oxf.xforms.processor.XFormsToXHTML$1.readImpl(XFormsToXHTML.java:80)
at org.orbeon.oxf.processor.ProcessorImpl$6.read(ProcessorImpl.java:995)
at org.orbeon.oxf.processor.ProcessorImpl$ProcessorOutputImpl.read(ProcessorImpl.java:1178)
at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:350)
at org.orbeon.oxf.processor.ProcessorImpl.readInputAsSAX(ProcessorImpl.java:355)
at org.orbeon.oxf.processor.xinclude.XIncludeProcessor.access$100(XIncludeProcessor.java:41)
...
and here is the log record from the HTTP access.log file for this request:
10.192.16.82 - baseqa20151x [19/Mar/2015:22:28:40 -0400] "GET /wl1213-test/baseqa20151/oasis2Portal/owsPortal/phs/billing-account-policy-inquiry-admin HTTP/1.1" 500 215530
Thanks for your help.