I am installing the datapusher service for CKAN.
CKAN has been configured to use an NGINX reverse proxy that routes client requests, following instructions here. SSL certificate is installed and configured in NGINX.
When trying to use the datapusher app to upload a file, it fails and Apache log gives this error:
Mon Apr 03 13:49:10.979179 2017] [:error] [pid 15468] 2017-04-03 13:49:10,979 CRITI [ckanext.datapusher.plugin] {'status_code': 403, 'message': 'An Error occurred while sending the job: 403 Client Error: Forbidden', 'details': u'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\\n<html><head>\\n<title>403 Forbidden</title>\\n</head><body>\\n<h1>Forbidden</h1>\\n<p>You don\\'t have permission to access /job\\non this server.</p>\\n<hr>\\n<address>Apache/2.4.7 (Ubuntu) Server at 127.0.0.1 Port 8800</address>\\n</body></html>\\n'}
When testing access to the datapusher's 8800 port through openssl this is the output:
[Mon Apr 03 13:49:10.981049 2017] [:error] [pid 15468] [remote 127.0.0.1:6855] Error - <type 'exceptions.TypeError'>: notify() takes exactly 3 arguments (2 given)
open:/etc/ckan> openssl s_client -connect 127.0.0.1:8800
CONNECTED(00000003)
140385459791520:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 275 bytes
The datapusher docs give a workaround for bypassing SSL here, using the SSL_Verify config. I tried this and there was no change.
I think that I need to either:
1. Force the nginx reverse proxy to allow SSL connections through port 8800 (in addition to 443). Or...
2. Configure ckan/datapusher/apache/nginx to bypass SSL/https on port 880.
Any suggestions?
I believe the 403 error is at the point that CKAN sends a request to DataPusher to ask it to load a particular resource. DataPusher is running on Apache only and thus is on HTTP (not HTTPS) so there should be no issue with SSL. Check your CKAN config is the default:
ckan.datapusher.url = http://127.0.0.1:8800/
DataPusher's SSL_VERIFY setting is for a later request - when datapusher makes a request to CKAN at ckan.site_url, which for you will go via nginx over HTTPS. You may need this setting, depending on whether the SSL in your python is compatible. Reading the code it suggests you need quotes and make sure the key is all caps. i.e. in your datapusher_settings.py:
SSL_VERIFY = 'False'
Related
I'm installing OpenVPN Access Server on a Google Cloud instance. Its webUI listens on port 943 using https. It has a self-signed certificate whose name doesn't match the server's hostname (10.150.0.2). I can't start an SSH tunnel. I'm looking for a way to troubleshoot the connection from the IAP service to my server.
The command I'm running is gcloud compute start-iap-tunnel vpn 943 --local-host-port=localhost:943 I receive the normal Testing if tunnel connection works message.
It errs out with ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 943)
If I add --log-http to the command invocation the relevant information follows (it looks like a normal req/resp cycle with a 200 that I assume is from my client to the IAP service):
Testing if tunnel connection works.
=======================
==== request start ====
uri: https://oauth2.googleapis.com/token
method: POST
== headers start ==
b'content-type': b'application/x-www-form-urlencoded'
b'user-agent': b'google-cloud-sdk gcloud/367.0.0 command/gcloud.compute.start-iap-tunnel invocation-id/db27de82264f47fcb63f6680afaa8327 environment/None environment-version/None interactive/False from-script/False python/3.7.9 term/xterm-256color (Macintosh; Intel Mac OS X 21.2.0)'
== headers end ==
== body start ==
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this request.
== body end ==
==== request end ====
---- response start ----
status: 200
-- headers start --
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Encoding: gzip
Content-Type: application/json; charset=utf-8
Date: Fri, 24 Dec 2021 02:11:52 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: scaffolding on HTTPServer2
Transfer-Encoding: chunked
Vary: Origin, X-Origin, Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 0
-- headers end --
-- body start --
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this response.
-- body end --
total round trip time (request+response): 0.246 secs
---- response end ----
----------------------
ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 943)
To my knowledge this is the limit of easily accessible troubleshooting for start-tap-tunnel.
Moving on to the local machine we can connect to 10.150.0.2:943 before puking a la certificate.
root#viongier:/usr/local/openvpn_as# wget https://10.150.0.2:943
--2021-12-24 02:01:47-- https://10.150.0.2:943/
Connecting to 10.150.0.2:943... connected.
ERROR: The certificate of ‘10.150.0.2’ is not trusted.
ERROR: The certificate of ‘10.150.0.2’ doesn't have a known issuer.
The certificate's owner does not match hostname ‘10.150.0.2’
It seems to me that my client happily connects to the IAP service which fails to connect to my server. I would expect to see an IAP error if it was erring out because of the cert. The only thing I can think of to test this is by generating a certificate whose issuer google likes. (LetsEncrypt for example.)
This message means that the backend does not have a socket open in the listening state. Common reasons are that no service has been started or a firewall is blocking the port.
To allow the Identity Aware Proxy into your VPC, allow traffic from 35.235.240.0/20.
ERROR: (gcloud.compute.start-iap-tunnel) While checking if a
connection can be made: Error while connecting [4003: 'failed to
connect to backend']. (Failed to connect to port 943)
This error means that the certificate provided does not match the address that the connection is made to:
ERROR: The certificate of ‘10.150.0.2’ is not trusted. ERROR: The
certificate of ‘10.150.0.2’ doesn't have a known issuer. The
certificate's owner does not match hostname ‘10.150.0.2’
Some clients, such as wget support ignoring SSL certificate validation. For wget see the --no-check-certificate flag.
Once you solve that problem you will run into another set of problems:
Under normal circumstances, you can not use HTTPS with tunnels. Tunnels are a form of man in the middle. There are tricks that can be employed, none of them secure.
Commercial SSL certificates do not support IP addresses only public domain names. You would need to create your own self-signed certificate, which would not be trusted or do not validate the certificate.
The last issue is that HTTPS endpoints require encryption negotiation from the client party. The start-iap-tunnel command does not initiate encryption (TLS negotiation). This command also does not do any form of certificate exchange and that is why you do not see an IAP error about certificates. This command only transfers data between the tunnel endpoints.
In summary, you cannot use HTTPS with TCP / SSH tunnels without deploying tricks and/or disabling features which defeats the purpose of HTTPS.
Allow IAP traffic through the firewall allowed my external client to connect to the internal port 943 via an IAP tunnel.
Allowing port 943 from 35.235.240.0/20 solved my problem.
More information is available at the GCP IAP docs
I'm trying to restrict an url called by Docusign event when a document is completed.
I want to only give access to this url by Docusign host or ip but i'm unable to do so because of my limited skills on Apache.
By following this documentation https://www.docusign.com/trust/security/esignature
I've tried to add this line in my vhost :
<LocationMatch "^/souscription/api/[^/].*/callback/.*$">
Require host docusign.com docusign.net
</LocationMatch>
But I have this error in apache log:
[Wed Jul 29 12:59:09.663648 2020] [authz_host:error] [pid 32671] [client 162.248.186.11:50836] AH01753: access check of 'docusign.com docusign.net' to /souscription/api/1.0/callback/118/completed failed, reason: unable to get the remote host name
What's wrong with my config ?
For Apache questions, use superuser.com
When building a listening server for receiving DocuSign webhook messages, filtering by IP is not recommended since it leads to a brittle installation that can fail at exactly the wrong time. Instead:
Use the combination of the Basic Authentication and HMAC features to assure yourself that the message really came from DocuSign.
Or better, use an intermediate PaaS service to queue the notification messages. The additional feature is that you can receive the notification messages from behind your firewall with no changes to the firewall. See the example repo and associated blog posts.
This works with Tomcat 8.5.50. However, with Tomcat 8.5.51, Apache cannot connect via AJP with the following error:
[Tue Mar 10 20:15:31.378937 2020] [proxy:error] [pid 42:tid 139841308157696] (111)Connection refused: AH00957: AJP: attempt to connect to 172.28.0.5:8009 (tomcatserver) failed
[Tue Mar 10 20:15:31.379336 2020] [proxy_ajp:error] [pid 42:tid 139841308157696] [client 192.168.0.1:58054] AH00896: failed to make connection to backend: tomcatserver
The Apache is on version 2.4.38:
Server version: Apache/2.4.38 (Debian)
Server built: 2019-10-15T19:53:42
The AJP connector in the server.xml has secretRequired="false". Everything is set up via Docker Compose.
The configuration for secretRequired isn't the only thing that changed:
From https://tomcat.apache.org/migration-85.html#Upgrading_8.5.x
In 8.5.51 onwards, the default listen address of the AJP Connector was changed to the loopback address rather than all addresses.
In 8.5.51 onwards, the requiredSecret attribute of the AJP Connector was deprecated and replaced by the secret attribute.
In 8.5.51 onwards, the secretRequired attribute was added to the AJP Connector. If set to true, the default, the AJP Connector will not
start unless a secret has been specified.
In 8.5.51 onwards, the allowedRequestAttributesPattern attribute was added to the AJP Connector. Requests with unrecognised attributes
will now be blocked with a 403.
Reference: AJP connector.
On top of that, the stock server.xml even has the AJPConnector commented, so it won't be active without being explicitly enabled.
Try adding allowedRequestAttributesPattern=".*" to the connector def.
Proceeding from where Olaf left off, follow these steps:
(1) You may omit the address attribute.
(2) Change the secretRequired attribute to secretRequired="true", or equivalently, leave it out. (The default value is True).
(3) Add a secret attribute to the workers.properties file and to the server.xml file. You may choose whatever secret you want, on condition that the values in both files match exactly.
(4) For the time being, add to the AJP connector the attribute allowedRequestAttributesPattern=".*", as T Cervenka suggests.
You should then end up with something like,
workers.properties
worker.list=worker1
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
worker.worker1.secret=F45A93BF-3AA7-4CB4-E49A-DB34573E4A25
server.xml
<Connector port="8009" protocol="AJP/1.3" maxThreads="500" secret="F45A93BF-3AA7-4CB4-E49A-DB34573E4A25" allowedRequestAttributesPattern=".*" />
The value of allowedRequestAttributesPattern must be a regular expression. It represents the request attributes passed from the reverse proxy to the AJP connector. See the Tomcat docs for details. https://tomcat.apache.org/tomcat-8.5-doc/config/ajp.html.
The regex value for allowedRequestAttributesPattern must be an exact match for the request attributes passed in the AJP protocol. Its default value (where you don't mention the attribute) is null: this is known to break requests. If in doubt, use the regex wildcard, ".*", as above.
I have configured Apache2 to use Client Certificate AUthentication using:
SSLVerifyClient require
It works, I can access my site with a valid Client Certificate.
However, when users connect to it without having a ClientCertificate installed, they get a confusing error from the browser.
(Chrome says 'ERR_SSL_PROTOCOL_ERROR', Firefox says 'ssl_error_handshake_failure_alert', Internet explorer only says 'Internet Explorer cannot display the Webpage'.
I want to show the users a custom ErrorDocument when they try to access without a valid Client Certificate.
The problem is that the site doesn't return a http error code, but aborts the request, so I can't use Apaches' 'ErrorDocument'.
last part of ssl_error_log is this:
[Wed Aug 31 11:11:57 2011] [info] [client 192.168.2.156] SSL library error 1 in handshake (server url:443)
[Wed Aug 31 11:11:57 2011] [info] SSL Library Error: 336105671 error:140890C7:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:peer did not return a certificate No CAs known to server for verification?
[Wed Aug 31 11:11:57 2011] [info] [client 192.168.2.156] Connection closed to child 1 with abortive shutdown (server url:443)
How do I return a valid ErrorDocument for this?
SSLVerifyClient optional
RewriteCond %{SSL:SSL_CLIENT_VERIFY} !=SUCCESS
RewriteRule ^/ http://localhost:8080/missing_cert.html [P,L]
I have been trying to follow the guidelines in this Microsoft article to authenticate
against Apache with Kerberos and AD. I have successfully tested the communication between the apache server and the AD server with kinit. However when I attempt to access a restricted page on the server with IE I get an Internal server error and the following appears in the apache error log.
[Wed Sep 24 14:18:15 2008] [debug] src/mod_auth_kerb.c(1483): [client 172.31.37.38] kerb_authenticate_user entered with user (NULL) and auth_type Kerberos
[Wed Sep 24 14:18:15 2008] [debug] src/mod_auth_kerb.c(1174): [client 172.31.37.38] Acquiring creds for HTTP/srvnfssol1.dev.local#DEV.LOCAL
[Wed Sep 24 14:18:15 2008] [error] [client 172.31.37.38] gss_acquire_cred() failed: Miscellaneous failure (see text) (Key table entry not found)
I have run a truss on the apache process and confirmed that it is in fact loading up the keytab file ok. I am wondering if there is something wrong with the format of the keytab file...
HTTP/srvnfssol1.dev.local#DEV.LOCAL
I am not sure what I am missing though. Or what other things to check.
Any suggestions?
Thanks
Peter
Ok. Keytabs are supposed to contain the Service principal name, in this case "HTTP/srvnfssol1.dev.local#DEV.LOCAL" and the encryption key. I see where the MS docs say just to echo that to a file, but I don't think that's right.
You'll need to use the ktpass utility to create the keytab. The MS docs are here.
In particular, you'll need to specify KRB5_NT_SRV_HST, and most of the rest of the options can be default.
Sample of it on my machine:
C:\>ktpass /out test.keytab /princ HTTP/srvnfssol1.dev.local#DEV.LOCAL
/ptype KRB5_NT_SRV_HST /pass *
Type the password for HTTP/srvnfssol1.dev.local:
Key created.
Output keytab to test.keytab:
Keytab version: 0x502
keysize 62 HTTP/srvnfssol1.dev.local#DEV.LOCAL
ptype 3 (KRB5_NT_SRV_HST) vno 1 etype 0x1 (DES-CBC-CRC)
keylength 8 (0xa7f1fb38041c199e)
If the active directory server is the KDC, you'll need to use the /map <name> argument, where <name> is the computer account in active directory representing the server.
Some details on how all this works. When you browse to the website it should respond with a WWW-Authenticate: Negotiate header, and your browser will send a request to the KDC (active directory server) to get a kerberos ticket for the service. The AD server will look up the encryption key for the ticket using the service principal name, and send an encrypted service ticket back to the browser. Once the browser has the service ticket, it'll reissue the HTTP request with an authenticate header containing the ticket. The apache server will look up its key in the keytab, decrypt the ticket, and grant access.
The "key table entry not found" error happens because apache isn't finding itself in the keytab. Can also happen if the name resolution/realms aren't set up right.
You should be able to see all the kerberos requests AP-REQ/AP-REP/TGS-REQ/TGS-REP using wireshark on the client, tcp or udp port 88.