We are just starting to use google cloud data fusion for our ETL.
We have a use case to hit a server(self-signed) and get some info using the http-plugin in the data fusion product.
In my pipeline, i am turning off SSL verification to accept any certificates but getting an error
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <on000120.aeo.ae.com> doesn't match any of the subject alternative names: []
Help text for the field also suggests to urn it off. See image.
Could someone please help?
I followed the instructions at https://www.freecodecamp.org/news/how-to-get-https-working-on-your-local-development-environment-in-5-minutes-7af615770eec/ to generate the key and crt files, so I had a file called v3.ext (which includes subjectAltName) and used that to generate the server.crt file. When I started my server with these files, I was able to deploy and run a pipeline using HTTP source from that local HTTPS server. How did you generate your crt and key? Would you please describe the plugins you're using?
I assume that issue that you've reported is not connected with Data Fusion runtime environment, but might be supposedly related the target HTTPS web server SSL/TLS configuration. Most of the time this issue occurs when the common name or SAN(Subject Alternative Name) of your SSL/TLS Certificate does not match the domain or address that you hit in HTTP call, hence this page is not listed in SAN's list and I can suppose that you didn't add any of domain names there.
As far as I know Data Fusion http-plugin, which is based on Java HTTP Client implementation, promotes HostnameVerifier class, that is used for URL's hostname and the server's identification hostname verification during establishing handshake between parties, you may take a look at the source code and find inherited method getDefaultHostnameVerifier() as well. Therefore, this can lead to the error, during http client intentions to launch SSL/TLS connection channel.
Even though, you turned of trust keystore certificate inspection in the plugin settings, the issue will persist because hostname validation still failing.
Related
I am quite new to ssl stuffs but I am afraid I can guess the final answer of the following problem/question:
We are building hardware (let's call them servers) that WILL have IP address modifications along there lifetime. Each Server must be reachable in a secured manner. We are planning to use a TLS 1.3 secured connection to perform some actions on the servers (update firmware, change configuration and so on). As a consequence we need to provide the server's with one certificate (each) so that they can state their identity. PKI issue is out of the scope of this question (we suppose) and we can take for granted that the clients and the servers will share a common trusted CA to ensure the SSL handshake goes ok. The server's will serve http connection on there configured (changeable) IP addresses only. There is no DNS involved on the loop.
We are wondering how to set the servers' certificates appropriately.
As IP will change, it cannot be used as the common name in the server's certificate.
Therefore, we are considering using something more persistent such as a serial number or a MAC address.
The problem is, as there is no DNS in the loop, the client can not issue http request to www.serialNumberOfServer.com and must connect to http://x.y.z.t (which will change frequently (at least frequently enough so that we don't issue a new server's certificate at each time))
If we get it right, ssl handshake requires to have the hostname (that's in the URL we are connecting to) matching either the commonName of the server's Certificate or one of its Subject's Alternative Name (SAN). Right? Here, it would be x.y.z.t.
So we think we are stucked in a situation in which the server cannot use it's IP to prove its identity and the client wants to use it exclusively to connect to the server.
Is there any work around?
Are we missing something?
Any help would be very (VERY) appreciated. Do not hesitated in cas you should need more detailed explanation!
For what it's worth, the development environment will be Qt using the QNetworkAccessManager/QSSlstuffs framework.
If you're not having the client use DNS at all, then you do have a problem. The right solution is to use DNS or static hostname lists (/etc/hosts, eg, on unix* or hosts.txt on windows eg.). That will let you set names appropriately.
If you can only use IP addresses, another option is to put all of your IP addresses into the certificate that the server might use. This is only doable if you have a reasonable small number of addresses that they might get assigned to.
Or you could keep a cache of certificates on the server with one address for each, and have part of the webserver start process to select the right certificate. Requires a bit more complex startup.
Edit: Finally, some SSL stacks (e.g. openssl) let you decide whether or not each particular verification error should be accepted as an error or that it can be ignored. This would let you override the errors on the client side. However, this is hard to implement properly and very prone to security issues if you don't bind the remote certificate properly it means you're subjecting yourself to man-in-the-middle or other attacks by blindly accepting any old certificate. I don't remember if Qt's SSL library gives you this level of flexibility or not (I don't believe so but didn't go pull up the documentation).
Went back on the subject 9 mont later!
Turns out there is an easy solution (at least with Qt framework)
Qt's QNetworkRequest::setPeerVerifyName does the job for us. It allows to connect to an host using its IP and verify a given CN during SSL handshake
See Qt's documentation extract below:
void QNetworkRequest::setPeerVerifyName(const QString &peerName)
Sets peerName as host name for the certificate validation, instead of the one used for the TCP connection.
This function was introduced in Qt 5.13.
See also peerVerifyName.
Just tested it positively right now
For my current project i need to implement http over tls at client ,for this i need a local server able to simulate this . is there any online or offline tool. which i can use and and see the handshake in wireshark .
For watching the TLS handshake you might not get far enough with wireshark. For such kind of monitoring you would ensure the security is low enough for wireshark to be able getting the session key from watching the handshake for decoding the later parts. Thus, you need to avoid any forward secrecy.
Otherwise any (https) server that is accessible (and willing to talk) to your client will do.
In case you are in an environment that is supported (e.g. any unix/linux), you might try using openssl. It allows setting up a server that will do handshake. And it will log the handshake such that you will be able looking at what is going on. This will eliminate the need for using wireshark for debugging.
For achieving this you need a Web Server accepting connection over TLS.I have achieved this on Apache Tomcat web Server.
TLS configuration needs to be done in the server.xml file present in config directory of Tomcat webserver. Connector tag needs to be added in the server.xml file that contains information like
TLS version to be used, port, List of supported Cipher Suites, Keystore path and password, Truststore path and password.
Any regular Rest client can be used to make a call like Postman client. But to use that over TLS/SSL certificate is needed to be installed in Chrome browser.
Hope this answers your doubt.
I am using the Let's Encrypt IIS client from https://github.com/Lone-Coder/letsencrypt-win-simple to generate a certificate for a server. Since the certificate is only valid for three months, I want it to auto-renew.
But the server for which I need that auto-renewing certificate is only bound to https:||mysubdomain.mydomain.com:443 and smtp:||mysubdomain.mydomain.com:25.
Both http:||mysubdomain.mydomain.com:80 and ftp:||mysubdomain.mydomain.com:21 point to a different server.
As you may have guessed, the error that is now thrown during the process is "The ACME server was probably unable to reach http:||mysubdomain.mydomain.com:80/.well-known/acme-challenge/abcdefgh...xyz".
It is completely clear to me why, but I can't fix it, because http:||mysubdomain.mydomain.com has to point to the other server. If the ACME server would try https:||mysubdomain.mydomain.com:443/.well-known/acme-challenge/abcdefgh...xyz, but ignore any certificate issue, he would successfully find the challenge.
Is there anything I can do, any feature I have overlooked, that would help me to get automated renewal working?
There are multiple options:
http-01
Redirect http://example.com/.well-known/acme-challenge/* to https://example.com/.well-known/acme-challenge/*, Boulder will happily follow any such redirect and ignore the provided certificate. That's the most simple way if you have access to the other server and can configure that redirect. It's a permanent redirect that you don't have to adjust, it'll be just fine every three months.
The option to use HTTPS directly has been removed due to security issues with some popular server software that uses the first host defined if some other virtual host doesn't define any HTTP host, which might lead to wrong issuances in multi-user environments aka shared hosting.
tls-sni-01
If you want to use just port 443, you can use another challenge type called tls-sni-01. But I think there's no client for Windows available yet that supports that challenge type.
dns-01
If you have control over the DNS via a simple API, you could also use the DNS challenge, it's completely independent of the port you can use.
Is this possible?
My understanding goes as far as this:
A non-proxy aware client will negotiate SSL directly with a listener, without every sending a CONNECT request to identify the destination host. A transparent proxy would need a destination host in order to forge a certificate with the hostname for the client.
I've read that some browser support the "server_name" extension in the client hello message which will identify the destination host and that if the extension is present this can be done. However, I'm unaware which browsers if any support this extension.
I would think that this should be possible but my efforts so far using squid and burp haven't been successful.
I understand that there's no way to obtain the destination host in the initial connection phase but I would think that with the correct configuration it would be possible to allow a forwarded connection in the initial phase then capture the returned certificate in order to read the destination then be able to inject the proxy's own CA-signed certificate at this point with the hostname derived from the legitimate certificate.
I think the best bet to get this working (if it's at all possible) is using squid's bump-server-first method http://wiki.squid-cache.org/Features/BumpSslServerFirst
I'm interested to hear if anyone has ever successfully gotten this to work.
We're having an issue with securing an intranet / internet website with SSL where
we can't know the qualified domain name in advance.
Basically, I'm trying to make a program that will be installed on a webserver
outside my direct control, to be accessable over intra- or internet. In either case
I want it to be secure via SSL (https). To do this, I would like to include and
install a SSL certificate on the target machine. My installer is fully prepackaged
and should not require any particular during- or postinstall intervention from my
end. Problem is, I can't know ahead of time the target machine's name or domain
name, so as far as I can tell the SSL connection will be returning warnings (or
worse?) when accessed, since the certificate I include will (must) have a different
name on it.
I really want to avoid those warnings, but I still want to keep it secure. Is there
any way to install a SSL connection without certificate warnings without the domain
name known ahead of time?
Thanks for any help you folks can give.
What you want to do is not possible. Here's why.
A certificate will include a set of names (Common Name, possibly along with Subject Alternative Names, possibly including wildcard names).
The client's web browser will do the following:
The user wanted to visit "https://myapp.mydomain.com/blog/posts/1".
The request is via SSL and the domain name in the request is "myapp.mydomain.com".
Get the certificate from the Web server.
Ensure that at least one of the names in the certificate is exactly equal to, or wildcard-matches, the domain name in the request.
Display the page.
Therefore, you need a certificate with the exact domain name (or a wildcard matching the exact domain name) by which the application will be used. And the certificate needs to be available at the same time as, or later than, the time when the exact domain name of the website becomes known, and cannot be made available any earlier.
You seem to be under the misapprehension that somehow a certificate can "create" or "install" an SSL connection. That is false. The Web server - Apache, IIS, Nginx, LigHTTPD, or whichever one you happen to use - is the program that knows how to every aspect of SSL connectivity. The certificate is just a file that the Web server sends to the client, without even opening or using in any way.
Additionally, the author of a webapp to be distributed is not responsible for creating or distributing certificates, and should not be under the misapprehension that he is responsible. Only the website maintainer should be responsible for obtaining a certificate for his website. As another person remarked, in your installation process or perhaps in a post-installation process, you may ask the person installing the webapp for a certificate. But that is the best you can do.
The best you can do is to buy a wildcard SSL certificate - but wait, it's not what you think. You still need to know the second-level domain (the TLD being ".com") ahead of time. You can effectively ask for a cert that covers *.foo.com - then any site, a.foo.com, b.foo.com will be covered. Of course, these certs are more expensive that FQDN certs because you are doing the buggers out of some extra coin.
-Oisin
Each of those sites should have their own SSL certificate. Why not prompt the user to provide the cert file during installation?
In most (if not all) cases, the SSL certificate is associated with the webserver (apache, IIS, etc.) and is not part of your application. It's up to the admin of the web server to install the certificate and not you as the author of the program.
If your installation program does have the ability to modify the web server configuration, and you are willing to have it use a self-signed certificate, you can script the creation of the certificate to allow the domain name to be input. However, I sense this is not really available to you. Also, a self-signed certificate will generally cause certificate warnings.
If I understand you correctly there might be a solution to your problem now. This solution won't help you, however, if you have no control over specifying what SSL certificates are served from the web server where your program is installed (as mentioned by someone else). If your program itself contains a web server you won't have this issue.
If you start with a trusted https website, you can make cross-domain TLS (SSL) XmlHttpRequests to the web servers that are running your application. This is made possible using the opensource Forge project. The project uses a TLS implementation written in JavaScript and a small Flash swf to handle the cross-domain requests. Your program will need to serve an XML Flash policy file that grants the trusted website access to the web server running the application.
Your program will also need to generate a self-signed SSL certificate and upload it to the trusted website. From there, each program's certificate can be included as trusted via the JavaScript TLS implementation. Alternatively, you can have your program upload its certificate to be signed by a CA you create, using a common or subject alternative name that is appropriate for your use (it doesn't have to be the domain name). Then you can use JavaScript to trust the CA certificate and look for the correct name on each certificate.
For more details check out the Forge project at github:
http://github.com/digitalbazaar/forge/blob/master/README
The links to the blog posts at the end provide more in-depth information about how it works.