I have a few questions about digital certificates:
1) As far as I understand, clients are able to verify server digital certificates, because clients contain a set of well known CA self-signed certificates on their machine. Is that enough for the whole verification to be complete or are there some additional remote calls that need to be made by the client to the actual CA (wherever it lives)? I read some things about certificate revocation (CRL and OCSP) which seemed to indicate that remote calls are necessary at some point or another.
2) When validating a server certificate what happens exactly on client-side? I've always explained it to myself that based on the CA in the server certificate the client finds the corresponding CA self-signed certificate on it's machine and uses the public key from it to decrypt the signature in the server certificate (so no remote calls here). The decrypted result is as far as I understand a digest of the server's public key. So the client hashes the public key in the server certificate to match it to the digest from the last step. If both match, then everything is ok and the client and server can exchange a symmetric key for further communication. Is that correct?
1) Technically, no, there are no remote calls required. Of course, it's a little more complex than that - if you're using OCSP, then yes, you will need to reach an OCSP server and it must be a remote call. If you use CRLs, you will need to make remote calls to fetch the CRL from the CRL Distribution Point [CDP] in the cert; but if the CRL is fetched and cached locally, and is not expired, then the cert checks are all local.
2) You are correct, the client-side 'chains to a trusted root'; there are more checks performed including key usage, date range checks and, as you noted, verifying the signatures of the certs. The final check is to make sure the root is available at the client.
I hope that helps.
When I open the SSL details of an internal application inside network (Intranet enterprise application) - I see 3 levels of certificates? What does each level mean?
There is no specific definition of each "level".
The HTTPS world works using a PKIX model: each entity (here specifically website and their hostname) is authenticated by some X.509 certificate.
For the trust to happen this server certificate needs to be signed by another certificate, called an authority certificate that you trust.
Browsers are shipped with hundreds of default authority certificates. The system is built under the assumption that you trust the entities behind those certificates fully (yes, without having at any point seen all this list or explicitely given consent; you can remove all of them from your browser's or even OS trust store but then you will not be able anymore to connect anywhere through HTTPS and you will need to re-add them one by one), which means you accept as valid (besides other technical checks like on dates, etc.) any certificate signed by these authority certificates.
But for multiple technical and non technical reasons, authorities often do not sign end certificates directly but uses intermediary certificates (the "middle" level in your case if you want).
It does not change anything from the model above as the trust flows from the root and when you connect to a server it is responding with its own certificate plus any other intermediaries needed to reach back any trusted root.
Note that the depth may vary. Some authorities may issue end certificate directly (so you will have 2 "levels" in the output), some may use one intermediary (frequent case), some may use a cascade of intermediaries (intermediaries through some technical points in their certificate can be restricted to sign only some specific certificates like under specific names and so on), this is all up the certificate authority that issued the given end certificate you are looking at.
And when you deal with "self-signed" certificates, you do not have levels anymore because the certificate is signed by itself which means it is its own certificate authority, so it is alone in the list of certificates.
In the beginning of an SSL query, the client sends a CLIENT_HELLO message.
The server replies with a certificate message that gives the chain of verifications going back to a known trusted agent.
Suppose for efficiency I wanted to store certificates locally for a new protocol. The current design in TLS is always to require getting the certificate. What could happen to a certificate that would require me to know?
I am trying to understand possible attack scenarios. Consider doing online banking, and suppose a certificate has been compromised. In such as case, the bank is not criminal, but they have been hacked and have to issue a new certificate. Is this reasonable?
If you consider that the bank itself is corrupt, then it seems to me there is no point in worrying about the certificate since they have your money and can just steal it. If the entity you are dealing with goes criminal, does the certificate matter?
Under what circumstances can certfiicates be revoked? I am trying to understand why SSL sends the certificate each time -- it seems really wasteful, but there is probably a good reason.
Would it be possible instead to keep all certificates stored on the client, but check a timestamp with a trusted server? It seems like one could at least send less data across the network
TLS Certificate message after `ServerHello is mandatory in mostly cases, so caching won't have any useful effect. See RFC5246
7.4.2. Server Certificate
When this message will be sent:
The server MUST send a Certificate message whenever the agreed- upon key exchange method uses certificates for authentication (this includes all key exchange methods defined in this document except DH_anon). This message will always immediately follow the ServerHello message.
TLS has its own methods to improve performance. When client sends a valid session_id in ClientHello the session can be resumed and the parties must proceed directly to the Finished messages
Also RFC5077 specifies how to resume sessions without server-side state
EDITED - added comments to specific questions
Suppose for efficiency I wanted to store certificates locally for a new protocol. The current design in TLS is always to require getting the certificate. What could happen to a certificate that would require me to know?
"always" is not correct. TLS sends certificates during handshake. Once the shared key is negotiated, the session can be resumed later by client using sessionid (the usual behaviour). Then, the server does not send the certification chain.
The server sends the certification chain. The client must verify that the presented certificate is reliable:
checking a digital signature performed with the private key of the server certificat
the certificate is issued by a trusted CA. Is supposed that client has a trust store with the root certificates of the certification authorities it trust. The client builds the certification chain presented by server until it finds the root certificate in local truststore
You can perfectly skip the sending of certificates from the server in the second step step if client has a copy of the server certificate in a local truststore
I am trying to understand possible attack scenarios. Consider doing online banking, and suppose a certificate has been compromised. In such as case, the bank is not criminal, but they have been hacked and have to issue a new certificate. Is this reasonable?
In this scenario the attacker could make a MITM attack. The certificate must be revoked by CA and client should check revocation. This is out of scope of TLS
If you consider that the bank itself is corrupt, then it seems to me there is no point in worrying about the certificate since they have your money and can just steal it. If the entity you are dealing with goes criminal, does the certificate matter?
Seems in this case the certificate is the least of the problems...
Under what circumstances can certfiicates be revoked?
Each CA stablish its own procedure. There is no a standard but there are "good practices": When certificate data changes (e.g email) or becomes invalid (Representative of a company), after a renewal revoke the older one, when key is compromised or certificate is lost
Would it be possible instead to keep all certificates stored on the client, but check a timestamp with a trusted server?
Yes it is possible as commented above: Verify a digital signature, verify revocation and stablish a refreshing mechanism
But if you're looking for performance comparing with TLS, the session resumption will probably have better results
Currently I am having a problem connecting to the server due to the following issue:
When I tried to connect to the server, it returned an error: MQRC_SSL_INITIALIZATION_ERROR
Upon closer analysis via WireShark, I found that the Client is attempting to connect to the server using SSL v2, while the server can only accept SSL V3, thus rejecting the connection.
I checked through the document, but am not able to find any information on
what SSL version the .Net client supports.
I would like to check whether the SSL version is controlled from the .Net MQ
client, and if so, how can we configure to make it connect via SSL v3?
Thanks.
I'm not sure I agree with your conclusion since WMQ has supported SSL V3.0 and TLS V1.0 since at least V6.0 and possibly earlier. This is more likely a mismatch of configurations between the client and server. The procedure I recommend to resolve SSL/TLS issues is as follows:
My method for debugging SSL connections on WMQ is to progress through the following sequence making sure each step works before advancing to the next:
Get the channel running without SSL. This validates that the channel names are spelled correctly, that a network route exists between the endpoints, that the QMgr's listener is running and that the client points to the right port. You'd be surprised how many times someone mis-keys a port or channel name.
Get the channel running with the SVRCONN definition set to SSLCAUTH(OPTIONAL). This performs an anonymous SSL connection similar to what your browser does. The QMgr presents a certificate to the client but the client is not obligated to send one back. This validates that the QMgr can find its certificate and that the client can find its trust store and properly validates the QMgr's cert. (Note: the QMgr will always request the client cert and the client will always send it if one is present. To perform this test, use a copy of the client's keystore that has the signer cert(s) but not the application's personal cert. Copy the keystore and delete the personal cert from the copy. Do NOT delete the original!)
Set the SVRCONN channel to SSLCAUTH(REQUIRED). This now requires the client to find its keystore (in the last step it required only its trust store) and to be able to find its certificate. It also requires the QMgr to be able to validate the client's cert.
Set up SSLPEER or CHLAUTH mapping rules to narrow the population of validated certificates that will be accepted on the channel.
The difference between steps #2 and #3 helps to isolate the problem by testing the SSL credential exchange in only one direction at a time. This allows you to identify whether the problem exists in the personal cert or the public cert and on which side of the channel. Nearly all problems are sorted out in these two steps.
UPDATE
Notes to respond to questions. There are two types of certificate used with SSL/TLS. The personal certificate contains the private key and is the one that doesn't get passed around. The public certificate is the one that contains the public key and can be given out freely. The private key is held in a keystore. The public keys (usually these are the CA's root and intermediate certs) are stored in a trust store. In some cases, these are separate files. For example, in Java and JMS the JSSE provider looks in the environment for variables that point to the keystore and to the trust store. It is possible in Java and JMS that the keystore and trust store variables point to the same file.
In the case of WebSphere MQ servers and clients other than Java, the keystore and trust store are combined into a single location. Often referred to as a kdb file, it is actually a CMS key database comprised of several files of which one is the KDB. In this case "keystore" is actually shorthand for a combined keystore and trust store. For the .Net client, set the keystore location and other SSL properties in the MQEnviornment.
In the SSL/TLS handshake, the server always sends its public certificate in response to a connections request. The client then must validate that certificate by first checking the signature and validity date, then looking in its trust store for the thing that signed the certificate. If the thing that signed the certificate is an intermediate signer cert (it has itself been signed by something) then the search continues up the signer cert chain until the root cert is reached. Assuming that the server is authenticated, the same procedure is applied in reverse by having the client present a cert and the server validating it.
When the process fails in Step #2 we can debug using knowledge of the process above. The QMgr must first find its cert in its keystore and present it to the client. If the QMgr cannot find its cert, the result is errors in the AMQERR01.LOG file stating this. Always look on the QMgr side first when things die in Step #2!
If the QMgr does find its cert then the next step is that client must be able to find its trust store and then within that trust store must find the necessary signer cert chain. If this fails, there should be errors on the client side to indicate that. For example, a common error when setting the client environment is to specify the entire file name, including the .kdb extension. When this happens the QMgr looks for [keystorename].kdb.kdb which doesn't exist. Another common error is that the personal certificate exists in the keystore but with the wrong label. Non-Java WMQ clients look for the certificate by label name constructed from the literal string ibmwebspheremq followed by the user ID in lower case. For example, if my user ID is TRob then my certificate label would be ibmwebspheremqtrob. Note that this is the certificate's label in the keystore and NOT the certificates Common Name or other field in the Distinguished Name.
Depending on the type of client in use, these may be in the Windows error log, local MQ error logs or other location. If you can't find client-side errors, WMQ tracing is also an option.
How does SSL work?
Where is the certificate installed on the client (or browser?) and the server (or web server?)?
How does the trust/encryption/authentication process start when you enter the URL into the browser and get the page from the server?
How does the HTTPS protocol recognize the certificate? Why can't HTTP work with certificates when it is the certificates which do all the trust/encryption/authentication work?
Note: I wrote my original answer very hastily, but since then, this has turned into a fairly popular question/answer, so I have expanded it a bit and made it more precise.
TLS Capabilities
"SSL" is the name that is most often used to refer to this protocol, but SSL specifically refers to the proprietary protocol designed by Netscape in the mid 90's. "TLS" is an IETF standard that is based on SSL, so I will use TLS in my answer. These days, the odds are that nearly all of your secure connections on the web are really using TLS, not SSL.
TLS has several capabilities:
Encrypt your application layer data. (In your case, the application layer protocol is HTTP.)
Authenticate the server to the client.
Authenticate the client to the server.
#1 and #2 are very common. #3 is less common. You seem to be focusing on #2, so I'll explain that part.
Authentication
A server authenticates itself to a client using a certificate. A certificate is a blob of data[1] that contains information about a website:
Domain name
Public key
The company that owns it
When it was issued
When it expires
Who issued it
Etc.
You can achieve confidentiality (#1 above) by using the public key included in the certificate to encrypt messages that can only be decrypted by the corresponding private key, which should be stored safely on that server.[2] Let's call this key pair KP1, so that we won't get confused later on. You can also verify that the domain name on the certificate matches the site you're visiting (#2 above).
But what if an adversary could modify packets sent to and from the server, and what if that adversary modified the certificate you were presented with and inserted their own public key or changed any other important details? If that happened, the adversary could intercept and modify any messages that you thought were securely encrypted.
To prevent this very attack, the certificate is cryptographically signed by somebody else's private key in such a way that the signature can be verified by anybody who has the corresponding public key. Let's call this key pair KP2, to make it clear that these are not the same keys that the server is using.
Certificate Authorities
So who created KP2? Who signed the certificate?
Oversimplifying a bit, a certificate authority creates KP2, and they sell the service of using their private key to sign certificates for other organizations. For example, I create a certificate and I pay a company like Verisign to sign it with their private key.[3] Since nobody but Verisign has access to this private key, none of us can forge this signature.
And how would I personally get ahold of the public key in KP2 in order to verify that signature?
Well we've already seen that a certificate can hold a public key — and computer scientists love recursion — so why not put the KP2 public key into a certificate and distribute it that way? This sounds a little crazy at first, but in fact that's exactly how it works. Continuing with the Verisign example, Verisign produces a certificate that includes information about who they are, what types of things they are allowed to sign (other certificates), and their public key.
Now if I have a copy of that Verisign certificate, I can use that to validate the signature on the server certificate for the website I want to visit. Easy, right?!
Well, not so fast. I had to get the Verisign certificate from somewhere. What if somebody spoofs the Verisign certificate and puts their own public key in there? Then they can forge the signature on the server's certificate, and we're right back where we started: a man-in-the-middle attack.
Certificate Chains
Continuing to think recursively, we could of course introduce a third certificate and a third key pair (KP3) and use that to sign the Verisign certifcate. We call this a certificate chain: each certificate in the chain is used to verify the next certificate. Hopefully you can already see that this recursive approach is just turtles/certificates all the way down. Where does it stop?
Since we can't create an infinite number of certificates, the certificate chain obviously has to stop somewhere, and that's done by including a certificate in the chain that is self-signed.
I'll pause for a moment while you pick up the pieces of brain matter from your head exploding. Self-signed?!
Yes, at the end of the certificate chain (a.k.a. the "root"), there will be a certificate that uses it's own keypair to sign itself. This eliminates the infinite recursion problem, but it doesn't fix the authentication problem. Anybody can create a self-signed certificate that says anything on it, just like I can create a fake Princeton diploma that says I triple majored in politics, theoretical physics, and applied butt-kicking and then sign my own name at the bottom.
The [somewhat unexciting] solution to this problem is just to pick some set of self-signed certificates that you explicitly trust. For example, I might say, "I trust this Verisign self-signed certificate."
With that explicit trust in place, now I can validate the entire certificate chain. No matter how many certificates there are in the chain, I can validate each signature all the way down to the root. When I get to the root, I can check whether that root certificate is one that I explicitly trust. If so, then I can trust the entire chain.
Conferred Trust
Authentication in TLS uses a system of conferred trust. If I want to hire an auto mechanic, I may not trust any random mechanic that I find. But maybe my friend vouches for a particular mechanic. Since I trust my friend, then I can trust that mechanic.
When you buy a computer or download a browser, it comes with a few hundred root certificates that it explicitly trusts.[4] The companies that own and operate those certificates can confer that trust to other organizations by signing their certificates.
This is far from a perfect system. Some times a CA may issue a certificate erroneously. In those cases, the certificate may need to be revoked. Revocation is tricky since the issued certificate will always be cryptographically correct; an out-of-band protocol is necessary to find out which previously valid certificates have been revoked. In practice, some of these protocols aren't very secure, and many browsers don't check them anyway.
Sometimes an entire CA is compromised. For example, if you were to break into Verisign and steal their root signing key, then you could spoof any certificate in the world. Notice that this doesn't just affect Verisign customers: even if my certificate is signed by Thawte (a competitor to Verisign), that doesn't matter. My certificate can still be forged using the compromised signing key from Verisign.
This isn't just theoretical. It has happened in the wild. DigiNotar was famously hacked and subsequently went bankrupt. Comodo was also hacked, but inexplicably they remain in business to this day.
Even when CAs aren't directly compromised, there are other threats in this system. For example, a government use legal coercion to compel a CA to sign a forged certificate. Your employer may install their own CA certificate on your employee computer. In these various cases, traffic that you expect to be "secure" is actually completely visible/modifiable to the organization that controls that certificate.
Some replacements have been suggested, including Convergence, TACK, and DANE.
Endnotes
[1] TLS certificate data is formatted according to the X.509 standard. X.509 is based on ASN.1 ("Abstract Syntax Notation #1"), which means that it is not a binary data format. Therefore, X.509 must be encoded to a binary format. DER and PEM are the two most common encodings that I know of.
[2] In practice, the protocol actually switches over to a symmetric cipher, but that's a detail that's not relevant to your question.
[3] Presumable, the CA actually validates who you are before signing your certificate. If they didn't do that, then I could just create a certificate for google.com and ask a CA to sign it. With that certificiate, I could man-in-the-middle any "secure" connection to google.com. Therefore, the validation step is a very important factor in the operation of a CA. Unfortunately, it's not very clear how rigorous this validation process is at the hundreds of CAs around the world.
[4] See Mozilla's list of trusted CAs.
HTTPS is combination of HTTP and SSL(Secure Socket Layer) to provide encrypted communication between client (browser) and web server (application is hosted here).
Why is it needed?
HTTPS encrypts data that is transmitted from browser to server over the network. So, no one can sniff the data during transmission.
How HTTPS connection is established between browser and web server?
Browser tries to connect to the https://payment.com.
payment.com server sends a certificate to the browser. This certificate includes payment.com server's public key, and some evidence that this public key actually belongs to payment.com.
Browser verifies the certificate to confirm that it has the proper public key for payment.com.
Browser chooses a random new symmetric key K to use for its connection to payment.com server. It encrypts K under payment.com public key.
payment.com decrypts K using its private key. Now both browser and the payment server know K, but no one else does.
Anytime browser wants to send something to payment.com, it encrypts it under K; the payment.com server decrypts it upon receipt. Anytime the payment.com server wants to send something to your browser, it encrypts it under K.
This flow can be represented by the following diagram:
I have written a small blog post which discusses the process briefly. Please feel free to take a look.
SSL Handshake
A small snippet from the same is as follows:
"Client makes a request to the server over HTTPS. Server sends a copy of its SSL certificate + public key. After verifying the identity of the server with its local trusted CA store, client generates a secret session key, encrypts it using the server's public key and sends it. Server decrypts the secret session key using its private key and sends an acknowledgment to the client. Secure channel established."
Mehaase has explained it in details already. I will add my 2 cents to this series. I have many blogposts revolving around SSL handshake and certificates. While most of this revolves around IIS web server, the post is still relevant to SSL/TLS handshake in general. Here are few for your reference:
SSL Handshake and IIS
Client certificate Authentication in SSL Handshake
Do not treat CERTIFICATES & SSL as one topic. Treat them as 2 different topics and then try to see who they work in conjunction. This will help you answer the question.
Establishing trust between communicating parties via Certificate Store
SSL/TLS communication works solely on the basis of trust. Every computer (client/server) on the internet has a list of Root CA's and Intermediate CA's that it maintains. These are periodically updated. During SSL handshake this is used as a reference to establish trust. For exampe, during SSL handshake, when the client provides a certificate to the server. The server will try to cehck whether the CA who issued the cert is present in its list of CA's . When it cannot do this, it declares that it was unable to do the certificate chain verification. (This is a part of the answer. It also looks at AIA for this.) The client also does a similar verification for the server certificate which it receives in Server Hello.
On Windows, you can see the certificate stores for client & Server via PowerShell. Execute the below from a PowerShell console.
PS Cert:> ls Location : CurrentUser StoreNames : {TrustedPublisher, ClientAuthIssuer, Root, UserDS...}
Location : LocalMachine StoreNames : {TrustedPublisher,
ClientAuthIssuer, Remote Desktop, Root...}
Browsers like Firefox and Opera don't rely on underlying OS for certificate management. They maintain their own separate certificate stores.
The SSL handshake uses both Symmetric & Public Key Cryptography. Server Authentication happens by default. Client Authentication is optional and depends if the Server endpoint is configured to authenticate the client or not. Refer my blog post as I have explained this in detail.
Finally for this question
How does the HTTPS protocol recognize the certificate? Why can't HTTP work with certificates when it is the certificates which do all the trust/encryption/authentication work?
Certificates is simply a file whose format is defined by X.509 standard. It is a electronic document which proves the identity of a communicating party.
HTTPS = HTTP + SSL is a protocol which defines the guidelines as to how 2 parties should communicate with each other.
MORE INFORMATION
In order to understand certificates you will have to understand what certificates are and also read about Certificate Management. These is important.
Once this is understood, then proceed with TLS/SSL handshake. You may refer the RFC's for this. But they are skeleton which define the guidelines. There are several blogposts including mine which explain this in detail.
If the above activity is done, then you will have a fair understanding of Certificates and SSL.