Recently, I have stumbled upon the basic understanding about PKI work-in-action process. I have looked at major articles about the principles but still I feel quite dumb to understand this process. I understand that PKI is not for „My blog„ but for the sake of simplicity lets walk through simple example “My e-Shop” (for example apache and php) and simple concepts. I wrote some statements that might be vague or even wrong but that’s what I want to know about the process of PKI:
“My e-Shop” as a company needs to be “certified” at some third party CA. That means I need to buy some kind of 1 year membership at that CA, and then, they will register “My e-Shop” at their systems and issue me some stuff like a certificate and a pair of unique public and private keys. Will I get some certificate-file?
Issued certificate is filled with my info and public key and stored in some file at my webserver. The certificate verifies that “My e-shop” is not a bureau of thieves.
Every time users access “My e-shop” through “https” their browsers “silently” checks the presented certificate of “My e-shop” with the one that is registered at CA.
What about users? Do their browsers generate local public+private keys when entering “My e-shop” via https?
When some user enters “My e-shop” via https the following happens: “My e-shop” (webserver) gets the public key (PK1) of the user. The server silently presents the certificate of “My e-shop” to the user, so the users gets the public key (PK2) of “My e-shop”. After some silent checking the browser of the user validates presented certiticate and a secure pipe is established.
When a user sends a request via secure pipe the requests is encrypted with public key of “My e-shop”. Then, web server decrypts the request with its private key. Then, web server sends encrypted response with a public key of the user. At the end, the browser of the user decrypts the response with his private key.
Taking these question by question:
(1) “My e-Shop” as a company needs to be
“certified” at some third party CA.
That means I need to buy some kind of
1 year membership at that CA, and
then, they will register “My e-Shop”
at their systems and issue me some
stuff like a certificate and a pair
of unique public and private keys.
Will I get some certificate-file?
Yes. Also, this varies from CA to CA - some CAs, a 1 year membership and a valid credit card is enough (not to hard to get, even if you happen to be a band of theives), some CAs require a certain amount of paperwork to verify that you are who you say you are. A CA is only as good as it's customer-verification process, so generally the more onerous the process, the better (and more expensive) the CA.
When you have paid your money and done your paperwork, then you and the CA will generate at least 2 things:
A public/private key pair. Often this is generated by YOU, not by the CA. The quality of how you have generated and stored your key pair is also a factor in determining how much another entity can trust your website. Generally, for identification certificates (like SSL server certs) it's a good idea to have the customer (My e-Shop) generate the key.
You'll send a certificate request in a standard format (example: PKCS10) to the CA. It will include a bunch of information about My e-Shop and the public key.
The CA company will check your paperwork and make sure you are you. Then it will use it's CA to digitally sign a certificate for you. That certificate has a bunch of the information you just provided, some extra info that the CA provider sets for you, your public key and the digital signature generated by cryptographically binding all the aforementioned data with the CA's private key.
You get back one of two things:
just the certificate (if you generated your own key pair)
the certificate and the key pair - if the CA generated the key pair for you
(2) Issued certificate is filled with my
info and public key and stored in
some file at my webserver. The
certificate verifies that “My
e-shop” is not a bureau of thieves.
yes... sort of. Your webserver will need both the signed certificate AND the key pair. The key pair is used as part of the SSL handshake to the end user, and that exchange proves that you are in posession of the key pair, and not some band of theives that got their hands on the certificate. This is why you want to protect the key pair.
(3) Every time users access “My e-shop”
through “https” their browsers
“silently” checks the presented
certificate of “My e-shop” with the
one that is registered at CA.
For some definition of "silent". A baseline browser checks that the certificate:
- is currently valid
- is signed by a CA that the browser trusts (or it either denies access or gives you an annoying popup)
Browser add ons will go farther and do things like full path checking (all the way to the self signed root) and certificate status checking. The higher the risk of the info being shared over the pipe, the more checking needs to happen.
(4) When some user enters “My e-shop”
via https the following happens: “My
e-shop” (webserver) gets the public
key (PK1) of the user. The server
silently presents the certificate of
“My e-shop” to the user, so the
users gets the public key (PK2) of
“My e-shop”. After some silent
checking the browser of the user
validates presented certiticate and
a secure pipe is established.
What about users? Do their browsers generate local public+private keys when entering “My e-shop” via https?
OK - stuff is going off the rails here.
For details, check the SSL protocol - the handshake section will give you the hard facts.
The browser and the webserver do pass back and forth information that lets them create a session encryption key. The browser does generate some starting material used to set up the session, but it is not "the public key of the user (PK1)". The user does not need to have PKI credentials UNLESS the server has been configured for client authentication. In a regular, run of the mill My e-Shop, the server has a PKI credential, the browser is generating some key material on the fly for just that one session. The server has absolutely no idea who the browser user is, SSL-wise - all of the user identification comes from a password or something else at the application level.
NOTE: This is for a regular commercial transaction. The higher the risk, the greater the protection. High risk transactions frequently require client authentication.
(5) When a user sends a request via
secure pipe the requests is
encrypted with public key of “My
e-shop”. Then, web server decrypts
the request with its private key.
Then, web server sends encrypted
response with a public key of the
user. At the end, the browser of the
user decrypts the response with his
private key.
Not quite. This is the weeds - a developer of a web application is relatively abstracted from all of this. But - the session content is not encrypted with asymmetric keys - that would be excruciatingly CPU intensive. The PKI of the server and the on-the-fly browser generated handshake data are used to exchange the session encryption key. Part of this is that the client selects and sends a secret key encrypted in the server's public key. Since only the server has the private key, only the server can decrypt the data sent by the client.
This secret key is symmetric (exactly which algorithm is also part of the handshake) and is used for the remainder of the session.
Your steps are incorrect on a couple of different levels. Let me restate them, hopefully in a clear manner.
Your site needs to have an SSL certificate from a trusted third-party CA (Verisign, etc.). You will need to purchase it from them. You will generate the public/private key pair, provide them with your public key (it is part of the certificate request) and they will generate the certificate.
The certificate contains the domain name of your site, your public key, and some other additional information. It does not verify anything, particularly not that your site isn't "a bureau of thieves". All it typically certifies is that you own the domain name for which you are requesting a certificate. It does not mean that your site is trustworthy by any means.
When a user accesses your site via HTTPS, your server will send the certificate down to the client's browser. At no time does the client communicate with the CA to validate your certificate.
3.1. Typically, end users (clients, customers) would not need their own certificates and private keys. This is called mutually-authenticated SSL, and involves both parties (your customers and your site) having a certificate. This is rarely (if ever) used in e-commerce sites.
When a user accesses your site via HTTPS, your server sends down the certificate to the client browser. The client will use this to perform an SSL Handshake. This handshake between the browser and your site is what establishes the secure tunnel. The specifics of the handshake protocol are more technical than you need.
When a request is sent to the server over the SSL tunnel, the server handles it the same way it would any other request. The only difference is that the communication between your customers and your server is encrypted. Your webserver will likely handle the decryption of the data from the request and present it to your webserver for processing. Likewise, the response data is returned back to the client in an encrypted form via the SSL Tunnel.
Better?
Regarding point 3:
As far as I know this is not completely correct. Since there is not really a kind of established live testing infrastructure (in comparison to services like DNS), the web server will send the complete chain of certificates to the browser.
On the other hand the browser or operating system is delivered with a set of trusted root certificates (which need to be updated sometimes, windows update, new browser releases, etc.)
And yes the browser will be supposed to generate an "untrusted" key pair, since under normal circumstances a standard user will not generate his or her own certificate.
Regarding point 5:
Also this is not completely the truth. Computation of asynchronous encryption requires a great amount of time. So the RSA algorithm is just used to exchange keys for faster synchronous encryption standards (i.e. AES, 3TES).
Related
What is the series of steps needed to securely verify a ssl certificate? My (very limited) understanding is that when you visit an https site, the server sends a certificate to the client (the browser) and the browser gets the certificate's issuer information from that certificate, then uses that to contact the issuerer, and somehow compares certificates for validity.
How exactly is this done?
What about the process makes it immune to man-in-the-middle attacks?
What prevents some random person from setting up their own verification service to use in man-in-the-middle attacks, so everything "looks" secure?
Here is a very simplified explanation:
Your web browser downloads the web server's certificate, which contains the public key of the web server. This certificate is signed with the private key of a trusted certificate authority.
Your web browser comes installed with the public keys of all of the major certificate authorities. It uses this public key to verify that the web server's certificate was indeed signed by the trusted certificate authority.
The certificate contains the domain name and/or ip address of the web server. Your web browser confirms with the certificate authority that the address listed in the certificate is the one to which it has an open connection.
Your web browser generates a shared symmetric key which will be used to encrypt the HTTP traffic on this connection; this is much more efficient than using public/private key encryption for everything. Your browser encrypts the symmetric key with the public key of the web server then sends it back, thus ensuring that only the web server can decrypt it, since only the web server has its private key.
Note that the certificate authority (CA) is essential to preventing man-in-the-middle attacks. However, even an unsigned certificate will prevent someone from passively listening in on your encrypted traffic, since they have no way to gain access to your shared symmetric key.
You said that
the browser gets the certificate's issuer information from that
certificate, then uses that to contact the issuerer, and somehow
compares certificates for validity.
The client doesn't have to check with the issuer because two things :
all browsers have a pre-installed list of all major CAs public keys
the certificate is signed, and that signature itself is enough proof that the certificate is valid because the client can make sure, on his own, and without contacting the issuer's server, that that certificate is authentic. That's the beauty of asymmetric encryption.
Notice that 2. can't be done without 1.
This is better explained in this big diagram I made some time ago
(skip to "what's a signature ?" at the bottom)
It's worth noting that in addition to purchasing a certificate (as mentioned above), you can also create your own for free; this is referred to as a "self-signed certificate". The difference between a self-signed certificate and one that's purchased is simple: the purchased one has been signed by a Certificate Authority that your browser already knows about. In other words, your browser can easily validate the authenticity of a purchased certificate.
Unfortunately this has led to a common misconception that self-signed certificates are inherently less secure than those sold by commercial CA's like GoDaddy and Verisign, and that you have to live with browser warnings/exceptions if you use them; this is incorrect.
If you securely distribute a self-signed certificate (or CA cert, as bobince suggested) and install it in the browsers that will use your site, it's just as secure as one that's purchased and is not vulnerable to man-in-the-middle attacks and cert forgery. Obviously this means that it's only feasible if only a few people need secure access to your site (e.g., internal apps, personal blogs, etc.).
I KNOW THE BELOW IS LONG, BUT IT IS DETAILED, YET SIMPLIFIED ENOUGH. READ CAREFULLY AND I GUARANTEE YOU'LL START FINDING THIS TOPIC IS NOT ALL THAT COMPLICATED.
First of all, anyone can create 2 keys. One to encrypt data, and another to decrypt data. The former can be a private key, and the latter a public key, AND VICERZA.
Second of all, in simplest terms, a Certificate Authority (CA) offers the service of creating a certificate for you. How? They use certain values (the CA's issuer name, your server's public key, company name, domain, etc.) and they use their SUPER DUPER ULTRA SECURE SECRET private key and encrypt this data. The result of this encrypted data is a SIGNATURE.
So now the CA gives you back a certificate. The certificate is basically a file containing the values previously mentioned (CA's issuer name, company name, domain, your server's public key, etc.), INCLUDING the signature (i.e. an encrypted version of the latter values).
Now, with all that being said, here is a REALLY IMPORTANT part to remember: your device/OS (Windows, Android, etc.) pretty much keeps a list of all major/trusted CA's and their PUBLIC KEYS (if you're thinking that these public keys are used to decrypt the signatures inside the certificates, YOU ARE CORRECT!).
Ok, if you read the above, this sequential example will be a breeze now:
Example-Company asks Example-CA to create for them a certificate.
Example-CA uses their super private key to sign this certificate and gives Example-Company the certificate.
Tomorrow, internet-user-Bob uses Chrome/Firefox/etc. to browse to https://example-company.com. Most, if not all, browsers nowadays will expect a certificate back from the server.
The browser gets the certificate from example-company.com. The certificate says it's been issued by Example-CA. It just so happens to be that Bob's OS already has Example-CA in its list of trusted CA's, so the browser gets Example-CA's public key. Remember: this is all happening in Bob's computer/mobile/etc., not over the wire.
So now the browser decrypts the signature in the certificate. FINALLY, the browser compares the decrypted values with the contents of the certificate itself. IF THE CONTENTS MATCH, THAT MEANS THE SIGNATURE IS VALID!
Why? Think about it, only this public key can decrypt the signature in such a way that the contents look like they did before the private key encrypted them.
How about man in the middle attacks?
This is one of the main reasons (if not the main reason) why the above standard was created.
Let's say hacker-Jane intercepts internet-user-Bob's request, and replies with her own certificate. However, hacker-Jane is still careful enough to state in the certificate that the issuer was Example-CA. Lastly, hacker-Jane remembers that she has to include a signature on the certificate. But what key does Jane use to sign (i.e. create an encrypted value of the certificate main contents) the certificate?????
So even if hacker-Jane signed the certificate with her own key, you see what's gonna happen next. The browser is gonna say: "ok, this certificate is issued by Example-CA, let's decrypt the signature with Example-CA's public key". After decryption, the browser notices that the certificate contents don't match at all. Hence, the browser gives a very clear warning to the user, and it says it doesn't trust the connection.
The client has a pre-seeded store of SSL certificate authorities' public keys. There must be a chain of trust from the certificate for the server up through intermediate authorities up to one of the so-called "root" certificates in order for the server to be trusted.
You can examine and/or alter the list of trusted authorities. Often you do this to add a certificate for a local authority that you know you trust - like the company you work for or the school you attend or what not.
The pre-seeded list can vary depending on which client you use. The big SSL certificate vendors insure that their root certs are in all the major browsers ($$$).
Monkey-in-the-middle attacks are "impossible" unless the attacker has the private key of a trusted root certificate. Since the corresponding certificates are widely deployed, the exposure of such a private key would have serious implications for the security of eCommerce generally. Because of that, those private keys are very, very closely guarded.
if you're more technically minded, this site is probably what you want: http://www.zytrax.com/tech/survival/ssl.html
warning: the rabbit hole goes deep :).
What is the series of steps needed to securely verify a ssl certificate? My (very limited) understanding is that when you visit an https site, the server sends a certificate to the client (the browser) and the browser gets the certificate's issuer information from that certificate, then uses that to contact the issuerer, and somehow compares certificates for validity.
How exactly is this done?
What about the process makes it immune to man-in-the-middle attacks?
What prevents some random person from setting up their own verification service to use in man-in-the-middle attacks, so everything "looks" secure?
Here is a very simplified explanation:
Your web browser downloads the web server's certificate, which contains the public key of the web server. This certificate is signed with the private key of a trusted certificate authority.
Your web browser comes installed with the public keys of all of the major certificate authorities. It uses this public key to verify that the web server's certificate was indeed signed by the trusted certificate authority.
The certificate contains the domain name and/or ip address of the web server. Your web browser confirms with the certificate authority that the address listed in the certificate is the one to which it has an open connection.
Your web browser generates a shared symmetric key which will be used to encrypt the HTTP traffic on this connection; this is much more efficient than using public/private key encryption for everything. Your browser encrypts the symmetric key with the public key of the web server then sends it back, thus ensuring that only the web server can decrypt it, since only the web server has its private key.
Note that the certificate authority (CA) is essential to preventing man-in-the-middle attacks. However, even an unsigned certificate will prevent someone from passively listening in on your encrypted traffic, since they have no way to gain access to your shared symmetric key.
You said that
the browser gets the certificate's issuer information from that
certificate, then uses that to contact the issuerer, and somehow
compares certificates for validity.
The client doesn't have to check with the issuer because two things :
all browsers have a pre-installed list of all major CAs public keys
the certificate is signed, and that signature itself is enough proof that the certificate is valid because the client can make sure, on his own, and without contacting the issuer's server, that that certificate is authentic. That's the beauty of asymmetric encryption.
Notice that 2. can't be done without 1.
This is better explained in this big diagram I made some time ago
(skip to "what's a signature ?" at the bottom)
It's worth noting that in addition to purchasing a certificate (as mentioned above), you can also create your own for free; this is referred to as a "self-signed certificate". The difference between a self-signed certificate and one that's purchased is simple: the purchased one has been signed by a Certificate Authority that your browser already knows about. In other words, your browser can easily validate the authenticity of a purchased certificate.
Unfortunately this has led to a common misconception that self-signed certificates are inherently less secure than those sold by commercial CA's like GoDaddy and Verisign, and that you have to live with browser warnings/exceptions if you use them; this is incorrect.
If you securely distribute a self-signed certificate (or CA cert, as bobince suggested) and install it in the browsers that will use your site, it's just as secure as one that's purchased and is not vulnerable to man-in-the-middle attacks and cert forgery. Obviously this means that it's only feasible if only a few people need secure access to your site (e.g., internal apps, personal blogs, etc.).
I KNOW THE BELOW IS LONG, BUT IT IS DETAILED, YET SIMPLIFIED ENOUGH. READ CAREFULLY AND I GUARANTEE YOU'LL START FINDING THIS TOPIC IS NOT ALL THAT COMPLICATED.
First of all, anyone can create 2 keys. One to encrypt data, and another to decrypt data. The former can be a private key, and the latter a public key, AND VICERZA.
Second of all, in simplest terms, a Certificate Authority (CA) offers the service of creating a certificate for you. How? They use certain values (the CA's issuer name, your server's public key, company name, domain, etc.) and they use their SUPER DUPER ULTRA SECURE SECRET private key and encrypt this data. The result of this encrypted data is a SIGNATURE.
So now the CA gives you back a certificate. The certificate is basically a file containing the values previously mentioned (CA's issuer name, company name, domain, your server's public key, etc.), INCLUDING the signature (i.e. an encrypted version of the latter values).
Now, with all that being said, here is a REALLY IMPORTANT part to remember: your device/OS (Windows, Android, etc.) pretty much keeps a list of all major/trusted CA's and their PUBLIC KEYS (if you're thinking that these public keys are used to decrypt the signatures inside the certificates, YOU ARE CORRECT!).
Ok, if you read the above, this sequential example will be a breeze now:
Example-Company asks Example-CA to create for them a certificate.
Example-CA uses their super private key to sign this certificate and gives Example-Company the certificate.
Tomorrow, internet-user-Bob uses Chrome/Firefox/etc. to browse to https://example-company.com. Most, if not all, browsers nowadays will expect a certificate back from the server.
The browser gets the certificate from example-company.com. The certificate says it's been issued by Example-CA. It just so happens to be that Bob's OS already has Example-CA in its list of trusted CA's, so the browser gets Example-CA's public key. Remember: this is all happening in Bob's computer/mobile/etc., not over the wire.
So now the browser decrypts the signature in the certificate. FINALLY, the browser compares the decrypted values with the contents of the certificate itself. IF THE CONTENTS MATCH, THAT MEANS THE SIGNATURE IS VALID!
Why? Think about it, only this public key can decrypt the signature in such a way that the contents look like they did before the private key encrypted them.
How about man in the middle attacks?
This is one of the main reasons (if not the main reason) why the above standard was created.
Let's say hacker-Jane intercepts internet-user-Bob's request, and replies with her own certificate. However, hacker-Jane is still careful enough to state in the certificate that the issuer was Example-CA. Lastly, hacker-Jane remembers that she has to include a signature on the certificate. But what key does Jane use to sign (i.e. create an encrypted value of the certificate main contents) the certificate?????
So even if hacker-Jane signed the certificate with her own key, you see what's gonna happen next. The browser is gonna say: "ok, this certificate is issued by Example-CA, let's decrypt the signature with Example-CA's public key". After decryption, the browser notices that the certificate contents don't match at all. Hence, the browser gives a very clear warning to the user, and it says it doesn't trust the connection.
The client has a pre-seeded store of SSL certificate authorities' public keys. There must be a chain of trust from the certificate for the server up through intermediate authorities up to one of the so-called "root" certificates in order for the server to be trusted.
You can examine and/or alter the list of trusted authorities. Often you do this to add a certificate for a local authority that you know you trust - like the company you work for or the school you attend or what not.
The pre-seeded list can vary depending on which client you use. The big SSL certificate vendors insure that their root certs are in all the major browsers ($$$).
Monkey-in-the-middle attacks are "impossible" unless the attacker has the private key of a trusted root certificate. Since the corresponding certificates are widely deployed, the exposure of such a private key would have serious implications for the security of eCommerce generally. Because of that, those private keys are very, very closely guarded.
if you're more technically minded, this site is probably what you want: http://www.zytrax.com/tech/survival/ssl.html
warning: the rabbit hole goes deep :).
I am a developer with very little experience of encryption. I am trying to learn more about Encryption and specifically SSL in my spare time.
Say a trusted company has a file (notepad) that contains a load of personal and confidential information. Say I wanted to ask them to send this information to a me to analyze. How would I do this?
My research is telling me that I would have to:
1) Create a self signed certificate
2) Generate the public and private key
3) Issue the public key to the trusted company and keep the private key
4) Ask the company to encrypt the information using the public key and then send it to me
5) Decrypt the file using the private key
Is that correct? How would I do this?
Should the certificate be signed by a Certificate Authority in this case as I would manually be issuing the public key to the trusted company (they know it comes from me).
As I said I am relatively new to this subject.
SSL is a protocol for encrypting network connections. It's not really relevant for manually encrypting and decrypting files. (It's also insecure and deprecated; TLS should be used instead.)
If you want to manually encrypt and decrypt files, yes, you could use a self-signed certificate and just give the public key to the company that'll send the file. However, you have to make sure to give them the public key in a way that can't be intercepted — such as physically handing it over on a flash drive, not emailing it — or you're vulnerable to a man-in-the-middle attack: someone could intercept the public key and give the company a different one instead. (If that happened, the company would unknowingly be encrypting the file for the attacker to decrypt, and the attacker would then re-encrypt the file with the real public key and send it along to you.)
If you're doing this manually and you deliver the public key in a secure way, you don't need to involve a certificate authority. The purpose of a certificate authority is to vouch for the authenticity of a certificate when it's delivered over an insecure channel. For example, when you visit stackoverflow.com, your browser gets a certificate (with a public key) from the web server. Since this certificate wasn't personally given to you by SO staff, how do you know that the web server is the real stackoverflow.com, not a man-in-the-middle attacker intercepting your connection in order to hijack your account? The website's certificate is signed by a certificate authority — DigiCert, currently — that says DigiCert has verified that it's real. Your browser is configured to trust that signature from DigiCert, because the browser developers have verified that the company is trustworthy.
PGP and GPG are good tools for securely exchanging encrypted files. They use a "web-of-trust" model based on securely exchanging certificates, rather than relying on a certificate authority.
I'm implementing a client with python's twisted that checks the server ssl certificate when connecting, following basically this recipe. I've seen in many HOWTOs such as this one the server checking the client's authenticity through a ssl certificate as well. Currently i authenticate my clients using an unique id and 1024 char string (they are automated clients without human interaction).
What I don't understand is what reason would I have to use the whole ssl thing for this instead of just sending the "password" to the server. After all the connection is already ssl encrypted, checking the server certificate and everything. This is a similar question but I want to know why people use ssl client certs and not just what is the best way to do it instead.
A client certificate restricts access to people authorized with certificates. Assuming your certificates are distributed and managed correctly, this makes it more difficult to connect from an unauthorized location (or say, a bot network), since you need more than just a username and password.
Client-side certificates are a potential part of a defense-in-depth strategy, if you are in an environment where you can manage client certificates.
Certificates are easy to revoke. Passwords can be stolen, but stealing a client side certificate would be much harder.
Using client certificate based mutual authentication prevents at least the following attacks/problems:
Phishing the password
Key logging the password
Shoulder surfing the password
Guessing the password
Password reuse on several services
Additionally, using client certs gives you the possibility to store client certificate (and the matching private key) on a smartcard, USB token or other hardware security module (HSM), thereby going from "something you know" (password) to "something you possess physically" (token, card) plus "something you know" (PIN). This is also called two-factor authentication.
In your specific case of using passwords as shared keys in a technical, system to system communication link, using certificates has two advantages:
scales better: with shared keys, every node has to share a different key/password with each other node, resulting in (n-1)! passwords, while with certificates, each node needs only one certificate and private key (n certificates plus a CA)
the possibility of storing the key on a HSM and thereby prevent it from being copied/stolen digitally.
The main advantage of client-side authentication (i.e. when server checks client certificate) is that if server gets compromised, the client's secret, which is private key for certificate, won't be compromised. Whereas if client uses credentials they could be compromised along with server.
Owning SSL certificates that are signed by a certificate authority means that the SSL certificate owners have gone through the hassle of being verified by the CA that the owner is who they say they are. For instance, if you have an ecommerce store called widgetsdeluxe.com and you have a certificate for the domain widgetsdeluxe.com that has been signed by Verisign, et. Al., shoppers will know that when they go to that site and the name on the certificate matches the actual domain name they went to, then they can trust that the information is secured and is coming from the widgetsdeluxe.com domain (this is to prevent spoofing and man-in-the-middle attacks).
How does SSL work?
Where is the certificate installed on the client (or browser?) and the server (or web server?)?
How does the trust/encryption/authentication process start when you enter the URL into the browser and get the page from the server?
How does the HTTPS protocol recognize the certificate? Why can't HTTP work with certificates when it is the certificates which do all the trust/encryption/authentication work?
Note: I wrote my original answer very hastily, but since then, this has turned into a fairly popular question/answer, so I have expanded it a bit and made it more precise.
TLS Capabilities
"SSL" is the name that is most often used to refer to this protocol, but SSL specifically refers to the proprietary protocol designed by Netscape in the mid 90's. "TLS" is an IETF standard that is based on SSL, so I will use TLS in my answer. These days, the odds are that nearly all of your secure connections on the web are really using TLS, not SSL.
TLS has several capabilities:
Encrypt your application layer data. (In your case, the application layer protocol is HTTP.)
Authenticate the server to the client.
Authenticate the client to the server.
#1 and #2 are very common. #3 is less common. You seem to be focusing on #2, so I'll explain that part.
Authentication
A server authenticates itself to a client using a certificate. A certificate is a blob of data[1] that contains information about a website:
Domain name
Public key
The company that owns it
When it was issued
When it expires
Who issued it
Etc.
You can achieve confidentiality (#1 above) by using the public key included in the certificate to encrypt messages that can only be decrypted by the corresponding private key, which should be stored safely on that server.[2] Let's call this key pair KP1, so that we won't get confused later on. You can also verify that the domain name on the certificate matches the site you're visiting (#2 above).
But what if an adversary could modify packets sent to and from the server, and what if that adversary modified the certificate you were presented with and inserted their own public key or changed any other important details? If that happened, the adversary could intercept and modify any messages that you thought were securely encrypted.
To prevent this very attack, the certificate is cryptographically signed by somebody else's private key in such a way that the signature can be verified by anybody who has the corresponding public key. Let's call this key pair KP2, to make it clear that these are not the same keys that the server is using.
Certificate Authorities
So who created KP2? Who signed the certificate?
Oversimplifying a bit, a certificate authority creates KP2, and they sell the service of using their private key to sign certificates for other organizations. For example, I create a certificate and I pay a company like Verisign to sign it with their private key.[3] Since nobody but Verisign has access to this private key, none of us can forge this signature.
And how would I personally get ahold of the public key in KP2 in order to verify that signature?
Well we've already seen that a certificate can hold a public key — and computer scientists love recursion — so why not put the KP2 public key into a certificate and distribute it that way? This sounds a little crazy at first, but in fact that's exactly how it works. Continuing with the Verisign example, Verisign produces a certificate that includes information about who they are, what types of things they are allowed to sign (other certificates), and their public key.
Now if I have a copy of that Verisign certificate, I can use that to validate the signature on the server certificate for the website I want to visit. Easy, right?!
Well, not so fast. I had to get the Verisign certificate from somewhere. What if somebody spoofs the Verisign certificate and puts their own public key in there? Then they can forge the signature on the server's certificate, and we're right back where we started: a man-in-the-middle attack.
Certificate Chains
Continuing to think recursively, we could of course introduce a third certificate and a third key pair (KP3) and use that to sign the Verisign certifcate. We call this a certificate chain: each certificate in the chain is used to verify the next certificate. Hopefully you can already see that this recursive approach is just turtles/certificates all the way down. Where does it stop?
Since we can't create an infinite number of certificates, the certificate chain obviously has to stop somewhere, and that's done by including a certificate in the chain that is self-signed.
I'll pause for a moment while you pick up the pieces of brain matter from your head exploding. Self-signed?!
Yes, at the end of the certificate chain (a.k.a. the "root"), there will be a certificate that uses it's own keypair to sign itself. This eliminates the infinite recursion problem, but it doesn't fix the authentication problem. Anybody can create a self-signed certificate that says anything on it, just like I can create a fake Princeton diploma that says I triple majored in politics, theoretical physics, and applied butt-kicking and then sign my own name at the bottom.
The [somewhat unexciting] solution to this problem is just to pick some set of self-signed certificates that you explicitly trust. For example, I might say, "I trust this Verisign self-signed certificate."
With that explicit trust in place, now I can validate the entire certificate chain. No matter how many certificates there are in the chain, I can validate each signature all the way down to the root. When I get to the root, I can check whether that root certificate is one that I explicitly trust. If so, then I can trust the entire chain.
Conferred Trust
Authentication in TLS uses a system of conferred trust. If I want to hire an auto mechanic, I may not trust any random mechanic that I find. But maybe my friend vouches for a particular mechanic. Since I trust my friend, then I can trust that mechanic.
When you buy a computer or download a browser, it comes with a few hundred root certificates that it explicitly trusts.[4] The companies that own and operate those certificates can confer that trust to other organizations by signing their certificates.
This is far from a perfect system. Some times a CA may issue a certificate erroneously. In those cases, the certificate may need to be revoked. Revocation is tricky since the issued certificate will always be cryptographically correct; an out-of-band protocol is necessary to find out which previously valid certificates have been revoked. In practice, some of these protocols aren't very secure, and many browsers don't check them anyway.
Sometimes an entire CA is compromised. For example, if you were to break into Verisign and steal their root signing key, then you could spoof any certificate in the world. Notice that this doesn't just affect Verisign customers: even if my certificate is signed by Thawte (a competitor to Verisign), that doesn't matter. My certificate can still be forged using the compromised signing key from Verisign.
This isn't just theoretical. It has happened in the wild. DigiNotar was famously hacked and subsequently went bankrupt. Comodo was also hacked, but inexplicably they remain in business to this day.
Even when CAs aren't directly compromised, there are other threats in this system. For example, a government use legal coercion to compel a CA to sign a forged certificate. Your employer may install their own CA certificate on your employee computer. In these various cases, traffic that you expect to be "secure" is actually completely visible/modifiable to the organization that controls that certificate.
Some replacements have been suggested, including Convergence, TACK, and DANE.
Endnotes
[1] TLS certificate data is formatted according to the X.509 standard. X.509 is based on ASN.1 ("Abstract Syntax Notation #1"), which means that it is not a binary data format. Therefore, X.509 must be encoded to a binary format. DER and PEM are the two most common encodings that I know of.
[2] In practice, the protocol actually switches over to a symmetric cipher, but that's a detail that's not relevant to your question.
[3] Presumable, the CA actually validates who you are before signing your certificate. If they didn't do that, then I could just create a certificate for google.com and ask a CA to sign it. With that certificiate, I could man-in-the-middle any "secure" connection to google.com. Therefore, the validation step is a very important factor in the operation of a CA. Unfortunately, it's not very clear how rigorous this validation process is at the hundreds of CAs around the world.
[4] See Mozilla's list of trusted CAs.
HTTPS is combination of HTTP and SSL(Secure Socket Layer) to provide encrypted communication between client (browser) and web server (application is hosted here).
Why is it needed?
HTTPS encrypts data that is transmitted from browser to server over the network. So, no one can sniff the data during transmission.
How HTTPS connection is established between browser and web server?
Browser tries to connect to the https://payment.com.
payment.com server sends a certificate to the browser. This certificate includes payment.com server's public key, and some evidence that this public key actually belongs to payment.com.
Browser verifies the certificate to confirm that it has the proper public key for payment.com.
Browser chooses a random new symmetric key K to use for its connection to payment.com server. It encrypts K under payment.com public key.
payment.com decrypts K using its private key. Now both browser and the payment server know K, but no one else does.
Anytime browser wants to send something to payment.com, it encrypts it under K; the payment.com server decrypts it upon receipt. Anytime the payment.com server wants to send something to your browser, it encrypts it under K.
This flow can be represented by the following diagram:
I have written a small blog post which discusses the process briefly. Please feel free to take a look.
SSL Handshake
A small snippet from the same is as follows:
"Client makes a request to the server over HTTPS. Server sends a copy of its SSL certificate + public key. After verifying the identity of the server with its local trusted CA store, client generates a secret session key, encrypts it using the server's public key and sends it. Server decrypts the secret session key using its private key and sends an acknowledgment to the client. Secure channel established."
Mehaase has explained it in details already. I will add my 2 cents to this series. I have many blogposts revolving around SSL handshake and certificates. While most of this revolves around IIS web server, the post is still relevant to SSL/TLS handshake in general. Here are few for your reference:
SSL Handshake and IIS
Client certificate Authentication in SSL Handshake
Do not treat CERTIFICATES & SSL as one topic. Treat them as 2 different topics and then try to see who they work in conjunction. This will help you answer the question.
Establishing trust between communicating parties via Certificate Store
SSL/TLS communication works solely on the basis of trust. Every computer (client/server) on the internet has a list of Root CA's and Intermediate CA's that it maintains. These are periodically updated. During SSL handshake this is used as a reference to establish trust. For exampe, during SSL handshake, when the client provides a certificate to the server. The server will try to cehck whether the CA who issued the cert is present in its list of CA's . When it cannot do this, it declares that it was unable to do the certificate chain verification. (This is a part of the answer. It also looks at AIA for this.) The client also does a similar verification for the server certificate which it receives in Server Hello.
On Windows, you can see the certificate stores for client & Server via PowerShell. Execute the below from a PowerShell console.
PS Cert:> ls Location : CurrentUser StoreNames : {TrustedPublisher, ClientAuthIssuer, Root, UserDS...}
Location : LocalMachine StoreNames : {TrustedPublisher,
ClientAuthIssuer, Remote Desktop, Root...}
Browsers like Firefox and Opera don't rely on underlying OS for certificate management. They maintain their own separate certificate stores.
The SSL handshake uses both Symmetric & Public Key Cryptography. Server Authentication happens by default. Client Authentication is optional and depends if the Server endpoint is configured to authenticate the client or not. Refer my blog post as I have explained this in detail.
Finally for this question
How does the HTTPS protocol recognize the certificate? Why can't HTTP work with certificates when it is the certificates which do all the trust/encryption/authentication work?
Certificates is simply a file whose format is defined by X.509 standard. It is a electronic document which proves the identity of a communicating party.
HTTPS = HTTP + SSL is a protocol which defines the guidelines as to how 2 parties should communicate with each other.
MORE INFORMATION
In order to understand certificates you will have to understand what certificates are and also read about Certificate Management. These is important.
Once this is understood, then proceed with TLS/SSL handshake. You may refer the RFC's for this. But they are skeleton which define the guidelines. There are several blogposts including mine which explain this in detail.
If the above activity is done, then you will have a fair understanding of Certificates and SSL.