Adding OpenSSL into existing app - ssl

I'm adding SSL support (currently pushing forward with OpenSSL) to an existing application. I've never done anything with cryptology before and after reading numerous articles and watching videos I'm still a little confused as to how I can implement it for my specific situation.
Our software is client/server, and the end-user purchases both and will install it on their premises.
My first bit of confusion is regarding certificates and private keys, and how to manage these. Should I have one certificate that get installed along with the app? Should each end-user have their own certificate generated? What about private keys? Do I bake the private key into the server binary? Or should there be a file with the private key?
I'm sure this is a solved problem, but I'm not quite sure where to look, or what to search for.
Thanks for any help and advice.

Adding OpenSSL into existing app
If all you need is an example of a SSL/TLS client, have a look at the OpenSSL's wiki and TLS Client example.
My first bit of confusion is regarding certificates and private keys, and how to manage these.
Yes, key management and distribution is the hardest problem in crpyto.
Public CAs have legally binding documents covering these practices. They are called Certification Practice Statements (CPS). You can have a lot of fun with them because the company lawyers tell you what you don't want to hear (or the marketing department refuses to tell you).
For example, here's an excerpt from Apple Inc. Certification Authority Certification Practice Statement:
2.4.2. CA disclaimers of warranties
To the extent permitted by applicable law, Subscriber agreements,
if applicable, disclaim warranties from Apple, including any
warranty of merchantability or fitness for a particular purpose.
2.4.3. CA limitations of liability
To the extent permitted by applicable law, Subscriber agreements,
if applicable, shall limit liability on the part of Apple and shall
exclude liability for indirect, special, incidental, and
consequential damages.
So, Apple is selling you a product with no warranty and that offers no liability!!! And they want you to trust them and give them money... what a racket! And its not just Apple - other CAs have equally obscene CPS'es.
Should I have one certificate that get installed along with the app?
It depends. If you are running your own PKI (i.e., you are the CA and control the root certificate), then distribute your root X509 certificate with you application and nothing else. There's no need to trust any other CAs, like Verisign or Startcom.
If you are using someone else's PKI (or the Internet's PKI specified in RFC 5280), then distribute only the root X509 certificate needed to validate the chain. In this case, you will distribute one CA's root X509 certificate for validation. You could potentially trust just about any certificate signed by that particular CA, however (and its likely to be in the 10's of thousands if you are not careful).
If you don't know in advance, then you have to do like browsers and pick a bunch of CAs to trust and carry around their root certificates for your application. You can grab a list of them from Mozilla, for example. You could potentially trust just about any certificate signed by all CAs, however (and its likely to be in the 10's of millions if you are not careful).
There's a lot more to using public CAs like browsers, and you should read through Peter Gutmann's Engineering Security. Pay particular attention to the Security Diversification strategies.
When the client connects to your server, your server should send its X509 certificate (the leaf certificate) and any intermediate certificates required to build a valid chain back to the root certificate you distribute.
Finally, you can get free SSL/TLS certificates trusted by most major browsers (including mobile) from Eddy Nigg at Startcom. He charges for the revocation (if needed) because that's where the cost lies. Other CAs charge you up front and pocket the proceeds if not needed.
Should each end-user have their own certificate generated?
That is possible, too. That's called client certificates or client authentication authentication. Ideally, you would be running your own PKI because (1) you control everything (including the CA operations) and don't need to trust anyone outside the organization; and (2) it can get expensive to have a commercial CA sign every user's certificate.
If you don't want to use client side certificates, please look into PSK (Preshared Keys) and SRP (Secure Remote Password). Both beat the snot out of classic X509 using RSA key transport. PSK and SRP do so because they provide mutual authentication and channel binding. In these systems, both the client and server know the secret or password and the channel is setup up; or one (or both) does not know and channel setup fails. The plain text username and password are never put on the wire as in RSA transport and basic_auth schemes. (I prefer SRP because its based on Diffie-Hellman, and have implemented it in a few systems).
What about private keys?
Yes, you need to manage the private keys associated with certificates. You can (1) store them in the filesystem with permissions or ACLs; (2) store them in a Keystore or Keychain like Android, Mac OS X, iOS, or Windows; (3) store them in an Hardware Security Module (HSM); or (4) store them remotely while keeping them online using Key Management Interop Protocol (KMIP).
Note: unattended key storage on a server is a problem without a solution. See, for example, Peter Gutmann's Engineering Security, page 368 under "Wicked Hard Problems" and "Problems without Solutions".
Do I bake the private key into the server binary?
No. You generate them when needed and then store them with the best protection you can provide.
Or should there be a file with the private key?
Yes, something like that. See above.
I'm sure this is a solved problem, but I'm not quite sure where to look, or what to search for.
I'm not sure I would really call it solved because of the key distribution problem.
And some implementations are just really bad, so you would likely wonder how the code passed for production.
The first thing you probably want (since your focusing on key management) is a treatment of "key management" and "key hierarchies".
You might also want some reference material. From the security engineering point of view, read Gutmann's Engineering Security and Ross Anderson's Security Engineering. From an implementation standpoint, grab a copy of Network Security with OpenSSL and SSL and TLS: Designing and Building Secure Systems.

Related

Can the ACME version 2 protocol be used to distribute SSL certificates (and keys) or only send new certificates?

ACME is used by some certificate authorities to process automated signing of certificate requests and issue resulting certificates.
An increasing number of applications (Eg Traefik) now have builtin support for ACME. This is very useful for managing certificate cycling without tracking where every deployed application stores its certificates.
However "free" certificates from letsencrypt and similar certificate authorities are not always appropriate for all applications. Such examples include some software requiring the exact same certificate being installed in multiple layers of an application's edge gateway (yes this does happen).
I'm wondering if, hypothetically, ACME v2 can be used to redistribute existing SSL certificates and associated private keys. Or is it only capable if responding to CSRs where the private key is never transferred between ACME server and client?
ACME is defined by RFC8555 currently.
There is nothing that prevents other CA to use it, besides Let's Encrypt, and indeed there are other CAs existing using it, and others adding it to their existing API.
The key associated to a certificate should never travel. The CA shouldn't know it. As such, there are no operations in ACME that would allow a CA to send the key, besides the certificate, and that is a good thing.
Retrieving a certificate is a core operation, see "7.4.2. Downloading the Certificate". It could happen separately/outside of a previous certificate request. The problem being then: how does the client know which URL to use to download a specific certificate (an information that it receives when doing the normal authorization/validation steps)?
There are other protocols to manage communication of cryptographic materials such as X509 certificates.
Still in ACME, you might be interested in RFC 8739 "Support for Short-Term, Automatically Renewed (STAR) Certificates in the Automated Certificate Management Environment (ACME)" which allows the CA to pre-generate certificates.
And the related RFC 9115 "An Automatic Certificate Management Environment (ACME) Profile for Generating Delegated Certificates",
Outside of ACME, but very similar, you have RFC 8894 "Simple Certificate Enrolment Protocol" (but still no transmission of private key).
But you are not clearly describing the problem you have around "redistribute existing SSL(sic) certificates and associated private keys". What else do you really need besides an URL that gives back that data if it is that you want to send back to client (but again: making the private key travel is, in general, a bad idea)? Is authenticating the client the problem? Discoverability of the URL to use to download a specific certificate/key?

Is my SSL certificate good enough for financial transactions on my shopping cart

I have an online shop and I've just installed a new SSL certificate and it was free. It does seem too good to be true. I'm a very cynical type of person.
I don't know about different types of SSL, but I just need to be able to accept payment data (I'm using a PayPal add-in on Opencart).
I got my certificate from letsencrypt and they don't explain much on there website.
But if you go to my website Gwenllian-retail you will see the certificate. Can I handle financial transactions with that?
If not what type of SSL do I need?
One does not need much money or complicated software to create valid SSL certificates. I could create my own with ease, if I wanted. In fact, I have done. There is no reason to think that LetsEncrypt certificates are somehow of a wrong kind.
The question is whether people will trust those certificates, and that comes back to whether they trust the Certificate Authority (CA) that signed them. If I sign my own certificate and present that to someone as proof of my identity then that other party has no more reason to trust that the data within accurately identify me than if I just told them directly.
LetsEncrypt serves as the CA for SSL certificates it provides. I have never relied on them for a certificate, but according to hosting company DreamHost, LetsEncrypt certificates are trusted by all major browsers. (LetsEncrypt makes the same claim about itself, too.)
Again, all this trust business is mostly about authentication: whether the entity that presents the certificate (your web site) is really the entity that it says it is. It is not about the nature or quality of the encryption with which the session is secured. That comes down to the capabilities of the two endpoints, and is largely independent of the certificate.
Let's Encrypt is a well known service backed up by many big players. So yes, it's OK to use it in on your site. BUT ! SSL certificate is not everything, it's only one of many shields to protect you application.

What SSL certificate do I need?

I'm developing software which will be deployed using clickonce (on the website foo.com), and which will then connect to my server using WCF with an encrypted transport
So I need an SSL certificate which will :
Identify my foo.com website has really being my website
Identify the exe I deploy using clickonce as being genuine
Identify my application server has really being my application server.
I also want my SSL certificate to be signed by an authority known to the public (ie, firefox or windows won't ask the user to install the authority's certificate first !)
What SSL certificate would you buy?
I've browsed the Verisign website, the "Secure Site EV" certificate costs 1150€ a year (the "Pro" version seems useful only for compatibility with older browsers)
It sounds like you're looking for two different types of certificates:
1 - SSL Certificate - for authentication of your website/application server.
2 - Code Signing Certificate - for integrity/authentication of the exe you deliver.
Typically those are two different certificates, with two different certificate profiles. At the very least, you need one certificate with two different key usages or extended key usages.
A few thoughts in no specific order:
Check your targeted browsers, they should each have a set of preconfigured root certificates - those are the most widely recognized public certificate sources. I'd probably check both Firefox and IE. Certificate vendors known to me as big names are - Versign, GeoTrust, RSA, Thawte, Entrust. But there's also GoDaddy and many others. Anything that comes in the delivered browser as a Trusted Root Certificate, will allow you to connect to your users without additional greif.
I suggest Googling for both "code signing certificate" and "SSL certificate".
How you configure your site will determine whether or not your website is validated or your authentication server is validated. If the certificate is stored on the apps server, then your user is getting SSL encryption all the way to the server. But many sites put the SSL certificate a little farther forward - like on a firewall, and then stage a collection of apps servers behind it. I don't see a security flaw in that, so long as the networking is carefully configured. To the outside users, both configurations will look the same - they'll get the lock on their browsers and a certificate that tells them that www.foo.com is offering it's credentials.
I'm seeing pretty great deals for SSL Certificates:
- GoDaddy - $12.99
- Register.com - $14.99
But they aren't necessarily code signing certifiates. For example, while GoDaddy's SSL Cert is $12.99, their code signing certs are $199.99! That's part of many certificate vendors business models - lure you in with cheap SSL Certs, and make you pay for code signing. A case could be made that code signing certificates are relatively higher liability. But also... they have to subsidize the cheap SSL certs somehow.
Theoretically, it should be possible to make a certificate that does both code signing and SSL, but I'm not sure you want that. If something should happen, it would be nice to be able to isolate the two functions. Also, I'm pretty sure you'd have to call the certificate vendors and ask if they did this, and if they don't, having them do it will likely jack up the price quite high.
As far as vendor, things to consider:
The technology is pretty much all the same. These days, aim for a minimum of 128 bit keys, I'd probably bump it up to 256, but I'm paranoid.
Beyond browser acceptabiliy, the only reason to pay more would be name recognition. Among the paranoid security wonks, I'd expect RSA, Thawte, Verisign and GeoTrust to have very good reputations. Probably EnTrust, too. This probably only matters if you are dealing with a security focused product. I think your average user will not be so aware.
From a security geek perspective - you're only as safe as the security of your Root CA (Certificate Authority). For the truly paranoid, the thing to do would be to dig into the background material of how the company hosts its root and issuing CAs, how are they physically securited? network security? personnel access control? Also - do they have public CRLs (Certificate Revocation Lists), how do you get a cert revoked? Do they offer OCSP (Online Certificate Status Protocol)? How do they check out certificate requestors to be sure they are giving the right cert to the right person? ... All this stuff really matters if you are offering something that must be highly secure. Things like medical records, financial managment applications, tax information, etc should be highly protected. Most web apps aren't so high risk and probably don't require this degree of scrutiny.
On that last bullet - if you dig into the Verisigns of the world - the very expensive certs - you're likely to see the value. They have a massive infrastructure and take the security of their CAs very seriously. I'm not so sure about the super-cheap hosting services. That said, if your risk is low, US$300 for an SSL Cert doesn't make much sense compared to US$12.99!!
So for web site / application servers you need an SSL certificate. You do not need an EV certificate. I've used ones from QuickSSL for this, as unlike some of the other cheap certificate providers they don't require the installation of an intermediate certificate on the server - that's a no-one for me.
For signing applications that's a different type of certificate altogether (kind of, it's still an X509 certificate, but the one you use for your web site is not one you can use to sign an application). You need an authenticode signing certificate from the likes of Verisign or Globalsign. These are a magnitude more expensive than a plain old SSL certificate and require you to be an incorporated company and produce those documents.

why do we trust SSL certificates?

A friend of mine asked me why we pay so much for SSL certificates if everyone could theoretically issue one. Why indeed? And how do we judge if the little lock in the browser is really trustworthy?
Certificates are cryptographically signed by something called a Certificate Authority(CA), and each browser has a list of CAs it implicitly trusts. These CAs are entities that have a set of cryptographic keys that can be used to sign any certificate, often for a fee. Any certificate signed by a CA in the trusted list will give a lock on a browser, because it's proven to be "trusted" and belongs to that domain.
You can self-sign a certificate, but the browser will warn you that the signer is not trusted, either by showing a big error box before allowing you in, or showing a broken lock icon.
In addition, even a trusted certificate will give an error if it's used for the wrong domain, or is modified to include another domain. This is ensured because the certificate includes the domains it is allowed to be used for, and it also has a cryptographic checksum/fingerprint that ensures its integrity.
This is not 100% safe at the moment, as there is the possibility to fake CA certificates that use MD5, see this link: http://www.phreedom.org/research/rogue-ca/. Though it has to be noted that this is pretty hard, as they exploited a weakness in an already existing CA, which may or may not have been closed by now.
In essence, we trust the certificates as much as we trust that our browser providers know how to select "proper" CAs. Those CAs are only trusted on virtue of their reputation, as a single misstep theoretically would be a very heavy blow on their trustworthiness if detected.
The whole CA business is amazing. I've purchased a couple of certificates from rapidssl.com, and all the "proof" they required was:
I could receive mail to the domain.
I could answer my phone.
That was it. Keep in mind, when trusting the little locks in the browser.
First, some background on strong public/private key cryptography, which SSL is based on:
A key has two parts, the private part and the public part. The public key can be used to encrypt material that requires the private key to decrypt. This allows the use of open communication channels to communicate securely.
One important aspect of public/private key cryptography is that the private key can be used to digitally sign a message which can be verified using the public key. This gives the receiver of a message the ability to verify concretely that the message they received was sent by the sender (the holder of the key).
The key to SSL certificates is that encryption keys themselves can be digitally signed.
A "certificate" is composed of a private/public key pair as well as digitally signed data. When someone buys an SSL certificate they generate a private/public key and submit the public key to a Certification Authority (CA) to be signed. The CA performs an appropriate level of due diligence on the buyer of the SSL certificate and signs the certificate with their private key. The SSL certificate will be bound to a particular website or set of websites and is essentially the CA indicating that they trust the owner of the private key of the certificate to be the proper owner of those websites.
The root certificates (public keys and other meta-data) for trusted CAs are included by default in major shipping browsers and operating systems (in windows, type "certmgr.msc" into a run prompt to see the certificate manager). When you connect to a web server using SSL the server will send you its SSL certificate including the public key and other meta data, all of which is signed by the CA. Your browser is able to verify the validity of the certificate, through the signature and the preloaded root certificates. This creates a chain of trust between the CA and the web server you are connecting to.
Because we have to trust someone.
Trusted SSL certificates have signatures of trusted authorities. For example, VeriSign has a deal with Microsoft, that their certificate is built in your browser. So you can trust every page with a VeriSign trusted certificate.
This graphic really picks the point:
RA = Registration Authority
CA = Certification Authority
VA = Validation Authority
Rough outline: A user applies for a
certificate with his public key at a
registration authority (RA). The
latter confirms the user's identity to
the certification authority (CA) which
in turn issues the certificate. The
user can then digitally sign a
contract using his new certificate.
His identity is then checked by the
contracting party with a validation
authority (VA) which again receives
information about issued certificates
by the certification authority.
If you are not using one of the accepted CAs people will get a message box when accessing the site talking about an untrusted certificate. That won't help to generate traffic to the site.
The lock only means that the site owner showed a CA some kind of proof that he really is who he claims to be. You must judge on your own if you trust that person/site.
It's like a stranger showing you a photo ID. Do you trust him more because you know for sure his name is John Doe? Probably not.
But when people you trust told you: "John Doe" is a good guy. The proof that the guy in front of you actually IS "John Doe", than you might choose to trust him as well.
Why? Because you're paying to ride along on someone elses reputation.... to vouch for you.
Its all about whose validating your claim to be you. Despite some of the documentaries Ive watched lately, and the recession, I'm still more likely to believe corporate America when they confirm your identity to me, than I am the Russian mafia. Even though both can just as easily issue certificates.
The amount you pay is basically just (how much it costs them to secure that reputation and/or suppress any security breaches) + (however much they can afford to gouge the market as a margin %).
Now the barriers to entry are quite high, cos its really expensive to earn that trust, so theres not a lot of competition. Therefore chances are the price isn't going to fall anytime soon.... unless Sony or GE etc decide to play.
You pay for a certificate so that when you go HTTPS (which you should for anything a little sensitive) your clients don’t get big warnings and go call your support saying that you have infected them & al…
Very little security, lot of FUD.
If you have the possibility of giving your clients your own certificate directly, do it. But it is a rare case.
Let's create an attack scenario.
Suppose the DNS was corrupted and https://facebook.com/ points to attacker's IP.
You sit down to your PC and open Facebook to loose few minutes on pointless scrolling. And then BANG, Certificate invalid error shows on your screen. Attacker signed https://facebook.com/ with his own cert to make sure no one will leave his copied facebook page because it's not encrypted so it looks suspicious. If browser wouldn't check certificate's authority, then attacker could sign corrupted page with his cert and you won't be aware you're connecting to the wrong IP.
So the attacker has 2 options to choose from:
Sign corrupted facebook page with his cert, so users will see an error.
Don't use https on his corrupted page.
Certificates are built on a chain of trust, and if let anyone be a signing authority, we would be implicitly trusting everyone. It's a bit scary today though, since there are over 200 so called "trusted authorities" whose certs are built into your browser!
There is one free CA that I know of though: StartCom. They issue free SSL certs, but they are only accepted in Firefox, not IE. (Not sure about Safari or Opera).
The other answers have explained the CA-system. The perspectives project aims to deploy a new approach to SSL, where you can choose whom to trust: http://perspectives-project.org/

Self-signed SSL Cert or CA? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
I would like to have the authentication and registration parts of my website encrypted (for obvious reason). This site is currently and older site which some friends and I started in middle school and still use today. I may or may not register it to be a Non-Profit Organization in the near future, but either way, a CA costs money and the organization doesn't have any and we are currently college kids.
Verisign is unreasonable and GoDaddy is $30/year. GoDaddy isn't too unreasonable, and I think their certs are accepted by most web browsers. The thing with GoDaddy is that I don't know why they have different SSL products (i.e.: why is it cheap to not verify me? does this have any implications on the cert and how the browser treats it if it just contains a domain name?)
Also, is there an issue with using my own cert? Could the login page be http, and have a line stating that I use a self-signed cert and here is it's fingerprint and then post the form to an https page? Safari's method isn't too bad or sound too scary. I'm afraid, however, that firefox 3's method will scare people away and give me a tonne of emails saying that my site is being hacked or something. I don't know how IE responds to self-signed certs. (There is also the issue of why pay for something I can create myself with no effort, but I'm not going to pose the philosophical part of it, this is a more practical question.)
In sum, do I give GoDaddy $30 a year or do I just tell people in a small paragraph what I'm doing and give the few people that will actually want my fingerprint it?
Edit: Some on a forum I was reading for more info mentioned that GoDaddy certs are only given if it's on a GoDaddy server, which this isn't. Two things: (1) is this true? and There are other CA's at about the same price, so the argument should still be the same.
There's a common misconception that self-signed certificates are inherently less secure than those sold by commercial CA's like GoDaddy and Verisign, and that you have to live with browser warnings/exceptions if you use them; this is incorrect.
If you securely distribute a self-signed certificate (or CA cert, as bobince suggested) and install it in the browsers that will use your site, it's just as secure as one that's purchased and is not vulnerable to man-in-the-middle attacks and cert forgery. Obviously this means that it's only feasible if only a few people need secure access to your site (e.g., internal apps, personal blogs, etc.).
In the interest of increasing awareness and encouraging fellow small-time bloggers like myself to protect themselves, I've written up a entry-level tutorial that explains the concepts behind certificates and how to safely create and use your own self-signed cert (complete with code samples and screenshots) here.
The SSL certificate solves two purposes: encryption of traffic (for RSA key exchange, at least) and verification of trust. As you know, you can encrypt traffic with (or without, if we're talking SSL 3.0 or TLS) any self-signed certificate. But trust is accomplished through a chain of certificates. I don't know you, but I do trust verisign (or at least Microsoft does, because they've been paid lots of money to get it installed in their operating systems by default), and since Verisign trusts you, then I trust you too. As a result, there's no scary warning when I go to such an SSL page in my Web browser because somebody that I trust has said you are who you are.
Generally, the more expensive the certificate, the more investigating that the issuing certificate authority does. So for the Extended Validation certificates, the requesters have to submit more documents to prove that they are who they say they are, and in return they get a bright, happy green bar in modern Web browsers (I think Safari doesn't do anything with it quite yet).
Finally, some companies go with the big boys like Verisign purely for the brand name alone; they know that their customers have at least heard of Verisign and so that for people shopping on their online store, their seal looks a little less sketch-ball than, say, GoDaddy's.
If the branding is not important to you or if your site is not prone to phishing attacks, then the cheapest SSL cert that you can buy that has its root installed in most Web browsers by default will be fine. Usually, the only verification done is that you must be able to reply to an e-mail sent to the DNS's administrative contact, thus "proving" that you "own" that domain name.
You can use those cheap-o certificates on non-GoDaddy servers, sure, but you'll probably have to install an intermediate certificate on the server first. This is a certificate that sits between your cheap-o $30 certificate and the GoDaddy "real deal" root certificate. Web browsers visiting your site will be like "hmm, looks like this was signed with an intermediate, you got that?" which requires may require an extra trip. But then it'll request the intermediate from your server, see that it chains up to a trusted root certificate that it knows about, and there is no problem.
But if you are not allowed to install the intermediate on your server (such as in a shared hosting scenario), then you are out of luck. This is why most people say that GoDaddy certs can't be used on non-GoDaddy servers. Not true, but true enough for many scenarios.
(At work we use a Comodo certificate for our online store, and a cheapo $30 GoDaddy cert to secure the internal connection to the database.)
Edited in italics to reflect erickson's insightful clarifications below. Learn something new every day!
Get a certificate from Let's Encrypt, a free CA this new decade, which is widely supported by browsers.
I haven't tried them yet, but StartCom was mentioned in a response to a similar question. Apparently you can get a one year certificate for free, and it's accepted by Firefox 3.
Even if you have to pay, I would suggest using a CA rather than self-signed certificates. Some people won't see your explanation, and a fake site could post their own fake certificate's fingerprint just like you propose. I doubt the average user knows what a certificate fingerprint is or how to check it.
Instead of creating a self-signed cert, create a self-signed CA, and sign your HTTPS certificate with that. It's easier to ask users to install a CA than a single server cert, and you can create new certs (eg. for subdomains, or to update expired certs) without users having to install a server cert again.
You can then decide later whether it's worth the $30 to switch from a cert signed by your own CA to the same cert signed by GoDaddy or whoever.
Either way, don't have an HTTP page with a form posted to HTTPS. The user cannot see that that's where it's going; they'd have to view source to check the form hasn't been hijacked to point elsewhere and no-one's going to do that. You would have to have an HTTP front page with the CA link and a separate link to the HTTPS login form.
Asking users to install a CA with a cert downloaded via plain HTTP is a bit naughty: if there were a man-in-the-middle they could replace your CA on the fly and hijack the ensuing HTTPS connections. The chances of this actually happening are quite low as it would have to be a targeted attack as opposed to plain old automated sniffing, but really you ought to be hosting the CA download link on some other HTTPS-protected service.
Customer acceptance is a matter only you can answer, knowing who your users are. Certainly the Firefox interface is excessively scary. If CAs like GoDaddy are down to $30 these days I would probably go for it; it used to be a lot, lot worse.
Assuming support on old and niche browsers is not much of an issue, just go for the cheapest CA available. You are supposed to be paying to have the CA properly verify who you are, but in practice that's not the way it works and it never has been, so paying extra for more thorough checks gets you almost nothing. Verisign's extortionate prices survive through corporate inertia alone.
CAs are there to receive money for doing nothing but owning a few hundred bits of private key. The identity-verifying stuff that was supposed to have been part of the CA mandate has been moved to EV certificates. Which are even more of a rip-off. Joy.
Self-signed certificates are insecure. Yes, really. "At least it's encrypted" isn't helping at all. From the article:
World-class encryption * zero authentication = zero security
If your website is for you and a few of your friends, then you might create your own CA and distribute your certificate to friends.
Otherwise either get a certificate from known CA (for free) or don't bother with self-signed certificates at all, because all you'd get is a false sense of security.
Why just encrypted traffic is not secure? You're always allowing the other end to decrypt your traffic (you have to, otherwise you'd be sending gibberish).
If you don't check who's on the other end, you're allowing anybody to decrypt your traffic. It doesn't make a difference if you send data to an attacker securely or not securely — the attacker gets the data anyway.
I'm not talking about checking whether e.g. paypal.com belongs to a trustworthy financial institution (that's a bigger problem). I'm talking about checking whether you're sending data to the paypal.com, or just to a van around the corner that sends a certificate saying "yeah, I'm like totally paypal.com and you have my word that it's true!"
I just finally broke down and switched my server from self-signed to a GoDaddy cert last night and it wasn't that big a deal, aside from their process not being as clear as it could be. $30/year is a reasonable cost and using the cert on a non-GoDaddy server is not an issue.
If you're going to be talking SSL to the public, get a real cert signed by a real CA. Even if you're working for minimum wage, you'll save more than $30/year worth of wasted time in dealing with user fears or distrust, and that's even before considering any possible lost revenue due to them being scared away from your site.
To answer your question about Internet Explorer, it will warn users about any site whose certificate is not signed by an IE-known (unfortunately called "trusted") CA. This includes for your own CA and for self-signed certificates. It will also make a warning if the domain in the certificate is not the one being accessed.
If this is a private site, you may not care so long as you are getting link-level encryption (and are you that fearful of someone sniffing your traffic?). If there is public access and you want SSL, then get a signed certificate from a recognized CA, as others have already advised.
If that van around the corner is capable of hijacking your internet connection already, you've got worse problems than self-signed certificates.
Banks should use client certificates for authentication. That would make it impossible for that van to do anything.... since it doesn't have the banks private key.
Self-signed certs are perfectly fine... assuming your internet connection hasn't been compromised. If your connection has been compromised... you're probably dogged anyway.