I would like to decode and store raw SSL/TLS data in order to prove something in the future, i think it will be better if i describe it by an example:
1) There is a remote REST endpoint that is accessible via SSL and provides some useful data. This endpoint is controlled by some third party and i have no control over it.
2) I issue a GET request to this endpoint and store the raw SSL traffic of the response in one file and then the decoded ssl traffic in another file
3) Additionally i save the SSL certificate of this endpoint in a separate file (i guess this operation would only need to be done once)
If i understand correctly when i have all the data from the steps above and in the future, when another third party asks me for a proof that there was certain data on the remote server at some point in the past i could:
1) Present them with the certificate i saved (step 3). Since this certificate is signed by a well known authority there is almost zero chance that i could forge it
2) Present them with the RAW SSL data along with a way to decode it using the certificate above
3) In the decoded REST response headers remote server time should also be present.
Therefore if i am not mistaken i would be able to prove, with almost no doubt, that at some point in the past there was that certain piece of data on the remote party's server even if the remote server itself is no longer operational.
Therefore i can use raw SSL data to prove that something existed on the remote server in past. And there is very little chance of someone challeging this claim because forging the response data or headers would mean that i was able to break the SSL encryption.
Is there a way to automate such scenarios? I mean is there a way to record raw SSL data in a form that would allow it to be easily decoded later?
I have tried using firefox/chrome browser along with the sslkeylog files and sniffing the traffic using wireshark - it actually works, but is there a way to automate it using some kind of REST library since using firefox and recording SSL keys each time can be troublesome - there may be a need to store thousands of these requests per hour and doing it manually would be a full time job, additionally i am not sure if there is a way to prove that these keys stored in ssskeylog files actually match the certificate of the remote server...
No, you cannot prove the server sent the data. For example, with TLS_RSA_WITH_AES_256_CBC_SHA256 as the ciphersuite, you can forge all data from both sides in an entire TLS session without even contacting the server. And the server side can also produce such a forgery. As noted by EJP, the only thing you can't forge is data that is signed. With RSA ciphersuites the only thing signed are the certificates, but they don't change and are not cryptographically bound to a TLS session.
Other ciphersuites may involve some signed handshake data, but TLS was not designed to provide non-repudiation so I doubt it does by accident.
Related
Disclaimer: Complete newbie trying to wrap my head around SSL.
I am developing a device using an ESP8266 which needs to connect securely to a known server for IOT purposes, we will develop and control the server endpoint as well as the ESP8266 based client (BearSSL etc), we will not control the SSL certificate updates on the hosted server and need to manage the changing certificate keys.
Using the SHA-1 fingerprint for the certificate installed on the server appears to be the most straightforward approach and will provide the basic security we need. The data we will be exchanging is not sensitive or mission critical, we just need to keep the web server happy going into the future.
I understand the need to update the SHA-1 fingerprint on the client when the server certificate updates and this would typically be done with a firmware update over a secure connection. Our use case will make this very difficult for various reasons, so I am trying to establish the best method for updating the fingerprint as it changes without requiring re-flashing/OTA updates.
What I don't understand is why there is a need to protect/hide/embed the fingerprint when any public user or hacker can visit our SSL server site and obtain the fingerprint through a browser or OpenSSL query. Can I simply not retrieve the current fingerprint (maybe encode it with our own basic encryption) from a known HTTP non SSL server perhaps running PHP which will obtain and calculate the current fingerprint of our SSL server for use by our IOT device ? Our device would query the HTTP server first, retrieve the fingerprint and store it in EEPROM until it expires, then simply re-obtain the new fingerprint as required. Then it goes off and talks to the SSL server.
So the crux of the question is, if a hacker can get the fingerprint straight from our SSL server, why would this be an unsafe approach, which I'm sure it is ?
I don't want to go down the trusted root CA with long expiry approach as our devices may need to run for 20-30 years and we'll need a device certificate update procedure regardless, and would prefer not to use ClientInsecure() if possible.
Assuming the non-SSL HTTP approach is no good, can anybody suggest an alternate automated method for retrieving the current fingerprint securely ? I have to assume our devices may get left in a cupboard or disconnected from Wifi for years at a time and need to automatically re-connect in the future without a firmware update.
Many thanks, and be gentle *8)
Your question may be removed as inappropriate for Stack Overflow but it's a really interesting one and I'm hoping you'll at least get a chance to see this answer.
First of all, there is absolutely no need to hide the fingerprint of the server's certificate. As you pointed out, anyone can get the fingerprint directly from the server itself.
If you're downloading the fingerprint from a different source in order to update your embedded device then it's not privacy you need, it's authentication - that you're getting it from the source you think you're getting it from - and integrity - that the fingerprint hasn't been corrupted or modified during transmission.
Which leads you to a chicken and egg problem. If you serve the updated fingerprint through non-HTTPS servers then it's vulnerable to modification and the servers are vulnerable to impersonation. If you serve it via HTTPS then you still have the issue of verifying the HTTPS server you're getting the fingerprint from.
You could use a pre-shared key to sign and verify the downloaded fingerprint. The embedded device would use a public key to decrypt a signed fingerprint, the server would have the private key to sign it. Then you face an entire new set of issues if the private key is every compromised - key revocation and distribution, which is part of the problem you're trying to skirt here with this whole process.
You're also going to want to do better than SHA-1. SHA-1 hasn't been considered cryptographically secure for years.
And in 20 - 30 years time, it's likely that whatever algorithm you're using will also cease to be cryptographically secure. Which means that you'll need to update the fingerprint algorithm over the course of decades.
Instead of using the fingerprint, you can embed in the device's firmware the top level certificate of the Certificate Authority that signed the server's certificate, but CA certificates will also expire well before 20-30 years elapse, and may also be revoked. If you embed the CA certificate then the web server will have to supply the embedded device with its entire certificate chain so that the device can verify signatures at each step, which on an ESP8266 may be very, very slow, even today.
In fact, it's quite likely that web servers 20-30 years from now won't use the same cyphers for SSL as they do today, and it's likely they won't continue to support the version of TLS (1.3) that's standard now. So you would need to be able to update your embedded software to TLS 1.8 or 2.0 or whatever the version will be that's needed 20-30 years from now. And the ESP8266 is not particularly fast at computing today's cyphers... it may be computationally impractical for it to compute the cyphers of decades in the future.
In fact, wifi 20-30 years from now will quite possibly not support hardware from today as well as wifi protocols evolve and also require updated cypher suites.
I'm also dubious that ESP8266's are likely to run continuously for 20 years without hardware failures. The main feature of the ESP8266 is that it's cheap, and cheap does not often correspond with reliability or longevity.
With much better performance, the ESP32 (still cheap) would stand a better chance to being able to compute the cyphers in use 20-30 years from now and support the future's wifi standards, but with its (and the ESP8266's) closed source wifi implementation you'd be at the mercy of Espressif to provide updates to its wifi stack 20 years from now, which I doubt will happen.
Problem:
I'm pretty new to this TLS thing and networking in general. But i'm trying to passively listen and decode TLS packets. I find it kind of frustrating that i cant decode packets on my network on my computer. The conclusion i got to is that is impossible because how TLS works. But in mathematics if you have enough of information you can often get the value of the unknown. In this instance i guess it might be the servers key that is the missing link.
Even if it would be cool to totally decrypt packets its not my main goal. What i specifically trying to do is to know if a specific twitch stream is watched on my computer/network. So if i type in the streamers name in to the program it checks if that stream is sent to my computer and responds with a boolean.
So i want your input guys and girls. Is there any way of achieve this without doing Man in the middle?
Thoughts:
I have been thinking about this problem and i got a couple of ideas (don't laugh at the possible stupidity, I'm not always very smart but its just some things that came to my mind):
(Comparing) Using my cert to send request to the twitch api and somehow compare the response and see if they have the same encrypted payload. Then i know this specific stream is watched.
(Decoding) feed cert (public and private key), decrypted message and encrypted message to some algorithm to get server key. My theory here is that i have enough information to get the server key. But my knowledge about ssl is limited and i have not considered handshakes and such.
(Decoding) Generate training data to an ai to teach it to decode ssl packets. Generating a lot of ssl packets with different keys and feed the cert and encrypted data to the ai.
I hope you understand what I'm trying to do. English is not my native language (sorry).
(Comparing) Using my cert to send request to the twitch api and somehow compare the response and see if they have the same encrypted payload. Then i know this specific stream is watched.
Data are encrypted with a symmetric key specific for this SSL/TLS session. Also a random initialization vector is used. Thus, even transfer of exactly the same data results in different encrypted data which means no comparison of the encrypted data will help.
(Decoding) feed cert (public and private key), decrypted message and encrypted message to some algorithm to get server key. My theory here is that i have enough information to get the server key. But my knowledge about ssl is limited and i have not considered handshakes and such.
If (the obsolete) RSA key exchange is used you would need to have the servers private key to decrypt which you don't have. With Diffie-Hellman key exchange even this private key would not be sufficient, but you need to have the master secret or pre-master secret of this specific SSL/TLS session (see the TLS standards for details what this is). This secret can only be found in the TLS client and TLS server for the time the TLS session is active - which means that you will not be able to use this either for decryption unless you have access to the internals of the client, in which case you might just look directly which stream they are viewing.
(Decoding) Generate training data to an ai to teach it to decode ssl packets. Generating a lot of ssl packets with different keys and feed the cert and encrypted data to the ai.
Properly encrypted data are more or less random and have no inherent structure which can be mapped to the original data or even parts of the data. AI will not magically find such a structure too. The only possible difference it might use to distinguish streams are the size and timing of the data - but only if these are specific for a stream which I doubt. So, most likely it is impossible this way too.
I find it kind of frustrating that i cant decode packets on my network on my computer. The conclusion i got to is that is impossible because how TLS works.
Indeed this is primary purpose of TLS. If correctly implemented you should have no way of decrypting transfered data.
There are even buggy and obsolete implementation on some servers where one can exploit some vulnerabilities, but - not for Twitch and not with your knowledge level only for buggy implementation
Generally you could consider TLS secure enough to ensure traffic integrity and confidentiality.
What i specifically trying to do is to know if a specific twitch stream is watched on my computer/network.
I'd provide some options to achieve the goal, however that would introduce dangerous vulnerabilities into your computer or network. (you have been warned)
Still you can sniff out the unencrypted traffic. So you can see e. g. DNS requests (you could see what hostnames are resolved, but not specific URL).
For your computer you could install a keylogger or get the requested URL from browser before it is encrypted.
Integrity of TLS rely on certification authorities. In theory could can create your own CA certificate and make it trusted on your computer or network. That could allow you mounting a man-in-the-middle attack (posing yourself as the target server with your own keys).
I am trying to see data (using tcpdump) which my browser sends to server which is using https protocol
tcpdump -i any -w /tmp/http.log
but application data is encrypted(as it was expected).
I am wondering is there a way to see data before it will be encrypted when the server is https?
EDIT: Encryption traffic is created by common web browsers like Firefox, Chrome, IE...
If you control the server, you can set it to permit the null cipher then force your client to use the same. The null cipher is just a fancy way of saying "unencrypted". This should NEVER be deployed, as even having it as an option in the ciphers list is HIGHLY insecure.
You could also add a trusted key to the client, and have the client use a proxy. The communication with the proxy uses the trusted key you created, and can look at the data before sending it on, encrypted with the key of the destination server. This is, effectively, a "Man in the Middle attack," and can be defeated by things like certificate pinning. Some companies use this to track employee computer usage (when used in that way, it's somewhat controversial).
Strictly speaking, both of those are attacks to get around the encryption, not looking at the data before it's encrypted. To see it before it's encrypted, you would, generally, have to modify either the client or server to record what it's sending (or maybe use a debugger), as generally the encryption is done by a library directly linked to the programme.
EDIT: the developer tools in Chrome and Firefox might be what you're looking for: if you click the page on the "network" tab (in chrome, I don't have FF up, but it has almost exactly the same thing) you can see almost all the aspects of the info being sent and received.
Just use Charles Proxy (free trial) on your computer. If the certificate is pinned this will not work bit that is probably not the case for a browser..
I developed (in CPP) a unique protocol over HTTP and I use it to communicate with my server.
Now, I want to establish SSL connection over my proprietary protocol to transfer the data itself.
Can I do it using OpenSSL? I searched the web and all I found is something related with BIO, but I didn't understood how to use it for my needs..
Anyway, the best solution for me will be a way I can pass OpenSSL my proprietary send & receive functions so all the communication itself will go only through my functions.
TNX ahead :)
Use BIO pairs. You can find an example in the ssltest.c program, search the source for bio_pair. The basic idea is that you treat the OpenSSL engine as a black box.
There are four things your code has to do:
When you receive encrypted data over the connection to the other side, you have to write it to the SSL engine's encrypted BIO.
When the SSL engine wants to send encrypted data to the other side, you have to read it from the SSL engine's encrypted BIO and transport it to the other side.
When you have plaintext you want to encrypt and send, you have to write it to the SSL engine's plaintext BIO.
When the SSL engine has plaintext it has decrypted for you, you have to read it from the SSL engine's plaintext BIO.
OpenSSL acts purely as an engine following the SSL protocol and moving data between the two BIOs. It does all the protocol negotiation and operations for you, so long as you keep all four of these data streams moving.
One caution I can give you is this -- do not assume any special relationship between these things. For example, you might have some plaintext you want to encrypt and send, and when you write it to the SSL engine's plaintext BIO, the SSL engine might not be able to make forward progress until it receives some encrypted data from the other side. Treat the SSL engine as a black box and do all these four things whenever possible . Do not try to "look through" the SSL engine and, for example, expect that because you handed the SSL engine some encrypted data it will have necessarily plaintext for you. It might, but it might also need to send encrypted data to the other side.
One other caution: the SSL engine has only one state. It does not have a read state and a write state. (Search this thread for "the nightmare scenario" if you want the ugly details.) This is most likely to bite you if you use an SSL connection with multiple threads and expect it to behave just like a TCP connection (where the read and write sides are independent except in the case of a fatal error or connection close).
Second option - a protocol that has its own messages and uses HTTP to
pass them between the client and the server.
If you're using HTTP to pass your own messages, using OpenSSL for SSL/TLS would imply that you'd need to write your own HTTP library library too.
Instead, use an HTTP library that supports HTTPS (most do), via OpenSSL or not. Exchanging your custom messages on top of HTTPS should be fairly transparent and similar to using plain HTTP. You'd just need to configure HTTPS normally.
I'm using SSL certificates in a client-server application; both the client and the server are using 2 certificates that will expire soon. Normally, you'd just replace the certificates with new ones, but this cannot happen at once because of the huge numbers of clients.
So, if only the servers and a part of the clients are updated, the rest of the clients won't be able to authenticate anymore.
A quick fix is to replace the binaries with a version which simply disregards expiry date of the certificates; the update of the clients can be done sequentially, as long as it will finish before the certificates expire.
Long term solutions I thought about:
use Puppet to push new certificates on clients
unfortunately, not viable because not all clients are/will be managed via Puppet
use a second set of certificates
if first set has expired, use the second one
in this manner, server will have new certificates, a part of clients will have new certificates and the rest of the clients will have old certificates, but everything works
client requests a new certificate from the server, if the current one has expired.
Are there any other solutions?
I assume you use the SSL certificates for an online SSL connection, like HTTPS, or SFTP.
The big question is: Do you still trust and want to use your server-side key! If so you can just re-issue the server certificate with a new expiration date still using the old key and thus extending the lifetime. The question is, if you still trust the old key or if it should be replaced. Old clients may still connect to you at that point.. You are still using the same public / private key pair, just made a new 'lifetime version' of the certificate for it. (That is what most public servers do..)
Using two sets of active SSL certificates for different keys on the server side is not really viable and only possible if you have good control over the handshake process on the client side and your server application supports it. The issue is that during an SSL negotiation, the server has to send its certificate first and the only indication it might get from the client is a ServerName extension during the ClientHello. (Assuming the client actually sends one). Otherwise the server is 'at a loss' on what the other side will or will not support. (There are some other extensions that might help for in indicating supported CA certificates, but your clients should support those).
The first is the most practical for the clients that do support it. Just renew their certificates (and perhaps keys) and push them. And you are done with those.
For the others, updating the client software and making sure they generate a new key and request a new certificate from the server when they need it (or in advance) might be the best solution.