Problem:
I'm pretty new to this TLS thing and networking in general. But i'm trying to passively listen and decode TLS packets. I find it kind of frustrating that i cant decode packets on my network on my computer. The conclusion i got to is that is impossible because how TLS works. But in mathematics if you have enough of information you can often get the value of the unknown. In this instance i guess it might be the servers key that is the missing link.
Even if it would be cool to totally decrypt packets its not my main goal. What i specifically trying to do is to know if a specific twitch stream is watched on my computer/network. So if i type in the streamers name in to the program it checks if that stream is sent to my computer and responds with a boolean.
So i want your input guys and girls. Is there any way of achieve this without doing Man in the middle?
Thoughts:
I have been thinking about this problem and i got a couple of ideas (don't laugh at the possible stupidity, I'm not always very smart but its just some things that came to my mind):
(Comparing) Using my cert to send request to the twitch api and somehow compare the response and see if they have the same encrypted payload. Then i know this specific stream is watched.
(Decoding) feed cert (public and private key), decrypted message and encrypted message to some algorithm to get server key. My theory here is that i have enough information to get the server key. But my knowledge about ssl is limited and i have not considered handshakes and such.
(Decoding) Generate training data to an ai to teach it to decode ssl packets. Generating a lot of ssl packets with different keys and feed the cert and encrypted data to the ai.
I hope you understand what I'm trying to do. English is not my native language (sorry).
(Comparing) Using my cert to send request to the twitch api and somehow compare the response and see if they have the same encrypted payload. Then i know this specific stream is watched.
Data are encrypted with a symmetric key specific for this SSL/TLS session. Also a random initialization vector is used. Thus, even transfer of exactly the same data results in different encrypted data which means no comparison of the encrypted data will help.
(Decoding) feed cert (public and private key), decrypted message and encrypted message to some algorithm to get server key. My theory here is that i have enough information to get the server key. But my knowledge about ssl is limited and i have not considered handshakes and such.
If (the obsolete) RSA key exchange is used you would need to have the servers private key to decrypt which you don't have. With Diffie-Hellman key exchange even this private key would not be sufficient, but you need to have the master secret or pre-master secret of this specific SSL/TLS session (see the TLS standards for details what this is). This secret can only be found in the TLS client and TLS server for the time the TLS session is active - which means that you will not be able to use this either for decryption unless you have access to the internals of the client, in which case you might just look directly which stream they are viewing.
(Decoding) Generate training data to an ai to teach it to decode ssl packets. Generating a lot of ssl packets with different keys and feed the cert and encrypted data to the ai.
Properly encrypted data are more or less random and have no inherent structure which can be mapped to the original data or even parts of the data. AI will not magically find such a structure too. The only possible difference it might use to distinguish streams are the size and timing of the data - but only if these are specific for a stream which I doubt. So, most likely it is impossible this way too.
I find it kind of frustrating that i cant decode packets on my network on my computer. The conclusion i got to is that is impossible because how TLS works.
Indeed this is primary purpose of TLS. If correctly implemented you should have no way of decrypting transfered data.
There are even buggy and obsolete implementation on some servers where one can exploit some vulnerabilities, but - not for Twitch and not with your knowledge level only for buggy implementation
Generally you could consider TLS secure enough to ensure traffic integrity and confidentiality.
What i specifically trying to do is to know if a specific twitch stream is watched on my computer/network.
I'd provide some options to achieve the goal, however that would introduce dangerous vulnerabilities into your computer or network. (you have been warned)
Still you can sniff out the unencrypted traffic. So you can see e. g. DNS requests (you could see what hostnames are resolved, but not specific URL).
For your computer you could install a keylogger or get the requested URL from browser before it is encrypted.
Integrity of TLS rely on certification authorities. In theory could can create your own CA certificate and make it trusted on your computer or network. That could allow you mounting a man-in-the-middle attack (posing yourself as the target server with your own keys).
Related
Disclaimer: Complete newbie trying to wrap my head around SSL.
I am developing a device using an ESP8266 which needs to connect securely to a known server for IOT purposes, we will develop and control the server endpoint as well as the ESP8266 based client (BearSSL etc), we will not control the SSL certificate updates on the hosted server and need to manage the changing certificate keys.
Using the SHA-1 fingerprint for the certificate installed on the server appears to be the most straightforward approach and will provide the basic security we need. The data we will be exchanging is not sensitive or mission critical, we just need to keep the web server happy going into the future.
I understand the need to update the SHA-1 fingerprint on the client when the server certificate updates and this would typically be done with a firmware update over a secure connection. Our use case will make this very difficult for various reasons, so I am trying to establish the best method for updating the fingerprint as it changes without requiring re-flashing/OTA updates.
What I don't understand is why there is a need to protect/hide/embed the fingerprint when any public user or hacker can visit our SSL server site and obtain the fingerprint through a browser or OpenSSL query. Can I simply not retrieve the current fingerprint (maybe encode it with our own basic encryption) from a known HTTP non SSL server perhaps running PHP which will obtain and calculate the current fingerprint of our SSL server for use by our IOT device ? Our device would query the HTTP server first, retrieve the fingerprint and store it in EEPROM until it expires, then simply re-obtain the new fingerprint as required. Then it goes off and talks to the SSL server.
So the crux of the question is, if a hacker can get the fingerprint straight from our SSL server, why would this be an unsafe approach, which I'm sure it is ?
I don't want to go down the trusted root CA with long expiry approach as our devices may need to run for 20-30 years and we'll need a device certificate update procedure regardless, and would prefer not to use ClientInsecure() if possible.
Assuming the non-SSL HTTP approach is no good, can anybody suggest an alternate automated method for retrieving the current fingerprint securely ? I have to assume our devices may get left in a cupboard or disconnected from Wifi for years at a time and need to automatically re-connect in the future without a firmware update.
Many thanks, and be gentle *8)
Your question may be removed as inappropriate for Stack Overflow but it's a really interesting one and I'm hoping you'll at least get a chance to see this answer.
First of all, there is absolutely no need to hide the fingerprint of the server's certificate. As you pointed out, anyone can get the fingerprint directly from the server itself.
If you're downloading the fingerprint from a different source in order to update your embedded device then it's not privacy you need, it's authentication - that you're getting it from the source you think you're getting it from - and integrity - that the fingerprint hasn't been corrupted or modified during transmission.
Which leads you to a chicken and egg problem. If you serve the updated fingerprint through non-HTTPS servers then it's vulnerable to modification and the servers are vulnerable to impersonation. If you serve it via HTTPS then you still have the issue of verifying the HTTPS server you're getting the fingerprint from.
You could use a pre-shared key to sign and verify the downloaded fingerprint. The embedded device would use a public key to decrypt a signed fingerprint, the server would have the private key to sign it. Then you face an entire new set of issues if the private key is every compromised - key revocation and distribution, which is part of the problem you're trying to skirt here with this whole process.
You're also going to want to do better than SHA-1. SHA-1 hasn't been considered cryptographically secure for years.
And in 20 - 30 years time, it's likely that whatever algorithm you're using will also cease to be cryptographically secure. Which means that you'll need to update the fingerprint algorithm over the course of decades.
Instead of using the fingerprint, you can embed in the device's firmware the top level certificate of the Certificate Authority that signed the server's certificate, but CA certificates will also expire well before 20-30 years elapse, and may also be revoked. If you embed the CA certificate then the web server will have to supply the embedded device with its entire certificate chain so that the device can verify signatures at each step, which on an ESP8266 may be very, very slow, even today.
In fact, it's quite likely that web servers 20-30 years from now won't use the same cyphers for SSL as they do today, and it's likely they won't continue to support the version of TLS (1.3) that's standard now. So you would need to be able to update your embedded software to TLS 1.8 or 2.0 or whatever the version will be that's needed 20-30 years from now. And the ESP8266 is not particularly fast at computing today's cyphers... it may be computationally impractical for it to compute the cyphers of decades in the future.
In fact, wifi 20-30 years from now will quite possibly not support hardware from today as well as wifi protocols evolve and also require updated cypher suites.
I'm also dubious that ESP8266's are likely to run continuously for 20 years without hardware failures. The main feature of the ESP8266 is that it's cheap, and cheap does not often correspond with reliability or longevity.
With much better performance, the ESP32 (still cheap) would stand a better chance to being able to compute the cyphers in use 20-30 years from now and support the future's wifi standards, but with its (and the ESP8266's) closed source wifi implementation you'd be at the mercy of Espressif to provide updates to its wifi stack 20 years from now, which I doubt will happen.
I would like to decode and store raw SSL/TLS data in order to prove something in the future, i think it will be better if i describe it by an example:
1) There is a remote REST endpoint that is accessible via SSL and provides some useful data. This endpoint is controlled by some third party and i have no control over it.
2) I issue a GET request to this endpoint and store the raw SSL traffic of the response in one file and then the decoded ssl traffic in another file
3) Additionally i save the SSL certificate of this endpoint in a separate file (i guess this operation would only need to be done once)
If i understand correctly when i have all the data from the steps above and in the future, when another third party asks me for a proof that there was certain data on the remote server at some point in the past i could:
1) Present them with the certificate i saved (step 3). Since this certificate is signed by a well known authority there is almost zero chance that i could forge it
2) Present them with the RAW SSL data along with a way to decode it using the certificate above
3) In the decoded REST response headers remote server time should also be present.
Therefore if i am not mistaken i would be able to prove, with almost no doubt, that at some point in the past there was that certain piece of data on the remote party's server even if the remote server itself is no longer operational.
Therefore i can use raw SSL data to prove that something existed on the remote server in past. And there is very little chance of someone challeging this claim because forging the response data or headers would mean that i was able to break the SSL encryption.
Is there a way to automate such scenarios? I mean is there a way to record raw SSL data in a form that would allow it to be easily decoded later?
I have tried using firefox/chrome browser along with the sslkeylog files and sniffing the traffic using wireshark - it actually works, but is there a way to automate it using some kind of REST library since using firefox and recording SSL keys each time can be troublesome - there may be a need to store thousands of these requests per hour and doing it manually would be a full time job, additionally i am not sure if there is a way to prove that these keys stored in ssskeylog files actually match the certificate of the remote server...
No, you cannot prove the server sent the data. For example, with TLS_RSA_WITH_AES_256_CBC_SHA256 as the ciphersuite, you can forge all data from both sides in an entire TLS session without even contacting the server. And the server side can also produce such a forgery. As noted by EJP, the only thing you can't forge is data that is signed. With RSA ciphersuites the only thing signed are the certificates, but they don't change and are not cryptographically bound to a TLS session.
Other ciphersuites may involve some signed handshake data, but TLS was not designed to provide non-repudiation so I doubt it does by accident.
I developed (in CPP) a unique protocol over HTTP and I use it to communicate with my server.
Now, I want to establish SSL connection over my proprietary protocol to transfer the data itself.
Can I do it using OpenSSL? I searched the web and all I found is something related with BIO, but I didn't understood how to use it for my needs..
Anyway, the best solution for me will be a way I can pass OpenSSL my proprietary send & receive functions so all the communication itself will go only through my functions.
TNX ahead :)
Use BIO pairs. You can find an example in the ssltest.c program, search the source for bio_pair. The basic idea is that you treat the OpenSSL engine as a black box.
There are four things your code has to do:
When you receive encrypted data over the connection to the other side, you have to write it to the SSL engine's encrypted BIO.
When the SSL engine wants to send encrypted data to the other side, you have to read it from the SSL engine's encrypted BIO and transport it to the other side.
When you have plaintext you want to encrypt and send, you have to write it to the SSL engine's plaintext BIO.
When the SSL engine has plaintext it has decrypted for you, you have to read it from the SSL engine's plaintext BIO.
OpenSSL acts purely as an engine following the SSL protocol and moving data between the two BIOs. It does all the protocol negotiation and operations for you, so long as you keep all four of these data streams moving.
One caution I can give you is this -- do not assume any special relationship between these things. For example, you might have some plaintext you want to encrypt and send, and when you write it to the SSL engine's plaintext BIO, the SSL engine might not be able to make forward progress until it receives some encrypted data from the other side. Treat the SSL engine as a black box and do all these four things whenever possible . Do not try to "look through" the SSL engine and, for example, expect that because you handed the SSL engine some encrypted data it will have necessarily plaintext for you. It might, but it might also need to send encrypted data to the other side.
One other caution: the SSL engine has only one state. It does not have a read state and a write state. (Search this thread for "the nightmare scenario" if you want the ugly details.) This is most likely to bite you if you use an SSL connection with multiple threads and expect it to behave just like a TCP connection (where the read and write sides are independent except in the case of a fatal error or connection close).
Second option - a protocol that has its own messages and uses HTTP to
pass them between the client and the server.
If you're using HTTP to pass your own messages, using OpenSSL for SSL/TLS would imply that you'd need to write your own HTTP library library too.
Instead, use an HTTP library that supports HTTPS (most do), via OpenSSL or not. Exchanging your custom messages on top of HTTPS should be fairly transparent and similar to using plain HTTP. You'd just need to configure HTTPS normally.
So here is my 'problem', I set up an FTP server thanks to vsftpd so that both login & data transfers should be encrypted.
Here is the interesting part of my vsftpd.conf file.
ssl_enable=YES
allow_anon_ssl=NO
require_ssl_reuse=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=YES
ssl_sslv3=YES
rsa_cert_file=/etc/vsftpd/vsftpd.pem
rsa_private_key_file=/etc/vsftpd/vsftpd.pem
ssl_ciphers=HIGH
I am using Filezilla as an FTP client, the connection is configured like this :
Protocol : FTP - File Transfer Protocol
Encryption : Require explicit FTP over TLS
Logon type: Normal
Some things to note :
Encryption : Plain FTP : does not work and I am happy with that.
(Response: 530 Non-anonymous sessions must use encryption.)
Encryption : Require implicit FTP over TLS : does not work either, the connection is refused by the server. I guess it is because I forced the SSL connection.
Now, once the (explicit) connection is established, Filezilla is showing a small lock icon at the bottom of the window saying The connection is encrypted. Click icon for details.
I wanted to make sure that the data transfers were indeed encrypted and not plain so I captured everything on my eth. card with Wireshark while downloading a file from my server to my computer.
Except that I can not find a single packet of SSL protocol, everything is TCP.
I am out of ideas on how to make sure the data transferred is encrypted, even if filezilla says so, and each time I google "vsftpd how to make sure data transfers are encrypted", the only answers I get is "ssl_enable=YES" or "Check the box Use SSL" ...
Thank you in advance for helping me !
After a little more research and especially after following the Complete walk through on http://wiki.wireshark.org/SSL, I have a better understanding of the whole thing.
I am answering to my own question hoping this will help someone someday, as long as what follows is correct...
Also writing this down is a good way for me, I think, to see if have clearly understood my problem. Any difficulties in writing this answer will prove me wrong.
First :
Typically, SSL uses TCP as its transport protocol.
SSL is wrapped in TCP, that is why I couldn't observe explicitly the SSL protocol while capturing packets.
When analyzing a TCP packet, I could only "Follow TCP stream" but not "Follow SSL stream" which mislead me into thinking the packet was not holding encrypted data. That is funny because the observable data was not human readable ... so encrypted.
To be able to decrypt it I had to provide wireshark the encryption key :
RSA keys list
This option specifies the bindings between an IP address, a port, a protocol and a decryption key.
Then, I could observe both encrypted / unencrypted data.
Also, after reading this on http://wiki.filezilla-project.org/ :
When you apply encryption to your FTP server the CPU will have to do many calculations to encrypt the data being sent and decrypt the data being received.
I simply decided to run the UNIX top command while downloading a file. I was able to observe a high CPU usage of the filezilla client process, contrary to a unencrypted data transfer. This was a second argument that confirmed the data transfered were indeed encrypted, and thus needed to be decrypted.
I have a client application that runs on my computer, which uses openssl to communicate with server using SSLv3. I need to see what they talk so I basically want to edit and recompile openssl library in order to acquire relevant encryption keys.
The application usually uses TLS-DHE-RSA-WITH-AES-256-CBC-SHA as cipher suite. Sometimes it uses aes128 instead of aes256 but that's not critical at this stage. I edited s3_enc.c file so far. At the end of ssl3_setup_key_block function I write the master key, key block(so the client-server write keys, mac keys and IV's), server and client randoms to a file. http://www.ietf.org/rfc/rfc2246.txt says if the stream cipher is exportable, client-server write keys are to be updated as final client-server write keys accordingly. I'm not sure if this cipher suite is considered exportable(whatever that means), but I create final write keys anyway.
The problem is whatever I tried, I couldn't successfully decrypt any packet which I acquired using wireshark. The output is meaningless no matter what and I'm stuck. I am considering checking the mac fields of packets after decryption in order to see whether decryption was correct. But that also seems to be a dead end.
Does anyone have any idea what I am missing, what I might be doing wrong?
If you can modify OpenSSL and make your client application use it, then all you need to do is make your modified OpenSSL dump the decrypted data to the file. Even easier alternative would be to make OpenSSL use one of NULL ciphersuites even when the application requests certain encryption.