I developed (in CPP) a unique protocol over HTTP and I use it to communicate with my server.
Now, I want to establish SSL connection over my proprietary protocol to transfer the data itself.
Can I do it using OpenSSL? I searched the web and all I found is something related with BIO, but I didn't understood how to use it for my needs..
Anyway, the best solution for me will be a way I can pass OpenSSL my proprietary send & receive functions so all the communication itself will go only through my functions.
TNX ahead :)
Use BIO pairs. You can find an example in the ssltest.c program, search the source for bio_pair. The basic idea is that you treat the OpenSSL engine as a black box.
There are four things your code has to do:
When you receive encrypted data over the connection to the other side, you have to write it to the SSL engine's encrypted BIO.
When the SSL engine wants to send encrypted data to the other side, you have to read it from the SSL engine's encrypted BIO and transport it to the other side.
When you have plaintext you want to encrypt and send, you have to write it to the SSL engine's plaintext BIO.
When the SSL engine has plaintext it has decrypted for you, you have to read it from the SSL engine's plaintext BIO.
OpenSSL acts purely as an engine following the SSL protocol and moving data between the two BIOs. It does all the protocol negotiation and operations for you, so long as you keep all four of these data streams moving.
One caution I can give you is this -- do not assume any special relationship between these things. For example, you might have some plaintext you want to encrypt and send, and when you write it to the SSL engine's plaintext BIO, the SSL engine might not be able to make forward progress until it receives some encrypted data from the other side. Treat the SSL engine as a black box and do all these four things whenever possible . Do not try to "look through" the SSL engine and, for example, expect that because you handed the SSL engine some encrypted data it will have necessarily plaintext for you. It might, but it might also need to send encrypted data to the other side.
One other caution: the SSL engine has only one state. It does not have a read state and a write state. (Search this thread for "the nightmare scenario" if you want the ugly details.) This is most likely to bite you if you use an SSL connection with multiple threads and expect it to behave just like a TCP connection (where the read and write sides are independent except in the case of a fatal error or connection close).
Second option - a protocol that has its own messages and uses HTTP to
pass them between the client and the server.
If you're using HTTP to pass your own messages, using OpenSSL for SSL/TLS would imply that you'd need to write your own HTTP library library too.
Instead, use an HTTP library that supports HTTPS (most do), via OpenSSL or not. Exchanging your custom messages on top of HTTPS should be fairly transparent and similar to using plain HTTP. You'd just need to configure HTTPS normally.
Related
Suppose I am sending "hello" to an api over ssl. My understanding is there would be a symmetric key exchange established over ssl and then the message "hello" will be encrypted using that symmetric key and sent over to the other server.
Now my question is, the next time I send a "hello 2", does the symmetric key exchange happen again? My guess would be that if it's a persistent connection, there would be no need for the key exchange again. Can someone confirm?
Meta: this doesn't appear to me to be programming, although it might be development and is mostly dupe How long does SSL connection between a client and a server persist? .
It depends on the application protocol used on top of SSL (which since 1999 is really TLS, although many things e.g. implementation classes still use the old name) and usually the implementations at both ends. For example, HTTP/1.1 defaults to connection persistence (which was often done in 1.0 as an extension called keep-alive), but either endpoint can change this by specifying connection: close, and even if the connection is kept open can choose to close it anytime later, perhaps after a minute or two, perhaps after a day or a week. The HTTPS implementation in browsers usually keeps connections open for a little while but has limits on the total connections open so those that haven't been used recently may need to be closed when others are opened. Other applications, libraries, and platforms vary. Other protocols also vary; for example an email agent using SMTPS would normally make a connection, transmit one or more emails, and then disconnect.
In addition, SSL-now-TLS through 1.2 supports session resumption, which allows the keyexchange (and other handshake results) performed on one connection to be saved (at both endpoints, or with the 'ticket' option, at client only) and reused on a new connection, for as long as the endpoint(s) agree; implementations usually call this session caching. See e.g. RFC 5246 section 7.3 specifically the part starting in the middle of page 36, and for one fairly common server (Apache) see SessionCache and SessionCacheTimeout directives. Resumption uses a new handshake but not a full keyexchange on that handshake.
However, this creates a security vulnerability if an endpoint's sesssion cache is compromised, so TLS 1.3 replaces it with a different method using dynamically created PSKs; see RFC 8446 section 2.2. This allows either a partial handshake (doing the actual keyexchange with [EC]DHE but authentication tied to the previous session by the PSK rather than full certificate-based authentication) which provides forward secrecy, or a minimal handshake (using the PSK both as the new initial secret and for authentication) which does not.
If you want an answer for specific software, and specific server(s), you need to look at the capabilities and configuration, and often also the current status, of that software and those server(s).
Problem:
I'm pretty new to this TLS thing and networking in general. But i'm trying to passively listen and decode TLS packets. I find it kind of frustrating that i cant decode packets on my network on my computer. The conclusion i got to is that is impossible because how TLS works. But in mathematics if you have enough of information you can often get the value of the unknown. In this instance i guess it might be the servers key that is the missing link.
Even if it would be cool to totally decrypt packets its not my main goal. What i specifically trying to do is to know if a specific twitch stream is watched on my computer/network. So if i type in the streamers name in to the program it checks if that stream is sent to my computer and responds with a boolean.
So i want your input guys and girls. Is there any way of achieve this without doing Man in the middle?
Thoughts:
I have been thinking about this problem and i got a couple of ideas (don't laugh at the possible stupidity, I'm not always very smart but its just some things that came to my mind):
(Comparing) Using my cert to send request to the twitch api and somehow compare the response and see if they have the same encrypted payload. Then i know this specific stream is watched.
(Decoding) feed cert (public and private key), decrypted message and encrypted message to some algorithm to get server key. My theory here is that i have enough information to get the server key. But my knowledge about ssl is limited and i have not considered handshakes and such.
(Decoding) Generate training data to an ai to teach it to decode ssl packets. Generating a lot of ssl packets with different keys and feed the cert and encrypted data to the ai.
I hope you understand what I'm trying to do. English is not my native language (sorry).
(Comparing) Using my cert to send request to the twitch api and somehow compare the response and see if they have the same encrypted payload. Then i know this specific stream is watched.
Data are encrypted with a symmetric key specific for this SSL/TLS session. Also a random initialization vector is used. Thus, even transfer of exactly the same data results in different encrypted data which means no comparison of the encrypted data will help.
(Decoding) feed cert (public and private key), decrypted message and encrypted message to some algorithm to get server key. My theory here is that i have enough information to get the server key. But my knowledge about ssl is limited and i have not considered handshakes and such.
If (the obsolete) RSA key exchange is used you would need to have the servers private key to decrypt which you don't have. With Diffie-Hellman key exchange even this private key would not be sufficient, but you need to have the master secret or pre-master secret of this specific SSL/TLS session (see the TLS standards for details what this is). This secret can only be found in the TLS client and TLS server for the time the TLS session is active - which means that you will not be able to use this either for decryption unless you have access to the internals of the client, in which case you might just look directly which stream they are viewing.
(Decoding) Generate training data to an ai to teach it to decode ssl packets. Generating a lot of ssl packets with different keys and feed the cert and encrypted data to the ai.
Properly encrypted data are more or less random and have no inherent structure which can be mapped to the original data or even parts of the data. AI will not magically find such a structure too. The only possible difference it might use to distinguish streams are the size and timing of the data - but only if these are specific for a stream which I doubt. So, most likely it is impossible this way too.
I find it kind of frustrating that i cant decode packets on my network on my computer. The conclusion i got to is that is impossible because how TLS works.
Indeed this is primary purpose of TLS. If correctly implemented you should have no way of decrypting transfered data.
There are even buggy and obsolete implementation on some servers where one can exploit some vulnerabilities, but - not for Twitch and not with your knowledge level only for buggy implementation
Generally you could consider TLS secure enough to ensure traffic integrity and confidentiality.
What i specifically trying to do is to know if a specific twitch stream is watched on my computer/network.
I'd provide some options to achieve the goal, however that would introduce dangerous vulnerabilities into your computer or network. (you have been warned)
Still you can sniff out the unencrypted traffic. So you can see e. g. DNS requests (you could see what hostnames are resolved, but not specific URL).
For your computer you could install a keylogger or get the requested URL from browser before it is encrypted.
Integrity of TLS rely on certification authorities. In theory could can create your own CA certificate and make it trusted on your computer or network. That could allow you mounting a man-in-the-middle attack (posing yourself as the target server with your own keys).
I would like to decode and store raw SSL/TLS data in order to prove something in the future, i think it will be better if i describe it by an example:
1) There is a remote REST endpoint that is accessible via SSL and provides some useful data. This endpoint is controlled by some third party and i have no control over it.
2) I issue a GET request to this endpoint and store the raw SSL traffic of the response in one file and then the decoded ssl traffic in another file
3) Additionally i save the SSL certificate of this endpoint in a separate file (i guess this operation would only need to be done once)
If i understand correctly when i have all the data from the steps above and in the future, when another third party asks me for a proof that there was certain data on the remote server at some point in the past i could:
1) Present them with the certificate i saved (step 3). Since this certificate is signed by a well known authority there is almost zero chance that i could forge it
2) Present them with the RAW SSL data along with a way to decode it using the certificate above
3) In the decoded REST response headers remote server time should also be present.
Therefore if i am not mistaken i would be able to prove, with almost no doubt, that at some point in the past there was that certain piece of data on the remote party's server even if the remote server itself is no longer operational.
Therefore i can use raw SSL data to prove that something existed on the remote server in past. And there is very little chance of someone challeging this claim because forging the response data or headers would mean that i was able to break the SSL encryption.
Is there a way to automate such scenarios? I mean is there a way to record raw SSL data in a form that would allow it to be easily decoded later?
I have tried using firefox/chrome browser along with the sslkeylog files and sniffing the traffic using wireshark - it actually works, but is there a way to automate it using some kind of REST library since using firefox and recording SSL keys each time can be troublesome - there may be a need to store thousands of these requests per hour and doing it manually would be a full time job, additionally i am not sure if there is a way to prove that these keys stored in ssskeylog files actually match the certificate of the remote server...
No, you cannot prove the server sent the data. For example, with TLS_RSA_WITH_AES_256_CBC_SHA256 as the ciphersuite, you can forge all data from both sides in an entire TLS session without even contacting the server. And the server side can also produce such a forgery. As noted by EJP, the only thing you can't forge is data that is signed. With RSA ciphersuites the only thing signed are the certificates, but they don't change and are not cryptographically bound to a TLS session.
Other ciphersuites may involve some signed handshake data, but TLS was not designed to provide non-repudiation so I doubt it does by accident.
So here is my 'problem', I set up an FTP server thanks to vsftpd so that both login & data transfers should be encrypted.
Here is the interesting part of my vsftpd.conf file.
ssl_enable=YES
allow_anon_ssl=NO
require_ssl_reuse=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=YES
ssl_sslv3=YES
rsa_cert_file=/etc/vsftpd/vsftpd.pem
rsa_private_key_file=/etc/vsftpd/vsftpd.pem
ssl_ciphers=HIGH
I am using Filezilla as an FTP client, the connection is configured like this :
Protocol : FTP - File Transfer Protocol
Encryption : Require explicit FTP over TLS
Logon type: Normal
Some things to note :
Encryption : Plain FTP : does not work and I am happy with that.
(Response: 530 Non-anonymous sessions must use encryption.)
Encryption : Require implicit FTP over TLS : does not work either, the connection is refused by the server. I guess it is because I forced the SSL connection.
Now, once the (explicit) connection is established, Filezilla is showing a small lock icon at the bottom of the window saying The connection is encrypted. Click icon for details.
I wanted to make sure that the data transfers were indeed encrypted and not plain so I captured everything on my eth. card with Wireshark while downloading a file from my server to my computer.
Except that I can not find a single packet of SSL protocol, everything is TCP.
I am out of ideas on how to make sure the data transferred is encrypted, even if filezilla says so, and each time I google "vsftpd how to make sure data transfers are encrypted", the only answers I get is "ssl_enable=YES" or "Check the box Use SSL" ...
Thank you in advance for helping me !
After a little more research and especially after following the Complete walk through on http://wiki.wireshark.org/SSL, I have a better understanding of the whole thing.
I am answering to my own question hoping this will help someone someday, as long as what follows is correct...
Also writing this down is a good way for me, I think, to see if have clearly understood my problem. Any difficulties in writing this answer will prove me wrong.
First :
Typically, SSL uses TCP as its transport protocol.
SSL is wrapped in TCP, that is why I couldn't observe explicitly the SSL protocol while capturing packets.
When analyzing a TCP packet, I could only "Follow TCP stream" but not "Follow SSL stream" which mislead me into thinking the packet was not holding encrypted data. That is funny because the observable data was not human readable ... so encrypted.
To be able to decrypt it I had to provide wireshark the encryption key :
RSA keys list
This option specifies the bindings between an IP address, a port, a protocol and a decryption key.
Then, I could observe both encrypted / unencrypted data.
Also, after reading this on http://wiki.filezilla-project.org/ :
When you apply encryption to your FTP server the CPU will have to do many calculations to encrypt the data being sent and decrypt the data being received.
I simply decided to run the UNIX top command while downloading a file. I was able to observe a high CPU usage of the filezilla client process, contrary to a unencrypted data transfer. This was a second argument that confirmed the data transfered were indeed encrypted, and thus needed to be decrypted.
I'm working on an iOS app, where it needs to connect to a web service that located on a web server. And it needs to give an arguments to some .php files that will receive these arguments via the POST method. The problem is that I don't want anyone to look/know these arguments, and what I've tried is that WireShark can discloses/sniffs the values of these arguments.
So, what is the suitable approach to secure the connection?, I thought about encrypting the data befor sending them to the server (using simle secrete/single key, since my data are not highly sensitive) and the same for fetching data from the server, if I'd go with this approach, then I'll need to implement my own classes, but I'm running out of time.
Thanks in advance :)
Https is an option here, but won't hide data from a determined attacker. An attacker can MITM himself to reverse engineer whatever protocol it is that you are using to communicate from iphone to server, and even modify requests in real time!
Here's a good example of how a researcher was able to reverse engineer's Apple's Game Center protocol and artificially set his own high score in an iOS app:
http://corte.si/posts/code/mitmproxy/tute-gamecenter/index.html
If you are concerned about this type of attack, then you should encrypt your payload using a pre-shared key and a symmetric cipher like AES-256 or blowfish.
Use https instead. Your data are not highly sensitive, so this should be enough. You may not even have to change your iPhone code (other than the URL change), although this depends on your server's certificate.