I'm currently working with the Amazon S3 API, and have a general wondering about the server-side integrity checks that can be done if you provide the MD5 hash during posting of an object.
I'm not sure I understand if the integrity check is required if you send the data (I'm assuming the object data you're posting also) via SSL/TLS, which provide their own support for data integrity in transit.
Should you send the digest regardless if you're posting over SSL/TLS? Isn't it superfluous to do so? Or is there something I'm missing?
Thanks.
Integrity checking provided by TLS provides no guarantees about what happens going into the TLS wrapper at the sender side, or coming out of it and being written to disk at the receiver.
So, no, it is not entirely superfluous because TLS is not completely end-to-end -- the unencrypted data is still processed, however little, on both ends of the connection... and any hardware or software that touches the unencrypted bits can malfunction and mangle them.
S3 gives you an integrity checking mechanism -- two, if you use both Content-MD5 and x-amz-content-sha256 -- and it seems unthinkable to try to justify bypassing them.
Related
I am developing a client server pair of applications to transfer files by streaming bytes over TCP/IP and the channel would use TLS always.
(Note: Due to certain OS related limitations SFTP or other such secure file transfer protocols cannot be used)
The application level protocol involves minimum but sufficient features to get the file to the other side.
I need to decide if the application level protocol needs to implement an integrity check (Ex: MD5).
Since TLS guarantees integrity, would this be redundant?
The use of TLS can provide you with some confidence that the data has not been changed (intentionally or otherwise) in transit, but not necessarily that the file that you intended to send is identical to the one that you receive.
There are plenty of other opportunities for the file to be corrupted/truncated/modified (such as when it's being read from the disk/database by the sender, or when it's written to disk by the receiver). Implementing your own integrity checking would help protect against those cases.
In terms of how you do the checking, if you're worried about malicious tampering then you should be checking a cryptographic signature (using something like GPG), rather than just a hash of the file. If you're going to use a hash then it's generally recommended to use a more modern algorithm such as a SHA-256 rather than the (legacy) MD5 algorithm - although most of the issues with MD5 won't affect you if you're only concerned about accidental corruption.
I'm using Server-Side Encryption with Customer-Provided Encryption Keys (SSE-C) to store some files. I want to download them but not decrypt them just yet. The use case is something like the Game of Thrones finale. I want cable operators to have the data but give them the key in the last second. But the decrypt headers are mandatory when the file is encrypted. Maybe I can toggle the mark that the file is encrypted?
For this application, you wouldn't use any variant of SSE.
SSE prevents your content from being stored on S3's internal disks in a form where accidental or deliberate compromise of those physical disks or their raw bytes -- however unlikely -- would expose your content to unauthorized personnel. That is fundamentally the purpose of all varieties of SSE. The variants center around how the keys are managed.
Server-side encryption is about data encryption at rest—that is, Amazon S3 encrypts your data at the object level as it writes it to disks in its data centers and decrypts it for you when you access it.
https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
SSE is decrypted by S3 and transiently re-encrypted using TLS for transmission on the network during the download. The final result in the client's hands is unencrypted.
For the application described, you would just upload the encrypted content to S3 without S3 being aware of the (external, already-applied) encryption.
If you also used some kind of SSE, that would be unrelated to the external encryption that you would also apply. Arguably, SSE would be somewhat redundant if the content is already encrypted before upload.
In fact, in the application described, depending on sensitivity and value of the content, each recipient would potentially have different keys and/or a slightly different source file (thus a substantially different encrypted file), so that the source of a leak could be identified by identifying which source variant was compromised.
For a personal MMO game project I am implementing a homebrew reliable UDP-based protocol in java. Given my current setup I beleive it would be relatively simple for a snooper to hijack a session, so in order to prevent this I am taking the opportunity to learn a little cryptology. Its very interesting.
I can successfully create a shared secret key between the client and server using a Diffie-Hellman key exchange (a very clever concept), but now I need to use this to guarantee the authenticity of the packets. My preliminary testing so far has shown that the couple of different ciphers Ive tried bloat the amount of data a bit, but I would like to keep things as small and fast as possible.
Given that I am only trying to authenticate the packet and not nessecarily conceal the entire payload, I have the idea that I could put an 8 byte session ID generated from the secret key into the packet header, encrypt the whole packet, and hash it back down to 8 bytes. I take the unencrypted packet and put the 8 byte hash into the place of the session ID and then send it off.
Would this be secure? It feels a little inelegant to encrypt the whole packet only to send it unencrypted - is there a better/faster way to achieve my goal? Please note I would like to do this myself since its good experience so Im not so interested in 3rd party libraries or other protocol options.
If both peers have access to a shared secret (which they should, since you're talking about Diffie-Hellman), you could simply store a hash of the datagram in its header. The receiver checks to see if it matches.
As an added security measure, you could also add a "challenge" field to your datagram and use it somewhere in the hashing process to prevent replays.
So this hash should cover:
The shared secret
A challenge
The contents of the datagram
EDIT
The "challenge" is a strictly incrementing number. You add it to your datagram simply to change the hash every time you send a new message. If someone intercepts a message, it cannot resend it: the receiver makes sure it doesn't accept it.
I need to implement a very secured Web Service using WCF. I have read a lot of documents about security in WCF concerning authorization, authentication, message encryption. The web service will use https, Windows Authentication for access to the WS, SQL Server Membership/Role Provider for user authentication and authorization on WS operations and finally message encryption.
I read in one of documents that it is good to consider security on each layer indenpendently, i.e. Transport Layer security must be thought without considering Message Layer. Therefore, using SSL through https in combination with message encryption (using public/private key encryption and signature) would be a good practice, since https concerns Transport Layer and message encryption concerns Message Layer.
But a friend told me that [https + message encryption] is too much; https is sufficient.
What do you think?
Thanks.
If you have SSL then you still need to encrypt your messages if you don't really trust the server which stores them (it could have its files stolen), so this is all good practice.
There comes a point where you have a weakest link problem.
What is your weakest link?
Example: I spend $100,000,000 defending an airport from terrorists, so they go after a train station instead. Money and effort both wasted.
Ask yourself what the threat model is and design your security for that. TLS is a bare minimum for any Internet-based communications, but it doesn't matter if somebody can install a keystroke logger.
As you certainly understand, the role of Transport-Level Security is to secure the transmission of the message, whereas Message-Level Security is about securing the message itself.
It all depends on the attack vectors (or more generally the purpose) you're considering.
In both cases, the security models involved can have to purposes: protection against eavesdropping (relying on encryption) and integrity protection (ultimately relying on signatures, since based on public-key cryptography in most cases).
TLS with server-certificate only will provide you with the security of the transport, and the client will know that the communication really comes from the server it expects (if configured properly, of course). In addition, if you use client-certificate, this will also guarantee the server that the communication comes from a client that has the private key for this client certificate.
However, when the data is no longer in transit, you rely on the security of the machine where it's used and stored. You might no longer be able to assert with certainty where the data came from, for example.
Message-level security doesn't rely on how the communication was made. Message-level signature allows you to know where the messages came from at a later date, independently of how they've been transferred. This can be useful for audit purposes. Message-level encryption would also reduce the risks of someone getting hold of the data if it's stored somewhere where some data could be taken (e.g. some intranet storage systems).
Basically, if the private key used to decrypt the messages has the same protection as the private key used for SSL authentication, and if the messages are not stored for longer time than the connection, in that case it is certainly overkill.
OTOH, if you've got different servers, or if the key is stored e.g. using hardware security of sorts, or is only made available by user input, then it is good advice to secure the messages themselves as well. Application level security also makes sense for auditing purposes and against configuration mistakes, although personally I think signing the data (integrity protection) is more important in this respect.
Of course, the question can also become: if you're already using a web-service that uses SOAP/WSDL, why not use XML encrypt/sign? It's not that hard to configure. Note that it does certainly take more processor time and memory. Oh, one warning: don't even try it if the other side does not know what they are doing - you'll spend ages explaining it and even then you run into trouble if you want to change a single parameter later on.
Final hint: use standards and standardized software or you'll certainly run into crap. Spend some time getting getting to know how things work, and make sure you don't accept ill formatted messages when you call verify (e.g. XML signing the wrong node or accepting MD5 and such things).
I want to secure the communication of a TCP-based program using a shared passphrase/key. The easiest way to do that without having to deal with block size, padding, ... is to directly use a stream cipher. Doing that way, the amount of data is not changed between clear and encrypted data and the modification is trivial.
Using only a stream cipher means that there is no authentication and I have always considered/heard that encryption without authentication is not secure enough and should not be used.
If adding authentication to a stream cipher is mandatory, we lose the simplicity that stream cipher has added because we must add an HMAC or use an authenticated encryption method (like crypto_secretbox from NaCl), there is a minimum message length, we must handle padding, ...
What would you recommend? Is it safe to only use stream cipher without authentication in some particular cases?
Using some kind of message authenticator is particularly important with stream ciphers, because the relationship between changes to the ciphertext and changes to the plaintext is so simple.
You can't just blindly go and apply the stream cipher without adding any extra information to the stream, anyway - remember the most important rule of stream ciphers:
NEVER RE-USE THE SAME KEYSTREAM
So unless you are only ever going to encrypt a single connection, and throw the passphrase away afterwards, you will need to generate a session key for each connection from the shared secret. This implies that you will need to send some extra information at the start of the connection, and since you're sending that anyway, sending a HMAC after each message should be no big deal.
Using a stream cipher because it seems simpler is usually a mistake, anyway. You mentioned crypto_secretbox from NaCl - I recommend using that, it will take care of the authentication and padding issues for you.
You could consider using AES in GCM-mode. That will give you a stream-cipher with built-in authentication.