We have a OpenSSL running on our embedded system, which is running ECOS OS. We are now upgrading our OpenSSL to 1.0.2 version. We have successfully ported and compiled the OpenSSL library. But when when we try to connect our device using SSL (via https), handshake fails with bad record mac alert always. We have enabled OpenSSL debug option, but unable to identify why its failing.
Have someone ported latest OpenSSL code to ECOS? Do we need to take of any special compilation flags with latest OpenSSL code for ECOS?
For reference, here is the relevant part of ssl3_get_record:
mac = rr->data + rr->length;
i=s->method->ssl3_enc->mac(s,md,0 /* not send */);
if (i < 0 || CRYPTO_memcmp(md, mac, (size_t)mac_size) != 0)
{
al=SSL_AD_BAD_RECORD_MAC;
SSLerr(SSL_F_SSL3_GET_RECORD,SSL_R_DECRYPTION_FAILED_OR_BAD_RECORD_MAC);
goto f_err;
}
After debugging we found that the random library (RAND) was failing for ECOS. There were lot of places in OpenSSL where it checks for random_bytes return type. Due to this failure, pre-master key decryption was failing. And incoming packets were not decrypted properly. Hence a BAD Mac records error was seen.
We also checked with our old ported code (0.9.6), RAND library was failing there also, but there we no return check for random_bytes and pseudo_rand_bytes. As a fix we made RAND to return success every time, and we can see SSL session being established fine with OpenSSL 1.0.2 version.
Related
Since Twitter changed their website design, I cannot get a set of tweets from any account by using built-in Zinc classes. It throws an error that says: ConnectionClosed: Connection closed while waiting for data
I am using Pharo 5, and I don't know how to tweak the ZnClient settings in order to keep the connection open or something to the purpose of getting the data.
testTwitter
| client |
self ensureSocketStreamFactory.
self isNativeSSLPluginPresent ifFalse: [ ^ self ].
(client := ZnClient new)
get: 'https://www.twitter.com/pharoproject'.
self assert: client isSuccess.
self assert: (client contents includesSubstring: 'Twitter').
client close
That's the test I have in place, it never passes, and throws the error mentioned above. What's missing here? I did a Ruby script using open-uri, openssl and Nokogiri and it fetched the tweets just fine. Perhaps it's a problem with the SSL connection itself?
The issue here is quite easy to answer, but you won't like it. Your issue is connected to the fact that the Twitter has deprecated support for TLS 1.0, TLS 1.1 on July 15/2019. Your pharo is using the deprecated TLS to connect. That is the reason why you are getting the timeout.
The solution?
You have to compile the new SSL/TLS support yourself which is not an easy task to do. You have to compile in at least TLS 1.2 to be able to connect again. There is lack of Pharo documentation how to compile support for new libraries. My guess is that you are using TLS 1.0 (see a note below) - since Pharo 6.1 (so your Pharo 5.x will have same or older libraries) has libgit2.so compiled against libssl.so.1.0.0 (which has dependency libcurl-gnutls.so.4) - If you update the libraries you can see that those support >= TLS 1.2.
Note:
This is connected to the issue which I have posted some time ago. Nobody upvoted it or answered so it got automatically deleted - you can vote to undelete it: https://stackoverflow.com/questions/51399321/getting-error-when-adding-ossubprocess-to-my-pharo-6-1-on-centos-7-4x (see the bottom of the post for the question). I don't have an answer for that as I have dedicated time to my Smalltalk/X project.
Or just switch to a newer Pharo. Adding your method to ZnHTTPSTest in Pharo 8 just works (tested on Pharo 8 build 686, Ubuntu 18.04.02 LTS with the stable vm in PharoLauncher)
I'm trying to bring an old TLS 1.0 implementation (that I did not write) up to date to speak TLS 1.2.
As a first step I integrated the TLS 1.1 change of putting the plaintext initialization vector in the record. That was no problem. It seemed to work well enough that I could read https://example.com in TLS 1.1, as well as SSL Labs viewMyClient.html.
Then I adapted to the TLS 1.2 change of the pseudorandom function to (for most practical purposes) P_SHA256 instead of the (more complex and bizarre) half and half MD5/SHA1 rigamarole. I did it wrong the first time and got an invalid MAC error, but it was more or less a typo on my part and I fixed it. Then the invalid MAC error went away.
But despite that, after sending the ClientKeyExchange->ChangeCipherSpec messages, I'm getting a "Decrypt Error" back from the server(s) (same Alert regardless, https://google.com or anything I try). I gather the ChangeCipherSpec message is encrypting just one byte, putting it into a message with padding and the MAC, etc.
If I tweak the MAC randomly by one byte, it goes back to complaining about invalid MAC. What confuses me is that the MAC itself is encrypted as part of GenericBlockCipher:
struct {
opaque IV[SecurityParameters.record_iv_length];
block-ciphered struct {
opaque content[TLSCompressed.length];
opaque MAC[SecurityParameters.mac_length]; // <-- server reads this fine!
uint8 padding[GenericBlockCipher.padding_length];
uint8 padding_length;
};
} GenericBlockCipher;
UPDATE: FWIW, I've added a Wireshark log of the failing 1.2 read of https://example.com, as well as a log of a functioning 1.1 session running what is the same code, not counting the P_SHA256 MAC update:
http://hostilefork.com/media/shared/stackoverflow/example-com-tls-1.2.pcapng (fails)
http://hostilefork.com/media/shared/stackoverflow/example-com-tls-1.1.pcapng (succeeds)
So, what exactly is it having trouble decrypting? The padding seems correct, as if add or subtract 1 to the byte I get an invalid MAC error. (The spec says "The receiver MUST check this padding and MUST use the bad_record_mac alert to indicate padding errors.", so that is to be expected.) If I corrupt the client-iv in the message from what I used to encrypt (just put a bad byte in the transmitted version), doing so also gives me Bad Record MAC. I'd expect that to wreck the decryption also.
So I'm puzzled on what could be the problem:
The server demonstrates discernment of valid MAC vs. not, so it must have decrypted. How's it getting the right MAC -and- having a decrypt error?
Cipher suite is an old one (TLS_RSA_WITH_AES_256_CBC_SHA) but I'm just tackling one issue at a time...and if I'm not mistaken, that shouldn't matter.
Does anyone with relevant experience have a theory of how TLS 1.2 could throw a wrench into code that otherwise works in TLS 1.1? (Perhaps someone who's done a similar updating to a codebase, and had to change more than the two things I've changed to get it working?) Am I missing another crucial technical change? What recourse do I have to find out what is making the server unhappy?
There's actually not anything wrong with the ChangeCipherSpec message. It's actually the Finished message that has the problem. It is complaining about the decrypted verify_data inside that message, which is not matching an expected hash (despite the encryption/decryption itself being correct).
But what's confusing in the Wireshark log is that the Finished message shows up on the same log line, but under the name "EncryptedHandshakeMessage" This makes it look like some kind of tag or label describing ChangeCipherSpec, but it's not. That message actually isn't encrypted at all.
TLS finished packet renamed encrypted handshake message
HTTPS over TLS - encrypted type
From the second link:
In practice, you will see unencrypted Client Hello, Server Hello, Certificate, Server Key Exchange, Certificate Request, Certificate Verify and Client Key Exchange messages. The Finished handshake message is encrypted since it occurs after the Change Cipher Spec message.
"Hoping someone has experience updating TLS 1.0 or 1.1 to 1.2, and might have seen a similar problem due to not changing more than the P_SHA256 MAC and bumping the version number"
They only mention two of the three places that you need to update the MD5/SHA1 combination in the "changes from TLS 1.1" section of RFC 5246:
The MD5/SHA-1 combination in the pseudorandom function (PRF) has been replaced with cipher-suite-specified PRFs. All cipher suites in this document use P_SHA256.
The MD5/SHA-1 combination in the digitally-signed element has been replaced with a single hash. Signed elements now include a field that explicitly specifies the hash algorithm used.
(Note: The second applies to certificates, and if you haven't gotten to certificate checking you wouldn't be at that point yet.)
What they don't mention in that section is the third place the MD5/SHA-1 combination changes, which is a hash used in the seed for the verify_data of the Finished message. However, this point is also a change from TLS 1.1, described much further down the document in section 7.4.9:
"Hash denotes a Hash of the handshake messages. For the PRF defined in Section 5, the Hash MUST be the Hash used as the basis for the PRF. Any cipher suite which defines a different PRF MUST also define the Hash to use in the Finished computation."
For a formal spec they're being a bit vague on "hash used as the basis for the PRF" (is it the HMAC or just the plain hash?) But it's the plain hash. So SHA256, unless the cipher suite's spec says otherwise.
(Note also the cipher suite can dictate the length of the verify_data as more than 12 bytes, though none mentioned in the spec do so.)
"What recourse do I have to find out what is making the server unhappy?"
YMMV. But what I did was just build OpenSSL as a static debug library, and linked it to a simple server. Then I added breakpoints and instrumentation to see what it was upset about. (GDB wasn't letting me step into the shared library, for some reason.)
Circa 30-Sep-2018, on a plain linux machine:
git://git.openssl.org/openssl.git
./config no-shared no-asm -g3 -O0 -fno-omit-frame-pointer -fno-inline-functions no-ssl2 no-ssl3
make
The simple server I used came from Simple TLS Server. Compile against the static library with:
gcc -g -O0 simple.c -o simple -lssl -lcrypto -ldl -lpthread
I followed the instructions for generating certificates here, but changed the AAs to localhost
openSSL sign https_client certificate with CA
Then I changed the cert.pem => rootCA.pem and key.pem => rootCA.key in the simple server code. I was able to do:
wget https://localhost:4433 --no-check-certificate
And successfully get back test as a response. So then it was just a matter of seeing where my client caused a failure.
I can think of 2 different situations that creates this problem:
Sending incorrect IV. IV affects only 1st block in decryption of CBC mode, so if your content is more than 16 bytes (AES block size), MAC part of your data will be decrypted correctly.
If you are using incorrect padding structure, you may get error in decryption(because padding verification fails), but content will be decrypted correctly.
I 'm working on esp8266 module these days.
The firmware on this module is nodemcu 0.9.6 and its language is lua.
I use mqtt protocol for my project and it works well.But TLS is necessary.
I don't know how to set tls on this firmware. I 've searched but documents about TLS on nodemcu website is not enough for me.
I know that I should use this instruction at connection time to broker:
m:connect(BROKER, 8883, 1 , function ( connection ) ... end )
and to put the CA file on module i should use something like this:
enter image description here
that 's what i read from nodemcu website --> https://nodemcu.readthedocs.io/en/master/en/modules/net/#netcertverify
but the problem is when i use this on my code the module doesn't understand it and gives error at upload time and runtime.
is this problem because of the version of my firmware? Upgrading will make it? Or I am just doing something wrong on my code?
I haven't seen sample code about putting CA file on module in other project so I guess my work is wrong.
if you have worked on TLS ( on nodemcu firmware ) I would be happy by your helps.
Build a recent firmware with SSL/TLS enabled.
Flash the binary to upgrade. Make sure all flashing parameters are exactly as documented and include the init data binary.
The rest of your code seems fine.
Take a look at this question and read through comments of the answer.
You may have to get esp_init_data_default.bin from latest sdk.
I used esptool.
esptool.py --port <PORT> write_flash -fm dio -fs 32m 0x00000 <BIN_PATH_AND_FILENAME> 0x3fc000 <PATH\>esp_init_data_default.bin
Please do upvote this post if you find it is useful. Thumbs up for #MarcelStör.
I have an application that supposedly runs well under Mono, but is having some problems on my system. In the meantime, I tried running it through WINE after using winetricks to install the proper version of .NET (winetricks dotnet452).
This worked great! The application hits Github to check for updates and manages the SSL/TLS connection flawlessly. Elsewhere while using it, it attempts to access another website, https://themoose.co.uk, but fails with an SSL/TLS error. The only reasonable difference I could find between that site and Github was that it uses an ECC cert as opposed to Github's more traditional RSA cert.
I also saw these lines in WINE's console output:
fixme:secur32:schan_get_cipher_algid Don't know CALG for encryption algorithm 2, returning 0
fixme:secur32:schan_imp_get_max_message_size Returning 1 << 14.
fixme:secur32:schan_get_cipher_algid Don't know CALG for encryption algorithm 2, returning 0
Googling these messages doesn't return anything useful.
The conclusion I am drawn to is that WINE doesn't support new ECC certificates, but I do not see that limitation documented anywhere! Am I going crazy, or is this an oversight in the documentation somewhere?
I am trying to fetch a file over HTTPS in Io language:
url := URL with("https://api.example.com")
url fetch println
And I get this:
Error_0x7f97e1509a80:
location = "/opt/local/lib/io/addons/Socket/io/URL.io:232"
message = "Protocol 'https' unsupported"
I was trying to find something on the net, but, as everybody knows, it's not easy because of the name. I only found this thread http://tech.groups.yahoo.com/group/iolanguage/message/10898 but that's quite old.
How can I get the HTTPS support in Io?
EDIT
I've found that there is a SecureSocket addon, a wrapper over OpenSSL, in Io's source. It wasn't installed when I did sudo port io install on my MacBook with Mountain Lion, though. I tried building it from source, but no luck. It didn't build for me on a Linux machine, either.
EDIT2
I just tried to build Io from source (git clone https://github.com/stevedekorte/io.git) again (using the included script build.sh) and it turned out that cmake did detect OpenSSL:
-- Found OpenSSL: /usr/lib/libssl.dylib;/usr/lib/libcrypto.dylib
But then the SecureSocket addon is not built. Its readme file: https://github.com/stevedekorte/io/tree/master/addons/SecureSocket says:
The DTLS1 bindings are not usable unless the patches in this file are
applied to OpenSSL 0.9.8e. However, this patch includes a
deactivation of the handshake retransmission code in d1_both.c,
making it unsuitable for production environments. I take no
responsibility, etc, etc. If you want to use it anyway, apply the
patches(gathered from various newsgroups and my own experimentation)
and uncomment the commented-out block of build.io. For what it's
worth, DTLS support in OpenSSL is new as of 0.9.8 and is pretty buggy
to begin with. It's a nice idea, but it doesn't seem to be
production ready at all yet. These bindings are no exception.
If you can't get io to do it your best option would be calling an external tool like wget or curl which can and then loading the file/result locally or returning it via a pipe.
For anybody else interested in another workaround, it should be possible to put stud in front of an Io program which will do the SSL stuff. I have not tested that myself yet.
stud - The Scalable TLS Unwrapping Daemon stud is a network proxy that
terminates TLS/SSL connections and forwards the unencrypted traffic to
some backend. It's designed to handle 10s of thousands of connections
efficiently on multicore machines.