TLS v1.3 error with certain endpoints - Description: Illegal Parameter (47) - ssl

Question

I am working with IBM on trying to find the cause for some handshake errors we are receiving randomly when connecting with certain endpoints. This is happening on an IBM i system using the GSKit SSL APIs. The IBM is acting as the client.
The error we are receiving during the handshake is 415 (Bad Peer).
In the trace we show this response from the remote server:
TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Illegal Parameter)
Content Type: Alert (21)
Version: TLS 1.2 (0x0303)
Length: 2
Alert Message
Level: Fatal (2)
Description: Illegal Parameter (47)
If we remove TLS 1.3 from the communications, the errors do not occur. If we add it back in, then the errors pop up communicating with Google and GoDaddy.
This only seemed to start happening recently after new SSL Certificates were installed at the endpoints. And, as I said, it's random. If I encounter the error, I can make the same request again without any error.
We weren't sure if it was because of how certain servers on their farms are configured differently from others, or if updates are propagating through them... but it's very odd.
We have isolated 2 ciphers that seem to be the issue. When removed, the errors seem to stop. They are:
*ECDHE_RSA_AES_256_GCM_SHA384 - this one seems to cause problems with Google
*AES_256_GCM_SHA384 - this one seems to cause problems with GoDaddy
The tech at IBM said that if we could somehow get in touch with someone at either place and recreate the issue, it would be a great help to figure out what is going on. Just hoping we can make that happen... if anyone knows anyone on the SSL support side of GoDaddy or Google that would be willing to work with myself and IBM to resolve this, it would be great.
Or, if anyone knows why it's happening, we're all ears. Right now the only option we have is turning off TLS v1.3 or removing these two ciphers. (and there may be more that affect other endpoints that we just don't know about yet).
TIA!
UPDATE
According to my tech, they are not following the RFC properly:
RFC 5246 (the last sentence):
The cipher suite list, passed from the client to the server in the
ClientHello message, contains the combinations of cryptographic
algorithms supported by the client in order of the client's
preference (favorite choice first). Each cipher suite defines a key
exchange algorithm, a bulk encryption algorithm (including secret key
length), a MAC algorithm, and a PRF. The server will select a cipher
suite or, if no acceptable choices are presented, return a handshake
failure alert and close the connection. If the list contains cipher
suites the server does not recognize, support, or wish to use, the
server MUST ignore those cipher suites, and process the remaining
ones as usual.
UPDATE 2
Here is a handshake that fails:
TLSv1.2 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.2 (0x0303)
Length: 549
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 545
Version: TLS 1.2 (0x0303)
Random: 16ab36ccafcb18074cf7ce0296745cb87a4ef732402fbdc273790f082d4844d1
Session ID Length: 0
Cipher Suites Length: 18
Cipher Suites (9 suites)
Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
Cipher Suite: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)
Compression Methods Length: 1
Compression Methods (1 method)
Extensions Length: 486
Extension: supported_versions (len=9)
Type: supported_versions (43)
Length: 9
Supported Versions length: 8
Supported Version: TLS 1.3 (0x0304)
Supported Version: TLS 1.2 (0x0303)
Supported Version: TLS 1.1 (0x0302)
Supported Version: TLS 1.0 (0x0301)
Extension: renegotiation_info (len=1)
Type: renegotiation_info (65281)
Length: 1
Renegotiation Info extension
Extension: ec_point_formats (len=2)
Type: ec_point_formats (11)
Length: 2
EC point formats Length: 1
Elliptic curves point formats (1)
EC point format: uncompressed (0)
Extension: signature_algorithms (len=22)
Type: signature_algorithms (13)
Length: 22
Signature Hash Algorithms Length: 20
Signature Hash Algorithms (10 algorithms)
Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
Extension: signature_algorithms_cert (len=20)
Type: signature_algorithms_cert (50)
Length: 20
Signature Hash Algorithms Length: 18
Signature Hash Algorithms (9 algorithms)
Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
Extension: supported_groups (len=12)
Type: supported_groups (10)
Length: 12
Supported Groups List Length: 10
Supported Groups (5 groups)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)
Supported Group: x25519 (0x001d)
Supported Group: secp521r1 (0x0019)
Supported Group: x448 (0x001e)
Extension: key_share (len=71)
Type: key_share (51)
Length: 71
Key Share extension
Client Key Share Length: 69
Key Share Entry: Group: secp256r1, Key Exchange length: 65
Group: secp256r1 (23)
Key Exchange Length: 65
Key Exchange: 047fbc26145d5067052eff17103285a7bdc30952cdbcea601491a6a08eca7d424484a9cb…
Extension: server_name (len=20)
Type: server_name (0)
Length: 20
Server Name Indication extension
Extension: extended_master_secret (len=0)
Type: extended_master_secret (23)
Length: 0
Extension: psk_key_exchange_modes (len=2)
Type: psk_key_exchange_modes (45)
Length: 2
PSK Key Exchange Modes Length: 1
PSK Key Exchange Mode: PSK with (EC)DHE key establishment (psk_dhe_ke) (1)
Extension: pre_shared_key (len=283)
Type: pre_shared_key (41)
Length: 283
Pre-Shared Key extension
Identities Length: 230
PSK Identity (length: 224)
Identity Length: 224
Identity: 000017719425bdded0ac21c8cd61318334f131527b4ef48f21ba8116523cf71681ae51fa…
Obfuscated Ticket Age: 68965835
PSK Binders length: 49
PSK Binders
The server then responds with
TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Illegal Parameter)
Content Type: Alert (21)
Version: TLS 1.2 (0x0303)
Length: 2
Alert Message
Level: Fatal (2)
Description: Illegal Parameter (47)

Answer 1

IBM released an APAR with PTF for V7R3 and V7R4.
The APAR number is MA49492
Regards Michael

Answer 2

You can review encryption protocols that Google Front End supports when communicating with clients in this whitepaper.
In this whitepaper, you will find more detail on encryption in transit for Google Cloud, including Google Cloud Platform and Google Workspace.
From table 1, you can check that the cipher ECDHE_RSA_AES_256_GCM_SHA384, which is having trouble connecting with Google, is not listed as compatible.
I have found this documentation from IBM that lists the supported CipherSpecs (I’m not quite sure if this applies to your IBM product). Also, there’s a troubleshooting TLS guide from IBM.
It is important to understand this information is not a comprehensive source for troubleshooting information, but rather a guide to aid in common problem resolution.
UPDATE
It's unclear if IBM's GSKit SSL APIs are generating the error, or if it is allowing the user to send invalid packets and then reporting the error.
Unsupported TLS 1.2 ciphers would be ignored in the handshake as stated here, but I believe there is just 5 ciphers in TLS 1.3 (see RFC 8446 - Cipher Suites and OpenSSL - Ciphersuites), and ECDHE_RSA_AES_256_GCM_SHA384 is not in them, so it would be strange if IBM's product allowed an invalid cipher to be requested.
The 1.3 cipher specs work fine with Google servers, like:
echo | openssl s_client -tls1_3 -connect SERVER:443 -ciphersuites TLS_CHACHA20_POLY1305_SHA256 | grep "New, TLSv1.3, Cipher"
As mentioned above, ECDHE_RSA_AES_256_GCM_SHA384 is not in the list of TLS 1.2 cipher specifications.
UPDATE 2
From the whitepaper, I noted:
TLS in the GFE is implemented with BoringSSL. Table 1 shows the encryption protocols that GFE supports when communicating with clients.
Protocols
Authentication
Key exchange
Encryption
Hash Functions
TLS 1.34
RSA 2048
Curve25519
AES-128-GCM
SHA384
TLS 1.2
ECDSA P-256
P-256 (NIST secp256r1)
AES-256-GCM
SHA256
The supported values above don't match with the Cipher Suite list provided shown on the handshake packet capture, note the different SHA values:
TLS_AES_128_GCM_SHA256 (0x1301)
TLS_AES_256_GCM_SHA384 (0x1302)
The values used are AES128 with SHA256, and AES256 with SHA384 but BoringSSL supports AES128 with SHA384 for TLS 1.3 and AES256 with SHA256 for TLS 1.2.
Looking at some more IBM docs for z/OS, I'm seeing a discrepancy with 0x1302 which showcases AES256 with SHA256.
For TLS V1.3, the following is allowed in the evaluated configurations:
Ciphers:
1301 - TLS_AES_128_GCM_SHA256
1302 - TLS_AES_256_GCM_SHA256
How are the SSL certificates used? Is it possible to share the link for those docs?
Throubleshoot
I would recommend to try the following:
Find out what is causing the randomness. Maybe there could be some propagation delay on your IBM system or the SSL distribution management.
Try changing 1302 from TLS_AES_256_GCM_SHA384 to TLS_AES_256_GCM_SHA256 and verify if matching z/OS docs works
Try changing 1301 from TLS_AES_128_GCM_SHA256 to TLS_AES_128_GCM_SHA384 and verify if matching BoringSSL works