I'm playing around with openssl s_client and I tried sending various payloads with/without padding and analysing the stack trace output. One case I don't understand is when I send empty bytes without padding I see the following output:
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
write to 0x107ede0 [0x109a7f3] (24 bytes => 24 (0x18))
0000 - 17 03 03 00 13 a2 03 0c-67 c0 aa 77 08 7d 37 cb ........g..w.}7.
0010 - 39 82 68 19 ba a2 39 bd-
I used a 16-byte aead tag cipher and since the content type field is 1 byte I would expect the total payload size to be 16 + 1 = 17 bytes. However, as evident in the stack trace openssl wrote a record with header 17 03 03 00 13 or 0x13 = 19 bytes. Why was 19 bytes sent instead of the expected 17 bytes? What are these two bytes?
Related
I am trying to implement a SSL client into my IoT project. I have copied the SSL_Client example I found in STM32Cube_FW_F7_V1.15.0 into my project and was able to compile succesfully. However the SSL handshake fails with -0x7780 MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE. I attach the console debug output:
. Seeding the random number generator... ok
. Loading the CA root certificate ... ok (1 skipped)
. Connecting to tcp/www.google.de/443... ok
. Setting up the SSL/TLS structure... ok
. Performing the SSL/TLS handshake...=> handshake
client state: 0
=> flush output
<= flush output
client state: 1
=> flush output
<= flush output
=> write client hello
client hello, max version: [3:3]
dumping 'client hello, random bytes' (32 bytes)
0000: e2 13 bf 6d 61 b6 fb a6 82 a4 59 f0 0b ef e9 03 ...ma.....Y.....
0010: 44 be de 3c 49 3d 39 56 51 60 3b b6 49 c4 17 50 D..<I=9VQ`;.I..P
client hello, session id len.: 0
dumping 'client hello, session id' (0 bytes)
client hello, add ciphersuite: c02b
client hello, got 1 ciphersuites (excluding SCSVs)
adding EMPTY_RENEGOTIATION_INFO_SCSV
client hello, compress len.: 1
client hello, compress alg.: 0
client hello, adding server name extension: www.google.de
client hello, adding signature_algorithms extension
client hello, adding supported_elliptic_curves extension
client hello, adding supported_point_formats extension
client hello, adding encrypt_then_mac extension
client hello, adding extended_master_secret extension
client hello, adding session ticket extension
client hello, total extension length: 62
=> write handshake message
=> write record
output record: msgtype = 22, version = [3:3], msglen = 111
dumping 'output record sent to network' (116 bytes)
0000: 16 03 03 00 6f 01 00 00 6b 03 03 e2 13 bf 6d 61 ....o...k.....ma
0010: b6 fb a6 82 a4 59 f0 0b ef e9 03 44 be de 3c 49 .....Y.....D..<I
0020: 3d 39 56 51 60 3b b6 49 c4 17 50 00 00 04 c0 2b =9VQ`;.I..P....+
0030: 00 ff 01 00 00 3e 00 00 00 12 00 10 00 00 0d 77 .....>.........w
0040: 77 77 2e 67 6f 6f 67 6c 65 2e 64 65 00 0d 00 0a ww.google.de....
0050: 00 08 04 03 04 01 03 03 03 01 00 0a 00 04 00 02 ................
0060: 00 15 00 0b 00 02 01 00 00 16 00 00 00 17 00 00 ................
0070: 00 23 00 00 .#..
=> flush output
message length: 116, out_left: 116
ssl->f_send() returned 116 (-0xffffff8c)
<= flush output
<= write record
<= write handshake message
<= write client hello
client state: 2
=> flush output
<= flush output
=> parse server hello
=> read record
=> fetch input
in_left: 0, nb_want: 5
in_left: 0, nb_want: 5
ssl->f_recv(_timeout)() returned 5 (-0xfffffffb)
<= fetch input
dumping 'input record header' (5 bytes)
0000: 15 03 03 00 02 .....
input record: msgtype = 21, version = [3:3], msglen = 2
=> fetch input
in_left: 5, nb_want: 7
in_left: 5, nb_want: 7
ssl->f_recv(_timeout)() returned 2 (-0xfffffffe)
<= fetch input
dumping 'input record from network' (7 bytes)
0000: 15 03 03 00 02 02 28 ......(
got an alert message, type: [2:40]
is a fatal alert message (msg 40)
mbedtls_ssl_handle_message_type() returned -30592 (-0x7780)
mbedtls_ssl_read_record() returned -30592 (-0x7780)
ERR
<= handshake
failed
! mbedtls_ssl_handshake returned -0x7780
Any help is greatly appreciated!
UPDATE:
The problem was the key exchange method. Only MBEDTLS_KEY_EXCHANGE_ECDHE_ECDSA_ENABLED was active. After i added MBEDTLS_KEY_EXCHANGE_ECDHE_RSA_ENABLED ( along with the needed MBEDTLS_RSA_C, MBEDTLS_PKCS1_V21 and MBEDTLS_PKCS1_V15) the handshake took place. Thanks a lot for pointing me to the right directon Gilles
The connection fails because the server decides to close the connection immediately after receiving the very first TLS message (ClientHello). It's sending the alert 40, which is “handshake failure”. Unfortunately, that's a generic “I don't like what I heard and I can't talk with you”, it doesn't give any information about what precisely it didn't like. Your TLS client is either buggy or, most likely, misconfigured and sent something that Google's server didn't accept.
Wireshark is helpful in diagnosing network protocol issues. Let's see what it says about the data sent by your client. I'm not a Wireshark expert so I did it the manual way, by starting Wireshark, telling it to listen to port 443 && host www.google.de and replaying the connection with
<clienthello.hex xxd -r -p | socket www.google.de 443 | xxd -p
Here's the dump of the ClientHello provided by Wireshark:
TLSv1.2 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.2 (0x0303)
Length: 111
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 107
Version: TLS 1.2 (0x0303)
Random: e213bf6d61b6fba682a459f00befe90344bede3c493d3956…
GMT Unix Time: Mar 11, 2090 20:50:05.000000000 CET
Random Bytes: 61b6fba682a459f00befe90344bede3c493d395651603bb6…
Session ID Length: 0
Cipher Suites Length: 4
Cipher Suites (2 suites)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
Compression Methods Length: 1
Compression Methods (1 method)
Compression Method: null (0)
Extensions Length: 62
Extension: server_name (len=18)
Type: server_name (0)
Length: 18
Server Name Indication extension
Server Name list length: 16
Server Name Type: host_name (0)
Server Name length: 13
Server Name: www.google.de
Extension: signature_algorithms (len=10)
Type: signature_algorithms (13)
Length: 10
Signature Hash Algorithms Length: 8
Signature Hash Algorithms (4 algorithms)
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: RSA (1)
Signature Algorithm: SHA224 ECDSA (0x0303)
Signature Hash Algorithm Hash: SHA224 (3)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: SHA224 RSA (0x0301)
Signature Hash Algorithm Hash: SHA224 (3)
Signature Hash Algorithm Signature: RSA (1)
Extension: supported_groups (len=4)
Type: supported_groups (10)
Length: 4
Supported Groups List Length: 2
Supported Groups (1 group)
Supported Group: secp224r1 (0x0015)
Extension: ec_point_formats (len=2)
Type: ec_point_formats (11)
Length: 2
EC point formats Length: 1
Elliptic curves point formats (1)
EC point format: uncompressed (0)
Extension: encrypt_then_mac (len=0)
Type: encrypt_then_mac (22)
Length: 0
Extension: extended_master_secret (len=0)
Type: extended_master_secret (23)
Length: 0
Extension: session_ticket (len=0)
Type: session_ticket (35)
Length: 0
Data (0 bytes)
Looking through this, most of the stuff is unsurprising except this:
Supported Groups (1 group)
Supported Group: secp224r1 (0x0015)
Secp224r1 is a little-used and officially deprecated curve. Very few servers accept it.
Reconfigure Mbed TLS to support secp256r1 instead. That's the de facto standard curve for resource-constrained devices for ECDH+ECDSA (either that, or Curve25519+Ed25519 for ECDH+EdDSA). In mbedtls/config.h, instead of listing MBEDTLS_ECP_DP_SECP224R1_ENABLED, list MBEDTLS_ECP_DP_SECP256R1_ENABLED. If you set MBEDTLS_ECP_MAX_BITS, make sure that it's set to 256 (or more).
I am trying to implement a SSL client into my IoT project. I have copied the SSL_Client example I found in STM32Cube_FW_F7_V1.15.0 into my project and was able to compile succesfully. However the SSL handshake fails with -0x7780 MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE. I attach the console debug output:
. Seeding the random number generator... ok
. Loading the CA root certificate ... ok (1 skipped)
. Connecting to tcp/www.google.de/443... ok
. Setting up the SSL/TLS structure... ok
. Performing the SSL/TLS handshake...=> handshake
client state: 0
=> flush output
<= flush output
client state: 1
=> flush output
<= flush output
=> write client hello
client hello, max version: [3:3]
dumping 'client hello, random bytes' (32 bytes)
0000: 88 d9 c4 b1 4f 82 ef a2 74 80 5c 6e 3f c4 29 ca ....O...t.\n?.).
0010: a4 8d 61 2b f6 37 ec 93 39 cb 7d d0 39 5a 67 9b ..a+.7..9.}.9Zg.
client hello, session id len.: 0
dumping 'client hello, session id' (0 bytes)
client hello, add ciphersuite: c02b
client hello, add ciphersuite: c031
client hello, add ciphersuite: c02d
client hello, add ciphersuite: 00a8
client hello, got 4 ciphersuites (excluding SCSVs)
adding EMPTY_RENEGOTIATION_INFO_SCSV
client hello, compress len.: 1
client hello, compress alg.: 0
client hello, adding server name extension: mbed TLS Server 1
client hello, adding signature_algorithms extension
client hello, adding supported_elliptic_curves extension
client hello, adding supported_point_formats extension
client hello, adding encrypt_then_mac extension
client hello, adding extended_master_secret extension
client hello, total extension length: 62
=> write handshake message
=> write record
output record: msgtype = 22, version = [3:3], msglen = 117
dumping 'output record sent to network' (122 bytes)
0000: 16 03 03 00 75 01 00 00 71 03 03 88 d9 c4 b1 4f ....u...q......O
0010: 82 ef a2 74 80 5c 6e 3f c4 29 ca a4 8d 61 2b f6 ...t.\n?.)...a+.
0020: 37 ec 93 39 cb 7d d0 39 5a 67 9b 00 00 0a c0 2b 7..9.}.9Zg.....+
0030: c0 31 c0 2d 00 a8 00 ff 01 00 00 3e 00 00 00 16 .1.-.......>....
0040: 00 14 00 00 11 6d 62 65 64 20 54 4c 53 20 53 65 .....mbed TLS Se
0050: 72 76 65 72 20 31 00 0d 00 0a 00 08 04 03 04 01 rver 1..........
0060: 03 03 03 01 00 0a 00 04 00 02 00 17 00 0b 00 02 ................
0070: 01 00 00 16 00 00 00 17 00 00 ..........
=> flush output
message length: 122, out_left: 122
ssl->f_send() returned 122 (-0xffffff86)
<= flush output
<= write record
<= write handshake message
<= write client hello
client state: 2
=> flush output
<= flush output
=> parse server hello
=> read record
=> fetch input
in_left: 0, nb_want: 5
in_left: 0, nb_want: 5
ssl->f_recv(_timeout)() returned 5 (-0xfffffffb)
<= fetch input
dumping 'input record header' (5 bytes)
0000: 15 03 03 00 02 .....
input record: msgtype = 21, version = [3:3], msglen = 2
=> fetch input
in_left: 5, nb_want: 7
in_left: 5, nb_want: 7
ssl->f_recv(_timeout)() returned 2 (-0xfffffffe)
<= fetch input
dumping 'input record from network' (7 bytes)
0000: 15 03 03 00 02 02 28 ......(
got an alert message, type: [2:40]
is a fatal alert message (msg 40)
mbedtls_ssl_handle_message_type() returned -30592 (-0x7780)
mbedtls_ssl_read_record() returned -30592 (-0x7780)
<= handshake
failed
! mbedtls_ssl_handshake returned -0x7780
I am thankfull for every hint in the right direction.
client hello, adding server name extension: mbed TLS Server 1
The client is using the SNI extension to indicate that it wants to talk to mbed TLS Server 1. The server on port 443 of www.google.de can respond as www.google.de, google.de and a bunch of other names that Google controls, but it does know about mbed TLS Server 1, so it sends a fatal alert indicating that it cannot complete the handshake.
You can use the sample client as is to talk to the sample server whose source code should be next to it. To contact another server, you need to change or remove the call to mbedtls_ssl_set_hostname.
We have 5 customers running the same WebService from Domino
This weekend we updated the customers servers with Domino 9.01. FP2 and the Poodle fixpack to be able to run TLS 1.0 incomming and outgoing.
4 Customers works perfect
1 Customer gets SSL errors for the outgoing Webservice (same errors as before we updated the servers), the incomming is working for TLS so we guess the updates for Poodle have worked as intended.
After setting som DEBUG_SSL parameters for one working and the failing server we got this logs
The failing row is
S_Read> nti_done return 0 bytes rc = 9
instead of intended
S_Read> nti_done return 5 bytes rc = 0
SSL_RCV> 00000000: 16 03 01 00 2E
I have searched google and nothing is there to understand what is missing
My guess is there is some problem with negotiating the cipher, but why and what to do for solving this matter.
I know there is some smart people out there ;-)
Log from failing server handshake
int_MapSSLError> Mapping SSL error 0 to 0 [SSLNoErr]
SSL_Handshake> Enter
SSL_Handshake> Current Cipher 0x0000 (Unknown Cipher)
SSLAdvanceHandshake Enter> Processed : 0 State: 4 (HandshakeClientIdle)
SSLAdvanceHandshake Enter> Processed : SSL_hello_request
SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeClientHello
SSLEncodeClientHello> We offered SSL/TLS version TLS1.0 (0x0301)
SSLAdvanceHandshake Exit> State : 5 (HandshakeServerHello)
S_Write> Enter len = 58
SSL_Xmt> 00000000: 16 03 01 00 35 01 00 00 31 03 01 54 A5 85 B7 4D '....5...1..T%.7M'
SSL_Xmt> 00000010: 15 80 11 80 C7 47 4D 1D 1D B1 89 5F F6 94 18 73 '....GGM..1._v..s'
SSL_Xmt> 00000020: C6 D3 7D 6A 15 92 A9 57 48 19 32 00 00 0A 00 2F 'FS}j..)WH.2..../'
SSL_Xmt> 00000030: 00 35 00 05 00 0A 00 04 01 00 '.5........'
S_Write> Switching Endpoint to sync
S_Write> Posting a nti_snd for 58 bytes
SSL_EncryptData> SSL not init exit
S_Write> Switching Endpoint to async
SSL_EncryptDataCleanup> SSL not init exit
S_Write> nti_done return 58 bytes rc = 0
S_Write> Exit, wrote 58 bytes
S_Read> Enter len = 5
S_Read> Switching Endpoint to sync
S_Read> Posting a nti_rcv for 5 bytes
SSL_RcvSetup> SSL not init exit
S_Read> Switching Endpoint to async
S_Read> nti_done return 0 bytes rc = 9
S_Read> nti_done return 0 bytes rc = 9 Event = 0x100
SSLSendAlert> Sending an alert of 0x0 (close_notify) level 0x2 (fatal)
SSL_Handshake> Changing SSL status from -6989 to -5000 to flush write queue
SSL_Handshake> After handshake state= 2 Status= -5000
SSL_Handshake> Exit Status = -5000
int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
...
Log from working server handshake
int_MapSSLError> Mapping SSL error 0 to 0 [SSLNoErr]
SSL_Handshake> Enter
SSL_Handshake> Current Cipher 0x0000 (Unknown Cipher)
SSLAdvanceHandshake Enter> Processed : 0 State: 4 (HandshakeClientIdle)
SSLAdvanceHandshake Enter> Processed : SSL_hello_request
SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeClientHello
SSLEncodeClientHello> We offered SSL/TLS version TLS1.0 (0x0301)
SSLAdvanceHandshake Exit> State : 5 (HandshakeServerHello)
S_Write> Enter len = 58
SSL_Xmt> 00000000: 16 03 01 00 35 01 00 00 31 03 01 54 A5 89 B3 A0 '....5...1..T%.3 '
SSL_Xmt> 00000010: 2B 75 D1 E9 D4 81 87 C3 5D 91 45 84 6A E2 47 9D '+uQiT..C].E.jbG.'
SSL_Xmt> 00000020: 76 BE 14 A8 A6 10 1C 06 FB 7D 8B 00 00 0A 00 2F 'v>.(&...{}...../'
SSL_Xmt> 00000030: 00 35 00 05 00 0A 00 04 01 00 '.5........'
S_Write> Switching Endpoint to sync
S_Write> Posting a nti_snd for 58 bytes
SSL_EncryptData> SSL not init exit
S_Write> Switching Endpoint to async
SSL_EncryptDataCleanup> SSL not init exit
S_Write> nti_done return 58 bytes rc = 0
S_Write> Exit, wrote 58 bytes
S_Read> Enter len = 5
S_Read> Switching Endpoint to sync
S_Read> Posting a nti_rcv for 5 bytes
SSL_RcvSetup> SSL not init exit
S_Read> Switching Endpoint to async
S_Read> nti_done return 5 bytes rc = 0
SSL_RCV> 00000000: 16 03 01 00 2E '.....'
S_Read> Exit, read 5 bytes
S_Read> Enter len = 46
S_Read> Switching Endpoint to sync
S_Read> Posting a nti_rcv for 46 bytes
SSL_RcvSetup> SSL not init exit
S_Read> Switching Endpoint to async
S_Read> nti_done return 46 bytes rc = 0
SSL_RCV> 00000000: 02 00 00 2A 03 01 54 7C 9D 24 4C B4 AD 62 4E 35 '...*..T|.$L4-bN5'
SSL_RCV> 00000010: 4C C3 B4 AB 34 6D 7D CB 8F 6B CC 80 00 FE 4C 4A 'LC4+4m}K.kL..~LJ'
SSL_RCV> 00000020: 77 87 CD 2E DF 98 04 10 13 29 0B 00 2F 00 'w.M._....)../.'
S_Read> Exit, read 46 bytes
SSLProcessProtocolMessage> Record Content: 22
SSLProcessHandshakeMessage Enter> Message: 2 State: 5 (HandshakeServerHello) Key Exchange: 0 Cipher: 0x0000 (Unknown Cipher)
SSLProcessHandshakeMessage Enter> Message: SSL_server_hello
SSLProcessServerHello> Server chose SSL/TLS version TLS1.0 (0x0301)
SSLProcessHandshakeMessage Exit> Message: 2 State: 5 (HandshakeServerHello) Key Exchange: 1 Cipher: 0x002F (RSA_WITH_AES_128_CBC_SHA)
SSLAdvanceHandshake Enter> Processed : 2 State: 5 (HandshakeServerHello)
SSLAdvanceHandshake Enter> Processed : SSL_server_hello
SSLAdvanceHandshake Exit> State : 8 (HandshakeCertificate)
SSL_Handshake> After handshake state= 8 Status= -5000
SSL_Handshake> Exit Status = -5000
int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
SSL_Handshake> Enter
SSL_Handshake> Current Cipher 0x002F (RSA_WITH_AES_128_CBC_SHA)
S_Read> Enter len = 5
S_Read> Switching Endpoint to sync
S_Read> Posting a nti_rcv for 5 bytes
SSL_RcvSetup> SSL not init exit
S_Read> Switching Endpoint to async
S_Read> nti_done return 5 bytes rc = 0
SSL_RCV> 00000000: 16 03 01 0E 9D '.....'
S_Read> Exit, read 5 bytes
S_Read> Enter len = 3741
....
/Stefan
PS: Here is the Java errors that come sfter the hand shake error
Error connecting to 'xxxxx' on port '443', SSL IO error. Remote session no longer responding.
at lotus.domino.axis.InternalFault.makeFault(Unknown Source)
at lotus.domino.axis.transport.http.HTTPSender.invoke(Unknown Source)
at lotus.domino.axis.strategies.InvocationStrategy.visit(Unknown Source)
at lotus.domino.axis.SimpleChain.doVisiting(Unknown Source)
at lotus.domino.axis.SimpleChain.invoke(Unknown Source)
at lotus.domino.axis.client.AxisClient.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invokeEngine(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.websvc.client.Call.invoke(Unknown Source)
I'm trying to read the xref table of a pdf version >= 1.5.
the xref table is an object:
58 0 obj
<</DecodeParms<</Columns 4/Predictor 12>>/Filter/FlateDecode/ID[<CB05990F613E2FCB6120F059A2BCA25B><E2ED9D17A60FB145B03010B70517FC30>]/Index[38 39]/Info 37 0 R/Length 96/Prev 67529/Root 39 0 R/Size 77/Type/XRef/W[1 2 1]>>stream
hÞbbd``b`:$AD`Ì ‰Õ Vˆ8âXAÄ×HÈ$€t¨ – ÁwHp·‚ŒZ$ìÄb!&F† .#5‰ÿŒ>(more here but can't paste)
endstream
endobj
as you can see
/FlatDecode
/Index [38 39], that is 39 entries in the stream
/W [1 2 1] that is each entry is 1 + 2 + 1 = 4 bytes long
/Root 39 0 R that is root object is number 39
BUT :
the decompressed stream is 195 bytes long (39 * 5 = 195). So the length of an entry is 4 or 5.
Here is the first inflated bytes
02 01 00 10 00 02 00 02 cd 00 02 00 01 51 00 02 00 01 70 00 02 00 05 7a 00 02
^^
if entry length is 4 then the root entry is a free object (see the ^^) !!
if the entry is 5: how to interpret the fields of one entry (reference is implicitly made to PDF Reference, chapter 3.4.7 table 3.16 ) ?
For object 38, the first of the stream: it seems, as it is of type 2, to be the 16 object of the stream object number 256, but there is no object 256 in my pdf file !!!
The question is: how shall I handle the 195 bytes ?
A compressed xref table may have been compressed with one of the PNG filters. If the /Predictor value is set to '10' or greater ("a Predictor value greater than or equal to 10 merely indicates that a PNG predictor is in use; the specific predictor function used is explicitly encoded in the incoming data")1, PNG row filters are supplied inside the compressed data "as usual" (i.e., in the first byte of each 'row', where the 'row' is of the width in /W).
Width [1 2 1] plus Predictor byte:
02 01 00 10 00
02 00 02 cd 00
02 00 01 51 00
02 00 01 70 00
02 00 05 7a 00
02 .. .. .. ..
After applying the row filters ('2', or 'up', for all of these rows), you get this:
01 00 10 00
01 02 ed 00
01 03 3e 00
01 04 ae 00
01 09 28 00
.. .. .. ..
Note: calculated by hand; I might have made the odd mistake here and there. Note that the PNG 'up' filter is a byte filter, and the result of the "up" filter is truncated to 8 bits for each addition.
This leads to the following Type 1 XRef references ("type 1 entries define objects that are in use but are not compressed (corresponding to n entries in a cross-reference table)."):2
#38 type 1: offset 10h, generation 0
#39 type 1: offset 2EDh, generation 0
#40 type 1: offset 33Eh, generation 0
#41 type 1: offset 4AEh, generation 0
#42 type 1: offset 928h, generation 0
1 See LZW and Flate Predictor Functions in PDF Reference 1.7, 6th Ed, Section 3.3: Filters.
2 As described in your Table 3.16 in PDF Ref 1.7.
In my project I need to know what a zlib header looks like. I've heard it's rather simple but I cannot find any description of the zlib header.
For example, does it contain a magic number?
zlib magic headers
78 01 - No Compression/low
78 9C - Default Compression
78 DA - Best Compression
Link to RFC
0 1
+---+---+
|CMF|FLG|
+---+---+
CMF (Compression Method and flags)
This byte is divided into a 4-bit compression method and a 4-
bit information field depending on the compression method.
bits 0 to 3 CM Compression method
bits 4 to 7 CINFO Compression info
CM (Compression method)
This identifies the compression method used in the file. CM = 8
denotes the "deflate" compression method with a window size up
to 32K. This is the method used by gzip and PNG and almost everything else.
CM = 15 is reserved.
CINFO (Compression info)
For CM = 8, CINFO is the base-2 logarithm of the LZ77 window
size, minus eight (CINFO=7 indicates a 32K window size). Values
of CINFO above 7 are not allowed in this version of the
specification. CINFO is not defined in this specification for
CM not equal to 8.
In practice, this means the first byte is almost always 78 (hex)
FLG (FLaGs)
This flag byte is divided as follows:
bits 0 to 4 FCHECK (check bits for CMF and FLG)
bit 5 FDICT (preset dictionary)
bits 6 to 7 FLEVEL (compression level)
The FCHECK value must be such that CMF and FLG, when viewed as
a 16-bit unsigned integer stored in MSB order (CMF*256 + FLG),
is a multiple of 31.
FLEVEL (Compression level)
These flags are available for use by specific compression
methods. The "deflate" method (CM = 8) sets these flags as
follows:
0 - compressor used fastest algorithm
1 - compressor used fast algorithm
2 - compressor used default algorithm
3 - compressor used maximum compression, slowest algorithm
ZLIB/GZIP headers
Level | ZLIB | GZIP
1 | 78 01 | 1F 8B
2 | 78 5E | 1F 8B
3 | 78 5E | 1F 8B
4 | 78 5E | 1F 8B
5 | 78 5E | 1F 8B
6 | 78 9C | 1F 8B
7 | 78 DA | 1F 8B
8 | 78 DA | 1F 8B
9 | 78 DA | 1F 8B
Deflate doesn't have common headers
The ZLIB header (as defined in RFC1950) is a 16-bit, big-endian value - in other words, it is two bytes long, with the higher bits in the first byte and the lower bits in the second.
It contains these bitfields from most to least significant:
CINFO (bits 12-15, first byte)
Indicates the window size as a power of two, from 0 (256 bytes) to 7 (32768 bytes). This will usually be 7. Higher values are not allowed.
CM (bits 8-11)
The compression method. Only Deflate (8) is allowed.
FLEVEL (bits 6-7, second byte)
Roughly indicates the compression level, from 0 (fast/low) to 3 (slow/high)
FDICT (bit 5)
Indicates whether a preset dictionary is used. This is usually 0.
(1 is technically allowed, but I don't know of any Deflate formats that define preset dictionaries.)
FCHECK (bits 0-4)
A checksum (5 bits, 0..31), whose value is calculated such that the entire value divides 31 with no remainder.*
Typically, only the CINFO and FLEVEL fields can be freely changed, and FCHECK must be calculated based on the final value. Assuming no preset dictionary, there is no choice in what the other fields contain, so a total of 32 possible headers are valid. Here they are:
FLEVEL: 0 1 2 3
CINFO:
0 08 1D 08 5B 08 99 08 D7
1 18 19 18 57 18 95 18 D3
2 28 15 28 53 28 91 28 CF
3 38 11 38 4F 38 8D 38 CB
4 48 0D 48 4B 48 89 48 C7
5 58 09 58 47 58 85 58 C3
6 68 05 68 43 68 81 68 DE
7 78 01 78 5E 78 9C 78 DA
The CINFO field is rarely, if ever, set by compressors to be anything other than 7 (indicating the maximum 32KB window), so the only values you are likely to see in the wild are the four in the bottom row (beginning with 78).
* (You might wonder if there's a small amount of leeway on the value of FCHECK - could it be set to either of 0 or 31 if both pass the checksum? In practice though, this can only occur if FDICT=1, so it doesn't feature in the above table.)
Following is the Zlib compressed data format.
+---+---+
|CMF|FLG| (2 bytes - Defines the compression mode - More details below)
+---+---+
+---+---+---+---+
| DICTID | (4 bytes. Present only when FLG.FDICT is set.) - Mostly not set
+---+---+---+---+
+=====================+
|...compressed data...| (variable size of data)
+=====================+
+---+---+---+---+
| ADLER32 | (4 bytes of checksum)
+---+---+---+---+
Mostly, FLG.FDICT (Dictionary flag) is not set. In such cases the DICTID is simply not present. So, the total hear is just 2 bytes.
The header values(CMF and FLG) with no dictionary are defined as follows.
CMF | FLG
0x78 | 0x01 - No Compression/low
0x78 | 0x9C - Default Compression
0x78 | 0xDA - Best Compression
More at ZLIB RFC
All answers here are most probably correct, however - if you want to manipulate ZLib compression stream directly, and it was produced by using gz_open, gzwrite, gzclose functions - then there is extra 10 leading bytes header before zlib compression steam comes - and those are produced by function gz_open - header looks like this:
fprintf(s->file, "%c%c%c%c%c%c%c%c%c%c", gz_magic[0], gz_magic[1],
Z_DEFLATED, 0 /*flags*/, 0,0,0,0 /*time*/, 0 /*xflags*/, OS_CODE);
And results in following hex dump: 1F 8B 08 00 00 00 00 00 00 0B
followed by zlib compression stream.
But there is also trailing 8 bytes - they are uLong - crc over whole file, uLong - uncompressed file size - look for following bytes at end of stream:
putLong (s->file, s->crc);
putLong (s->file, (uLong)(s->in & 0xffffffff));