Apache is executing 404 urls twice [closed] - apache

Question

Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 2 months ago.
Improve this question
While trying to debug slow apache response times (apparently due to rules in .htaccess) I found a different issue.
Apache seems to execute 404 urls twice.
Here are the ssl_request_logs. The first number in the logs is the time in (microseconds), it goes down significantly when I disable .htaccess but thats another discussion.
253125 [18/Dec/2022:12:19:18 +0000] 1.1.1.1 www.example.com "GET /testing404 HTTP/1.1" 1018 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 404
202625 [18/Dec/2022:12:19:18 +0000] 1.1.1.1 www.example.com "GET /testing404 HTTP/1.1" 1018 "Mozilla/5.0,(Macintosh; Intel Mac OS X 10_15_7),AppleWebKit/537.36,(KHTML, like Gecko),Chrome/108.0.0.0,Safari/537.36" 404
While 200 (and 301's) show up just once
358050 [18/Dec/2022:12:22:00 +0000] 1.1.1.1 www.example.com "GET /account-login HTTP/1.1" 13844 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" 200
The below are the ssl_error_logs when trying a 404 and using the debug log level. I tried trace8 but not sure all the extra data can help diagnose. I also renamed the .htaccess so its disabled and wont fill the log with even more unnecessary data. (removed text like pid info, times, IP's to get under character limit)
[Sun Dec 18 516095 ] [ssl:info] AH01964: Connection to child 64 established (server www.example.com:443)
[Sun Dec 18 516302 ] ssl_engine_rand.c(126): Server: Seeding PRNG with 144 bytes of entropy
[Sun Dec 18 516581 ] ssl_engine_kernel.c(2206): OpenSSL: Handshake: start
[Sun Dec 18 516668 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: before SSL initialization
[Sun Dec 18 516739 ] ssl_engine_io.c(2406): OpenSSL: read 5/5 bytes from BIO#7f7a84007b70 [mem: 7f7a8400d433]
[Sun Dec 18 516768 ] ssl_engine_io.c(2406): OpenSSL: read 314/314 bytes from BIO#7f7a84007b70 [mem: 7f7a8400d438]
[Sun Dec 18 516783 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: before SSL initialization
[Sun Dec 18 516839 ] [ssl:debug] ssl_engine_kernel.c(2397): AH02043: SSL virtual host for servername www.example.com found
[Sun Dec 18 516996 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS read client hello
[Sun Dec 18 517291 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write server hello
[Sun Dec 18 517383 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write change cipher spec
[Sun Dec 18 517427 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: TLSv1.3 write encrypted extensions
[Sun Dec 18 517696 ] ssl_engine_io.c(2406): OpenSSL: write 4096/4096 bytes to BIO#7f7a84004aa0 [mem: 7f7a840166f0]
[Sun Dec 18 517746 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write certificate
[Sun Dec 18 520249 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: TLSv1.3 write server certificate verify
[Sun Dec 18 520305 ] ssl_engine_io.c(2406): OpenSSL: write 494/494 bytes to BIO#7f7a84004aa0 [mem: 7f7a840166f0]
[Sun Dec 18 520399 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write finished
[Sun Dec 18 520415 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: TLSv1.3 early data
[Sun Dec 18 619351 ] ssl_engine_io.c(2406): OpenSSL: read 5/5 bytes from BIO#7f7a84007b70 [mem: 7f7a84019853]
[Sun Dec 18 619448 ] ssl_engine_io.c(2406): OpenSSL: read 1/1 bytes from BIO#7f7a84007b70 [mem: 7f7a84019858]
[Sun Dec 18 619461 ] ssl_engine_io.c(2406): OpenSSL: read 5/5 bytes from BIO#7f7a84007b70 [mem: 7f7a84019853]
[Sun Dec 18 619470 ] ssl_engine_io.c(2406): OpenSSL: read 69/69 bytes from BIO#7f7a84007b70 [mem: 7f7a84019858]
[Sun Dec 18 619543 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: TLSv1.3 early data
[Sun Dec 18 619667 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS read finished
[Sun Dec 18 619698 ] ssl_engine_kernel.c(2210): OpenSSL: Handshake: done
[Sun Dec 18 619733 ] [ssl:debug] ssl_engine_kernel.c(2259): AH02041: Protocol: TLSv1.3, Cipher: TLS_AES_256_GCM_SHA384 (256/256 bits)
[Sun Dec 18 619841 ] [socache_shmcb:debug] mod_socache_shmcb.c(510): AH00831: socache_shmcb_store (0x01 -> subcache 1)
[Sun Dec 18 619879 ] [socache_shmcb:debug] mod_socache_shmcb.c(864): AH00847: insert happened at idx=0, data=(0:32)
[Sun Dec 18 619888 ] [socache_shmcb:debug] mod_socache_shmcb.c(869): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/209
[Sun Dec 18 619898 ] [socache_shmcb:debug] mod_socache_shmcb.c(531): AH00834: leaving socache_shmcb_store successfully
[Sun Dec 18 619911 ] ssl_engine_kernel.c(2066): Inter-Process Session Cache: request=SET status=OK id=0140749c6670cdfacb34513b315d9251af355e0b6829622ef21ca6bef4d8c69b timeout=300s (session caching)
[Sun Dec 18 619939 ] ssl_engine_io.c(2406): OpenSSL: write 79/79 bytes to BIO#7f7a84004aa0 [mem: 7f7a840166f0]
[Sun Dec 18 620002 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write session ticket
[Sun Dec 18 620047 ] [socache_shmcb:debug] mod_socache_shmcb.c(510): AH00831: socache_shmcb_store (0x97 -> subcache 23)
[Sun Dec 18 620070 ] [socache_shmcb:debug] mod_socache_shmcb.c(864): AH00847: insert happened at idx=0, data=(0:32)
[Sun Dec 18 620078 ] [socache_shmcb:debug] mod_socache_shmcb.c(869): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/209
[Sun Dec 18 620107 ] [socache_shmcb:debug] mod_socache_shmcb.c(531): AH00834: leaving socache_shmcb_store successfully
[Sun Dec 18 620116 ] ssl_engine_kernel.c(2066): Inter-Process Session Cache: request=SET status=OK id=9765b1a6423b5ddaca014e359ecad943fba6da0efc0700fc68ca6cefa7dd3431 timeout=300s (session caching)
[Sun Dec 18 620134 ] ssl_engine_io.c(2406): OpenSSL: write 79/79 bytes to BIO#7f7a84004aa0 [mem: 7f7a840166f0]
[Sun Dec 18 620172 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write session ticket
[Sun Dec 18 61 ] [http2:trace1] h2_c1.c(218): h2_h2, process_conn
[Sun Dec 18 620247 ] [http2:trace1] h2_c1.c(226): h2_h2, process_conn, new connection using protocol 'http/1.1', direct=0, tls acceptable=1
[Sun Dec 18 620256 ] [http2:trace1] h2_c1.c(282): h2_h2, declined
[Sun Dec 18 620311 ] ssl_engine_io.c(2406): OpenSSL: read 5/5 bytes from BIO#7f7a84007b70 [mem: 7f7a84007ea3]
[Sun Dec 18 620334 ] ssl_engine_io.c(2406): OpenSSL: read 1265/1265 bytes from BIO#7f7a84007b70 [mem: 7f7a84007ea8]
[Sun Dec 18 620363 ] [core:trace5] protocol.c(713): Request received from client: GET /testing404page HTTP/1.1
[Sun Dec 18 620528 ] [ssl:debug] ssl_engine_kernel.c(422): AH02034: Initial (No.1) HTTPS request received for child 64 (server www.example.com:443)
[Sun Dec 18 620560 ] http_request.c(436): Headers received from client:
[Sun Dec 18 620569 ] Host: www.example.com
[Sun Dec 18 620576 ] Connection: close
[Sun Dec 18 620584 ] Accept: text/html, application/xhtml+xml, application/xml; q=0.9, image/avif, image/webp, image/apng, */*; q=0.8, application/signed-exchange; v=b3; q=0.9
[Sun Dec 18 620592 ] User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
[Sun Dec 18 620599 ] Accept-Encoding: identity
[Sun Dec 18 620605 ] Accept-Language: en-US, en; q=0.9
[Sun Dec 18 620612 ] Cookie: _ga=GA1.1.929352060.1671367811; _ga_LN3Q7T4VE2=GS1.1.1671367811.1.0.1671367815.0.0.0
[Sun Dec 18 620620 ] Upgrade-Insecure-Requests: 1
[Sun Dec 18 620627 ] CDN-ServerId: 1027
[Sun Dec 18 620633 ] CDN-ServerZone: ISR
[Sun Dec 18 620648 ] CDN-Host: www.example.com
[Sun Dec 18 620655 ] CDN-ConnectionId: 626360907
[Sun Dec 18 620662 ] CDN-MobileDevice: false
[Sun Dec 18 620669 ] X-Forwarded-Proto: https
[Sun Dec 18 620676 ] BunnyCDN-LBKey: j0YiR5gUfYiZmQrWSLV6opVewPUZFwJD21K1W9dkKWg4EnYSjbS5c9
[Sun Dec 18 620683 ] sec-ch-ua: \\"Not?A_Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"108\\", \\"Google Chrome\\";v=\\"108\\"
[Sun Dec 18 620691 ] sec-ch-ua-mobile: ?0
[Sun Dec 18 620698 ] sec-ch-ua-platform: \\"macOS\\"
[Sun Dec 18 620705 ] sec-fetch-site: none
[Sun Dec 18 620712 ] sec-fetch-mode: navigate
[Sun Dec 18 620718 ] sec-fetch-user: ?1
[Sun Dec 18 620725 ] sec-fetch-dest: document
[Sun Dec 18 620732 ] CDN-RequestCountryCode: IL
[Sun Dec 18 620738 ] CDN-RequestId: 2ec409327996c82006a7bd1489fd2b69
[Sun Dec 18 620745 ] CDN-LoopCount: 1
[Sun Dec 18 620752 ] CDN-ProxyVer: 1.03
[Sun Dec 18 620759 ] Via: BunnyCDN
[Sun Dec 18 620765 ] X-Real-IP: 80.178.138.38
[Sun Dec 18 620772 ] X-Forwarded-For: 80.178.138.38
[Sun Dec 18 620779 ] BunnyCDN-OriginPathBase64: L3Rlc3Rpbmc0MDRwYWdl
[Sun Dec 18 620786 ] CDN-RequestUrl:
[Sun Dec 18 621203 ] [authz_core:debug] mod_authz_core.c(818): AH01626: authorization result of Require all granted: granted
[Sun Dec 18 621226 ] [authz_core:debug] mod_authz_core.c(818): AH01626: authorization result of <RequireAny>: granted
[Sun Dec 18 621235 ] [core:trace3] request.c(362): request authorized without authentication by access_checker_ex hook: /testing404page
[Sun Dec 18 627213 ] [core:info] AH00128: File does not exist: /var/www/html/testing404page
[Sun Dec 18 627305 ] [http:trace3] http_filters.c(1132): Response sent with status 404, headers:
[Sun Dec 18 627317 ] [http:trace5] http_filters.c(1139): Date: Sun, 18 Dec 12:50:51 GMT
[Sun Dec 18 627325 ] [http:trace5] http_filters.c(1142): Server: Apache/2.4.54 (codeit) OpenSSL/1.1.1q+quic
[Sun Dec 18 627334 ] http_filters.c(961): Content-Length: 196
[Sun Dec 18 627341 ] http_filters.c(961): Connection: close
[Sun Dec 18 627348 ] http_filters.c(961): Content-Type: text/html; charset=iso-8859-1
[Sun Dec 18 627371 ] ssl_engine_io.c(1856): coalesce: have 0 bytes, adding 200 more (buckets=1)
[Sun Dec 18 627812 ] ssl_engine_io.c(1856): coalesce: have 200 bytes, adding 196 more (buckets=1)
[Sun Dec 18 627837 ] ssl_engine_io.c(1917): coalesce: passing on 396 bytes
[Sun Dec 18 627882 ] ssl_engine_io.c(2406): OpenSSL: write 418/418 bytes to BIO#7f7a84004aa0 [mem: 7f7a84219233]
[Sun Dec 18 628017 ] ssl_engine_io.c(2406): OpenSSL: write 24/24 bytes to BIO#7f7a84004aa0 [mem: 7f7a84219233]
[Sun Dec 18 628048 ] ssl_engine_kernel.c(2225): OpenSSL: Write: SSL negotiation finished successfully
[Sun Dec 18 628060 ] [ssl:debug] ssl_engine_io.c(1151): AH02001: Connection closed to child 64 with standard shutdown (server www.example.com:443)
[Sun Dec 18 638828 ] [ssl:info] AH01964: Connection to child 128 established (server www.example.com:443)
[Sun Dec 18 639010 ] ssl_engine_rand.c(126): Server: Seeding PRNG with 144 bytes of entropy
[Sun Dec 18 639234 ] ssl_engine_kernel.c(2206): OpenSSL: Handshake: start
[Sun Dec 18 639331 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: before SSL initialization
[Sun Dec 18 639396 ] ssl_engine_io.c(2406): OpenSSL: read 5/5 bytes from BIO#7f7a84007b70 [mem: 7f7a8400d433]
[Sun Dec 18 639432 ] ssl_engine_io.c(2406): OpenSSL: read 308/308 bytes from BIO#7f7a84007b70 [mem: 7f7a8400d438]
[Sun Dec 18 639449 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: before SSL initialization
[Sun Dec 18 639502 ] [ssl:debug] ssl_engine_kernel.c(2397): AH02043: SSL virtual host for servername www.example.com found
[Sun Dec 18 639689 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS read client hello
[Sun Dec 18 639990 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write server hello
[Sun Dec 18 640102 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write change cipher spec
[Sun Dec 18 640141 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: TLSv1.3 write encrypted extensions
[Sun Dec 18 640454 ] ssl_engine_io.c(2406): OpenSSL: write 4096/4096 bytes to BIO#7f7a84004aa0 [mem: 7f7a840166f0]
[Sun Dec 18 640534 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write certificate
[Sun Dec 18 643242 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: TLSv1.3 write server certificate verify
[Sun Dec 18 643321 ] ssl_engine_io.c(2406): OpenSSL: write 478/478 bytes to BIO#7f7a84004aa0 [mem: 7f7a840166f0]
[Sun Dec 18 643421 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write finished
[Sun Dec 18 643446 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: TLSv1.3 early data
[Sun Dec 18 747121 ] ssl_engine_io.c(2406): OpenSSL: read 5/5 bytes from BIO#7f7a84007b70 [mem: 7f7a84019853]
[Sun Dec 18 747211 ] ssl_engine_io.c(2406): OpenSSL: read 1/1 bytes from BIO#7f7a84007b70 [mem: 7f7a84019858]
[Sun Dec 18 747231 ] ssl_engine_io.c(2406): OpenSSL: read 5/5 bytes from BIO#7f7a84007b70 [mem: 7f7a84019853]
[Sun Dec 18 747241 ] ssl_engine_io.c(2406): OpenSSL: read 53/53 bytes from BIO#7f7a84007b70 [mem: 7f7a84019858]
[Sun Dec 18 747274 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: TLSv1.3 early data
[Sun Dec 18 747353 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS read finished
[Sun Dec 18 747376 ] ssl_engine_kernel.c(2210): OpenSSL: Handshake: done
[Sun Dec 18 747411 ] [ssl:debug] ssl_engine_kernel.c(2259): AH02041: Protocol: TLSv1.3, Cipher: TLS_AES_128_GCM_SHA256 (128/128 bits)
[Sun Dec 18 747607 ] [socache_shmcb:debug] mod_socache_shmcb.c(510): AH00831: socache_shmcb_store (0x4d -> subcache 13)
[Sun Dec 18 747665 ] [socache_shmcb:debug] mod_socache_shmcb.c(864): AH00847: insert happened at idx=0, data=(0:32)
[Sun Dec 18 747683 ] [socache_shmcb:debug] mod_socache_shmcb.c(869): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/194
[Sun Dec 18 747697 ] [socache_shmcb:debug] mod_socache_shmcb.c(531): AH00834: leaving socache_shmcb_store successfully
[Sun Dec 18 747710 ] ssl_engine_kernel.c(2066): Inter-Process Session Cache: request=SET status=OK id=4d936df371aa4cf9fcdf61d23982017034ab7cec351ec23dd1da80ed6e672f2a timeout=300s (session caching)
[Sun Dec 18 747764 ] ssl_engine_io.c(2406): OpenSSL: write 79/79 bytes to BIO#7f7a84004aa0 [mem: 7f7a840166f0]
[Sun Dec 18 747826 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write session ticket
[Sun Dec 18 747889 ] [socache_shmcb:debug] mod_socache_shmcb.c(510): AH00831: socache_shmcb_store (0x66 -> subcache 6)
[Sun Dec 18 747941 ] [socache_shmcb:debug] mod_socache_shmcb.c(864): AH00847: insert happened at idx=0, data=(0:32)
[Sun Dec 18 747959 ] [socache_shmcb:debug] mod_socache_shmcb.c(869): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/194
[Sun Dec 18 747983 ] [socache_shmcb:debug] mod_socache_shmcb.c(531): AH00834: leaving socache_shmcb_store successfully
[Sun Dec 18 748002 ] ssl_engine_kernel.c(2066): Inter-Process Session Cache: request=SET status=OK id=661ae4bb6e6fe52f898a8956c4644e3fe30126ec61ce49825836bdd963c61f2b timeout=300s (session caching)
[Sun Dec 18 748040 ] ssl_engine_io.c(2406): OpenSSL: write 79/79 bytes to BIO#7f7a84004aa0 [mem: 7f7a840166f0]
[Sun Dec 18 748106 ] ssl_engine_kernel.c(2215): OpenSSL: Loop: SSLv3/TLS write session ticket
[Sun Dec 18 748159 ] [http2:trace1] h2_c1.c(218): h2_h2, process_conn
[Sun Dec 18 748195 ] [http2:trace1] h2_c1.c(226): h2_h2, process_conn, new connection using protocol 'http/1.1', direct=0, tls acceptable=1
[Sun Dec 18 748205 ] [http2:trace1] h2_c1.c(282): h2_h2, declined
[Sun Dec 18 748270 ] ssl_engine_io.c(2406): OpenSSL: read 5/5 bytes from BIO#7f7a84007b70 [mem: 7f7a84007ea3]
[Sun Dec 18 748293 ] ssl_engine_io.c(2406): OpenSSL: read 1791/1791 bytes from BIO#7f7a84007b70 [mem: 7f7a84007ea8]
[Sun Dec 18 748331 ] [core:trace5] protocol.c(713): Request received from client: GET /testing404page HTTP/1.1
[Sun Dec 18 748543 ] [ssl:debug] ssl_engine_kernel.c(422): AH02034: Initial (No.1) HTTPS request received for child 128 (server www.example.com:443)
[Sun Dec 18 748576 ] http_request.c(436): Headers received from client:
[Sun Dec 18 748585 ] CDN-GzipEnabled: true
[Sun Dec 18 748592 ] CDN-BrotliEnabled: true
[Sun Dec 18 748600 ] CDN-RequestUrl: https://www.example.com/testing404page
[Sun Dec 18 748607 ] CDN-CacheKey: luakey_eb91c56d-6_149_/testing404page
[Sun Dec 18 748614 ] Accept: text/html,application/xhtml+xml,application/xml; q=0.9,image/avif,image/webp,image/apng,*/*; q=0.8,application/signed-exchange; v=b3; q=0.9
[Sun Dec 18 748628 ] User-Agent: Mozilla/5.0,(Macintosh; Intel Mac OS X 10_15_7),AppleWebKit/537.36,(KHTML, like Gecko),Chrome/108.0.0.0,Safari/537.36
[Sun Dec 18 748636 ] Accept-Language: en-US,en; q=0.9
[Sun Dec 18 748643 ] Upgrade-Insecure-Requests: 1
[Sun Dec 18 748650 ] CDN-ServerId: 1027
[Sun Dec 18 748657 ] CDN-ServerZone: ISR
[Sun Dec 18 748664 ] CDN-Host: www.example.com
[Sun Dec 18 748671 ] CDN-ConnectionId: 626360907
[Sun Dec 18 748677 ] CDN-MobileDevice: false
[Sun Dec 18 748684 ] X-Forwarded-Proto: https
[Sun Dec 18 748691 ] BunnyCDN-LBKey: j0YiR5gUfYiZmQrWSLV6opVewPUZFwJD21K1W9dkKWg4EnYSjbS5c9
[Sun Dec 18 748698 ] sec-ch-ua: \\"Not?A_Brand\\";v=\\"8\\", \\"Chromium\\";v=\\"108\\", \\"Google Chrome\\";v=\\"108\\"
[Sun Dec 18 748706 ] sec-ch-ua-mobile: ?0
[Sun Dec 18 748713 ] sec-ch-ua-platform: \\"macOS\\"
[Sun Dec 18 748720 ] sec-fetch-site: none
[Sun Dec 18 748726 ] sec-fetch-mode: navigate
[Sun Dec 18 748733 ] sec-fetch-user: ?1
[Sun Dec 18 748740 ] sec-fetch-dest: document
[Sun Dec 18 748746 ] CDN-RequestCountryCode: IL
[Sun Dec 18 748754 ] CDN-RequestId: 2ec409327996c82006a7bd1489fd2b69
[Sun Dec 18 748761 ] CDN-LoopCount: 1
[Sun Dec 18 748767 ] CDN-ProxyVer: 1.03
[Sun Dec 18 748774 ] Via: BunnyCDN
[Sun Dec 18 748781 ] X-Real-IP: 80.178.138.38
[Sun Dec 18 748788 ] X-Forwarded-For: 80.178.138.38
[Sun Dec 18 748799 ] BunnyCDN-SentToOptimizer: false
[Sun Dec 18 748806 ] BunnyCDN-StorageToken: 2YQhY50fYNCjlYTHZY0MqQepFPIuYmptHWQZKqwE
[Sun Dec 18 748814 ] BunnyCDN-CachePath: __bcdn_perma_cache__/pullzone__eb91c56d-6__149/___testing404page___/
[Sun Dec 18 748820 ] BunnyCDN-CacheKey: ___file___
[Sun Dec 18 748827 ] BunnyCDN-StorageZone: tedty
[Sun Dec 18 748834 ] BunnyCDN-PermaCache: true
[Sun Dec 18 748841 ] BunnyCDN-RequestUri: L3Rlc3Rpbmc0MDRwYWdl
[Sun Dec 18 748847 ] BunnyCDN-OriginPort: 443
[Sun Dec 18 748854 ] BunnyCDN-OriginScheme: https
[Sun Dec 18 748861 ] BunnyCDN-OriginPath: %2Ftesting404page
[Sun Dec 18 748868 ] BunnyCDN-OriginPathBase64: L3Rlc3Rpbmc0MDRwYWdl
[Sun Dec 18 748875 ] BunnyCDN-OriginHostHeader: www.example.com
[Sun Dec 18 748881 ] BunnyCDN-OriginHost: 192.237.172.37
[Sun Dec 18 748888 ] Host: www.example.com
[Sun Dec 18 748895 ] Accept-Encoding: gzip, deflate, br
[Sun Dec 18 749107 ] [authz_core:debug] mod_authz_core.c(818): AH01626: authorization result of Require all granted: granted
[Sun Dec 18 749130 ] [authz_core:debug] mod_authz_core.c(818): AH01626: authorization result of <RequireAny>: granted
[Sun Dec 18 749139 ] [core:trace3] request.c(362): request authorized without authentication by access_checker_ex hook: /testing404page
[Sun Dec 18 755757 ] [core:info] AH00128: File does not exist: /var/www/html/testing404page
[Sun Dec 18 755853 ] [http:trace3] http_filters.c(1132): Response sent with status 404, headers:
[Sun Dec 18 755874 ] [http:trace5] http_filters.c(1139): Date: Sun, 18 Dec 12:50:51 GMT
[Sun Dec 18 755883 ] [http:trace5] http_filters.c(1142): Server: Apache/2.4.54 (codeit) OpenSSL/1.1.1q+quic
[Sun Dec 18 755891 ] http_filters.c(961): Content-Length: 196
[Sun Dec 18 755909 ] http_filters.c(961): Content-Type: text/html; charset=iso-8859-1
[Sun Dec 18 755931 ] ssl_engine_io.c(1856): coalesce: have 0 bytes, adding 181 more (buckets=1)
[Sun Dec 18 755950 ] ssl_engine_io.c(1856): coalesce: have 181 bytes, adding 196 more (buckets=1)
[Sun Dec 18 755959 ] ssl_engine_io.c(1917): coalesce: passing on 377 bytes
[Sun Dec 18 756002 ] ssl_engine_io.c(2406): OpenSSL: write 399/399 bytes to BIO#7f7a84004aa0 [mem: 7f7a84217463]
[Sun Dec 18 756050 ] ssl_engine_io.c(2415): OpenSSL: I/O error, 5 bytes expected to read on BIO#7f7a84007b70 [mem: 7f7a84217463]
[Sun Dec 18 12:50:56.761328 ] ssl_engine_io.c(2415): OpenSSL: I/O error, 5 bytes expected to read on BIO#7f7a84007b70 [mem: 7f7a84217463]
[Sun Dec 18 12:50:56.761470 ] ssl_engine_io.c(2406): OpenSSL: write 24/24 bytes to BIO#7f7a84004aa0 [mem: 7f7a84217463]
[Sun Dec 18 12:50:56.761586 ] ssl_engine_kernel.c(2225): OpenSSL: Write: SSL negotiation finished successfully
[Sun Dec 18 12:50:56.761613 ] [ssl:debug] ssl_engine_io.c(1151): AH02001: Connection closed to child 128 with standard shutdown (server www.example.com:443)
If it helps here are the enabled modules.
Loaded Modules:
core_module (static)
so_module (static)
http_module (static)
access_compat_module
actions_module
alias_module
allowmethods_module
auth_basic_module
auth_digest_module
authn_anon_module
authn_core_module
authn_dbd_module
authn_dbm_module
authn_file_module
authn_socache_module
authz_core_module
authz_dbd_module
authz_dbm_module
authz_groupfile_module
authz_host_module
authz_owner_module
authz_user_module
autoindex_module
brotli_module
cache_module
cache_disk_module
cache_socache_module
data_module
dbd_module
deflate_module
dir_module
dumpio_module
echo_module
env_module
expires_module
ext_filter_module
filter_module
headers_module
include_module
info_module
log_config_module
logio_module
macro_module
mime_magic_module
mime_module
negotiation_module
remoteip_module
reqtimeout_module
request_module
rewrite_module
setenvif_module
slotmem_plain_module
slotmem_shm_module
socache_dbm_module
socache_memcache_module
socache_redis_module
socache_shmcb_module
status_module
substitute_module
suexec_module
unique_id_module
unixd_module
userdir_module
version_module
vhost_alias_module
watchdog_module
dav_module
dav_fs_module
dav_lock_module
lua_module
mpm_worker_module
proxy_module
lbmethod_bybusyness_module
lbmethod_byrequests_module
lbmethod_bytraffic_module
lbmethod_heartbeat_module
proxy_ajp_module
proxy_balancer_module
proxy_connect_module
proxy_express_module
proxy_fcgi_module
proxy_fdpass_module
proxy_ftp_module
proxy_http_module
proxy_hcheck_module
proxy_scgi_module
proxy_uwsgi_module
proxy_wstunnel_module
ssl_module
systemd_module
cgid_module
http2_module
php5_module
proxy_http2_module
pagespeed_module
It's a dedicated server/IP and only one domain.
I've played around with the <VirtualHost>'s to see if thats the issue but nothing worked.
Here is my httpd.conf (cleaned up for the post)
ServerRoot "/etc/httpd"
Listen 80
Include conf.modules.d/*.conf
User apache
Group apache
<Directory />
AllowOverride All
</Directory>
DocumentRoot "/var/www/html"
<Directory "/var/www">
AllowOverride None
# Allow open access:
Require all granted
</Directory>
<IfModule dir_module>
DirectoryIndex index.html
</IfModule>
<Directory "/var/www/html">
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>
<Files ".ht*">
Require all denied
</Files>
ErrorLog "logs/error_log"
LogLevel error
<IfModule logio_module>
# You need to enable mod_logio.c to use %I and %O
LogFormat "%D %h %l %u %t %{Host}i \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
</IfModule>
<IfModule log_config_module>
#LogFormat "%D %h %l %u %t %{Host}i \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
#LogFormat "%D %h %l %u %t %{Host}i \"%r\" %>s %b" common
</IfModule>
<IfModule alias_module>
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
</IfModule>
#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/www/cgi-bin">
AllowOverride None
Options None
Require all granted
</Directory>
<IfModule mime_module>
TypesConfig /etc/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
</IfModule>
AddDefaultCharset UTF-8
<IfModule mime_magic_module>
MIMEMagicFile conf/magic
</IfModule>
EnableSendfile on
# Supplemental configuration
#
# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf
Protocols h2 http/1.1
And the ssl.conf (also cleaned up)
Listen 443 https
SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
#SSLRandomSeed startup file:/dev/random 512
#SSLRandomSeed connect file:/dev/random 512
#SSLRandomSeed connect file:/dev/urandom 512
SSLCryptoDevice builtin
#SSLCryptoDevice ubsec
<VirtualHost *:443>
DocumentRoot "/var/www/html"
<IfModule mod_ssl.c>
ErrorLog logs/ssl_error_log
LogLevel debug trace5
</IfModule>
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
BrowserMatch "MSIE [2-5]"nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log "%D %t %{X-Forwarded-For}i %{Host}i \"%r\" %b \"%{User-Agent}i\" %>s"
ServerName www.example.com
ServerAlias example.com
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/www.example.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.example.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/www.example.com/chain.pem
</VirtualHost>
Could this madness be normal and I'm missing the logic behind it?
Perhaps there is a better way to debug and find the issue?
Any insight or suggestions would be very much appreciated.
Edit 1
Here is the php.conf - I had it renamed (disabled) during the above tests but it seems I need it enabled for my admin panel. Probably doesn't make a difference but thought to post anyway.
AddHandler php5-script .php
AddType text/html .php
#
# Add index.php to the list of files that will be served as directory
# indexes.
#
DirectoryIndex index.php
php_value session.save_handler "files"
php_value session.save_path "/var/lib/php/session"
php_value soap.wsdl_cache_dir "/var/lib/php/wsdlcache"
Edit 2
This is confirmed not the CDN but I did find another clue. The duplicate entries for 404 pages in apache only show up when you access with the domain. If I access the URL with the IP , just 1 request on apache.

Answer 1

This was in fact the CDN. They make 2 requests when they first encounter a URL. This is due to the way their CDN and push style cache is setup. I'll have to cut down on all the apache re-write rules to lower the server response times.