Why do some HTTPS / TLS (might be TLS 1.3 only?) connections take so long to show up in Squid Proxy's access.log?
Running Squid Proxy 4.15 on Rocky Linux (have tried other Linux variants with earlier Squid versions). First thought certain sites were not going through the configured proxy (Firefox HTTP / HTTPS proxy settings), but then observed that the response time was in minutes and learned that Squid only writes to the access log once the request completes. Here's a sample:
07/Sep/2022:15:12:56 dns_time=- url="www.virustotal.com:443" duration_ms=173168 bytes=644252 mime=- src=##.##.##.##:50251 dest=74.125.34.46:443 http_method=CONNECT status=200 http_user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0" squid_req_hier=TCP_TUNNEL:HIER_DIRECT
The browser experience when visiting this site is normal and when observing network connectivity via Firefox debug, there are no in-process network calls that correlate to the 2.8m response time logged by Squid within access.log. When troubleshooting or trying to get an understanding of proxy traffic via access.log entries, this delay makes it quite difficult to derive patterns or test proper connectivity without waiting long periods of time. I've combed through http://www.squid-cache.org/Doc/config/logformat/ and other config items under http://www.squid-cache.org/Doc/config/ and cannot identify anything to log this entry more quickly or figure what is taking so long.
Here is my Squid configuration:
acl SSL_ports port 443
acl Safe_ports port 443
acl CONNECT method CONNECT
http_access allow !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow all
http_access deny all
http_port 3128
tls_outgoing_options min-version=1.2
logformat splunk_fun %tg dns_time=%dt url="%ru" duration_ms=%tr bytes=%st mime=%mt src=%>a:%>p dest=%<a:%<p http_method=%rm status=%>Hs http_user_agent="%{User-Agent}>h" squid_req_hier=%Ss:%Sh
access_log daemon:/var/log/squid/access.log splunk_fun
Any insight into lengthy HTTPS / TLS response times (not impacting end-user browser performance) would be greatly appreciated.
Related
I’m trying to host an apache webserver on a phone with termux, and the final goal is to make this server mobile by using 4G.To achieve this I thought it was possible to use a reverse SSH connection (to set a connection from the phone, wherever he is on internet) to a proxy in my home. The DNS of my website will redirect to this (reverse) proxy, which will redirect to my phone webserver, by using the reverse SSH connection.
For now, the phone server is on the same wifi network as the proxy, not in 4G.
So this phone connects to a computer which acts as a proxy, by connecting with reverse SSH on port 8082. From this computer, when I search http://localhost:8082 on internet on the computer, it works, I have access to the website.
However I can’t access the website from another device on the local network or from internet (I tried http://IPproxyComputer:8082 and I tried from a device on internet http://MyPublicIP with a port redirection from 80 to IPproxyComputer :8082 on my router). I don’t have any firewall on.
Here is a diagram to clarify
I didn’t find anyone trying to host a webserver with this config but someone explain how to give access to a device with reverse SSH and a squid proxy here.
So I thought this is what I must do, but with a different configuration for Squid because in my case, the request comes from someone on internet or local network to the webserver, so Squid must listen on a port, let say 8081 and redirect to 8082 where the SSH connection is, but I failed to set this up.
I need help to undersand two things:
-Why is the page accessible from the proxy with http://localhost:8082, but not from a device on the local network with http://IpproxyComputer:8082 ? Is it because of an authorization (and is it possible to change it?). Or is it just impossible to do this kind of connection ?
Is it a solution to use squid ? How to set this up ?
Here is the reverse SSH command I used from the phone webserver :
ssh -R 8080:localhost:8081 UsrProxyComputer#IPproxyComputer
Here is the config I tried for squid (it is the first time I use it and I am really a beginner in networks)
Define the listening port and default site
# Declare that virtual hosts will be used for allowing the reverse proxy
http_port 8081 accel vhost
# First we will configure the servers in our system
cache_peer 127.0.0.1 parent 8082 0 proxy-only name=InterRedir
# Create an additional ACL for local network access
acl localip src 192.168.0.0/24
http_access allow localip
cache_peer_access InterRedir allow localip
# Next we will map domains to the specific systems
# 1) This is done by creating an ACL for the domain
# 2) Then granting http access to it to allow the connection
# to get through.
# 3) Then mapping an acl to the specific server
# MyWebSite.com
acl MyWebSite_acl dstdomain MyWebSite.Com
http_access allow MyWebSite_acl
cache_peer_access InterRedir allow MyWebSite_acl
# Additional ACL definitions
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
acl purge method PURGE
acl CONNECT method CONNECT
# Restrictions
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny all
# Disable caching
cache deny all
I Have configured Both Apache and Wildfly to use AJP in order to achieve the Proxy so i can produce Access-logs
Acccess Logs are Enabled from management profile of Wildfly
Port is listening on 8009
Apache has a Virtual Host listening on port 80
configured with the ProxyPass command.
Access Pattern is configured
and the result is receiving logs that have been proxied but not the original client ip is diplayed but only the loopback of 127.0.0.1
So am asking about a way to reveal the client Ip that requests the Apache Server.
syntax is okay and %a does display the 127.0.0.1
Thanks!
I have tried multiple access patterns (%a,%h,{i,xxx}...
I have tried both X-Forward-For ,X-Forwarded-For on Apache PreserveHost On etc..
I have tried tcpdump the port 8009 receiving 0 packets on the monitoring
It seems that it is possible to get Apache server to Proxy and Manage SSL handshake on https requests and service them as 'http' thru another server behind it.
I have configured an apache server for ProxyPass using following configuration
SSLProxyEngine On
ProxyPass / http://localhost:8081/
ProxyPassReverse / http://localhost:8081/
I am able to get all all traffic to the apache server that is listening to port 8080 direct and serve by the localhost:8081 server so
http://localhost:8080/hi is being correctly served by http://localhost:8081/hi
However the following does not work :
http**s**://localhost:8080/hi to be served by http://localhost:8081/hi
Apache is trying to pass the https:// traffic to the 8081 server, without managing the SSL handshake.
Your Apache listener on port 8080 is an http listener, not an https listener. You can't handle both types of traffic on the same port. If you want to handle SSL traffic, you'll need to set up a new context on another port with SSLEngine On and all the other normal SSL configuration (certificate, key, etc).
This question has one version of this configuration.
Also this post.
Currently, I am running a website which is running on Tomcat which has Apache in front and varnish for caching. Tomcat is running on port 8080 and port 81 is used as varnish back end. I need to implement a feature for which I need to know the IP address of the client's machine visiting the website. I've tried to access the IP address by the header X-FORWARDED-FOR.
When the website is accessed from port 81 (i.e. http://mywebsite.com:81/) I can get the IP address of the client's machine but unfortunately when accessing through default port 80 (i.e. http://mywebsite.com/) it's returning the localhost IP address (127.0.0.1). Can anyone suggest what can be the worked around to get the user's real IP address from port 80 as well?
Your setup, if I understood correctly, is as follows:
Varnish (port 80) -> Apache (port 81) -> Tomcat (port 8080)
And you would like the actual remote IP to show up as REMOTE_ADDR on the Tomcat server.
Varnish appends X-Forwarded-For header by default, so that's already sorted. To get Apache to pass the actual remote IP to Tomcat, you should install reverse proxy add forward module for Apache (mod_rpaf). mod_rpaf does exactly what you're looking for. The appropriate config for Apache would be:
<IfModule !rpaf_module>
LoadModule rpaf_module modules/mod_rpaf-2.0.so
</IfModule>
<IfModule rpaf_module>
RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1
RPAFheader X-Forwarded-For
</IfModule>
After enabling the module Tomcat should see the correct REMOTE_ADDR header as well as the HTTP_X_FORWARDED_FOR header.
I have apache httpd that I want to proxy to two different tomcat servers.
I see this:
http://tomcat.apache.org/connectors-doc-archive/jk2/proxy.html
But that is only for one tomcat server. What if I had one server running on 8081 in addition to a tomcat running at 8080?
There's an easier way to setup load balancing using mod_proxy_balancer. Simply list the tomcat servers under a balancer list than put that balancer in your ProxyPass:
<Proxy balancer://mycluster>
BalancerMember http://tomcat1:8080/
BalancerMember http://tomcat2:8081/
</Proxy>
ProxyPass /test balancer://mycluster
Apache httpd two out-of-the-box options for proxying to any number of backend Tomcat instances:
mod_proxy_http
mod_proxy_ajp
They are configured identically to each other, except that the former uses the HTTP protocol for communication and the latter uses the AJP protocol and URLs that start with ajp:// instead of http:// for the backend server. Both can be configured for load-balancing, failover, etc. in the same way. You can proxy to completely separate Tomcat instances (i.e. no load-balancing: just separate backends) by providing separate proxy configuration for separate URL spaces (e.g. /app1 -> Tomcat1 and /app2 -> Tomcat2) or you can configure the two (or more) backend instances for load-balancing, etc.
Specifically, look at the documentation for the following httpd configuration directives:
<Proxy>
BalanceMember
ProxyPass
ProxyPassReverse
You can find documentation for all of these here:
http://httpd.apache.org/docs/2.2/mod/mod_proxy.html (General)
http://httpd.apache.org/docs/2.2/mod/mod_proxy_http.html (HTTP)
http://httpd.apache.org/docs/2.2/mod/mod_proxy_ajp.html (AJP)
http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html (load-balancer)
If you want to use the AJP protocol and you have more complex configuration needs, you can also use mod_jk (not mod_jk2, which is an old, dead, abandoned, completely irrelevant project, now). You can find out more about mod_jk on the Tomcat site here: http://tomcat.apache.org/connectors-doc/
mod_jk has a radically different configuration procedure and a lot more AJP-specific options than mod_proxy_ajp.
The (short) documentation you mentioned in your original post (from the old mod_jk2 docs) points to Apache httpd's mod_proxy_ajp and mod_proxy_balancer modules (though it points to the unstable httpd 2.1, which was the bleeding-edge at the time that documentation was written). You were on the right track: you just needed to keep reading. You can definitely proxy to as many backend instances of Tomcat as you want with any of the modules described here.
You can install HAProxy on either 3rd server which will work as LB to both of them or you can install HAProxy on any one of them and then do following configuration.
To install HAProxy (if you're running Ubuntu/Debain distro)
$ sudo apt-get install haproxy
# Setup config file in /etc/haproxy/haproxy.cnf per requirement
# change /etc/default/ to Enabled = 1 and restart haproxy service
after setup do following mods in config:
$ sudo vim /etc/haproxy/haproxy.cfg
global
maxconn 4096
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen webcluster *:80
mode http
stats enable
stats auth us3r:passw0rd
balance roundrobin
option httpchk HEAD / HTTP/1.0
option forwardfor
cookie LSW_WEB insert
option httpclose
server web01 192.168.0.1:8080 cookie LSW_WEB01 check
server web02 192.168.0.2:8081 cookie LSW_WEB02 check
Once done, restart HAProxy service by:
$ sudo service haproxy restart
Here 192.168.0.1 and 192.168.0.2 can be your two servers one can be running on port 8080 and another can be on 8081.
Ref. Post: http://www.leaseweblabs.com/2011/07/high-availability-load-balancing-using-haproxy-on-ubuntu-part-1/
You will also find online help if you will google about how to setup haproxy on your linux distribution if you're not using Ubuntu/Debain. But yes you can bet on it as it's proven tool for the job.