I have a Java EE Application ,Where i want to monitor some of my HTTP request response & I want to write those req,resp into a file.Also it should be in user control i.e When ever the user wants to monitor he can switch on/off in a application. How to do that?? There are several other tools which we can monitor ,but i don't want it manually.
You can set up a proxy server, like the infamous Burp Proxy does. Here is a small example.
You can also easily read the HTTP requests and responses in this manner, writing them to a file like you want.
Related
I am using a proxy and have set AUTO_AUTOTHROTTLE_ENABLED to True. I had the impression that scrapy throttles the sites which I am crawling, instead it seems like scrapy throttles requests to proxy itself. How do I throttle requests to sites instead of proxy?
Update: I am manually setting proxy in meta while making each request, instead of using the proxy middleware.
I don't think this is possible to do solely from the spider side. By looking at the throttling algorithm and at the AutoThrottle extension source code, you can see that the delay being used is the time difference between sending a request and getting back a response. Everything that happens in between is added up to this delay (including the proxy delay).
To further verify this, consider the steps:
AutoThrottle is using latency information from the response, found
in the response.meta['download_latency] (see here)
The latency information ('download_latency') is set in the dedicated callback once the download is completed, by subtracting the start time from the current time (see here).
The start time is actually set just before the download agent is instructed to download the request, which means everything in between is added up to the final latency (see here).
If you want to actually throttle according to target latency through a proxy, this will have to be handled by the proxy itself. I suggest using some of the managed proxy pool solutions like Crawlera.
I have all the requests going to access.log in weblogic server, I need to stop logging few request patterns. Is there any possibility ?
I already customized the access loggers with CustomELFLogger and seems to be there is no option to stop the logs not to go to access.log file.
Any other thoughts ?
There is no feature to filter the particular requests but a workaround would be , implement the customelflogger and replace cs-uri filed with custom class.
In the custom class, just put a condition for the specific requests that needs to be filtered and leave the rest.
It is working as expected.
I am kind of confused. So pls go easy on me. Take any standard web application implemented with mvc, like codeigniter or rails. The scripts gets executed only when a browser sends request right. So when a user logs in and sends request the server recieves it and sends him response.
Now consider the scenario where apart from the regular application i also need something like a backend process. For example a script which checks whether a bidding time is closed and sends the mail to the bidder that the bidding is closed and chooses the bid winner. Now all these actions has to be done automatically as soon as the bidding time ends.
Now if this script is part of a regular app then it should be triggered by the client(browser) but i dont want that to happen. This should be like a bot script which must run on the server checking the DB for events and patterns like this.
How do i go about doing something like this. Also is this possible to have this implemented on a regular shared or dedicated hosting where we dont have shell access but only ftp access.
You'd have to write your script as a standalone program and either have it run continuously in the background or have cron (or some other scheduling service; also only works if you're only interested in time-based events) execute it for you.
There are probably hosts that have shell-less ways to do this (fancy GUI interfaces for managing background processes or something,) but your run of the mill web host with only FTP access definitely doesn't.
You need a cron job, it's easy to set up on linux. That cron job will either call the command line version of PHP with your script or create a local HTTP request with curl or wget.
If you don't have access then you need an external site that automatically generates periodic HTTP requests. A cheap one is setcronjob.
I am using IBM HTTP Server 6.1 / Apache 2.0.47. I would like to pull a specific piece of data out of all requests coming through the HTTP server and if it exists log the found data along with the target URL. It needs to be as efficient as possible.
Is a filter appropriate or a handler?
Does a filter/handler exist that I can configure and use as is or do I need to write something? How do I configure, or write this?
Thanks.
You could use mod_security apache module , which have a good audit log tool SecAuditLog (log all headers), that you can declench by http status. You'll find as well fine filters, that will maybe fits your needs.
And do not hesitate to ask servfault gurus on that.
What I am trying to achieve is to have Apache's mod_proxy_balancer check if a request was already made using a Memcache store.
Basically:
Streaming media request comes in.
Check if streaming media has already been served with Memcache.
If so, can that streaming media server handle another request.
If so send request to said streaming media server.
If not send request to the next streaming media server in line.
Store key:value pair in Memcache.
My questions are:
Does mod_proxy_balancer already do this in some way?
Is there anyway to make Apache a content-aware load balancer?
Any other suggestions would be greatly appreciated too, other software, other approach, etc.
Cheers.
Looking at 'mod_proxy_balancer.c'; one could, as suggested in the comments in the file, add additional lbmethods. Something along the lines of "bymemcached_t" or "bymemcached_r" where the t and r endings denote the "bytraffic" and "byrequests" methods respectively. We would do our pseudo code above and if not found proceed to the other methods and save the result in the memcached store.
In my research I came across HAProxy which does exactly what I want from its documentation using the balance algorithm option of 'uri' just not using Memcached. Which is fine for my purposes.