I’m trying to get a mod_perl2 application ported to AWS. As part of the port I thought I’d move from Debian Squeeze to Wheezy with the latest stable mod_perl & Apache2 combination.
The application works right up to the point I try and write JSON responses to the client. At this point, each request is canceled on the client and on the server I get the error
Apache2::RequestIO::print: (103) Software caused connection abort
whenever I write to the client, i.e.:
$self->req->print($output);
I’ve tried tcpdumping the response to the client, and I can see it being written out, but no response is received on the client end and it just barfs chips. I can’t find any information on how to get around this.
I found quite a few people asking about this question on the net without many answers. The solution to my problem was very specific but I thought I’d post what I did anyway, it may help someone.
The client was canceling the request before the response was fully written, which was crapping out Apache::RequestIO (for reasons I still don’t know).
I couldn’t work out why I was seeing this behavior.
By using tcpdump I could see that data was being written out to the client – and it looked fine.
By inspecting the page in Chrome and looking at the network stack, I could see that my request for data was being canceled after no response was received (which was odd because the code worked fine on other servers and I could see the response was being written). Debugging was may harder because with Apache crashing out with an error in print IO I couldn’t check if the bytes written equaled the bytes of data. I wasn’t sure if something was getting stuck on the server side.
So, I changed the Content-Type of the response from application/json to text/html, so that I could query the page and just look at the actual response as text. Once I did that, I could see that the response was fine.
I started to look for other causes, and I found that in the migration to the new server, I’d missed altering some URLs in the DB to point to the new server, which meant my application was trying to get some data from the old DB.
This in turn was causing a load of timing issues, which was causing my problems. Once I fixed the config, the problems went away.
Related
I’m a long-time reader here but a newbie at posting a question. Hopefully I’ll cover everything you guys need to hopefully help.
Background information:
We are running ColdFusion 10 on two servers that are load balanced (I’m not sure how they are load balanced – they are not clustered and are not using sticky sessions, this much I know). Unfortunately, I do not have access to our CF server admin at all; I have to rely on others.
I’ve implemented a punch out system that allows our users to connect to a vendor’s site to shop, then returns their items to our cart on our site. This has been working in our development servers without any issues. Everything worked well when we tested this in production as well. However, when we moved it into production last week, we started getting an error, but only when the code was running off of ONE of the load balanced servers. The error we received back from the vendor site stated that the error detail was: “I/O Exception: Server Key”. All of the research I conducted led me to believe that our CF servers needed the vendors cert (it is an https connection), so I told this to our server guy. He reinstalled the certs (he had said that they were there) and that did seem to solve the problem. I was successfully able to punch out to our vendor site from both of our load balanced servers.
We did a bit more testing (which all seemed fine) and then put it back into production this morning only to have the same issue occur. On one of the servers, this is working and on the other one it is not. My server guy tells me that the vendor certs are currently in place in the ColdFusion keystore.
Here is the cfhttp call I’m using:
<cfhttp url="#vendorURL#" method="POST" throwOnError="no" result="returnedObj">
<cfhttpparam type="XML" name="xmlPunchoutData" value="#trim(RequestPunchoutXML)#" />
</cfhttp>
Where ‘RequestPunchoutXML’ has a xml structure requesting a punch out from the vendor.
This looks possibly related: ColdFusion 10 - CFHTTP - Random peer not authenticated on SSL calls (cacerts file updated) but the error I'm getting isn't this one, though I think that they are probably related.
Questions: Any idea what is going on here? Could a badly set up load balancer be the issue here? Is it possible that the cfhttp call is starting from one of the servers and getting the response returned to the other? Could there be some reason that the certs are failing? Is this some other issue altogether that I have not yet identified? Any thoughts/ideas/suggestions would be greatly helpful.
Thanks in advance,
Janice
So I've got a problem where a small percentage of incoming requests are resulting in "400 bad request" errors and I could really use some input. At first I thought they were just caused by malicious spiders, scrapers, etc. but they seem to be legitimate requests.
I'm running Apache 2.2.15 and mod_perl2.
The first thing I did was turn on mod_logio and interestingly enough, for every request where this happens the request headers are between 8000-9000 bytes, whereas with most requests it's under 1000. Hmm.
There are a lot of cookies being set, and it's happening across all browsers and operating systems, so I assumed it had to be related to bad or "corrupted" cookies somehow - but it's not.
I added \"%{Cookie}i\" to my LogFormat directive hoping that would provide some clues, but as it turns out half the time the 400 error is returned the client doesn't even have a cookie. Darn.
Next I fired up mod_log_forensic hoping to be able to see ALL the request headers, but as luck would have it nothing is logged when it happens. I guess Apache is returning the 400 error before the forensic module gets to do its logging?
By the way, when this happens I see this in the error log:
request failed: error reading the headers
To me this says Apache doesn't like something about the raw incoming request, rather than a problem with our rewriting, etc. Or am I misunderstanding the error?
I'm at a loss where to go from here. Is there some other way that I can easily see all the request headers? I feel like that's the only thing that will possibly provide a clue as to what's going on.
We set a lot of cookies and it turns out we just needed to bump up LimitRequestFieldSize which defaults to 8190. Hope this helps someone else some day...
I am using a java raw HTTP client to connect to Shopify API (specifically, using Play Framework with the non-defualt sync driver which is actually the JDK's default driver).
My application usually manages to connect successfully and convert the temporary access token into a permanent one by calling the /admin/oauth/access_token endpoint.
However, sometimes I get this error result from the API:
Generic Error(400)
{"error":"invalid_request"}
I haven't been able to reproduce the issue with my test stores - I've tried installing a fresh store, reinstalling existing stores after uninstalling, I'm not sure why this call sometimes fail and how to debug it. The API call still continues to succeed for some stores using our application.
Some things that I am doing:
Even if the URL of the store is on a custom domain, I'm always using the https://foo.myshopfiy.com/admin/oauth/access_token URL and not the URL of the custom domain, to prevent a redirect.
I am always using an https URL and never an http one, again to prevent a redirect (we noticed a few issues with redirect with the Java HTTP client, so we aim to have zero redirects)
A thread I found about this error suggest possible problems with our SSL certificates, however I don't think this is my problem because some requests work for us, and the result of running openssl on our machine does't show any issues.
How should I proceed? Open a support ticket with Shopify?
FYI, I see that this specific problem only started yesterday on Feb 19 2013, so it might be a temporary issue.
FYI, the problem was caused by reusing a temporary access code.
Our fault - Shopify could have been more clear in their error message though.
I have some WCF services which are running great locally; client can consume them and the server is putting data in the DB as expected. The problem is that when I deploy these to a staging machine, all I can see are HTTP 500 errors.
How do I start debugging the problem?
Given that it's only on staging and not on my local dev machine, I assume it's an IIS configuration problem somewhere.
When I use Fiddler to see what's being sent and what the response is, I can see (as expected) correct request data, and only a 500 as the response -- no further details.
I'm pretty green to WCF and IIS, so it's probably something obvious; I've used aspnet_iisreg, deployed my .svc file and all the built DLLs/files from bin; maybe I missed something.
I looked in the IIS logs, but they're pretty skimpy; no error information there, either (or maybe I'm looking in the wrong place?)
(More important than solving the specific problem is figuring out how to see enough details about errors so I can work through problems myself.)
Edit: I of course checked the event logs first -- and surprisingly, didn't find any mention of the exceptions. So I assume that the service is at least being invoked, and that something is faulting in the middle.
The first place to look for errors is event log on the server. There should be basic information why request was not processed. If it is WCF related you can turn on WCF tracing and check for more details in generated logs.
Add:
<httpErrors errorMode="Detailed"/>
In Web.config under:
<system.webServer>
And see what's happening in more details.
'Http 500 Internal Server Error' might occur if your Service Account's password got expired. Please make sure that you don't have any issues with Service Account which is running the app pool on IIS.
It turns out that the server was returning a 500 because of a huge dataset returned; WCF puts some limitations on the size of data (and strings) you can return, to prevent DOS attacks. I solved the problem by increasing the limits, and decreasing the size of data returned (where applicable).
I have a WCF service that among other bindings also uses WebHttpBinding for JSON inputs/results.
I made a custom IErrorHandler implementation in order to be able to set the StatusCode to 400 when something goes wrong and also return a JSON understandable message. It´s the straight implementation that you can find everywhere (nice way described here).
My problem is: when I test it locally using Visual Studio Web Development Server (Cassini) it works perfectly. However, when I deploy it to my test server (Windows 2008 with standard config for IIS and everything else) it does not work.
When I call it and debug with Firebug I get a HttpStatusCode 200 as a return and no response text. With Fiddler I get a HttpStatusCode 504 and no return at all. However, the behavior I expected (and what happens locally) is a call to the error callback of the ajax call with the responseText set.
I debugged it remotely and everything looks just fine. The execution pipeline is OK and all the classes are called as they should be just like they are locally, except it does not work.
Any suggestions? I´m pretty much out of options here to figure this out.
Thanks a lot!
if firebug and fiddler are giving different results, what happens if you telnet to it directly and perform a request (Something like:)
GET /VirtualDirectoryAndGetData HTTP/1.1
HOST: example.com
[carriage return]
It wouldn't surprise me if you're somehow getting odd headers/formatting back (to explain why firebug/fiddler disagree)
Another thing to test would be publishing to your dev machine to see if it's a machine-specific issue or a server vs dev webserver issue.
If it's happening anywhere outside VS, you might also try commenting out the lines where you set
rmp.StatusCode = System.Net.HttpStatusCode.BadRequest;
rmp.StatusDescription = "Bad request";
This may indicate whether it's a response code issue or an error handler issue.
If you can edit your question to include the results (with sensitive info removed), we'll see if we can track it down further.
Edit: after looking at the question again, it may well be that the server is erroring before it can send ANY response. FF might assume 200 by default, whereas ie might assume 504 (Gateway Timeout). This is total speculation but is possible. Do you see anything in the event logs?
I had a similar issue which I was able to solve. Take a look at the IIS settings. Details on how I overcame the issue are in this post: IErrorHandler returning wrong message body when HTTP status code is 401 Unauthorized