I have some WCF services which are running great locally; client can consume them and the server is putting data in the DB as expected. The problem is that when I deploy these to a staging machine, all I can see are HTTP 500 errors.
How do I start debugging the problem?
Given that it's only on staging and not on my local dev machine, I assume it's an IIS configuration problem somewhere.
When I use Fiddler to see what's being sent and what the response is, I can see (as expected) correct request data, and only a 500 as the response -- no further details.
I'm pretty green to WCF and IIS, so it's probably something obvious; I've used aspnet_iisreg, deployed my .svc file and all the built DLLs/files from bin; maybe I missed something.
I looked in the IIS logs, but they're pretty skimpy; no error information there, either (or maybe I'm looking in the wrong place?)
(More important than solving the specific problem is figuring out how to see enough details about errors so I can work through problems myself.)
Edit: I of course checked the event logs first -- and surprisingly, didn't find any mention of the exceptions. So I assume that the service is at least being invoked, and that something is faulting in the middle.
The first place to look for errors is event log on the server. There should be basic information why request was not processed. If it is WCF related you can turn on WCF tracing and check for more details in generated logs.
Add:
<httpErrors errorMode="Detailed"/>
In Web.config under:
<system.webServer>
And see what's happening in more details.
'Http 500 Internal Server Error' might occur if your Service Account's password got expired. Please make sure that you don't have any issues with Service Account which is running the app pool on IIS.
It turns out that the server was returning a 500 because of a huge dataset returned; WCF puts some limitations on the size of data (and strings) you can return, to prevent DOS attacks. I solved the problem by increasing the limits, and decreasing the size of data returned (where applicable).
Related
Our users are restless. They keep complaining about woolly, unmeasurable stuff, particularly slowness, without giving specifics, which of course makes it very difficult to track down.
Nonetheless, it is quite possible that they are right, that there are server calls that are taking way too long to come back. So I want to put some kind of sniffer on the web site (we're using ASP.NET MVC 4 on IIS7) that will log any call that takes more than n seconds to turn around, or that returns more than x megabytes of data, along with all request parameters, the response size, and maybe a certain amount of response data.
I haven't a clue how to do this, though. Any suggestions?
here is my take on this:
FRT
While you can use failed request tracing to log slow requests, in my experience is more useful for finding out why a request fails before it hits your application, rather than why its running slowly. 9/10 times its going to simply show you that the slowdown is in your code somewhere.
Log Parser
Yes you can download and analyze iis logs. I use Log Parser Lizard to do the analysis - its a great gui over log parser. Here's a sample of how you might query slow requests over 1000ms:
SELECT
To_String(To_timestamp(date, time), 'dd/MM/yyyy hh:mm:ss') As Time,
cs-uri-stem, cs-uri-query, cs-method, time-taken, cs-bytes, sc-status
FROM
'C:\inetpub\logs\LogFiles\W3SVC1\u_ex140721.log'
WHERE
time-taken > 1000
ORDER BY time-taken desc
New Relic
My recommendation - go easy on yourself and sign up for a free trial. No I don't work for them, but I've used their APM product a lot. Install the agent on the server - set it up. In 10 mins you will be amazed at the data you see about the site. Trust me.
Its designed to work in production environments and gives you amazing depth of info on what's running slow, down to the database query and stack traces. Its pure awesome. Once its setup wait for the next user complaint, log in and look at traces for the time frame.
When your pro trial ends, you can still get valuable data on the free tier, but it will only keep last 24 hours. We purchased licenses -expensive yes, but worth every cent. Why? Time taken to identify root causes was reduced by an order of magnitude, we can get proactive by looking at what is number 2, 3 and 4 on the slow requests list and working those before they become big problems, and finally the alerting makes us much more responsive when things were going wrong.
Code it
You could roll you own. This blog uses Mvc ActionFilters to do the logging. You could also use an HttpModule similar to this post. The nice thing about this approach is you can compile and implement the module separately from your application, and then just drop in the dll and update web.config to wire up the module. I would be wary of these approaches for a very busy site. Also, getting the right level of detail to fully identify the root is challenging.
View Requests
As touched on by Appleman1234, IIS has a little known feature to look at requests currently executing. Its handy for the 'hey its running slow right now' situation. You can use appcmd.exe or the IIS gui to do it. You will need to install the 'Request Monitor' IIS feature for this to work. This approach is ok for rudimentary narrowing of the problem, but does not show you whats running slowly in your controller.
There are various ways you can do this:
Failed Requests Tracing(FRT) – formerly known as Failed Request Event Buffering (FREB) with custom failure condition of takes over a certain time to load / run
Logging request information with IIS logging functionality and then using a tool like LogParserStudio
Using tools like Fiddler or IISMonitor on the IIS server to capture request information
For FRT the official documentation is available here and information how to capture dumps for long running process is avaliable here
For logging request information in IIS information about log file analysis is located here
For information on configuring Fiddler to capture IIS requests find information here
A summary of the steps in the linked resources is provided below.
For FRT
From IIS Manager for a given site,In the Actions pane, under Configure, click Failed Request Tracing and enter desired values in dialog box to enable Failed Request Tracing.
From IIS Manager for a given site, under IIS click Failed Request Tracing Rules, in order to define rules of failure for a given request. In the Actions pane, click Add and follow the wizard.
The logs will go in the directory you specify and are viewable in a web broswer.
For IIS logging
Logging is enabled by default on IIS
From IIS Manager for a given site,under IIS click Logging, and in the Actions Pane, click Enable to enable logging if it isn't already.
From IIS Manager for a given site,under IIS click Logging, and then configure as desired and click apply.
Install LogParser, .Net 4.x and LogParserStudio (if you need additional steps see here
Open LogParserStudio and add logs to it, you then can use SQL queries to get information from the log files.
For Fiddler
You need to change the user that IIS runs as to a user that can launch applications, like Fiddler (instead of Network Service), and then launch Fiddler with that user.
Also see Monitor Activity on a Web Server (IIS 7) for further information.
I'm about a month away developing my silverlight application (this is my first). Everything went rather smoothly until today, when out of the blue I started getting this message:
An error occurred while trying to make a request to URI 'http://localhost:2682/Services/Authentication/LoginService.svc'. This could be due to attempting to access a service in a cross-domain way without a proper cross-domain policy in place, or a policy that is unsuitable for SOAP services. You may need to contact the owner of the service to publish a cross-domain policy file and to ensure it allows SOAP-related HTTP headers to be sent. This error may also be caused by using internal types in the web service proxy without using the InternalsVisibleToAttribute attribute. Please see the inner exception for more details.
I'm using WCF Services and this issue never appeared until now.
I've added a clientdomain.xml and clientaccesspolicy.xml file to my [projectname].web folder, and re-wrote them about a 1000 different ways.
I've also used Fiddler and it shows me that the error is on both those files, the error is
[Fiddler] The socket connection to localhost failed. ErrorCode: 10061. No connection could be made because the target machine actively refused it 127.0.0.1:2682
I've searched the error "10061" and it has to do with socket definition. But I couldn't find any solution to that.
Don't know if it has anything to do with it, but my "ASP.net Development Server" port is 6939.
Keep in mind that the app has NOT been deployed, so this is only happening locally. I'm using MS VS 2010 and MS SQL Server 2008.
Am I doing anything wrong or is this a silverlight issue??
On a last note, I haven't changed anything on Port, socket or service configuration. Last thing I was doing was editing a XAML file on client side and and the app started throwing me this error.
Need help, can't do anything until this is solved!!!!
Thanks.
i think you are using you app on localhost and a dynamic port is getting assigned and this port is not fixed and every run and that causes the refuse problem. if you want to fix this, create a solid url for example,
http://localhost/apps/Services/Authentication/LoginService.svc
Well, last night, just before I went to bed, I noticed something odd. In my "ServiceReferences.ClientConfig" file, the endpoint ports for each one of my services where diferent from the ones the silverlight machine used, so going on a hunch (and because I was reaching my sanity breakpoint) I decided to eliminate all my Service References and re-add them again.
I worked... go figure. Still don't know why this happened and if anyone could shed some light on the subject, I would appreciate it. It's kinda of annoying having to re-add all my services references. Right now I have only 6 of them, but in the near future they may go over 20, and if this happens again... well, it's going to be a real pain...
Thanks
I have a problem running a Java (using Metro) client against a .NET STS and secured web service. However, when I run my .NET based client, it always works.
As you probably already know, when a JAX-WS client is ran, it requests metadata from the service during runtime (even though it already has ran wsimport during design time). However, it seems that this runtime metadata request is where my problem is.
The problem that I'm facing is that during the runtime requests of metadata, some WSDL's exported by my WCF service caused the Java client to just 'hang' during the mex requests. When it hangs, it doesn't even get to the point of issuing the RST request.
For example, I can get to a spot where I have 9 [OperationContract] attributes and it works. But when I add a 10th service method, it doesn't work. However, if I remove one of the 9, then it works. I know there isn't a problem with a particular method because I can mix and match and the same pattern holds.
I can't seem to deduce a pattern or reason for why some WSDL's work and some don't. I strongly doubt there is a limit to the number of service methods. However, could this be a problem with "overall complexity" with the exported WSDL?
Does anyone have any ideas? Have anyone run into this issue before?
If more information is needed, I can glady post it. I'm just trying to keep the initial post a manageable length.
I will also add that I'm running the STS and secured web serivce in .NET 4, and they are based on WIF (so I don't have to worry about security settings). My .NET client is also .NET 4. On the Java side, I'm using Netbeans 6.9.1 with Metro 2.1 running Glassfish 3. I've verified that I receive the same issue running on Metro 2.0.
Please see WSDL Requests with Metro/JAX-WS/WSIT During Runtime for a detailed answer.
Following the example given, there're chances that MaxMessageReceivedSize or MaxStringContentLength limit is reached. Have you tried increasing the values of MaxMessageReceivedSize and MaxStringContentLength for binding? You can try enabling WCF traces, there'd be a warning logged if any such limit is reached.
I have a WCF service that has been working flawlessly for 3 months. It is consumed by local clients on the same server hosting the WCF service and local network clients. It uses SSL and basic authentication for security.
A few nights ago, the local client (local network clients not affected) started receiving 404 errors whenever it tried to use the service. I am able to open a browser on the server hosting the WCF and view the WSDL and even call the "put" command and get the expected "method not allowed". I have confirmed that no software or hardware changes have been made to the hosting server. I have confirmed that the SSL key is valid. I have confirmed that the permissions for the Application Pool are sufficient. I have confirmed that no firewall is running. The only thing odd is the IIS log showing that the first post does not contain the basic authentication user. However, the next line in the log does and shows a 200 response. I am not entirely sure that log is not normal. See below. I was hoping somebody could give me another place to research to find the problem. Please let me know.
2010-08-28 10:30:03 192.168.100.100 POST /protected/Service_Name_Here.svc/put - 443 - 192.168.100.100 - 401 2 5 2
2010-08-28 10:30:03 192.168.100.100 POST /protected/Service_Name_Here.svc/put - 443 User_Name_Here 192.168.100.100 - 200 0 0 5
EDIT: The local client that is throwing the error is transferring large files to the WCF service. The local network clients are transferring small files and not throwing the error. I found this link that suggests that the default transferMode="Buffered" will throw a 404 for files above 20 MB file. The fix for this person was to change the transferMode="Streamed". However, the "Streamed" setting only allows 1 parameter to be passed to the WCF service. I have multiple parameters so I need to find a fix for "buffered" mode.
The fix for this person was to change the transferMode="Streamed". However, the "Streamed" setting only allows 1 parameter to be passed to the WCF service. I have multiple parameters so I need to find a fix for "buffered" mode.
Sounds like that's the correct fix, however the caveat is that streamed mode requires custom message contracts; you can't use the "RPC" style that WCF pushes as a default for operations. If you need to provide more than one parameter in a streamed mode transfer, simply add them to your custom message contract.
Here's a nice discussion on the subject from Microsoft.
If you have problems with message size be aware that there are 3 levels of configuring accepted request size for IIS:
WCF - default max message size 65KB (maxReceivedMessageSize)
ASP.NET runtime hosting WCF - default max request size is 4MB (maxRequestLength)
IIS 7 with request filtering installed - default max request size about 28MB (maxAllowedContentLength)
If WCF rejects your message you will probably get meaning full error but for ASP.NET and IIS you will get exactly HTTP 404.
Streaming will not help you unless you change your operations.
I have a WCF service that among other bindings also uses WebHttpBinding for JSON inputs/results.
I made a custom IErrorHandler implementation in order to be able to set the StatusCode to 400 when something goes wrong and also return a JSON understandable message. It´s the straight implementation that you can find everywhere (nice way described here).
My problem is: when I test it locally using Visual Studio Web Development Server (Cassini) it works perfectly. However, when I deploy it to my test server (Windows 2008 with standard config for IIS and everything else) it does not work.
When I call it and debug with Firebug I get a HttpStatusCode 200 as a return and no response text. With Fiddler I get a HttpStatusCode 504 and no return at all. However, the behavior I expected (and what happens locally) is a call to the error callback of the ajax call with the responseText set.
I debugged it remotely and everything looks just fine. The execution pipeline is OK and all the classes are called as they should be just like they are locally, except it does not work.
Any suggestions? I´m pretty much out of options here to figure this out.
Thanks a lot!
if firebug and fiddler are giving different results, what happens if you telnet to it directly and perform a request (Something like:)
GET /VirtualDirectoryAndGetData HTTP/1.1
HOST: example.com
[carriage return]
It wouldn't surprise me if you're somehow getting odd headers/formatting back (to explain why firebug/fiddler disagree)
Another thing to test would be publishing to your dev machine to see if it's a machine-specific issue or a server vs dev webserver issue.
If it's happening anywhere outside VS, you might also try commenting out the lines where you set
rmp.StatusCode = System.Net.HttpStatusCode.BadRequest;
rmp.StatusDescription = "Bad request";
This may indicate whether it's a response code issue or an error handler issue.
If you can edit your question to include the results (with sensitive info removed), we'll see if we can track it down further.
Edit: after looking at the question again, it may well be that the server is erroring before it can send ANY response. FF might assume 200 by default, whereas ie might assume 504 (Gateway Timeout). This is total speculation but is possible. Do you see anything in the event logs?
I had a similar issue which I was able to solve. Take a look at the IIS settings. Details on how I overcame the issue are in this post: IErrorHandler returning wrong message body when HTTP status code is 401 Unauthorized