We were informed from the PCI compliance team that we failed because "verbose reports" are on. I have never heard of this before. We have an IIS6 server. We are not running crystal reports or anything of the sort. I can't find any item in II6 called verbose reports. Does anyone know what verbose reports are and more importantly how to turn them off?
The documentation for verbosity in IIS 6 is located at http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/d87ac473-eddb-4f7e-9a91-ce9f168b5635.mspx?mfr=true
They can be managed via a Provider file. Documentation located at: http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/991e07c3-4143-48cc-ab39-8d49065a8388.mspx?mfr=true
They are probably reffering to IIS (6) Logggin Verbosity Levels
Level 5 is Verbose :
IIS providers report detailed information about requests
The issue was a piece of software on the server called "Event Sentry" (not sure if that is spelled correctly). I haven't got the full details yet from the server guy and I will add more when I know more. However, I figured it was best to tell the issue in case anyone else has the issue.
Related
I'm trying to diagnose a request failure scenario caused by a possible concurrent access to HttpContext. The thing is that the issue is reproducible only on a server and that too when put under load.
A simple dotnet-dump collect output will not help because that only gives the state of process at a particular point in time. What I'm trying to find out is if a dump can be triggered when a particular condition is met. The intent is to collect detailed stack trace at the point of the exception.
For example, take dump when a null ref exception is thrown when a particular property on a particular type is accessed.
As of now, I've tried to make the app code as lean as possible. Got rid of unnecessary middleware in request pipeline. But the stack trace I see I get of exception is hardly of any use.
Well, based on your comment, It seems you are looking for tool like debug-diag for linux machine. You could consider ProcDump for linux version which similar to debug-diag tool but for linux as no other suitable tool available. However, it has few fature limitaions. Please follow the below insturctions if that meet your requirement.
Installation guideline:
Please check here for details instruction
Minimum Requirements:
Ubuntu 16.04 LTS
Minimum OS:
Red Hat Enterprise Linux / CentOS 7
Fedora 29
Note:
Please note that, Curreent version does not have full feature parity with Windows version of ProcDump, specifically, stay alive functionality, and custom performance counters.
For more details, you could visit our official document as well.
Our users are restless. They keep complaining about woolly, unmeasurable stuff, particularly slowness, without giving specifics, which of course makes it very difficult to track down.
Nonetheless, it is quite possible that they are right, that there are server calls that are taking way too long to come back. So I want to put some kind of sniffer on the web site (we're using ASP.NET MVC 4 on IIS7) that will log any call that takes more than n seconds to turn around, or that returns more than x megabytes of data, along with all request parameters, the response size, and maybe a certain amount of response data.
I haven't a clue how to do this, though. Any suggestions?
here is my take on this:
FRT
While you can use failed request tracing to log slow requests, in my experience is more useful for finding out why a request fails before it hits your application, rather than why its running slowly. 9/10 times its going to simply show you that the slowdown is in your code somewhere.
Log Parser
Yes you can download and analyze iis logs. I use Log Parser Lizard to do the analysis - its a great gui over log parser. Here's a sample of how you might query slow requests over 1000ms:
SELECT
To_String(To_timestamp(date, time), 'dd/MM/yyyy hh:mm:ss') As Time,
cs-uri-stem, cs-uri-query, cs-method, time-taken, cs-bytes, sc-status
FROM
'C:\inetpub\logs\LogFiles\W3SVC1\u_ex140721.log'
WHERE
time-taken > 1000
ORDER BY time-taken desc
New Relic
My recommendation - go easy on yourself and sign up for a free trial. No I don't work for them, but I've used their APM product a lot. Install the agent on the server - set it up. In 10 mins you will be amazed at the data you see about the site. Trust me.
Its designed to work in production environments and gives you amazing depth of info on what's running slow, down to the database query and stack traces. Its pure awesome. Once its setup wait for the next user complaint, log in and look at traces for the time frame.
When your pro trial ends, you can still get valuable data on the free tier, but it will only keep last 24 hours. We purchased licenses -expensive yes, but worth every cent. Why? Time taken to identify root causes was reduced by an order of magnitude, we can get proactive by looking at what is number 2, 3 and 4 on the slow requests list and working those before they become big problems, and finally the alerting makes us much more responsive when things were going wrong.
Code it
You could roll you own. This blog uses Mvc ActionFilters to do the logging. You could also use an HttpModule similar to this post. The nice thing about this approach is you can compile and implement the module separately from your application, and then just drop in the dll and update web.config to wire up the module. I would be wary of these approaches for a very busy site. Also, getting the right level of detail to fully identify the root is challenging.
View Requests
As touched on by Appleman1234, IIS has a little known feature to look at requests currently executing. Its handy for the 'hey its running slow right now' situation. You can use appcmd.exe or the IIS gui to do it. You will need to install the 'Request Monitor' IIS feature for this to work. This approach is ok for rudimentary narrowing of the problem, but does not show you whats running slowly in your controller.
There are various ways you can do this:
Failed Requests Tracing(FRT) – formerly known as Failed Request Event Buffering (FREB) with custom failure condition of takes over a certain time to load / run
Logging request information with IIS logging functionality and then using a tool like LogParserStudio
Using tools like Fiddler or IISMonitor on the IIS server to capture request information
For FRT the official documentation is available here and information how to capture dumps for long running process is avaliable here
For logging request information in IIS information about log file analysis is located here
For information on configuring Fiddler to capture IIS requests find information here
A summary of the steps in the linked resources is provided below.
For FRT
From IIS Manager for a given site,In the Actions pane, under Configure, click Failed Request Tracing and enter desired values in dialog box to enable Failed Request Tracing.
From IIS Manager for a given site, under IIS click Failed Request Tracing Rules, in order to define rules of failure for a given request. In the Actions pane, click Add and follow the wizard.
The logs will go in the directory you specify and are viewable in a web broswer.
For IIS logging
Logging is enabled by default on IIS
From IIS Manager for a given site,under IIS click Logging, and in the Actions Pane, click Enable to enable logging if it isn't already.
From IIS Manager for a given site,under IIS click Logging, and then configure as desired and click apply.
Install LogParser, .Net 4.x and LogParserStudio (if you need additional steps see here
Open LogParserStudio and add logs to it, you then can use SQL queries to get information from the log files.
For Fiddler
You need to change the user that IIS runs as to a user that can launch applications, like Fiddler (instead of Network Service), and then launch Fiddler with that user.
Also see Monitor Activity on a Web Server (IIS 7) for further information.
I'm testing the AppFabric cache feature in my website deployed from ASP.net MVC 4. In the web.config of my website, if I set the security level to anything except no security (None/None), like this
<securityProperties mode="Transport" protectionLevel="EncryptAndSign" />
then, getting cache from the method below will throw exception,
DataCacheFactory.GetDefaultCache()
Also, in the PowerShell Cache Management, I've already set the protection level to the same level (Transport/EncryptAndSign).
But getting cache still failed.
Exception message is as following,
ErrorCode(ERRCA0017):SubStatus(ES0006):There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable... )
The situation only happens in my deployed website. But if I run my website in Visual Studio debug mode, cache would response successfully no matter what the security level is set to.
OK! I finally solved the tricky issue. This tough thing nearly made me contribute my 50 bounties (almost half) !
After some survey, I found the first clue,
Security Error in APPFabric when changed the security mode from None to Transport?
(Note, I can't easily google it out from some keywords, but the result I want was easily dug out from Msdn AppFabric Cache forum. I thought google search can help me to find any answer.)
I got the same exception message as the first link. But Rahul's answer didn't work. However, Rahul's link revealed a second clue, that is, use Tracelog.exe to track the connection problem.
From the track log in Event Viewer, I found there are some error events saying my server's authority isn't accessible, so I searched in the third clue, Cache Administration PowerShell commands for setting permission in PowerShell.
In the third clue, thank the author, mark_og who really saved me ! You taught us how to set the permission in IIS like this,
Grant-CacheAllowedClientAccount "IIS APPPOOL\[AppPoolName]"
it's the key answer which solves my problem !
We have scaled-out some portions of our ASP.NET app to run on one server, and other portions to run on another server (& under a subdomain).
The two servers share (SQL Server) Session. We used this MS article to create a tiny HTTP Module to sync app domain name between the two servers (sans the cookie domain code, which can be configured in the web.config. I later found this CodeProject article which is essentially the same.)
Everything's working well, except for a small issue: deployment changes or web.config tweaks require a manual app pool recycle (the auto-recycle no longer works - instead we get the "web server is currently unavailable / hit refresh" error).
I tried moving the app domain naming code from the HTTP Module into the Application_Start section of the Global.asax (maybe this is a better place for it?) - but received the same problem.
I know that one solution is to hard-code the app name in one of the SQL Server Session stored procedures; but am a bit hesitant to do this.
Edit: The app is ASP.NET 3.5 under IIS 6.0 (thanks #Chris & #bzlm)
You should check if proper Recycling Events are turned on in IIS, maybe this can help http://support.microsoft.com/kb/332088
Update. We opened a tech support case with Microsoft about this. After a week or so of back & forth, they said they had reproduced the issue in their environment and understand the cause (a timing issue deep inside the ASP.NET internals) - but that there is no resolution that they're aware of. I complained that the HTTP module is Microsoft code, but they said that this code is under "FAST PUBLISH" terms - intended to help & advise customers; yet not warranted.
Ah well. We now just manually recycle the app pool after making a web.config change.
I've got a class library doing all my NHibernate stuff. It also handles all the mapping using Fluent NHibernate - no mapping files to deploy.
This class library is consumed by a number of apps, including a Windows Service running on my computer. Although it works fine in all my web apps, the Windows Service gets this when it tries to use NHibernate:
An invalid or incomplete configuration was used while creating a SessionFactory. Check PotentialReasons collection, and InnerException for more detail.
at FluentNHibernate.Cfg.FluentConfiguration.BuildSessionFactory()
at Kctc.NHibernate.KctcSessionFactory.get_SessionFactory() in C:\Kctc\Trunk\Kctc.NHibernate\KctcSessionFactory.cs:line 28
...more stack trace...
I have checked for an InnerException and there doesn't appear to be one. I have no idea what the PotentialReasons collection is, and Google doesn't seem to be forthcoming either.
This is my dev machine, so when I'm working on my web apps they run locally (i.e. using the web server in Visual Studio). The fact that the Windows Service and my dev web apps are running on this same machine suggest it's not to do with trust settings or what have you.
Can anyone suggest what I should try? This is one of those ones where I'm so stumped I can't even think of how to get more information about the problem.
Just a wild guess. NHibernate picks up the hibernate.cfg.xml file from the execution directory. Did you configure the execution directory of the service that it can find this file?
I've found out what the problem is. The Service did not deploy with the required NHibernate.ByteCode.LinFu.dll.
I appear to have an ongoing problem with the Visual Studio compiler not always copying indirect dependencies (i.e. dlls required by class libraries required by the app) into the output folder during the build. I should have thought of this sooner really.
Thanks for racking your brains on my behalf guys.
I bet the name of the connection string is missing from the app.config. For me that message is almost exclusively a missing connection string.
Are you targeting the same database or could it be some sort of schema mismatch between databases?
Could it be authentication issues on the service like you use windows authentication where it can't be used (or the sql authentication that doesn't work)?
It's hard to tell when there is no code, just an exception!
EDIT Are you ever using HttpContext, HostingEnvironment or anything else specific to "web"?