I have WCF service which takes a post request and JSON. The service then use the C# backgroundworker class to further parse JSON and update DB.
But when I post JSON that is larger than 7KB, the worker is crashing. I am not getting any exceptions or errors in the application logs, but when I look at system logs , I get this,
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-WAS" Guid="{524B5D04-133C-4A62-8362-64E8EDB9CE40}" EventSourceName="WAS" />
<EventID Qualifiers="32768">5011</EventID>
<Version>0</Version>
<Level>3</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2014-06-16T18:51:21.000000000Z" />
<EventRecordID>74571</EventRecordID>
<Correlation />
<Execution ProcessID="0" ThreadID="0" />
<Channel>System</Channel>
<Computer>xxxxx</Computer>
<Security />
</System>
<EventData>
<Data Name="AppPoolID">YYYYYYYY</Data>
<Data Name="ProcessID">ZZZZZZZ</Data>
<Binary>6D000780</Binary>
</EventData>
</Event>
Could this be a memory leak? How can I find the exact cause if my application log does not show any errors?
PS: The service is working perfect in local and staging environments we have. But live server has a different configuration.
Related
I have two almost identical web api services programmed in C# and installed on the same Windows 2008 Server on IIS 6.1. When I do web api calls to them, they both work just fine. I am using log4net for logging purposes. One of them, however, does not always log. They both have the same exact log4net configuration, which is:
<log4net>
<root>
<level value="DEBUG" />
<appender-ref ref="RollingLogFileAppender" />
</root>
<appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
<file value="C:\LOGS\SomeFolder\" />
<appendToFile value="true" />
<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
<preserveLogFileNameExtension value="true" />
<rollingStyle value="Date" />
<datePattern value="'WebApi.One.'yyyy-MM-dd'.log'" />
<staticLogFileName value="false" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%-5level [%thread][%date{dd-MM-yy HH:mm:ss,fff}] %logger - %message%newline" />
</layout>
</appender>
</log4net>
The only difference in the configuration is the datePattern which has a slightly different name so that they do not log to the same exact file.
For the web api where logging works as expected a new log file is created each day as soon as a web api call comes in. For the web api where logging does not seem to work the new log file is not created. However, if I make a web api call from a browser on the same server as it is installed on, then logging starts. After the logging has started for that day it continues fine (even with web api calls coming from other machines). But the next day no new file is created.
I am unable to see what the difference is. Surely there must be something I am not thinking of that makes these two web apis behave differently when it comes to logging. Remember, both of the services work fine, it is just the logging that is not working for one of them.
Any suggestions?
Edit 1:
After adding diagnostics as suggested by Peter I can see that the access to the path is denied:
log4net:ERROR Could not create Appender [RollingLogFileAppender] of type [log4net.Appender.RollingFileAppender]. Reported error follows.
System.UnauthorizedAccessException: Access to the path 'C__LOGS_WebApi.One_' is denied.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.Threading.Mutex.MutexTryCodeHelper.MutexTryCode(Object userData)
at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
at System.Threading.Mutex.CreateMutexWithGuaranteedCleanup(Boolean initiallyOwned, String name, Boolean& createdNew, SECURITY_ATTRIBUTES secAttrs)
at System.Threading.Mutex..ctor(Boolean initiallyOwned, String name, Boolean& createdNew, MutexSecurity mutexSecurity)
at System.Threading.Mutex..ctor(Boolean initiallyOwned, String name)
at log4net.Appender.RollingFileAppender.ActivateOptions()
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseAppender(XmlElement appenderElement)
log4net:ERROR Appender named [RollingLogFileAppender] not found.
I don't know why though.
Edit 2:
I checked solutions in these StackOverflow answers:
What am I doing wrong with thie log4net implementation?
log4net: Error on loading custom appender
But still no progress.
This looked very similar, but I could see no solution there (although maybe there is one).
Edit 3:
Folder permissions:
According to the exception message, the process does not have enough rights to write to the location of the log file.
In my experience with dealing with IIS, changing the security permissions on the folder where the log file is stored so that the application/process has the rights to modify the log file(s) usually works.
To confirm/troubleshoot start by giving everyone full control on the folder and testing that the file can be modified.
If that works then you can confirm it is a permission issue.
I usually give IIS_IUSRS the following permission on the Log folder where I store my logs and archive.
Read
Write
Modify
To limit possible attack vectors I try to give the process as few permission as needed to perform its function. Adding and removing permissions and then testing that it still functions as desired.
It looks like a security issue, to debug this you should enable debugging for log4net:
<appSettings>
<add key="log4net.Internal.Debug" value="true"/>
</appSettings>
And
<system.diagnostics>
<trace autoflush="true">
<listeners>
<add
name="textWriterTraceListener"
type="System.Diagnostics.TextWriterTraceListener"
initializeData="C:\tmp\log4net.txt" />
</listeners>
</trace>
</system.diagnostics>
in your configuration.
This way you are able to see why the creation of the file fails.
log4net faq
I've search many sites and tried several different opinions. But I still could not solve it. Here are the things I did right now:
In Global.asax at Application_Startup I give the file path and startup the Log4Net.
Right after I start log4net, I write a log that says "Application has
stared'"
Currently, there is only 1 worker in IIS for the WCF application The
IIS user has access to WRITE, MODIFY and READ privileges
The problem:
When I invoke a method of service directly (without doing the 2.
step below), No Logs is written
On a browser, I write te WCF url and hit ENTER, Log4Net creates the
folder and the files (files are EMPTY at this point).
If I make requests and invoke the methods (doing the 1st step), now
Log4Net writes the logs.
The actual problem:
After the 3rd step, (lets say we waited without any invokes of the
WCF methods around 10 minutes or more), the invoking DOES NOT CREATE
Log4Net Text logs ANYMORE.
Sometimes, if I repeat the 2nd step, it begins writing the logs
again. But there is no coherent results.
Here is the Config.xml:
<?xml version="1.0" encoding="utf-8"?>
<log4net>
<appender name="ProcessInfo_FileAppender" type="log4net.Appender.RollingFileAppender">
<file type="log4net.Util.PatternString" value="L:\LOGs\ProcessInfo\ProcessInfo_[%processid].txt" />
<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
<appendToFile value="true" />
<rollingStyle value="Composite" />
<maxSizeRollBackups value="200" />
<maximumFileSize value="30MB" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%thread] - %message%newline" />
</layout>
</appender>
<logger name="ProcessInfo">
<levelMin value="ERROR" />
<levelMax value="INFO" />
<appender-ref ref="ProcessInfo_FileAppender" />
</logger>
<root></root>
</log4net>
I have other WCF projects which have no problem even with multiple Workers. (I used the exact same IIS and Log4Net xml configuration with them). Also, as I mentioned on the title, I have a WebSite who has exact same logging codes (they both using a common 3rd party dll which I wrote) and has NO PROBLEM of writing Log4Net text logging at all.
Please help.
Thanks.
The problem is not in your logging configuration, you should try to enable log4net internal debugging. This will tell you why the logging stops. I guess there is some code that reconfigures your logging to load configuration from your web.config which is not there.
<configuration>
...
<appSettings>
<add key="log4net.Internal.Debug" value="true"/>
</appSettings>
...
<system.diagnostics>
<trace autoflush="true">
<listeners>
<add
name="textWriterTraceListener"
type="System.Diagnostics.TextWriterTraceListener"
initializeData="C:\tmp\log4net.txt" />
</listeners>
</trace>
</system.diagnostics>
</configuration>
Log4net FAQ
I'm trying to cluster a pair of servers with a shared Infinispan cache (Replicated Asynchronously). One always starts successfully, and registers itself properly with the JDBC database. When the other starts, it registers properly with the database, and I see a bunch of chatter between them, then, while waiting on a response from the second server, I get
`org.infinispan.commons.CacheException: Initial statue transfer timed out`
I think it's just an issue of configuration, but I'm not sure how to debug my configuration issues. I've spent several days configuring and re-configuring my Infinispan XML, and my JGroups.xml:
Infinispan:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:6.0"
xsi:schemaLocation="urn:infinispan:config:6.0 http://www.infinispan.org/schemas/infinispan-config-6.0.xsd
urn:infinispan:config:remote:6.0 http://www.infinispan.org/schemas/infinispan-cachestore-remote-config-6.0.xsd"
xmlns:remote="urn:infinispan:config:remote:6.0"
>
<!-- *************************** -->
<!-- System-wide global settings -->
<!-- *************************** -->
<global>
<shutdown hookBehavior="DEFAULT"/>
<transport clusterName="DSLObjectCache">
<properties>
<property name="configurationFile" value="jgroups.xml"/>
</properties>
</transport>
<globalJmxStatistics enabled="false" cacheManagerName="Complex.com"/>
</global>
<namedCache name="ObjectCache">
<transaction transactionMode="TRANSACTIONAL" />
<locking
useLockStriping="false"
/>
<invocationBatching enabled="true"/>
<clustering mode="replication">
<async asyncMarshalling="true" useReplQueue="true" replQueueInterval="100" replQueueMaxElements="100"/>
<stateTransfer fetchInMemoryState="true" />
</clustering>
<eviction strategy="LIRS" maxEntries="500000"/>
<expiration lifespan="86400000" wakeUpInterval="1000" />
</namedCache>
<default>
<!-- Configure a synchronous replication cache -->
<locking
useLockStriping="false"
/>
<clustering mode="replication">
<async asyncMarshalling="true" useReplQueue="true" replQueueInterval="100" replQueueMaxElements="100"/>
<stateTransfer fetchInMemoryState="true" />
</clustering>
<eviction strategy="LIRS" maxEntries="500000"/>
<expiration lifespan="86400000" wakeUpInterval="1000" />
<persistence>
<cluster remoteCallTimeout="60000" />
</persistence>
</default>
</infinispan>
Jboss.xml:
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-3.0.xsd">
<!-- Default the external_addr to #DEADBEEF so we can see errors coming through
on the backend -->
<TCP
external_addr="${injected.external.address:222.173.190.239}"
receive_on_all_interfaces="true"
bind_addr="0.0.0.0"
bind_port="${injected.bind.port:12345}"
conn_expire_time="0"
reaper_interval="0"
sock_conn_timeout="20000"
tcp_nodelay="true"
/>
<JDBC_PING
datasource_jndi_name="java:jboss/datasources/dsl/control"
/>
<MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK
external_addr="${injected.external.address:222.173.190.239}"
bind_addr="0.0.0.0"
/>
<FD timeout="10000" max_tries="5"/>
<VERIFY_SUSPECT timeout="1500"
bind_addr="0.0.0.0"
/>
<pbcast.NAKACK use_mcast_xmit="false"
retransmit_timeouts="300,600,1200,2400,4800"
discard_delivered_msgs="true"/>
<UNICAST3 ack_batches_immediately="true"
/>
<RSVP ack_on_delivery="true"
throw_exception_on_timeout="true"
timeout="1000"
/>
<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
max_bytes="400000"/>
<pbcast.GMS print_local_addr="true" join_timeout="5000"
view_bundling="true" view_ack_collection_timeout="5000"/>
<FRAG2 frag_size="60000"/>
<pbcast.STATE_SOCK
bind_port="54321"
external_addr="${injected.external.address:222.173.190.239}"
bind_addr="0.0.0.0"
/>
<pbcast.FLUSH timeout="1000"/>
</config>
I've tried, frankly, every configuration option I can think of, and I'm not sure why the replication keeps timing out. All communication between these servers is wide open. Sorry to just dump so much XML, but I'm not even sure how to collect more information.
Continued exploration indicated that Infinispan was pushing logs to the server.log, but - due to my configuration, this was not duplicated on the console. Further inspection revealed that I left a single element in my cache objects unserializable - making it impossible for it to be written to the wire and transferred. The logs are very specific, making this actually a very easy problem to track down once I realized where the logs were being written.
If you come here from the future, my advice is to just tail every single log you can on the working server, and see what comes up.
Enviroment: IIS 7.5. .NET 4.5.1. WINDOWS 2008 R2 SP1 Integrated Mode
I have an MVC application working in enviroment as above. Processing of some of my requests takes about 15 seconds. It has nothing to do with controller's action activities or view rendering. It happens also for actions returning view with one constant string or for JSON result actions.
In controller action's log I can see that the request begins it's execution about 10 second later than I can see it in fiddler on the browser side.
I've analyzed the request with Failed Requests Tracing and it turns out that the request gets stuck somewhere in the guts of IIS. This is the only request this time, there is a lot of free memory and processor time.
It turns out that the event that consumes about 10 seconds is always the event that uses ExtensionlessUrlHandler-Integrated-4.0 . Then the next event appears about 10 seconds later.
The common thing for these failerd request is tha they are issued as AJAX requests, but at the time they are processed there are no other open requests so concurency is not the issue.
What could be a reason for such delays?
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="WWW Server" Guid="{3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83}"/>
<EventID>0</EventID>
<Version>1</Version>
<Level>4</Level>
<Opcode>43</Opcode>
<Keywords>0x0</Keywords>
<TimeCreated SystemTime="2014-03-28T15:44:19.663Z"/>
<Correlation ActivityID="{00000000-0000-0000-0703-0080000000FC}"/>
<Execution ProcessID="9308" ThreadID="3504"/>
<Computer>COMP1</Computer>
</System>
<EventData>
<Data Name="ContextId">{00000000-0000-0000-0703-0080000000FC}</Data>
<Data Name="OldHandlerName"></Data>
<Data Name="NewHandlerName">ExtensionlessUrlHandler-Integrated-4.0</Data>
<Data Name="NewHandlerModules">ManagedPipelineHandler</Data>
<Data Name="NewHandlerScriptProcessor"></Data>
<Data Name="NewHandlerType">System.Web.Handlers.TransferRequestHandler</Data>
</EventData>
<RenderingInfo Culture="en-US">
<Opcode>HANDLER_CHANGED</Opcode>
</RenderingInfo>
<ExtendedTracingInfo xmlns="http://schemas.microsoft.com/win/2004/08/events/trace">
<EventGuid>{D42CF7EF-DE92-473E-8B6C-621EA663113A}</EventGuid>
</ExtendedTracingInfo>
</Event>
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="WWW Server" Guid="{3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83}"/>
<EventID>0</EventID>
<Version>1</Version>
<Level>5</Level>
<Opcode>1</Opcode>
<Keywords>0x100</Keywords>
<TimeCreated SystemTime="2014-03-28T15:44:28.977Z"/>
<Correlation ActivityID="{00000000-0000-0000-0703-0080000000FC}"/>
<Execution ProcessID="9308" ThreadID="3504"/>
<Computer>COMP1</Computer>
</System>
<EventData>
<Data Name="ContextId">{00000000-0000-0000-0703-0080000000FC}</Data>
<Data Name="ModuleName">HttpCacheModule</Data>
<Data Name="Notification">1</Data>
<Data Name="fIsPostNotification">false</Data>
</EventData>
<RenderingInfo Culture="en-US">
<Opcode>NOTIFY_MODULE_START</Opcode>
<Keywords>
<Keyword>RequestNotifications</Keyword>
</Keywords>
<freb:Description Data="Notification">BEGIN_REQUEST</freb:Description>
</RenderingInfo>
<ExtendedTracingInfo xmlns="http://schemas.microsoft.com/win/2004/08/events/trace">
<EventGuid>{002E91E3-E7AE-44AB-8E07-99230FFA6ADE}</EventGuid>
</ExtendedTracingInfo>
</Event>
I'm using Windows 8. In drive C 's root directory (C:) there are 27 WCF Trace file; 26 of them are named like <guid>Traces.svclog and one is Traces.svclog. I'm trying to find out which applications are creating these files; because I don't have any application that uses wcf trace described here.
And here is my C:\ directory's view:
I've looked all of the traces: there are only 2 unique trace and the rest is exact copies of them. Here comes the details from one of them;
<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>589828</EventID>
<Type>3</Type>
<SubType Name="Information">0</SubType>
<Level>8</Level>
<TimeCreated SystemTime="2012-10-15T06:37:57.4222685Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" />
<Execution ProcessName="Microsoft.VisualStudio.Web.Host" ProcessID="7104" ThreadID="78" />
<Channel />
<Computer>MYCOMP</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Information">
<TraceIdentifier>http://msdn.microsoft.com/tr-TR/library/System.ServiceModel.Activation.WebHostCompilation.aspx</TraceIdentifier>
<Description>ASP.Net hosted compilation.</Description>
<AppDomain>7f594460-3-129947566686622036</AppDomain>
<Source>System.ServiceModel.Activation.ServiceParser/17864371</Source>
<ExtendedData xmlns="http://schemas.microsoft.com/2006/08/ServiceModel/StringTraceRecord">
<VirtualPath>/BI/DataService.svc</VirtualPath>
</ExtendedData>
</TraceRecord>
</DataItem>
</TraceData>
</ApplicationData>
</E2ETraceEvent>
So the C:\ directory is filling with rubbish traces. I've installed WCF&WF samples, and maybe that caused this. Can you help me on this; to find out which application is doing this?
Search your drive for app.config files that have systems.diagnostics sections that enable trace on the system.servicemodel tracesource.