Log4Net on multiple nodes behind loadbalancer overwrite previous day rolling files - asp.net-core

We are using the RollingFileAppender in a web-applicfation running behind a load-balaner on multiple nodes.
We noticed that logs of previous days often are very small, only a few lines, while the current log is large. It is not 100% consistent, about 1 in 5 previous logs apear to be a full log (using 2 nodes).
We figured that both nodes must be renaming log.log to the previous date with a minimal timespan between them. The last node to do so will actually overwrite the previous log with the new logfile created moments earlier by the first node.
This is our curated config:
<log4net>
<appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
<lockingModel type="log4net.Appender.FileAppender+MinimalLock"/>
<file value="\\shared-path\log.log" />
<datePattern value="yyyy-MM-dd.'txt'"/>
<staticLogFileName value="true"/>
<appendToFile value="true"/>
<rollingStyle value="Date"/>
<maxSizeRollBackups value="10"/>
<maximumFileSize value="20MB"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%thread] %message%newline"/>
</layout>
</appender>
<root>
<level value="ALL"/>
<appender-ref ref="RollingLogFileAppender"/>
</root>
</log4net>
Is there any simple way to prevent this from hapening?

The problem can be solved by setting <staticLogFileName value="false"/>.
In that case it will not rename log.log to the formatted archive name, but simply create a new file for the new log.
A disadvantage of this solution is that any Tail or monitoring tool will now have to check the directory for the newest file instead of just going for the static log.log.
Edit:
This is not a viable solution. We are missing chunks of logs, the time sometimes goes back 1 minute so log4net seems to be caching before flushing to the file...
See also this answer to the question Logging web services operations behind load balancer

Related

Running Gatling tests not showing testFile.log on IntelliJ

We have an issue where whenever our Gatling performance tests are run the .log file that should generate at the root of the folder is not there.
This is my whole logback file if anyone able to help please.
<contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
<resetJUL>true</resetJUL>
</contextListener>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<!-- <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter> -->
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%-5level] %logger{15} - %msg%n%rEx</pattern>
<immediateFlush>true</immediateFlush>
</encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>testFile.log</file>
<append>false</append>
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%-5level] %logger{15} - %msg%n%rEx</pattern>
</encoder>
</appender>
<!-- uncomment and set to DEBUG to log all failing HTTP requests -->
<!-- uncomment and set to TRACE to log all HTTP requests -->
<logger name="io.gatling.http.engine.response" level="TRACE" />
<root level="WARN">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE" />
</root>
Thank you very much.
Update
It seems the issue may be with IntelliJ itself as noticed we can see the file when going directly to the finder.
Disabling custom plugins should help. Seems one of configuration was corrupted.
There's a good chance the file is simply not generated where you expect it. Try setting an absolute path instead to verify.

Log4net: The process cannot access the file because it is being used by another process

I have an asp.net website that is utilizing log4net. Due to several third party libraries (like NHibernate), there are several appenders across multiple threads. IIS is setup so there should be only one process. I have confirmed this by utilizing the process id in the filename.
Web.config Excerpt:
<appender name="AppAppender" type="log4net.Appender.RollingFileAppender">
<file type="log4net.Util.PatternString" value="Logs\App_[%processid]_.log" />
<appendToFile value="true" />
<rollingStyle value="Composite" />
<datePattern value="yyyy-MM-dd" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="10MB" />
<staticLogFileName value="false" />
<preserveLogFileNameExtension value="true" />
<threshold value="DEBUG" />
<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value=" %date %-5level [%property{SessionId}] - %message%newline" />
</layout>
</appender>
When recycling the app pool in IIS I sometimes get these errors:
log4net:ERROR [RollingFileAppender] ErrorCode: GenericFailure. Unable to acquire lock on file App_[6844]_2014-12-18.log. Access to the path 'App_[6844]_2014-12-18.log' is denied.
My question, is are some of these error inevitable because its multi-threaded, or if configured properly should they never occur.
Additional Info:
I'm using log4net 1.2.13 (installed over nuget)
Frozen Mountain's WebSync (3.5.1.3), also note that this library directly references log4net 1.2.11, I used a bindingRedirect to 1.2.13
NHibernate version 3.1.0.400, it was the first version to not use log4net 1.2.10
**NHibernate can easily be upgraded if we think thats the problem. WebSync would be more difficult.

Why does debug output show in WinDbg but not VS2010?

We have .NET 4 application that does some logging.
When the application is launched from Studio, there's no output in the Debug view.
When the application is launched from WinDbg, the logging is shown as expected.
I've read that there was a change in .NET 4 preventing the logging from showing up in both Studio and WinDbg, but it doesn't seem to show up in Studio at all now. Before the solution was upgraded, it did log as expected in VS2008.
Why is the Debug output showing up correctly when launched from WinDbg but not when launched from VS2010?
This was due to a misconfigured log4net.config file.
Added the <appender> block and <appender-ref> block to reference said appender.
<log4net>
...
<appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender" >
<layout type="log4net.Layout.PatternLayout">
<param name="Header" value="[Header]\r\n" />
<param name="Footer" value="[Footer]\r\n" />
<param name="ConversionPattern" value="%d [%t] %-5p %m%n" />
</layout>
</appender>
<root>
...
<appender-ref ref="ConsoleAppender" />
</root>
</log4net>

NHibernate - see SQL without all the other guff

So I'm using log4net to write log output to the trace. Show sql is specified in the configuration file. I seem to have to set the log output level to DEBUG to get the SQL output, but DEBUG also produces pages and pages of other guff I have to scroll past.
Can I get the SQL without the guff?
Thanks
David
You can add a logger for NHibernate.SQL in the log4net config block, like so:
<logger name="NHibernate.SQL" additivity="false">
<level value="DEBUG" />
<appender-ref ref="ConsoleAppender" />
</logger>
With the appender-config
<appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date - %message%newline"/>
</layout>
</appender>
(replace this with whatever you prefer, like rollingFileAppender)
Another option is using a tool like NHibernate Profiler.

Intermittent log4net RollingFileAppender locked file issue

We are seeing an intermittent issue on development and production machines whereby our log files are not getting logged to.
When running in development and debugging using Visual Studio we get the following log4net error messages in the VS output window:
log4net:ERROR [RollingFileAppender] Unable to acquire lock on file C:\folder\file.log.
The process cannot access the file 'C:\folder\file.log' because it is being used by another process.
log4net:ERROR XmlConfigurator: Failed to find configuration section 'log4net' in the application's .config file.
Check your .config file for the <log4net> and <configSections> elements.
The configuration section should look like:
<section
name="log4net"
type="log4net.Config.Log4NetConfigurationSectionHandler,log4net" />
Our current workaround for the issue is to rename the last log file. We would of course expect this to fail (due to the aforementioned file lock), but it normally doesn't. Once or twice the rename has failed due to a lock from the aspnet_wp.exe process.
Our log4net configuration section is shown below:
<log4net>
<appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
<file value="C:\folder\file.log"/>
<appendToFile value="true" />
<datePattern value="yyyyMMdd" />
<rollingStyle value="Date" />
<maximumFileSize value="10MB" />
<maxSizeRollBackups value="100" />
<layout type="log4net.Layout.PatternLayout">
<header value="[Header]
"/>
<footer value="[Footer]
"/>
<conversionPattern value="%date %-5level %logger ${COMPUTERNAME} %property{UserHostAddress} [%property{SessionID}] - %message%newline"/>
</layout>
</appender>
<root>
<level value="INFO"/>
<appender-ref ref="RollingLogFileAppender"/>
</root>
</log4net>
As mentioned, we are seeing this intermittently on machines, but once the issue happens it persists.
Try adding
<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
to your <appender /> element. There is some performance impact because this means that log4net will lock the file, write to it, and unlock it for each write operation (as opposed to the default behavior, which acquires and holds onto the lock for a long time).
One implication of the default behavior is that if you're using it under a Web site that is being executed under multiple worker processes running on the same machine, each one will try to acquire and hold onto that lock indefinitely, and two of them are just going to lose. Changing the locking model to the minimal lock works around this issue.
(When debugging, ungraceful terminations and spinning up lots of new worker processes is exactly the type of thing that's likely to happen.)
Good luck!
Also be aware of the log4net FAQ:
How do I get multiple process to log to the same file?
Before you even start trying any of the alternatives provided, ask
yourself whether you really need to have multiple processes log to the
same file, then don't do it ;-).
FileAppender offers pluggable locking models for this usecase but all
existing implementations have issues and drawbacks.
By default the FileAppender holds an exclusive write lock on the log
file while it is logging. This prevents other processes from writing
to the file. This model is known to break down with (at least on some
versions of) Mono on Linux and log files may get corrupted as soon as
another process tries to access the log file.
MinimalLock only acquires the write lock while a log is being written.
This allows multiple processes to interleave writes to the same file,
albeit with a considerable loss in performance.
InterProcessLock doesn't lock the file at all but synchronizes using a
system wide Mutex. This will only work if all processes cooperate (and
use the same locking model). The acquisition and release of a Mutex
for every log entry to be written will result in a loss of
performance, but the Mutex is preferable to the use of MinimalLock.
If you use RollingFileAppender things become even worse as several
process may try to start rolling the log file concurrently.
RollingFileAppender completely ignores the locking model when rolling
files, rolling files is simply not compatible with this scenario.
A better alternative is to have your processes log to
RemotingAppenders. Using the RemoteLoggingServerPlugin (or
IRemoteLoggingSink) a process can receive all the events and log them
to a single log file. One of the examples shows how to use the
RemoteLoggingServerPlugin.
If you have
<staticLogFileName value="true" />
<rollingStyle value="Date" />
<datePattern value="yyyyMMdd" />
and add
<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
then there will be an error while the rolling happens.
The first process will create the new file and the rename the current file.
Then next proces will do the same and take the newly created file and overwrite the newly renamed file.
Resulting in the logfiel for the last day being empty.