Apache: Directory index forbidden by Options directive - apache

Question

We are running 4 Wordpress network installations on a Windows Server 2008 R2 VPS, with Apache 2.2.17 and PHP 5.3.10 and for some reason we are regularly getting this (sample) error:
Error log
[Thu Feb 16 15:01:59 2012] [error] [client x.x.x.x] Directory index forbidden by Options directive: C:/_webserver/_www/wp/www/
Access log
host x.x.x.x - - [17/Feb/2012:12:59:23 +0200] "GET / HTTP/1.1" 403 306 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.2; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; .NET4.0C; MATM)"
The error "Directory index forbidden" usually means that a directory is trying to be accessed, but there is no file (according to the options directive) to be displayed and directory listing is forbidden. This however is not the case here. The error refers to the folder C:/_webserver/_www/wp/www/, which is the webroot for the project, and has always had an index.php. Also, httpd.conf is set-up with: DirectoryIndex index.html index.php
Seeing as how the error occurs in Apache, I think it's highly unlikely that this can be caused by either PHP or Wordpress.
The tough thing is that we don't know how to reproduce the error, so it is hard for us to test this.
What can we do to find out what the issue could be? Can it have anything to do with the set-up of Apache (seems like a redundant question). Can it have anything to do with the file already being read by Apache? Is there some way we can get more info about this issue?
I would welcome any help to help me solve this nasty case.
UPDATE
These are the modules that I currently have in use
LoadModule deflate_module modules/mod_deflate.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule cache_module modules/mod_cache.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule alias_module modules/mod_alias.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule dir_module modules/mod_dir.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule php5_module "c:/_webserver/_server/php-5.3.10-Win32-VC9-x86/php5apache2_2.dll"
Options directives:
<Directory />
Options FollowSymLinks ExecCGI
AllowOverride None
Order deny,allow
Allow from all
</Directory>
<Directory "C:/Program Files (x86)/Apache Software Foundation/Apache2.2/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
httpd-vhosts.conf looks like this:
NameVirtualHost *:80
<VirtualHost *:80>
<Directory "C:/_webserver/_www/sites/www">
Options FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
Include "C:/_webserver/_www/sites/htaccess.conf"
DocumentRoot "C:/_webserver/_www/sites/www"
ServerName xxx
ServerAlias xxx
CustomLog logs/sites.access.log mycombined
ErrorLog logs/sites.error.log
</VirtualHost>
I have 5 virtual hosts set-up like this, with each their own error and access log. The projects don't use a .htaccess, but this is set-up statically via the conf for performance.
The server runs on windows, so the MPM set-up is a little limited
# WinNT MPM
# ThreadsPerChild: constant number of worker threads in the server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_winnt_module>
ThreadsPerChild 1750
MaxRequestsPerChild 0
</IfModule>
Final update
Well, I decided to turn off Apache caching completely, and since then, had no more errors. Unfortunately I haven't had too much time this week to do proper testing, but at least I know where the issue lies. And with a not so busy server, no caching is alright for now. I might be back in a while :-)

Answer 1

This is certainly something hard to debug, occasional bugs are the worst ones :-)
My first thoughts were "internal dumy connections" related issues, but that would not show you an IE8-beta signature in access.log.
So I found three links that you may investigate:
A serverFault intermittent 403 problem, related to mod_negotiation and SSI
Another serverFault intermittent 403, related to MaxClients reached, check that you do not reach MaxClients when that error appears.
A deep analysis of intermittent 403 related to mod_cache
From that I think this kind of problem is a little like drug interactions. So the first thing to do is:
Check the modules loaded in your apache configuration and remove (comment Load lines) the ones you do not need at all (and you will have a faster Apache if you never did it before!).
Build a test environment, for modules that you are still using in production (where removing it would make your application crash). You'll need to be able to reproduce the bug with wget or ab or any other massive HTTP requests tool.
try to inactive modules, one by one, until the problem disappears.
Modules that usually make strange behaviors are:
mod_negotiation (with the related Option Multiviews). Apache is then trying to serve alternate files, after a negociation with the browsers headers. That may break your RewriteRules or interact badly with other modules. That usually lead to some unattented responses to malformed queries, I always remove that module.
mod_include : Server Side Includes (with related option Includes), also known as SSI. Who really needs that?
mod_cache & mod_disk_cache , really, that's a very old school things, you'd better try using Varnish or any other reverse proxy caches
mod_rewrite: the swiss knife, but are you sure you didn't wrote a very strange Rule somewhere?
mod_dir : check you do not have DirectorySlash Off, that may interact badly with some other module doing strange things
mod_isapi : reading the doc may lead you to some hints. For me I have the feeling this is an experimental support, on heavy load I'm quite sure strange things could happen.
mod_proxy : remove it if you do not need it
UPDATE: (after configuration details)
Reading your configuration I saw several little mistakes (unrelated):
<Directory /> I don't think this will work on Windows, as your root is c: not /. But I'm maybe wrong. At least you do not need an allow from all here, quite unsecure.
If you do not use .htaccess files set the AllowOverride None everywhere, especially in a <Directory "C:">, to avoid seeking for theses files from the root directory.
Now for your problem. I do not see any mod_cache related option in your configuration (but maybe you have some included files in subdirectories of apache configuration that use mod_cache directives. If you do not use any of the mod_cache directives you can suspend that module without any risk.

Answer 2

If it's intermittent, it's fair to guess that someone is intermittently deleting and replacing index.php.
Re comments -- this does not require 'sabotage'. If you simply redeploy some style of applications with a running webserver, or restore a backup, there is a window of time when Apache might see a this directory but no file.