Frankly, I don’t know for sure which particular package was the culprit. There was nothing logged that pointed to any issues – indeed any and all logs I went through indicated all was well. I also could not find a single thing online that resembled what I was experiencing. I have managed my own servers for about 10 years now. I’m far from a competent Linux admin but I usually know enough to get by or just enough to get out of trouble, so I became severely frustrated when a seemingly simple upgrade could result in all my sites being made unavailable.

I started by stripping back my Apache install to next to nothing – which brought back a single static site i set up as a test. Then line-by-line I reintroduced various modules, includes and various directive lines. At the very bottom I had my mod_jk include – which by the way serves over half of my sites.

As soon as I reintroduced mod_jk it dumped. After much fiddling, rebuilding mod_jk, installing from rpm, downgrading packages, tinkering with various mod_jk parameters, using strace on startup (great article on debugging with strace here) I finally put it down to the inclusion of the following:

<Location /*/META-INF/*>
    Deny From All
</Location>
<Location /*/WEB-INF/*>
    Deny From All
</Location>

The above tells Apache as a global directive to disallow any access to the protected Java web application directories WEB-INF and META-INF.

The lines above have been a part of my apache/mod_jk config for as long as I remember. I haven’t yet had a good look as to why the above is no longer acceptable – I’m just relieved to have everything back up. I would have expected Apache to complain about configuration errors, issue the relevant message and fail to start just like it does for most other issues. Not so in this case and furthermore it silently takes down the whole server instance.

I didn’t find any references to this anywhere – perhaps this will help someone equally confused as I was

After some serious digging (it drove me a little nuts) and a plethora of less than helpful tips and references I stumbled across this:  Webalizer – Web Server Statistics “how to”.

Apparently it has something to do with the way browsers accessing the site report their name and that some have multiple parens (‘(‘ and ‘)’) in it. This had the affect of confusing the mangle agent that reports agent names accessing the site. The fix was specifying a MangleAgents value other than 1 (zero or 2 worked fine). I set it to 2 and all is well – I now have all my stats working again.