Obviously something was different between the two servers, but what was it? My first thought was to check permissions of the Application Pools, so after a quick check of the virtual folder properties, I was looking at the Identity tab of the responsible pool. Here is a screenshot of what I saw on the first box with identifiable traits redacted to protect the guilty:
Upon seeing that this AppPool was running as "Local System", my eyes bugged out of my head a bit. If you are not aware of the Predefined users available in this drop down box, here is a quick synopsis from Microsoft TechNet:
"By default, application pools operate under the Network Service account, which has low-level user access rights. That is, this account provides better security against attackers or malicious users who might attempt to take over the computer on which the World Wide Web Publishing Service (WWW service) is running. The LocalService account also has low-level user access rights, which is helpful for situations that do not require access to resources on remote computers. You can configure application pools to run as LocalSystem, which is an account with more user rights than the Network Service or LocalService account. However, be mindful that running an application pool under an account with increased user rights presents a high security risk." - technet.microsoft.comIn other words, an AppPool run as "Local System" is a bad idea. That being said, this was naturally the box where the event log entry was working. Too bad it took a sledgehammer to do it! *
Moving over to the other server, the Predefined drop drown was set to the default "Network Service". I could have just switched it to "Local System" and called it a day, but we already established that wasn't a good idea. The question now was, how could we make this work while leaving it set to "Network Service"?
After researching (aka Google-Fu) a bit, I ran across a question on Stack Overflow that covered a similar situation. The answer provided by DOK referenced the following quote from Microsoft MSDN:
"Least privileged accounts have sufficient permissions to be able to write records to the event log by using existing event sources. However, they do not have sufficient permissions to create new event sources." - msdn.microsoft.comThis helpful information pointed me in the right direction. From previous experience, I already knew the location where Windows stores the event source names. If you fire up RegEdit, you will find them all listed in the following location: "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog". Under this key are keys for each class of event log and under them their respective event sources.
The issue was resolved by manually adding the key under the Application key. This was a great solution because we did not have to change the "broken" server from "Network Service". In reality, the server wasn't broken at all and it can be argued that the server that was "working" initially was actually broken. It has since been switched to "Network Service" as well.
And now it is time for the moral of the story if it hasn't hit home yet-- always follow a policy of least privilege. There have been so many times I have encountered people who practically give away the keys to the castle in order to get a program working. Don't take the easy way out. Do a little troubleshooting. Identify the root cause of the issue. Compromising security is not worth the risk.
* This particular server is running IIS 6, so the quoted TechNet article is specific to that version. As of IIS 7 there is now an additional option for "ApplicationPoolIdentity". This user is unique to the specific AppPool rather than being a shared account like the other options. This is inherently more secure and is thus the default and recommended option.
