Change log rotation for Apache

[donkirkby]

A few minutes after 11am on Feb 8, the Kive web server rolled over its log file at /var/log/kive/kive_web.log, because it had reached 15MB. Unfortunately, the rollover seems to have happened many times, because files kive_web.log.1 through kive_web.log.9 all have time stamps around that time and sizes around 1.3MB. The same entries appear in many of the files.
According to an article I found by Justin Montgomery, the problem is that Apache launches several wsgi worker processes for a Django app, and all of them are writing to the same log file. They can't coordinate to make the log rotation safe, so they all rotate around the same time, and each of them creates a separate copy of the log file.
Justin recommends scheduling the logrotate command instead of trying to do this inside Python. We could either launch this from the purge command, or schedule a separate task. Once an hour seems like a reasonable schedule.
Also investigate whether we can log to the standard apache logs.

[donkirkby]

Here's a test script that can reproduce the problem when you run it against the CentOS Vagrant box:

import requests

while True:
    response = requests.post('http://localhost:8080/login',
                             data=dict(username='bogus', password='fake'))
    assert response.status_code == 500, response.status_code

This generates a steady stream of error messages in the log file, and it rolls over several times when it gets to the size limit.
You can make it happen faster, by reducing the log file size in settings.py. I found 500KB was a good size.

[donkirkby]

Configuration changes are described in the INSTALL file, with an example in the vagrant scripts.

We also removed the current log rotation by deleting the /etc/logrotate.d/httpd, because that requires a restart of the httpd service when the files rotate. Removed the KIVE_LOG entry from /etc/sysconfig/httpd, so the error messages would go to the standard /var/log/httpd/error_log.