Log Rotation

The following is a script running once a day on a CentOS box to manage logs
Whenever possible, applications and machines are outputting logs to the NAS. Centralized logging allows us to manage them in one location and also helps with troubleshooting problems.

There are 2 log volumes. "Hot" Logs keeps logs created or modified within the past 24 hours. Archived Logs keeps logs for 60 days. This is very helpful when troubleshooting an active production issue because the relevant logs are very easy to find. Sifting through archived logs can take some time.

The script also does a gzip on the archived logs volume to conserve space. Logs will typically compress 80-95%. The script also removes empty folders to keep the log volumes clean.

Before running the entire script, run the rsync. It is key to get all the logs to archives before starting any deletion.

#!/bin/bash

HOTLOGS=/mounts/logs/
ARCHIVELOGS=/mounts/logsarchived/

# Delete any file older than 2 days from HOT Logs
echo "$(date): Deleting expired files:"
find $HOTLOGS -daystart -mtime +2 -type f -print -exec /bin/rm -vf {} \;

echo Geo log maintenance script
# Copy HOT to Archived
#rsync -aqO $HOTLOGS $ARCHIVELOGS --exclude $HERMOD
rsync -aqO --exclude "*.xml" $HOTLOGS $ARCHIVELOGS

# gzip files in Archived Logs older than 1 day
echo "$(date): Compressing old files:"
find $ARCHIVELOGS -type f -daystart -mtime +1 ! -name "*.gz" -print -exec /bin/gzip -v -f -S ".$(date +%F).gz" {} \;

# Delete any file older than 60 days from Archived Logs
echo "$(date): Deleting expired files:"
find $ARCHIVELOGS -daystart -mtime +60 -type f -name "*.gz" -print -exec /bin/rm -vf {} \;

# Remove empty HOT Logs directories
echo "$(date): Deleting empty directories:"
find $HOTLOGS -depth -type d -empty -print -exec /bin/rmdir {} \;

# Remove empty Archived Logs directories
echo "$(date): Deleting empty directories:"
find $ARCHIVELOGS -depth -type d -empty -print -exec /bin/rmdir {} \;

echo $(date) Done