Log Rotation

The following is a script running once a day on a CentOS box to manage logs
Whenever possible, applications and machines are outputting logs to the NAS. Centralized logging allows us to manage them in one location and also helps with troubleshooting problems.

There are 2 log volumes. "Hot" Logs keeps logs created or modified within the past 24 hours. Archived Logs keeps logs for 60 days. This is very helpful when troubleshooting an active production issue because the relevant logs are very easy to find. Sifting through archived logs can take some time.

The script also does a gzip on the archived logs volume to conserve space. Logs will typically compress 80-95%. The script also removes empty folders to keep the log volumes clean.

Before running the entire script, run the rsync. It is key to get all the logs to archives before starting any deletion.

#!/bin/bash

HOTLOGS=/mounts/logs/
ARCHIVELOGS=/mounts/logsarchived/

# Delete any file older than 2 days from HOT Logs
echo "$(date): Deleting expired files:"
find $HOTLOGS -daystart -mtime +2 -type f -print -exec /bin/rm -vf {} \;

echo Geo log maintenance script
# Copy HOT to Archived
#rsync -aqO $HOTLOGS $ARCHIVELOGS --exclude $HERMOD
rsync -aqO --exclude "*.xml" $HOTLOGS $ARCHIVELOGS

# gzip files in Archived Logs older than 1 day
echo "$(date): Compressing old files:"
find $ARCHIVELOGS -type f -daystart -mtime +1 ! -name "*.gz" -print -exec /bin/gzip -v -f -S ".$(date +%F).gz" {} \;

# Delete any file older than 60 days from Archived Logs
echo "$(date): Deleting expired files:"
find $ARCHIVELOGS -daystart -mtime +60 -type f -name "*.gz" -print -exec /bin/rm -vf {} \;

# Remove empty HOT Logs directories
echo "$(date): Deleting empty directories:"
find $HOTLOGS -depth -type d -empty -print -exec /bin/rmdir {} \;

# Remove empty Archived Logs directories
echo "$(date): Deleting empty directories:"
find $ARCHIVELOGS -depth -type d -empty -print -exec /bin/rmdir {} \;

echo $(date) Done

Robocopy and Delete

@echo on
setlocal
set TODAY=%year%-%month%-%day% //create a variable called TODAY for the date
set LOGFILE=c:\sqlBackup-%TODAY%.log //set log file location
Set SOURCE=\\nas\backups\prod //set source of files to be copied
Set Target=\\nas\backups\prod2 //set destination of files to be copied
cd "Program Files\Windows Resource Kits\Tools" //navigate to location of robocopy executable
robocopy.exe %SOURCE% %TARGET% /MIR //execute robocopy with variables
echo Y|del %SOURCE%\*.* //delete all files from source
endlocal

Windows SQL Express Backup Script with Extras

This script was created to perform a backup on MS SQL 2005 Express. The lite version of MS SQL does not have built in maintenance tasks. Then robocopy moves the files from the box to a CIFS share. Then a forfiles line is used to delete old copies of the backups.

sqlcmd -S servername\SQLinstanceName -E -Q "EXEC sp_BackupDatabases @backupLocation='d:\backups\', @backupType='F'"
robocopy.exe d:\backups \\nas\backups\servername /mir
Forfiles -p d:\backups -s -m *.* -d -2 -c "Cmd /C del @FILE"