Linux Setup

These are some commands and setting to run when setting up an Linux box

Install GNOME
Ubuntu 11.10 installs with Unity - I'm still not a fan
sudo apt-get install gnome-shell

Restart the machine and then select from the available X configurations - I like GNOME Classic

SSH Alias

Edit .bashrc to create SSH aliases for fast access to other Linux instances
alias servername='ssh domain\\username@servername.domain.com'
The name of the alias can be anything you like - if there are a lot of servers, the name of the server will probably work the best

source .bashrc to reload changes without a reboot

Create RSA Key Pair

Start in home directory
mkdir .ssh
chmod 700 .ssh
ssh-keygen -t rsa

I usually leave the passphrase blank because I guard my private key

Make backups of the key pair

Other options-C "username@domain.com" - This will set the email address assigned to the public key, by default it will use the username and server name where it is being created

-D 4096 This creates a 4096 bit key, 2048 is set by default and strong enough in most situations

Add SSH Public Key to Server
Access your home directory on the Server
mkdir .ssh
vi .ssh/authorized_keys
Copy/paste the public key
chmod 600 .ssh/authorized_keys

Ruby on RHEL/CentOS/Amazon Linux

By default, Amazon Linux AMI uses Ruby 1.8.6 - development prefers version 1.9.2

Remove preinstalled Ruby

sudo yum erase ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs

Install Development tools

sudo yum install openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel

Download and Build Ruby 1.9.2

wget ftp://ftp.ruby-lang.org//pub/ruby/1.9/ruby-1.9.2-p290.tar.gz tar xzvf ruby-1.9.2-p290.tar.gz cd ruby-1.9.2-p290 ./configure make make install gem install bundler

Add additional packages

yum install freetds freetds-devel git nginix

Add web group and directory

sudo groupadd web

sudo usermod -a -G web nginx

sudo mkdir /var/www

sudo chgrp web /var/www

sudo chmod g+ws /var/www

BgInfo

BgInfo
A Sysinternals tool for displaying information on the desktop - http://technet.microsoft.com/en-us/sysinternals/bb897557

Inside of GPO editor, User Configuration > Policies > Windows Settings > Scripts > Logon
Add script to GPO called bginfo.bat (Make it one line in file, multiple lines for ease of reading documentation)

\\myDomain.com\sysvol\myDomain.com\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\

USER\Scripts\Logon\Bginfo.exe \\myDomain.com\sysvol\myDomain.com\Policies\

{31B2F340-016D-11D2-945F-00C04FB984F9}\USER\Scripts\Logon\bgconfig.bgi 

/Timer:0 /NoLicPrompt

Customize bgconfig.bgi to specifications desired
/Timer:0 = no delay in deploying the wallpaper
/NoLicPrompt = skips the license prompt

Troubleshooting
After joining the domain and rebooting, BgInfo might not be displayed
From cmd, run gpupdate /force and reboot
BgInfo is displayed multiple times
Launch regedit and make the following changes
Delete HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Desktop
Remove the value, keep the key of HKEY_CURRENT_USER\Software\Winternals\BGInfo\Wallpaper
Compare registries of working and misbehaving servers to fix inconsistencies

Ubuntu 11.10 Over the Edge

Last week Ubuntu 11.10 was released.  I was running 11.04 and decided I would upgrade.  I started with Ubuntu at 8.04 and have had successfull, pain free updates since that time.  The streak died with 11.10.  The specific issue was the nVidia driver.  And in my case, it was made worse by my dual monitor setup.  My laptop would only boot about a third of the time even if the additional monitors were not connected.  I was not happy.

The other issue I am dealing with is Unity.  Ubuntu 11.04 saw Gnome replaced by Unity as the X extension.  I did not like Unity.  I like a taskbar and want to be able to access my programs fast.  It sure seems Ubuntu wanted to make their OS look like a Mac.  If I want a Mac, I could buy a Mac.  Why did Canonical change something that had been successfull and was not broken.  In 11.04, I was able to revert to Gnome 2 Classic and keep going.  I was annoyed, but it was not a deal breaker.

I made a valiant effort to keep Ubuntu going.  I tried loading older versions and experimented with 32 and 64 bit installations.  It is very important for me to stay on the latest release.  I have been in too many work situations where old technology cannot be upgraded and for one reason or another.  This is a pain to manage. 

I probably could have lived with Unity, but I still faced the first issue, my laptop would not consistently boot with multiple monitors and a nVidia driver.  I looked at a number of distros and decided to try Fedora.   I wanted a widely used distro that is consistently updated and supported.  I also wanted something dependable.  I run Linux on my work computer because it helps me be better at my job.  A hobbyist distro would not cut it.  I thought about CentOS, but it really is best for servers.  I've used OpenSuse in the past and was not a big fan of YaST.  Fedora, here I come.

Fedora is available as a Live CD that can be installed to the hard drive.  The install was straight forward and fast.  I really liked the option to encrypt the entire hard drive instead of just the home drive.  And it was nice to have yum as the package manager.  I needed to use my Google stills to figure out how to get nVidia, Skype, Flash, and Virtualbox up and running.  These were much easier in Ubuntu.

My biggest issue with Fedora was the look and feel.  And that goes back to Gnome 3.  It is cut from the same cloth as Unity and I do not like using it.  It might be great for a new user or for a tablet.  But for a power user doing multiple tasks at the same time, it slowed me down.

So where do I go next?  The latest version of Ubuntu will not consistently boot on my machine.  Fedora works, but it is very hands on.  Gnome 3 and Unity both stink.  I'm giving serious consideration to going back to Windows.  I've used 7 at home and in a virutal machine and been very impressed.  It is fast, reliable, and the user interface is consistent and can be customized for a power user.  Granted Gnome and Unity can be customized, but it is not an easy process and it is time consuming.

My biggest gripe with Windows is the lack of a native SSH client.  Putty works, but it's not as easy as Linux command line.  So I think I'm headed to Windows 7 and a Ubuntu virtual machine. 

AWStats

I have been on a quest to use AWStats to gather statistics from IIS logs.  I ran into a big roadblock with Elastic Load Balancers and external IP addresses.  I documented the solutions to those issues in previous posts.  The new ELB is in place and it's time for payoff from a lot of hard work.  Here are notes about AWStats setup.


Basic setup is here: http://awstats.sourceforge.net/docs/awstats_setup.html
Kept the setup as close to default as possible

conf file Edits
To add multiple logs - use logresolvemerge.pl

LogFile="/usr/bin/logresolvemerge.pl /mounts/logs/iis/server01/W3SVC1/

u_ex%YY-0%MM-0%DD.log /mounts/logs/iis/server02/W3SVC1/

u_ex%YY-0%MM-0%DD.log /mounts/logs/iis/server03/W3SVC1/u_ex%YY-0%MM-0%DD.log|"

Using Default IIS LogFormat might not work - use this

LogFormat="date time s-sitename s-computername s-ip cs-method cs-uri-stem

cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie)

cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken"

Delete some of the comments at the top of the file to make edits easier
Delete the localhost file if it is not needed
Keep awstats.model.conf for its comments and instructions for editing

html files
perl awstats.pl -config=mysite -update for first log analysis
Files are stored in /var/www/awstats
A awstats.conf files is created for Apache
Create a cronjob for hourly updating
00 * * * * /usr/bin/awstats_updateall.pl now -awstatsprog=/var/www/awstats/awstats.pl
Files cannot be viewed without calling the awstats perl script
Example - http://yourserver/awstats/awstats.pl?=config=services

Elastic Load Balancer(ELB) - SSL termination at the ELB with Backend Authentication

Wildcard Certificate
AWS requires certificates to be in PEM format for uploading

This example uses a wildcard certificate generated by IIS and issued by GoDaddy.  There are better ways to create a certificate for an ELB which can be found in Amazon's documentation

1) Export the certificate as a PFX with the private key and check "Include all certificates in the certification path if possible" - use a password
2) Use openssl on a Linux box to run the following commands
3) openssl pkcs12 -in filename.pfx -nocerts -out key.pem
The command will request the Import Password
Then it will request a PEM Password - Use the same password for ease
4)openssl pkcs12 -in filename.pfx -clcerts -nokeys -out cert.pem
The command will request the Import Password
5)openssl rsa -in key.pem -out server.key
The command will request the PEM password created in the first command

Example

[root@util02 testing]# openssl pkcs12 -in godaddy.pfx -nocerts -out key.pem
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
[root@util02 testing]# openssl pkcs12 -in godaddy.pfx -clcerts -nokeys -out cert.pem
Enter Import Password:
MAC verified OK
[root@util02 testing]# openssl rsa -in key.pem -out server.key
Enter pass phrase for key.pem:
writing RSA key

Upload the certificate
This can be done by the Management Console or via command line
When copying the certificates, be sure to include the headers and footers
If using command line, the cert.pem has unnecessary certificate details at the beginning of the file which will cause the import to fail

http://www.xdevsoftware.com/blog/post/Upload-IIS-SSL-Certificate-into-Amazon-Elastic-Load-Balancer.aspx

Backend Certificates
Certificates can be self signed, 'fakes' and should have expiration date of 2039
See previous post for self signed certs
Export the certificate from IIS
Use openssl on a Linux box to run the following commands
openssl pkcs12 -in certificate.pfx -out certificate.pem -nodesOpen
Import via Management Console or command line

Create Load Balancer
Load Balancer Protocol HTTPS
Port 443
Instance Protocol HTTPS
Port 443
Choose wildcard certificate
Ciphers - Leave as default - ELBSample-ELBDefaultNegotiationPolicy
Select Enable Backend Authentication
Create Health Check HTTPS:443/application/validation.aspx - Leave other settings as default
Add instances
Setup DNS
ELBs have A records like this: loadbalancer-374828799.us-east-1.elb.amazonaws.com (A Record)
Create a CNAME stagingserviceslb.futurehealthsoftware.com with stagingservices-374828799.us-east-1.elb.amazonaws.com

External IP Addresses into IIS Log Files
There are a couple of options

If the application is non WCF - use the F5 ISAPI filter
More details - http://devcentral.f5.com/weblogs/Joe/archive/2009/12/23/x-forwarded-for-http-module-for-iis7-source-included.aspx

If the application is WCF - use the ARR Helper Module
More details - http://blogs.iis.net/anilr/archive/2009/03/03/client-ip-not-logged-on-content-server-when-using-arr.aspx

Self Signed Certificates

Amazon Linux has the structure installed by default

Create the serial database and index file  
cd /etc/pki/CA 
echo '100001' >serial
touch index.txt

The openssl.cnf file is located at /etc/pki/tls It can left as default, it is 2048 bit encryption

Create the private key and root certificate
openssl req -new -x509 -extensions v3_ca -keyout private/cakey.pem -out cacert.pem -days 9999 -config /etc/pki/tls/openssl.cnf

This will create a root cert with expiration in 2039
Generate a password
Use the following for information
Country Name - 2 Letter Abbreviation = US
State or Province Name - Full Name of State = Nebraska
City or Locality - Full City Name = Omaha
Organization Name - Company Name = Acme Inc
Organizational Unit - IT Common Name - Name of the Authority = Company Name Certificate Authority
Email Address

Create a CSR using the Information from the Root Certificate

To generate a certificate
openssl ca -config /etc/pki/tls/openssl.cnf -policy policy_anything -days 9999 -infiles /etc/pki/CA/newcerts/CSRfile

The cert will be displayed on screen

Note
IIS7 will generate a self signed cert in a couple of clicks, but the certs are only valid for 1 year. Expired certs have caused problems with load balancers in the past

Helpful outside link
http://www.flatmtn.com/article/setting-openssl-create-certificates

MSSQL Template for Cacti

The templates can be found here

Follow the instructions except for step 7
Mem Cache is not needed, but there is a "{" at the end of line 29 and a "}" at the end of line 73 that are needed, comment the other syntax in those lines

Give them a few minutes to populate

Sendmail Relay and SES

Simple Email Service (SES)
Documentation is here: http://docs.amazonwebservices.com/ses/latest/DeveloperGuide/

The API code for SES needs to be installed on the sendmail server
Amazon Linux has the code installed at /opt/aws/bin

There are instructions in the SES documentation for integrating with Sendmail
The documentation instructs edits to the sendmail.cf file
Best practices are to edit the sendmail.mc file and then run the make command

Add the following line to the end of sendmail.mc

Maws-email, P=/opt/aws/ses/ses-send-email.pl, F=mDFMuXn, U=mailuser, S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP, A=ses-send-email.pl -r -k /opt/aws/aws-credentials -e https://email.us-east-1.amazonaws.com   -f $f $u

P = location of SES scripts - edit as needed
F,S,R,A,f,u = Sendmail options - do not change these
U = mailuser is a local user, create on the instance with no remote access rights
k = location of credential file - edit as needed
-e = SES servers, can be changed but not necessary

After adding the line, run make -C /etc/mail
Restart Sendmail - /etc/init.d/sendmail restart

Follow the remainder of the AWS SES instructions for sendmail setup

Sendmail Changes
Edit /etc/mail/access
Add the following to enable devices within the internal network with 10.* IP addresses to send mail to the relay server
Connect:10 RELAY
Compile the access.db file
makemap hash /etc/mail/access.db < /etc/mail/access
Restart Sendmail - /etc/init.d/sendmail restart

Edit sendmail.mc
Make sure the following line is commented out - the dnl does the commenting
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl

Make sure the following lines are active and changed to FH
MASQUERADE_AS(`yourdomain.com')dnl
FEATURE(masquerade_envelope)dnl

After any and all of these changes, run make -C /etc/mail
Restart Sendmail - /etc/init.d/sendmail restart

Testing
PMP has a built in SMTP test utility
Troubleshooting - Review /var/log/maillog

Notes
Documentation for sendmail is scarce or very hard to follow. Sendmail has been around for years and is mature software. But it is opensource and by my definition still a bit confusing. To get sendmail working like I wanted, I used the Googler but never found anything I would consider Bookmark worthy.

After getting the setup working, there are very few steps and it seems easy. But there was not much to start with and there was quite a bit of trial and error.

Turn off Screen Saver via GPO

This should be done for cloud/virtual servers to save CPU/resources

User Configuration -> Policies -> Administrative Templates -> Control Panel -> Display
Set Screen Saver to disabled

Cacti

Cacti is an open source performance monitor for IT infrastructure. It will run on Linux or Windows and stands up very well against commercial equivalents. Be mindful that open source can mean lots of tinkering time. I have used Cacti for many years but was recently given an opportunity to set it up for the first time. There were a few quirks in the process I want to remember for the future to speed up with installation. This is not an exhaustive how-to doc as I was able to find everything I needed via Google searches. But it was not all in one place so I wanted to put this together.

I prefer Linux for Cacti - PHP and MySQL are significantly easier to setup

Follow Official Cacti Installation Document
http://www.cacti.net/downloads/docs/html/install_unix.html

Notes
The newest version of PHP typically does not require any of the changes from the Cacti Installation Document, but go through the checklist regardless
Some of the MySQL commands will require --password forcing a password prompt
Common issue, no graphs being created - run the poller.php command as cactiuser from the command prompt - this will generate output with errors
Go through the errors - found one of the php files did not have proper MySQL credentials
Timezone issue - FH runs in UTC, but for reading Cacti graphs, having current time is convienient

Do not change any of the settings until graphs are being populated, once that is complete, change the poller to spine and set the polling interval to 1 minutes

Spine
Follow Official Cacti Spine Installation Document - http://cacti.net/spine_install_rhlnx.php
When changing to Spine, clear the poller cache - http://www.cacti.net/downloads/docs/html/scripts.html#CLI_REBUILD_POLLER_CACHE

1 Minute Polling
Change Settings page to 1 minute and change crontab to * * * * *
Also must update data templates - http://docs.cacti.net/manual:087:3_templates.1_data_template
Step changes to 60
Heartbeat changes to 120

Import Windows Graph Templates
The bundled Windows graphs that come preinstalled are average at best, import CPU, memory, I/O, and disk graphs from Cacti Forums - http://forums.cacti.net/viewtopic.php?f=12&t=29832
The thread is many pages long - there are updated versions of the templates, find the newest post from the author and download the attachments
There are instructions on the first post
These graphs require SNMP Informant STD version

Import IIS Graph Templates
Available from Cacti Forums - http://forums.cacti.net/viewtopic.php?f=12&t=12464
Import via the GUI and add to the Windows Template

Ubuntu Menu Configuration

Ubuntu has started shipping with the close, maximize, and minimize buttons in the upper left corner of the menu bar. I cannot get used to it and prefer them on the traditional right side.

A quick How To

1 Press ALT-F2 and run gconf-editor
2 Go to apps --> metacity --> general
3 Select and right-click button_layout
4 Click Edit Key
5 Replace with menu:minimize,maximize,close
6 Click OK and it's done

Log Rotation

The following is a script running once a day on a CentOS box to manage logs
Whenever possible, applications and machines are outputting logs to the NAS. Centralized logging allows us to manage them in one location and also helps with troubleshooting problems.

There are 2 log volumes. "Hot" Logs keeps logs created or modified within the past 24 hours. Archived Logs keeps logs for 60 days. This is very helpful when troubleshooting an active production issue because the relevant logs are very easy to find. Sifting through archived logs can take some time.

The script also does a gzip on the archived logs volume to conserve space. Logs will typically compress 80-95%. The script also removes empty folders to keep the log volumes clean.

Before running the entire script, run the rsync. It is key to get all the logs to archives before starting any deletion.

#!/bin/bash

HOTLOGS=/mounts/logs/
ARCHIVELOGS=/mounts/logsarchived/

# Delete any file older than 2 days from HOT Logs
echo "$(date): Deleting expired files:"
find $HOTLOGS -daystart -mtime +2 -type f -print -exec /bin/rm -vf {} \;

echo Geo log maintenance script
# Copy HOT to Archived
#rsync -aqO $HOTLOGS $ARCHIVELOGS --exclude $HERMOD
rsync -aqO --exclude "*.xml" $HOTLOGS $ARCHIVELOGS

# gzip files in Archived Logs older than 1 day
echo "$(date): Compressing old files:"
find $ARCHIVELOGS -type f -daystart -mtime +1 ! -name "*.gz" -print -exec /bin/gzip -v -f -S ".$(date +%F).gz" {} \;

# Delete any file older than 60 days from Archived Logs
echo "$(date): Deleting expired files:"
find $ARCHIVELOGS -daystart -mtime +60 -type f -name "*.gz" -print -exec /bin/rm -vf {} \;

# Remove empty HOT Logs directories
echo "$(date): Deleting empty directories:"
find $HOTLOGS -depth -type d -empty -print -exec /bin/rmdir {} \;

# Remove empty Archived Logs directories
echo "$(date): Deleting empty directories:"
find $ARCHIVELOGS -depth -type d -empty -print -exec /bin/rmdir {} \;

echo $(date) Done

Robocopy and Delete

@echo on
setlocal
set TODAY=%year%-%month%-%day% //create a variable called TODAY for the date
set LOGFILE=c:\sqlBackup-%TODAY%.log //set log file location
Set SOURCE=\\nas\backups\prod //set source of files to be copied
Set Target=\\nas\backups\prod2 //set destination of files to be copied
cd "Program Files\Windows Resource Kits\Tools" //navigate to location of robocopy executable
robocopy.exe %SOURCE% %TARGET% /MIR //execute robocopy with variables
echo Y|del %SOURCE%\*.* //delete all files from source
endlocal

Windows SQL Express Backup Script with Extras

This script was created to perform a backup on MS SQL 2005 Express. The lite version of MS SQL does not have built in maintenance tasks. Then robocopy moves the files from the box to a CIFS share. Then a forfiles line is used to delete old copies of the backups.

sqlcmd -S servername\SQLinstanceName -E -Q "EXEC sp_BackupDatabases @backupLocation='d:\backups\', @backupType='F'"
robocopy.exe d:\backups \\nas\backups\servername /mir
Forfiles -p d:\backups -s -m *.* -d -2 -c "Cmd /C del @FILE"