Bash Script Output logging

Bash Script Output logging

Searching for logging technic's while running bash command/scripts we ran across these to elegant ways to ensure all logging is captured when running a script.


Quick and dirty : (Credit to christian)

to get the ssh output to your logfile, you have to redirect stderr to stdout. you can do this by appending 2>&1 after your bash script.

it should look like this:

#!/bin/bash
(
...
) 2>&1 | tee ...

when this does not display the mesages in the right order, try to add another subshell:

#!/bin/bash
((
...
) 2>&1) | tee ...

Super Nice: (Credit to post by: nicerobot )

I generally put something similar to the following at the beginning of every script (especially if it'll run as a daemon):

#!/bin/bash
exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>log.out 2>&1
# Everything below will go to the file 'log.out':

Explanation:

1. exec 3>&1 4>&2
   Saves file descriptors so they can be restored to whatever they were before redirection or used themselves to output to whatever they were before the following redirect.
2. trap 'exec 2>&4 1>&3' 0 1 2 3
   Restore file descriptors for particular signals. Not generally necessary since they should be restored when the sub-shell exits.
3. exec 1>log.out 2>&1
   Redirect stdout to file log.out then redirect stderr to stdout. Note that the order is important when you want them going to the same file. stdout must be redirected before stderr is redirected to stdout.

From then on, to see output on the console (maybe), you can simply redirect to &3. For example,

echo "$(date) : part 1 - start" >&3

will go to wherever stdout was directed, presumably the console, prior to executing line 3 above.

Powershell Invoke-WebRequest to Jenkins

Scenario: Jenkins with authentication turned on and a user wanting to use Powershell to remotely execute a job

These 2 links provide most of the background and give examples for WGET. Jenkins authentication being managed by Crowd in this scenario

https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API
https://wiki.jenkins-ci.org/display/JENKINS/Authenticating+scripted+clients


Invoke-WebRequest -Uri $url -Headers @{"Authorization" = "Basic "+[System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($User+":"+$Password ))}

Linux: Checking Disk Space

Scenario: The monitoring system sends an email stating a Linux server is running low on disk space. I often had a hard time figuring out what part of the filesystem was growing and needed attention. I've used this a few times with good success.

cd /path/to/some/where
du -hsx * | sort -rh | head -10

Keep drilling down to find the root cause of the disk space issue.

Jenkins Slave Automatic Start

Scenario: Jenkins Master running on Windows with a Jenkins Slave running on CentOS

Problem: The Jenkins Slave is not automatically connecting after a restart of the slave server, master server, or master Jenkins service

Solution: In early 2013 Jenkins introduced a security token fix outlined here

The thread provides many options for connecting. I decided the path of least resistence was to enable anonymous connect/disconnect for slaves. This is acceptable in my environment because each Jenkins instance is behind a firewall and I feel the security risk is minimal. This might not always be the case, but in this situation, it fixed the problem quickly.

I created a jenkins-slave service in /etc/init.d based on this article. I made a few modifications to the script, but it's working great.

AWS SimpleDB IAM Access

We have nodejs applications running in AWS EC2. Instances should be disposable. We setup the apps to log to AWS SimpleDB. But creating a user to access SimpleDB was more difficult then I expected. In IAM, I created a user and applied two policies using the policy generator:

{
  "Statement": [
    {
      "Action": [
        "sdb:*"
      ],
      "Resource": [
        "arn:aws:sdb:*:1111222333:domain/node*"
      ],
      "Effect": "Allow"
    }
  ]
}

{
  "Statement": [
    {
      "Action": [
        "sdb:ListDomains"
      ],
      "Resource": [
        "arn:aws:sdb:*:1111222333:*"
      ],
      "Effect": "Allow"
    }
  ]
}

The number is the AWS account number
The key for success was the second policy for Listing Domains

This is an excellent way to connect and manage SimpleDB from the Chrome browser
https://chrome.google.com/webstore/detail/sdbnavigator/ddhigekdfabonefhiildaiccafacphgg?hl=en-US

Statsd + Graphite Server Configuration

Thanks to Eric for most of this work, he did the heavy lifting and then I documented https://github.com/etsy/statsd/

Yum
yum install python-pip graphite-web python-carbon python-whisper graphite nodejs npm mysql-server mysql-client MySQL-python

User
Create user nodejs to run application
vi /etc/sudoers
Add: nodejs ALL = NOPASSWD: /usr/bin/node
Comment: #Defaults requiretty

Node
mkdir /opt/statsd
cd /opt/statsd
npm install nodeunit
npm install temp
npm install underscore
chown -R nodejs:nodejs /opt/statsd

Mysql
mysql -u root -p

create database graphite;
create user 'graphite'@'localhost' identified by 'password';
grant all on graphite.* to 'graphite'@'localhost';

cd /etc/graphite-web
Edit local_settings.py
Change Database Configuration settings, uncomment and edit to the following

DATABASES = {
    'default': {
        'NAME': 'graphite',
        'ENGINE': 'django.db.backends.mysql',
        'USER': 'graphite',
        'PASSWORD': 'password',
        'HOST': 'localhost',
        'PORT': '3306'
    }
}

Graphite
Run the following
/usr/lib/python2.6/site-packages/graphite/manage.py syncdb
/usr/lib/python2.6/site-packages/graphite/manage.py createsuperuser

Graphite will connect to Apache by default to root directory
service httpd restart
Open Browser and go to http:// and should see graphite

Statsd Config Download statsd from Github and extract to /opt/statsd

Copy exampleConfig.js to Config.js and add the following

{
  "graphitePort": 2003,
  "graphiteHost": "127.0.0.1",
  "address": "0.0.0.0",
  "port": 8125,
  "flushInterval": 10000,
  "deleteIdleStats": false,
  "deleteGauges": false,
  "deleteTimers": false,
  "deleteSets": false,
  "deleteCounters": false,
  "graphite": {
    "legacyNamespace": true,
    "globalPrefix": "stats",
    "prefixCounter": "counters",
    "prefixTimer": "timers",
    "prefixGauge": "gauges",
    "prefixSet": "sets"
  }

Firewall
Open UPD port in ipTables/Firewall
iptables -I INPUT 10 -m state --state NEW -p udp --dport 8125 -j ACCEPT (10 is the line number, change per ipTables open ports)
/sbin/service iptables save
service iptables restart

Upstart
Create /etc/init/statsd

#!upstart
description "statsd"

env PROGRAM_NAME="statsd"
env FULL_PATH="/opt/statsd"
env FILE_NAME="stats.js"
env NODE_PATH="/usr/bin/node"
env USERNAME="nodejs"

start on runlevel [2345]
stop on runlevel [016]

script
    echo $$ > /var/run/$PROGRAM_NAME.pid
    cd $FULL_PATH       
    exec sudo -u $USERNAME $NODE_PATH $FULL_PATH/$FILE_NAME /opt/statsd/dbs2Config.js >> /var/log/$PROGRAM_NAME.sys.log 2>&1
end script

Start statsd
initctl start statsd

Cacti LDAP Authentication with Active Directory

Cacti 0.8.8a
Windows 2008 R2 Domain Level

yum install php-ldap

Settings -> Authentication -> LDAP Authentication

LDAP used for admins only, graph readers will use the guest account

Guest User: No User
User Template: admin
Server: FQDN of domain controller
Port Standard: 389
Port SSL: 636
Protocol Version: Version 3
Referrals: Disabled
Mode: Specific Searching
Distinguished Name: Blank
Require Group Membership: Not Checked
Group Distinguished Name: CN=cacti_admins,OU=groups_users,DC=foo,DC=domain,DC=com
Group Member Attribute: member
Group Member Type: Distinguished Name
Search Base: ou=groups_users,DC=foo,DC=domain,DC=com
Search Filter: (&(objectclass=user)(objectcategory=user)(userPrincipalName=*))
Search Distinguished Name: ldaplookupuser@foo.domain.com
Search Password: *********

Tips: Some users report issues trying to connect via local and LDAP in the same browser session, log out to test