Sysadmin Blog

Wednesday, April 15, 2015

GIT

Create new GIT repo on the server:
add user git
create athorized_keys for git
mkdir /data/git
ln -s /data/git /git
cd /git
mkdir ops.git
cd ops.git
git --bare init

Friday, April 10, 2015

Built-in puppet variables in templates:

# This file is controlled by Puppet
# /etc/puppet/puppet.conf
# <%= scope.lookupvar('::osfamily') %> <%= scope.lookupvar('::operatingsystemmajrelease') %>

Example of template and file:

file { "/etc/puppet/puppet.conf":
owner => "root",
group => "root",
mode => 644,
content => template("puppet/puppet.conf.erb"),
notify => Service["puppet"],
}

file { "/etc/sysconfig/puppet":
owner => "root",
group => "root",
mode => 644,
source => "puppet:///modules/puppet/puppet",
notify => Service["puppet"],
}

Hieradata

cat hiera.yaml
---
:backends:
- yaml
:yaml:
:datadir: /etc/puppet/hieradata
:hierarchy:
- "%{::clientcert}"
- "nodes/%{::fqdn}"
- "%{::environment}"
- "location/%{::location}"
- defaults

Example of location yaml

cat location/mylocation.yaml
---
dns::search: myl.location.com location.com
dns::servers:
- 192.168.0.100
- 192.168.0.101
ssh::group_key: AAAAB3NzaC1yc........

Monday, March 16, 2015

How to setup E-mail relay

Gmail rejects e-mail from my server.

Solution: E-mail relay through Mandrill (MailChimp service)

1. Create account with Mandrill (it's free)
2. Generate some API key on Mandrill website
3. Setup your Postfix
yum -y install postfix cyrus-sasl-plain cyrus-sasl-md5

4. Modify /etc/postfix/main.cf
relayhost = smtp.mandrillapp.com
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
smtp_sasl_mechanism_filter = AUTH LOGIN
smtp_sasl_security_options =

5. Put your credentials in /etc/postfix/sasl_passwd file
smtp.mandrillapp.com yourusername@yourdomain.com:yourapikeyhere

6. Create sasl db readable by postfix
postmap sasl_passwd
chmod 600 sasl_passwd
chown postfix:postfix sasl_passwd.db

Your maillog should look similar to this:
Mar 15 23:45:05 tm1 postfix/master[29546]: daemon started -- version 2.6.6, configuration /etc/postfix
Mar 15 23:45:41 tm1 postfix/pickup[29548]: 34B98580496: uid=500 from=<user>
Mar 15 23:45:41 tm1 postfix/cleanup[29678]: 34B98580496: message-id=<20150316064541.34B98580496@mail.tagmap.me>
Mar 15 23:45:41 tm1 postfix/qmgr[29549]: 34B98580496: from=<user@mail.tagmap.me>, size=419, nrcpt=1 (queue active)
Mar 15 23:45:42 tm1 postfix/smtp[29680]: 34B98580496: to=<user@gmail.com>, relay=smtp.mandrillapp.com[54.70.134.182]:25, delay=1.3, delays=0.02/0.01/1.1/0.24, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 5A3F038028A)
Mar 15 23:45:42 tm1 postfix/qmgr[29549]: 34B98580496: removed

Tuesday, December 23, 2014

LVM

Logical Volume Manager

How to expand existing root partition using LVM

Add a second physical drive. Scan the system, no reboot needed:
echo "- - -" > /sys/class/scsi_host/host1/scan

Create Logic Group and Volume:

pvcreate datavg /dev/sdb1
vgcreate datavg /dev/sdb1
lvcreate -l 100%FREE -n lvdata datavg

# vgdisplay
--- Volume group ---
VG Name datavg
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 59.99 GiB
PE Size 4.00 MiB
Total PE 15358
Alloc PE / Size 15358 / 59.99 GiB
Free PE / Size 0 / 0
VG UUID zWlhGf-YDZa-27xv-aH0t-F6fW-Hs1e-TY3d8t

mkfs.ext3 -m 0 /dev/datavg/lvdata
tune2fs -c0 /dev/datavg/lvdata

Expand existing root LVM partition

pvdisplay
pvcreate /dev/sdb1
vgextend root_partition_name /dev/sdb1
lvextend -l +100%FREE /dev/root_partition_name/root

resize2fs /dev/root_partition_name/root

If you need to reduce it back:

vgreduce -a root_partition_name

Thursday, September 11, 2014

HTTP POST into a form from curl

I want to post some data from file.txt into a form on the website. Curl can do it from command line:

curl -X POST -d @filename.txt -u usernamehere:passwordhere https://server.company.com/session/sync.jsp

The content of filename.txt is pairs of name=value separated by &. E.g.

id=1&options=doit&field3=Submit

Friday, September 05, 2014

Code pre-formatted text to Blog

To paste code or pre-formatted text into Blog use pre tags (mind those spaces)

< pre >
< / pre >

Puppet logs on CentOS

By default Puppet sends logs to /var/log/messages Modify /etc/sysconfig/puppet file to send them to /var/log/puppet/ E.g.

# The puppetmaster server
PUPPET_SERVER=puppetmaster

# If you wish to specify the port to connect to do so here
#PUPPET_PORT=8140

# Where to log to. Specify syslog to send log messages to the system log.
PUPPET_LOG=/var/log/puppet/puppet.log

# You may specify other parameters to the puppet client here
#PUPPET_EXTRA_OPTS=--waitforcert=500

Wednesday, August 13, 2014

Change password without root on a list of servers

Some companies are strict on passwords and force you to to change your password every 30 days, no repeating passwords, the passwords have to be long, contain special characters, etc. Here is a script that you can add as a cronjob assuming you have your ssh keys setup.

You'd need a file "list.txt" with the list of IPs and "oldpass.txt" file with the current password. You can just set a cronjob to do reset password on a regular basis (every 25 days for example).

#!/bin/bash
# Generate random new password 24 characters long
newpass=`mkpasswd -l 30 -d 3 -C 5 -s 3`
# Get old password from oldpass file
oldpass=`cat oldpass.txt`
echo Old: $oldpass
echo New: $newpass

# ssh to server and reset password
for server in `cat list.txt`
  do
    echo "Changing $server"
    ssh -t $server 'passwd <<EOF
'$oldpass'
'$newpass'
'$newpass'
EOF'
  done

Sort IP addresses

cat list.txt |sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n

Friday, June 27, 2014

Enable port on Arista switch

Similar to Cisco, enabling port 35 on switch1 for example.

switch1>en
switch1#show interfaces Ethernet 35
switch1#configure
switch1(config)#interface Ethernet 35
switch1(config-if-Et35)#no shutdown
switch1(config-if-Et35)#end

Save your changes:
switch1#copy running-config startup-config

Tuesday, June 17, 2014

Bash - cross-reference two files

I have two files: list_to_remove and current_list. I want to remove all servers from list_to_remove from the current list.

for i in $(cat list_to_remove); do grep -v "$i.company.com" current_list > temp && mv temp current_list; done

Thursday, June 05, 2014

How to replace a section of text file bash script

Goal:
I want to replace a section of hibernate.cfg.xml with new servers. I need to remove servers between
<property name="hibernate.memcached.servers">
server1 server2 server3
</property>

and replace them with mem1 mem2 mem3

cat replace.sh


#!/bin/bash

awk  'BEGIN {A = 1};//{A=0; print "\t\t\n\t\t\t\
mem1.atl.company.com:11212 mem2.atl.company:11212 mem3.atl.company.com:11212 mem4.atl.company.com:11212 mem5.atl.company.com:11212 mem6.atl.company:11212 mem148.atl.company.com:11212 mem149.atl.company.com:11212 mem150.atl.company.com:11212\
\n\t\t"};/.*/ { if ( A == 1) print $0};/<\/property>/{A=1}; ' $1

Run script against any xml file

./replace.sh hibernate.cfg.xml > hibernate.cfg.xml.new

Thursday, May 29, 2014

Puppet manifest for multiple servers

Puppet supports regular expressions.

E.g.
Run puppet manifest on all the web servers:
node /^web.*$/ {
class { 'server_web': }
}

Run on a few (db0-9)
node /^db[0-9]\..*$/ {
class { 'server_db': }
}

Wednesday, May 14, 2014

bash script sum

Bash script exercise.
Add variable to the result of the sum. E.g.
1 1 2 3 5 8 13 21, etc.

#!/bin/bash
x=1
y=1
ans=0

while [ $ans -lt 100 ]
#while true
do
ans=$(( x + y ))
echo $x + $y = $ans
x=$y
y=$ans
done

Monday, May 12, 2014

Bash "for loop"
How to ping multiple selected servers in selected multiple racks.

for rack in 1 3 4 6 9 {14..27}; do echo $rack; for server in {46..48}; do fping "serv"$server"rack"$rack |grep unreachable |tee -a /tmp/down; done; done

Friday, May 02, 2014

Number of hits from server in bash

Let's say I have a list of servers and I want to calculate how many hits did I have from "client.com" on April 28, 2014

#!/bin/bash
total=0

for server in $(cat server_list)
do
    echo $server
    echo "Current count = "$total
    servercount=$(ssh $server "zcat /var/log/httpd/access.2014-04-28.log.gz |grep client.com |wc -l")
    echo "Server count ="$servercount
    total=$(($total + $servercount))
done

echo "Total count = "$total

Monday, April 21, 2014

Multiple DNS master servers

It is perfectly fine to have multiple master DNS servers as long as you keep them all in sync. Here is an example how to make changes to DNS zones in GIT and push the changes to Bind servers.

DNS push script uses dedicated "sysadmin" account. SSH keys have to be in place.

#!/bin/bash
# OPS branch have to be checked out to ~/git/ops/ to use this script
# I assume you have sudo privileges

usage(){
        echo -e Usage:\\n$0 datacenter
        exit 1
}

if [[ -z "$1" ]]
then
usage
exit 1
fi

datacenter=$1

cd ~/git/ops/config/sjc/named/var/named/data/
# Build reverse DNS zones
/usr/bin/mkrdns -root ~/git/ops/config/sjc/named/ ~/git/ops/config/sjc/named/etc/named.conf
# Checking syntax
for zone in `ls`
do
    test=$(named-checkzone $zone $zone |grep OK)
    if [ "$test" != 'OK' ]
      then
      echo "Zone $zone syntax is wrong!"
      exit 1
    else
      echo "Syntax is fine"
    fi
done

echo "pushing to $datacenter"

# Execute remote dns change
sudo su - sysadmin -c "ssh -t -o StrictHostKeyChecking=no mgr1.$datacenter.company.com \"~/git/ops/scripts/dnspull.sh\""
echo "======================================================"
echo "mgr1.$datacenter.company.com is done"
echo "======================================================"
sudo su - sysadmin -c "ssh -t -o StrictHostKeyChecking=no mgr2.$datacenter.company.com \"~/git/ops/scripts/dnspull.sh\""
echo "======================================================"
echo "mgr2.$datacenter.company.com is done"
echo "======================================================"

DNS pull script will pull your changes from GIT, apply the changes, restart named, test if resolution is working properly (in this case "mgrclust1" server), and push the change to the next server.

#!/bin/bash
# Run it on the actual DNS (mgr1,2) server as user "sysadmin"

# I'm relying on resolv to get DC
DC=$(cat /etc/resolv.conf |grep search |awk '{print $2}' |awk -F. '{print $1}')
echo "=============================================="
echo "Pulling zones for $DC"
echo "=============================================="

# Pull from GITolite
cd ~/git/ops/
git pull origin master

# Copy zones over
rsync -av config/$DC/named/etc/ /var/named/chroot/etc/
rsync -av --delete config/$DC/named/var/named/ /var/named/chroot/var/named/

# Fix permissions
chown sysadmin:named /var/named/chroot/etc/named.conf
chown -R sysadmin:named /var/named/chroot/var/named/

# Restart named, look for OK
named_test=$(sudo /etc/init.d/named restart |grep Starting |awk '{print $4}')
echo $named_test

if [ "$named_test" != 'OK' ]
then
    echo "Named restart failed!"
    exit 1
else
    echo "Named restarted"
fi

# Try to resolve mgrclust1
dns_test=$(host mgrclust1 localhost |grep mgrclust1 |awk '{print $4}' |awk -F. '{print $4}')
if [ "$dns_test" != '1' ]
then
echo "I can't resolve mgrclust1!"
exit 1
fi

Monday, December 02, 2013

Kickstart

Menu config file example

/tftpboot/pxelinux.cfg/default

default vesamenu.c32
Menu Background sm_bk.png
Menu Title Boot Menu

label install
   menu label ^Ubuntu-10.04
   kernel ubuntu-installer10/amd64/linux
   append vga=normal initrd=ubuntu-installer10/amd64/initrd.gz ks=http://10.10.4.58/ks10.cfg

label install
   menu label ^Ubuntu-12.04
   menu default
   kernel ubuntu-installer/amd64/linux
   append vga=normal initrd=ubuntu-installer/amd64/initrd.gz ks=http://10.10.4.58/ks12.cfg

LABEL Centos-5.8
    MENU LABEL ^Centos-5.8
    KERNEL centos58/vmlinuz
    APPEND initrd=centos58/initrd.img ramdisk_size=100000 ip=dhcp ksdevice=bootif ks=http://10.10.23.201/ks/ks.cfg
    IPAPPEND 2

LABEL Centos-6
    MENU LABEL ^Centos-6
    KERNEL centos6/vmlinuz
    APPEND initrd=centos6/initrd.img ip=dhcp ksdevice=bootif ks=http://10.10.23.201/ks/ks_co6.cfg

LABEL Centos-6-p1
    MENU LABEL ^Centos-6 p1p1
    KERNEL centos6/vmlinuz
    APPEND initrd=centos6/initrd.img ip=dhcp ksdevice=bootif ks=http://10.10.23.201/ks/ks_co6p1.cfg

LABEL Centos-6-em1
    MENU LABEL ^Centos-6 em1
    KERNEL centos6/vmlinuz
    APPEND initrd=centos6/initrd.img ip=dhcp ksdevice=bootif ks=http://10.10.23.201/ks/ks_co6em1.cfg

label expert
   menu label ^Expert install
   kernel ubuntu-installer/amd64/linux
   append priority=low vga=normal initrd=ubuntu-installer/amd64/initrd.gz --

label cli-expert
   menu label Command-^line expert install
   kernel ubuntu-installer/amd64/linux
   append tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false priority=low vga=normal initrd=ubuntu-installer/amd64/initrd.gz --

label rescue
   menu label ^Rescue mode
   kernel ubuntu-installer/amd64/linux
   append vga=normal initrd=ubuntu-installer/amd64/initrd.gz rescue/enable=true -- quiet

label Local_drive
   localboot 0
   menu label ^Local Drive

prompt 0
timeout 60

Kickstart config file example:

/srv/kickstart/cs6ks.cfg

#platform=x86, AMD64, or Intel EM64T
install
key --skip
lang en_US.UTF-8
# Forces the cmdline installer to be used (debugging)
#cmdline
# System authorization information
auth --useshadow --enablemd5
# System bootloader configuration
bootloader --location=mbr
# Clear the Master Boot Record
zerombr
# Partition clearing information
clearpart --all --initlabel
#ignoredisk --only-use=sda,sdc,sdd
# Use text mode install
text
# Firewall configuration
firewall --disabled
# Run the Setup Agent on first boot
firstboot --disable
# System keyboard
keyboard us
# System language
#lang en_US
# Installation logging level
logging --level=info
# Use network installation
url --url=http://10.10.23.201/centos/6/os/x86_64/
# Network information
network --bootproto=dhcp --device=eth0 --onboot=on
# Reboot after installation
reboot
#Root password
rootpw --iscrypted $1...1

# SELinux configuration
selinux --disabled
# Do not configure the X Window System
skipx
# System timezone
timezone America/Los_Angeles
# Install OS instead of upgrade
install
# Disk partitioning information
part /boot --bytes-per-inode=4096 --fstype="ext3" --ondisk=sda --size=500

# Web servers use 8G of swap
#part swap --bytes-per-inode=4096 --fstype="swap" --ondisk=sda --size=8000
# Changing swap for testing
part swap --bytes-per-inode=4096 --fstype="swap" --ondisk=sda --size=1000
# The rest goes to root
part / --bytes-per-inode=4096 --fstype="ext3" --grow --ondisk=sda --size=1

# Use WF Repo
repo --name=sm_base --baseurl=http://10.10.23.201/centos/6/os/x86_64/
repo --name=sm_updates --baseurl=http://10.10.23.201/centos/6/updates/x86_64/

%packages
@core
ntp
openssh-clients
openssh-server
strace
oprofile
screen
sysstat
lynx
-bluez-gnome
-bluez-libs
-bluez-utils
-ccid
-coolkey
-conman
-cups
-dosfstools
-ifd-egate
-irda-utils
-NetworkManager
-oddjob
-oddjob-libs
-pcsc-lite
-pcsc-lite-libs
-ppp
-redhat-lsb
-rp-pppoe
-stunnel
-wpa_supplicant
-ypbind
-yp-tool

%post --log=/root/my-post-log
# Remove all i386/i686 packages, we only want x86_64 packages.
rpm -e $(rpm -qa --queryformat='%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n' | grep '\.i[3456]86$')

# XFS support
yum -y install kmod-xfs xfs*

# Disable services typically not needed.
for service in iptables ip6tables rpcgssd rpcidmapd iscsi iscsid ; do
        echo "Disabling some unneeded services."
        chkconfig $service off
        echo "Services disabled."
done

echo "exclude=kernel* *.i?86" >> /etc/yum.conf
rm -rf /etc/yum.repos.d/*
wget http://vmks.corp.example.com/sm6.repo
mv sm6.repo /etc/yum.repos.d/sm.repo
ntpdate -u ntp1
ntpdate -u ntp1
ntpdate -u ntp1
/etc/init.d/ntpd start
chkconfig ntpd on

# Converting from DHCP to static IPs
KSDEVICE=`LANG=C /sbin/route -n | awk '/^0.0.0.0/ { print $8 }'`
IP=`LANG=C /sbin/ifconfig $KSDEVICE | /bin/awk '/inet/ && !/inet6/{sub(/addr:/, ""); print $2}'`
HOSTNAME=`LANG=C /usr/bin/host $IP | /bin/awk '{sub(/\.$/, ""); print $5}' |cut -d"." -f1`
NETMASK=`LANG=C /sbin/ifconfig $KSDEVICE | /bin/awk '/inet/ && !/inet6/ {sub(/Mask:/, ""); print $4}'`
NETWORK=`LANG=C /bin/ipcalc $IP -n $NETMASK | /bin/cut -d\= -f2`
GATEWAY=`LANG=C echo $NETWORK | awk -F'.' '{print $1"."$2"."$3"."$4+1}'`
HWADDR=`LANG=C /sbin/ifconfig $KSDEVICE | /bin/awk '/HWaddr/ { print $5 }'`

cat << EOF > /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=$HOSTNAME
GATEWAY=$GATEWAY
EOF

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-$KSDEVICE
DEVICE=$KSDEVICE
BOOTPROTO=static
IPADDR=$IP
NETMASK=$NETMASK
ONBOOT=yes
HWADDR=$HWADDR
EOF

# chef-client

if [ ! -e /etc/chef ]; then
        mkdir /etc/chef
fi

cat > /etc/chef/client.rb << EOCLRB
log_level        :info
log_location     STDOUT
chef_server_url "http://chefserver:4000"
validation_client_name "chef-validator"
# Using default node name (fqdn)
EOCLRB
chmod 600 /etc/chef/client.rb

cat > /etc/chef/validation.pem << EOVALPEM
-----BEGIN RSA PRIVATE KEY-----
MIIE...Wg==

-----END RSA PRIVATE KEY-----
EOVALPEM
chmod 600 /etc/chef/validation.pem

# Install Chef packages
yum -y install rubygem-chef
chkconfig chef-client on
yum -y update

Monday, November 25, 2013

Apache Active Directory authentication

If your Linux server is setup with AD authentiction you can add Apache AD auth as well.

AD LDAP authentication

Install OpenLDAP packages:

yum -y install openldap-clients openldap-devel

Comment everything from /etc/openldap/ldap.conf except the following line:

TLS_REQCERT never

Make sure that you can run a simple LDAP query against domaincontroller.yourdomain.com

ldapsearch -x -LLL -D 'Igor Grinkin' -H ldaps://domaincontroller.yourdomain.com -W -b "CN=Domain Admins,CN=Users,DC=corp,DC=yourdomain,DC=com" -L cn=*

Modify vhost.

<Directory "/usr/share/icinga/">
   Options None
   AllowOverride All
   Order allow,deny
   Allow from all
   AuthName "Please enter your Corp AD credentials"
   AuthType Basic
   AuthLDAPURL "ldap://domaincontroller.yourdomain.com:389/OU=YourDomain,DC=corp,DC=domaincontroller,DC=com?samAccountName?sub?(objectCategory=person)"
   AuthLDAPBindDN "cn=joinaccount,CN=Users,DC=corp,DC=yourdomain,DC=com"
   AuthLDAPBindPassword "joinaccount password"
   AuthBasicProvider ldap
   AuthzLDAPAuthoritative off
   Require valid-user
</Directory>

Monday, November 04, 2013

Chef OS versions fork

On CentOS-5.8 the service is portmap. In CentOS-6.4 it got obsolete and replaced by rpcbind. Here is the case statement for Chef.

# Start portmap or rpcbind
service "port" do
  case node["platform_version"]
    when "5.8"
      service_name "portmap"
    when "6.4"
      service_name "rpcbind"
  end
  action [ :start, :enable ]
end