HTTP POST into a form from curl

I want to post some data from file.txt into a form on the website. Curl can do it from command line:

curl -X POST -d @filename.txt -u usernamehere:passwordhere https://server.company.com/session/sync.jsp

The content of filename.txt is pairs of name=value separated by &. E.g.

id=1&options=doit&field3=Submit

Code pre-formatted text to Blog

To paste code or pre-formatted text into Blog use pre tags (mind those spaces)

< pre >
< / pre >

Puppet logs on CentOS

By default Puppet sends logs to /var/log/messages Modify /etc/sysconfig/puppet file to send them to /var/log/puppet/ E.g.

# The puppetmaster server
PUPPET_SERVER=puppetmaster

# If you wish to specify the port to connect to do so here
#PUPPET_PORT=8140

# Where to log to. Specify syslog to send log messages to the system log.
PUPPET_LOG=/var/log/puppet/puppet.log

# You may specify other parameters to the puppet client here
#PUPPET_EXTRA_OPTS=--waitforcert=500

Change password without root on a list of servers

Some companies are strict on passwords and force you to to change your password every 30 days, no repeating passwords, the passwords have to be long, contain special characters, etc.  Here is a script that you can add as a cronjob assuming you have your ssh keys setup.

You'd need a file "list.txt" with the list of IPs and "oldpass.txt" file with the current password.  You can just set a cronjob to do reset password on a regular basis (every 25 days for example).

#!/bin/bash
# Generate random new password 24 characters long
newpass=`mkpasswd -l 30 -d 3 -C 5 -s 3`
# Get old password from oldpass file
oldpass=`cat oldpass.txt`
echo Old: $oldpass
echo New: $newpass

# ssh to server and reset password
for server in `cat list.txt`
  do
    echo "Changing $server"
    ssh -t $server 'passwd <<EOF
'$oldpass'
'$newpass'
'$newpass'
EOF'
  done

Sort IP addresses

cat list.txt |sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n

Enable port on Arista switch

Similar to Cisco, enabling port 35 on switch1 for example.

switch1>en
switch1#show interfaces Ethernet 35
switch1#configure
switch1(config)#interface Ethernet 35
switch1(config-if-Et35)#no shutdown
switch1(config-if-Et35)#end

Save your changes:
switch1#copy running-config startup-config

Bash - cross-reference two files

I have two files: list_to_remove and current_list.  I want to remove all servers from list_to_remove from the current list.

for i in $(cat list_to_remove); do grep -v "$i.company.com" current_list > temp && mv temp current_list; done

How to replace a section of text file bash script

Goal:
I want to replace a section of hibernate.cfg.xml with new servers.  I need to remove servers between
                <property name="hibernate.memcached.servers">
server1 server2 server3
                </property>

and replace them with mem1 mem2 mem3

cat replace.sh

#!/bin/bash awk 'BEGIN {A = 1};//{A=0; print "\t\t\n\t\t\t\ mem1.atl.company.com:11212 mem2.atl.company:11212 mem3.atl.company.com:11212 mem4.atl.company.com:11212 mem5.atl.company.com:11212 mem6.atl.company:11212 mem148.atl.company.com:11212 mem149.atl.company.com:11212 mem150.atl.company.com:11212\ \n\t\t"};/.*/ { if ( A == 1) print $0};/<\/property>/{A=1}; ' $1

Run script against any xml file

./replace.sh hibernate.cfg.xml > hibernate.cfg.xml.new

Puppet manifest for multiple servers

Puppet supports regular expressions.

E.g.
Run puppet manifest on all the web servers:
node /^web.*$/ {
  class { 'server_web': }
}

Run on a few (db0-9)
node /^db[0-9]\..*$/ {
  class { 'server_db': }
}

bash script sum

Bash script exercise.
Add variable to the result of the sum.  E.g.
1 1 2 3 5 8 13 21, etc.


#!/bin/bash
x=1
y=1
ans=0

while [ $ans -lt 100 ]
#while true
do
  ans=$(( x + y ))
  echo $x + $y = $ans
  x=$y
  y=$ans
done

Bash "for loop"
How to ping multiple selected servers in selected multiple racks.

for rack in 1 3 4 6 9 {14..27}; do echo $rack; for server in {46..48}; do fping "serv"$server"rack"$rack |grep unreachable |tee -a /tmp/down; done; done

Number of hits from server in bash

Let's say I have a list of servers and I want to calculate how many hits did I have from "client.com" on April 28, 2014

#!/bin/bash
total=0

for server in $(cat server_list)
  do
    echo $server
    echo "Current count = "$total
    servercount=$(ssh $server "zcat /var/log/httpd/access.2014-04-28.log.gz |grep client.com |wc -l")
    echo "Server count ="$servercount
    total=$(($total + $servercount))
done

echo "Total count = "$total

Multiple DNS master servers

It is perfectly fine to have multiple master DNS servers as long as you keep them all in sync.  Here is an example how to make changes to DNS zones in GIT and push the changes to Bind servers.


DNS push script uses dedicated "sysadmin" account.  SSH keys have to be in place.


#!/bin/bash
# OPS branch have to be checked out to ~/git/ops/ to use this script
# I assume you have sudo privileges


usage(){
        echo -e Usage:\\n$0 datacenter
        exit 1
}

if [[ -z "$1" ]]
then
  usage
  exit 1
fi

datacenter=$1

cd ~/git/ops/config/sjc/named/var/named/data/
# Build reverse DNS zones
/usr/bin/mkrdns -root ~/git/ops/config/sjc/named/ ~/git/ops/config/sjc/named/etc/named.conf
# Checking syntax
for zone in `ls`
  do
    test=$(named-checkzone $zone $zone |grep OK)
    if [ "$test" != 'OK' ]
      then
      echo "Zone $zone syntax is wrong!"
      exit 1
    else
      echo "Syntax is fine"
    fi
done

echo "pushing to $datacenter"

# Execute remote dns change
sudo su - sysadmin -c "ssh -t -o StrictHostKeyChecking=no mgr1.$datacenter.company.com \"~/git/ops/scripts/dnspull.sh\""
echo "======================================================"
echo "mgr1.$datacenter.company.com is done"
echo "======================================================"
sudo su - sysadmin -c "ssh -t -o StrictHostKeyChecking=no mgr2.$datacenter.company.com \"~/git/ops/scripts/dnspull.sh\""
echo "======================================================"
echo "mgr2.$datacenter.company.com is done"
echo "======================================================"

DNS pull script will pull your changes from GIT, apply the changes, restart named, test if resolution is working properly (in this case "mgrclust1" server), and push the change to the next server.

#!/bin/bash
# Run it on the actual DNS (mgr1,2) server as user "sysadmin"


# I'm relying on resolv to get DC
DC=$(cat /etc/resolv.conf |grep search |awk '{print $2}' |awk -F. '{print $1}')
echo "=============================================="
echo "Pulling zones for $DC"
echo "=============================================="

# Pull from GITolite
cd ~/git/ops/
git pull origin master

# Copy zones over
rsync -av config/$DC/named/etc/ /var/named/chroot/etc/
rsync -av --delete config/$DC/named/var/named/ /var/named/chroot/var/named/

# Fix permissions
chown sysadmin:named /var/named/chroot/etc/named.conf
chown -R sysadmin:named /var/named/chroot/var/named/

# Restart named, look for OK
named_test=$(sudo /etc/init.d/named restart |grep Starting |awk '{print $4}')
echo $named_test

if [ "$named_test" != 'OK' ]
  then
    echo "Named restart failed!"
    exit 1
  else
    echo "Named restarted"
fi

# Try to resolve mgrclust1
dns_test=$(host mgrclust1 localhost |grep mgrclust1 |awk '{print $4}' |awk -F. '{print $4}')
if [ "$dns_test" != '1' ]
  then
  echo "I can't resolve mgrclust1!"
  exit 1
fi

Kickstart

Menu config file example

/tftpboot/pxelinux.cfg/default

default vesamenu.c32
Menu Background sm_bk.png
Menu Title Boot Menu

label install
   menu label ^Ubuntu-10.04
   kernel ubuntu-installer10/amd64/linux
   append vga=normal initrd=ubuntu-installer10/amd64/initrd.gz ks=http://10.10.4.58/ks10.cfg

label install
   menu label ^Ubuntu-12.04
   menu default
   kernel ubuntu-installer/amd64/linux
   append vga=normal initrd=ubuntu-installer/amd64/initrd.gz ks=http://10.10.4.58/ks12.cfg

LABEL Centos-5.8
    MENU LABEL ^Centos-5.8
    KERNEL centos58/vmlinuz
    APPEND initrd=centos58/initrd.img ramdisk_size=100000 ip=dhcp ksdevice=bootif ks=http://10.10.23.201/ks/ks.cfg
    IPAPPEND 2

LABEL Centos-6
    MENU LABEL ^Centos-6
    KERNEL centos6/vmlinuz
    APPEND initrd=centos6/initrd.img ip=dhcp ksdevice=bootif ks=http://10.10.23.201/ks/ks_co6.cfg

LABEL Centos-6-p1
    MENU LABEL ^Centos-6 p1p1
    KERNEL centos6/vmlinuz
    APPEND initrd=centos6/initrd.img ip=dhcp ksdevice=bootif ks=http://10.10.23.201/ks/ks_co6p1.cfg

LABEL Centos-6-em1
    MENU LABEL ^Centos-6 em1
    KERNEL centos6/vmlinuz
    APPEND initrd=centos6/initrd.img ip=dhcp ksdevice=bootif ks=http://10.10.23.201/ks/ks_co6em1.cfg

label expert
   menu label ^Expert install
   kernel ubuntu-installer/amd64/linux
   append priority=low vga=normal initrd=ubuntu-installer/amd64/initrd.gz --

label cli-expert
   menu label Command-^line expert install
   kernel ubuntu-installer/amd64/linux
   append tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false priority=low vga=normal initrd=ubuntu-installer/amd64/initrd.gz --

label rescue
   menu label ^Rescue mode
   kernel ubuntu-installer/amd64/linux
   append vga=normal initrd=ubuntu-installer/amd64/initrd.gz rescue/enable=true -- quiet

label Local_drive
   localboot 0
   menu label ^Local Drive

prompt 0
timeout 60

Kickstart config file example:

/srv/kickstart/cs6ks.cfg

#platform=x86, AMD64, or Intel EM64T
install
key --skip
lang en_US.UTF-8
# Forces the cmdline installer to be used (debugging)
#cmdline
# System authorization information
auth  --useshadow  --enablemd5
# System bootloader configuration
bootloader --location=mbr
# Clear the Master Boot Record
zerombr
# Partition clearing information
clearpart --all --initlabel
#ignoredisk --only-use=sda,sdc,sdd
# Use text mode install
text
# Firewall configuration
firewall --disabled
# Run the Setup Agent on first boot
firstboot --disable
# System keyboard
keyboard us
# System language
#lang en_US
# Installation logging level
logging --level=info
# Use network installation
url --url=http://10.10.23.201/centos/6/os/x86_64/
# Network information
network --bootproto=dhcp --device=eth0 --onboot=on
# Reboot after installation
reboot
#Root password
rootpw --iscrypted $1...1

# SELinux configuration
selinux --disabled
# Do not configure the X Window System
skipx
# System timezone
timezone  America/Los_Angeles
# Install OS instead of upgrade
install
# Disk partitioning information
part /boot --bytes-per-inode=4096 --fstype="ext3" --ondisk=sda --size=500

# Web servers use 8G of swap
#part swap --bytes-per-inode=4096 --fstype="swap" --ondisk=sda --size=8000
# Changing swap for testing
part swap --bytes-per-inode=4096 --fstype="swap" --ondisk=sda --size=1000
# The rest goes to root
part / --bytes-per-inode=4096 --fstype="ext3" --grow --ondisk=sda --size=1

# Use WF Repo
repo --name=sm_base --baseurl=http://10.10.23.201/centos/6/os/x86_64/
repo --name=sm_updates --baseurl=http://10.10.23.201/centos/6/updates/x86_64/

%packages
@core
ntp
openssh-clients
openssh-server
strace
oprofile
screen
sysstat
lynx
-bluez-gnome
-bluez-libs
-bluez-utils
-ccid
-coolkey
-conman
-cups
-dosfstools
-ifd-egate
-irda-utils
-NetworkManager
-oddjob
-oddjob-libs
-pcsc-lite
-pcsc-lite-libs
-ppp
-redhat-lsb
-rp-pppoe
-stunnel
-wpa_supplicant
-ypbind
-yp-tool

%post --log=/root/my-post-log
# Remove all i386/i686 packages, we only want x86_64 packages.
rpm -e $(rpm -qa --queryformat='%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n' | grep '\.i[3456]86$')

# XFS support
yum -y install kmod-xfs xfs*

# Disable services typically not needed.
for service in iptables ip6tables rpcgssd rpcidmapd iscsi iscsid ; do
        echo "Disabling some unneeded services."
        chkconfig $service off
        echo "Services disabled."
done

echo "exclude=kernel* *.i?86" >> /etc/yum.conf
rm -rf /etc/yum.repos.d/*
wget http://vmks.corp.example.com/sm6.repo
mv sm6.repo /etc/yum.repos.d/sm.repo
ntpdate -u ntp1
ntpdate -u ntp1
ntpdate -u ntp1
/etc/init.d/ntpd start
chkconfig ntpd on

# Converting from DHCP to static IPs
KSDEVICE=`LANG=C /sbin/route -n | awk '/^0.0.0.0/ { print $8 }'`
IP=`LANG=C /sbin/ifconfig $KSDEVICE | /bin/awk '/inet/ && !/inet6/{sub(/addr:/, ""); print $2}'`
HOSTNAME=`LANG=C /usr/bin/host $IP | /bin/awk '{sub(/\.$/, ""); print $5}' |cut -d"." -f1`
NETMASK=`LANG=C /sbin/ifconfig $KSDEVICE | /bin/awk '/inet/ && !/inet6/ {sub(/Mask:/, ""); print $4}'`
NETWORK=`LANG=C /bin/ipcalc $IP -n $NETMASK | /bin/cut -d\= -f2`
GATEWAY=`LANG=C echo $NETWORK | awk -F'.' '{print $1"."$2"."$3"."$4+1}'`
HWADDR=`LANG=C /sbin/ifconfig $KSDEVICE | /bin/awk '/HWaddr/ { print $5 }'`

cat << EOF > /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=$HOSTNAME
GATEWAY=$GATEWAY
EOF

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-$KSDEVICE
DEVICE=$KSDEVICE
BOOTPROTO=static
IPADDR=$IP
NETMASK=$NETMASK
ONBOOT=yes
HWADDR=$HWADDR
EOF

# chef-client

if [ ! -e /etc/chef ]; then
        mkdir /etc/chef
fi

cat > /etc/chef/client.rb << EOCLRB
log_level        :info
log_location     STDOUT
chef_server_url  "http://chefserver:4000"
validation_client_name "chef-validator"
# Using default node name (fqdn)
EOCLRB
chmod 600 /etc/chef/client.rb

cat > /etc/chef/validation.pem << EOVALPEM
-----BEGIN RSA PRIVATE KEY-----
MIIE...Wg==

-----END RSA PRIVATE KEY-----
EOVALPEM
chmod 600 /etc/chef/validation.pem

# Install Chef packages
yum -y install rubygem-chef
chkconfig chef-client on
yum -y update

Apache Active Directory authentication

If your Linux server is setup with AD authentiction you can add Apache AD auth as well.

AD LDAP authentication

Install OpenLDAP packages:

yum -y install openldap-clients openldap-devel

Comment everything from /etc/openldap/ldap.conf except the following line:

TLS_REQCERT never

Make sure that you can run a simple LDAP query against domaincontroller.yourdomain.com

ldapsearch -x -LLL -D 'Igor Grinkin' -H ldaps://domaincontroller.yourdomain.com -W -b "CN=Domain Admins,CN=Users,DC=corp,DC=yourdomain,DC=com" -L cn=*

Modify vhost.

<Directory "/usr/share/icinga/">
   Options None
   AllowOverride All
   Order allow,deny
   Allow from all
   AuthName "Please enter your Corp AD credentials"
   AuthType Basic
   AuthLDAPURL "ldap://domaincontroller.yourdomain.com:389/OU=YourDomain,DC=corp,DC=domaincontroller,DC=com?samAccountName?sub?(objectCategory=person)"
   AuthLDAPBindDN "cn=joinaccount,CN=Users,DC=corp,DC=yourdomain,DC=com"
   AuthLDAPBindPassword "joinaccount password"
   AuthBasicProvider ldap
   AuthzLDAPAuthoritative off
   Require valid-user
</Directory>

Chef OS versions fork

On CentOS-5.8 the service is portmap. In CentOS-6.4 it got obsolete and replaced by rpcbind.  Here is the case statement for Chef.

# Start portmap or rpcbind
service "port" do
  case node["platform_version"]
    when "5.8"
      service_name "portmap"
    when "6.4"
      service_name "rpcbind"
  end
  action [ :start, :enable ]
end

Convert MS Word document to HTML

Convert a long Word doc with simple formatting to html.

1.  Save the doc as Web page, filtered with html extenstion

Some special characters will not look right, like ' and "

2.  Substitute those with vi editor

:1,$s/\%x92/'/g
:1,$s/\%x93/"/g
To insert Copyright Symbol  , insert
&copy;

Linux sed example - find a replace string in /etc/fstab.  Changing "sdb1" to "VolGroup00/local"

for i in `cat lists/dblist`; do ssh -t $i "sudo sed -i 's/sdb1/VolGroup00\/local/' /etc/fstab"; done

Linux parted from command line.  Remove partition number 1 on sdb.

parted -s /dev/sdb -- rm 1

MySQL cheatsheet

Login to the database called "testdb" as "root" with "coolpass" password:
# mysql -u root -pcoolpass testdb

Let's give user "testuser" all privileges with password "coolpass"
mysql> GRANT ALL ON *.* to testuser@'localhost' IDENTIFIED BY 'coolpass';
mysql> FLUSH PRIVILEGES;

Show me all the recent users:
mysql> select * from appusers order by UserId desc limit 2;

Show me all messages that were sent today:
mysql> select * from messagecenter where DateTime like '2013-08-16%' order by DateTime desc;

My SQL logging.  Let's see what mysql database is doing.
mysql> show processlist;
The number of rows will give you an idea how busy is the db.

If you want to watch history, you can send output to a log file.
First let's find out where the log is:
mysql> SHOW VARIABLES LIKE "general_log%";
+------------------+------------------------+
| Variable_name    | Value                  |
+------------------+------------------------+
| general_log      | OFF                    |
| general_log_file | /var/lib/mysql/db2.log |
+------------------+------------------------+
2 rows in set (0.00 sec)

Let's enable logging:
mysql> SET GLOBAL general_log = 'ON';

Let's see what it's doing:
tail -f /var/lib/mysql/db2.log

Turn off loggin:
mysql> SET GLOBAL general_log = 'OFF';

Extending partitions on Linux VMware virtual machines

Extending partitions on Linux VMware virtual machines

 I had to increase the hard drive size on one of the VMs and read this article:

http://www.techrepublic.com/blog/smb-technologist/extending-partitions-on-linux-vmware-virtual-machines/

In the comments people say that you can't do that if you are trying to expand primary root partition.  Not true.  Here is how:

1. Shut down VM and increase the hard drive size
2. If that Option is disabled consolidate your snapshots (delete them except the last one) 
3. Download Linux rescue CD iso
4. Mount iso as CDROM and boot your VM from it.
5. Use gparted utility that comes with Linux rescue CD to resize partition to max size
6. Reboot VM