AWS Terraform

If you ever had to manage AWS Amazon environment you probably start by clicking through AWS console - Web GUI, then you quickly realize that it won't scale, because all your clicks are not repeatable and there is no code review of some sort.

Next logical step it to use aws cli, but then you'd need to create a bunch of scripts to manage it.  There are a couple of options for scalable and systematic approach - one is Cloud Formation from Amazon, another cool solution is Terraform.

Terraform allows you to manage pretty much any AWS resource from a single configuration file that could be sitting in your GIT repo.  This is truly "infrastructure as a code" solution.

Here is an example of how to create a simple ec2 instance with your parameters.

resource "aws_instance" "my-server" {
    ami                         = "ami-93742ea3"
    availability_zone           = "us-west-2a"
    ebs_optimized               = false
    instance_type               = "t2.micro"
    monitoring                  = false
    key_name                    = "your-ssh-access-key"
    subnet_id                   = "subnet-99999999"
    security_group_ids      = ["sg-ae33333", "sg-bb33333"]
    associate_public_ip_address = true
    private_ip                  = "10.10.10.50"
    source_dest_check           = true

    root_block_device {
        volume_type           = "gp2"
        volume_size           = 10
        iops                  = 30
        delete_on_termination = true
    }

    tags {
        "Name" = "my-server"
    }
}

As you can see above, some values like subnet and security group are hardcoded since we are dealing with existing infrastructure, and those resources were created by hand.

It's easy enough to refactor those into a separate file called "variables.tf", so that you could at least use them by readable human names.

variable "access_key" {}
variable "secret_key" {}
# Default AWS region is West
variable "aws_region" {
  default = "us-west-2"
}
# Security Groups
variable "sg" {
    default = {
        internal_ping = "sg-0444444"
        internal_ssh = "sg-444444444"
   }
}

Now you can start using those variables and you terraform template will be more readable
  security_group_id      = "${var.sg.internal_ping},${var.sg.internal_ssh},${var.sg.internal_http}"

Here is another example on how Terraform can manage a single DNS entry assuming that you have all your zones in that variables.tf file
resource "aws_route53_record" "my-server-A" {
    zone_id = "${var.dns_zone.company_com}"
    name    = "my-server.company.com"
    type    = "A"
    records = ["10.10.10.50"]
    ttl     = "600"
}

Active Directory authentication for Linux

Add the following lines to your kickstart config.

# Join AD domain
yum -y install pbis-open
/usr/bin/domainjoin-cli setname ${hostname}.corp.yourdomain.com
/usr/bin/domainjoin-cli join --ou OU=LinuxServers,OU=Internal,DC=corp,DC=yourdomain,DC=com corp.yourdomain.com joinaccount joinpassword
/opt/pbis/bin/config AssumeDefaultDomain true
/opt/pbis/bin/config LoginShellTemplate /bin/bash
/opt/pbis/bin/config HomeDirTemplate %H/%D/%U


Simple shell script to do the same:
joinad.sh

#!/bin/bash
if [ -z "$1" ]
  then
    echo "Usage: joinad.sh yourservername"
    exit 1
fi

echo "Joining $1"
ssh -t $1 "domainjoin-cli join --disable hostname --ou OU=LinuxServers,OU=Internal,DC=corp,DC=yourdomain,DC=com corp.yourdomain.com joinaccount joinpassword; /opt/pbis/bin/config AssumeDefaultDomain true; /opt/pbis/bin/config LoginShellTemplate /bin/bash; /opt/pbis/bin/config HomeDirTemplate %H/%U"

Terminal/Console

Screen

Below is some customization for screen utility.
cat .screenrc

# SSH agent link
setenv SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock

# Number of lines
defscrollback 10000

# for ctrl right and left arrows
bindkey ^[[1;5D prev
bindkey ^[[1;5C next

bindkey "^[[D" prev # ctrl-left
bindkey "^[[C" next # ctrl-right

# To remove splits
bind X remove

# Window list at the bottom. hostname, centered tabs and redmarked active windows:
#hardstatus alwayslastline
#hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %d/%m %{W}%c %{g}]'
#caption     always        "%{+b rk}%H%{gk} |%c %{yk}%d.%m.%Y | %72=Load: %l %{wk}"
hardstatus alwayslastline "%?%{yk}%-Lw%?%{wb}%n*%f %t%?(%u)%?%?%{yk}%+Lw%?"

To keep your ssh-agent running in screen session add rc file:
cat .ssh/rc
if test "$SSH_AUTH_SOCK" ; then
  ln -sf $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi

VMware CLI management

I'm trying to use command line interface as much as I can.  Below are some examples of VMware CLI management.

Get a list of all VMs:

vim-cmd vmsvc/getallvms

Take a look at template:

/vmfs/volumes/541abff4-d8f5aafc-5d95-002590e90bb0 # /vmfs/volumes/datastore1/vmwa
re-ovftool/ovftool prve-loadtest.ova

Deploy VM from Windows ovf tool:

C:\Program Files (x86)\VMware\VMware OVF Tool>ovftool --disableVerification --no
SSLVerify -dm=thick --datastore=datastore1 --name=loadtest1  --net:"VM Net
work"="VM Network" c:\Users\igrinkin\loadtest.ova vi://root:root_password@hypervisor1
Opening OVA source: c:\Users\igrinkin\loadtest.ova
Opening VI target: vi://root@10.107.130.29:443/
Deploying to VI: vi://root@10.107.130.29:443/
Transfer Completed
The manifest validates
Warning:
 - No manifest entry found for: 'loadtest1-disk1.vmdk'.
Completed successfully

Copy VM to another ESX server

ssh hypervisor1
cd /vmfs/volumes/datastore1/
scp -r /vmfs/volumes/datastore1/testvm/ hypervisor2:/vmfs/volumes/datastore1/

Open hypervisor2 in vSphere

Configuration - Storage - right-click on datastore1 - Browse to "testvm" folder - click on testvm.vmx - Add to inventory

Remove VM from hypervisor1 to cleanup

You can easily script it to emulate what Vcenter does.

Add a second hard drive to VM without reboot

Add second drive to live server.

Run

echo "- - -" > /sys/class/scsi_host/host2/scan

fdisk should show you the second drive.  Create partition and format.

mkfs.ext3 -m0 /dev/sdb1
tune2fs -c0 -i0 /dev/sdb1

Using mysql module in puppet manifest

I downloaded mysql module from Puppet Labs.

Here is an example of using it in graphite module.

class graphite {

    $graphite_packages = ["graphite-web","MySQL-python","python-carbon","python-whisper"]
    package { $graphite_packages: ensure => installed }

    $graphite_services = ["httpd","carbon-cache"]
    service { $graphite_services: ensure => running, enable => true }

    file { "/etc/graphite-web/local_settings.py":
           owner => "root",
           group => "root",
           mode  => 644,
           source => "puppet:///modules/graphite/local_settings.py",
           notify  => Service["httpd"],
    }

    file { "/etc/carbon/storage-schemas.conf":
           owner => "root",
           group => "root",
           mode  => 644,
           source => "puppet:///modules/graphite/storage-schemas.conf",
           notify  => Service["httpd"],
    }

    file { "/root/graph.sql":
           owner => "root",
           group => "root",
           mode  => 644,
           source => "puppet:///modules/graphite/graph.sql",
           notify  => Service["mysqld"],
    }

    include '::mysql::server'

    mysql::db { 'mydb':
      dbname         => 'graphite',
      user           => 'graphite',
      password       => 'graphitepassword',
      host           => 'localhost',
      grant          => ['ALL'],
      sql            => '/root/graph.sql',
      import_timeout => 900,
    }

}

How to: puppet - hiera - foreman enc

Here is an example of a puppet setup with hiera and Foreman as ENC (external node classifier)

Let's say I want to manage my /etc/resolv.conf using puppet.  Since I have multiple datacenters, I want to point linux clients to the closest DNS server.

I want my puppet templates to be generic, so that I don't have to touch it again.  All hard-coded data (like IP addresses) goes into hiera.

Puppet

I separate puppet classes by modules for convenience.  Here is how puppet manifest looks like:

cat puppet/modules/dns/manifests/init.pp

class dns ( $dns_search = hiera("dns::search"),
            $dns_servers = hiera("dns::servers")) {

    file { "/etc/resolv.conf":
           owner => "root",
           group => "root",
           mode  => 644,
           content => template("dns/resolv.conf.erb"),
    }
}
Basically, I'm saying that "dns_search" and "dns_servers" variables will come from hiera data.

Here is the template that puppet will apply:
cat puppet/modules/dns/templates/resolv.conf.erb

# This file is controlled by Puppet
search <%= dns_search %>
<% @dns_servers.each do |server| -%>
nameserver <%= server %>
<% end -%>

Hiera

Hiera configuration file goes by location:
cat /etc/puppet/hiera.yaml

---
:backends:
  - yaml
:yaml:
  :datadir: /etc/puppet/hieradata
:hierarchy:
  - "%{::clientcert}"
  - "nodes/%{::fqdn}"
  - "%{::environment}"
  - "location/%{::location}"
  - defaults

Location hiera file for Redwood City will look like:
cat puppet/hieradata/location/rwc.yaml

---
dns::search: rwc.mycompany.com mycompany.com
dns::servers:
 - 192.168.0.2
 - 192.168.0.3
 - 10.10.0.2

Foreman

You don't have to use Foreman but it gives you a nice GUI, dashboard and can easily be used as ENC to create puppet host groups and configuration groups.

Foreman installation was pretty straightforward from:
http://theforeman.org/manuals/1.1/quickstart_guide.html

foreman-installer --foreman-db-type=mysql

I used mysql database and my own certificate that was signed by my own CA.
cat /etc/puppet/foreman.yaml

---
:url: "https://foreman.mycompany.com"
:ssl_ca: "/etc/pki/tls/certs/mycompanyca.crt"
:ssl_cert: "/etc/pki/tls/certs/foreman.crt"
:ssl_key: "/etc/pki/tls/private/foreman.key"
:user: ""
:password: ""
:puppetdir: "/var/lib/puppet"
:puppetuser: "puppet"
:facts: true
:timeout: 10
:threads: null

GIT

Create new GIT repo on the server:
add user git
create athorized_keys for git
mkdir /data/git
ln -s /data/git /git
cd /git
mkdir ops.git
cd ops.git
git --bare init

Puppet notes

Built-in puppet variables in templates:

# This file is controlled by Puppet
# /etc/puppet/puppet.conf
# <%= scope.lookupvar('::osfamily') %> <%= scope.lookupvar('::operatingsystemmajrelease') %>

Example of template and file:

    file { "/etc/puppet/puppet.conf":
           owner => "root",
           group => "root",
           mode  => 644,
           content => template("puppet/puppet.conf.erb"),
           notify  => Service["puppet"],
    }

    file { "/etc/sysconfig/puppet":
           owner => "root",
           group => "root",
           mode  => 644,
           source => "puppet:///modules/puppet/puppet",
           notify  => Service["puppet"],
    }

Hieradata

cat hiera.yaml
---
:backends:
  - yaml
:yaml:
  :datadir: /etc/puppet/hieradata
:hierarchy:
  - "%{::clientcert}"
  - "nodes/%{::fqdn}"
  - "%{::environment}"
  - "location/%{::location}"
  - defaults


Example of location yaml

cat location/mylocation.yaml
---
dns::search: myl.location.com location.com
dns::servers:
 - 192.168.0.100
 - 192.168.0.101
ssh::group_key: AAAAB3NzaC1yc........

How to setup E-mail relay

Gmail rejects e-mail from my server.

Solution: E-mail relay through Mandrill (MailChimp service)

1.  Create account with Mandrill (it's free)
2.  Generate some API key on Mandrill website
3.  Setup your Postfix
 yum -y install postfix cyrus-sasl-plain cyrus-sasl-md5

4.  Modify /etc/postfix/main.cf
relayhost = smtp.mandrillapp.com
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
smtp_sasl_mechanism_filter = AUTH LOGIN
smtp_sasl_security_options =

5.  Put your credentials in /etc/postfix/sasl_passwd file
smtp.mandrillapp.com yourusername@yourdomain.com:yourapikeyhere

6.  Create sasl db readable by postfix
postmap sasl_passwd
chmod 600 sasl_passwd
chown postfix:postfix sasl_passwd.db


Your maillog should look similar to this:
Mar 15 23:45:05 tm1 postfix/master[29546]: daemon started -- version 2.6.6, configuration /etc/postfix
Mar 15 23:45:41 tm1 postfix/pickup[29548]: 34B98580496: uid=500 from=<user>
Mar 15 23:45:41 tm1 postfix/cleanup[29678]: 34B98580496: message-id=<20150316064541.34B98580496@mail.tagmap.me>
Mar 15 23:45:41 tm1 postfix/qmgr[29549]: 34B98580496: from=<user@mail.tagmap.me>, size=419, nrcpt=1 (queue active)
Mar 15 23:45:42 tm1 postfix/smtp[29680]: 34B98580496: to=<user@gmail.com>, relay=smtp.mandrillapp.com[54.70.134.182]:25, delay=1.3, delays=0.02/0.01/1.1/0.24, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 5A3F038028A)
Mar 15 23:45:42 tm1 postfix/qmgr[29549]: 34B98580496: removed

LVM

Logical Volume Manager

How to expand existing root partition using LVM

Add a second physical drive.  Scan the system, no reboot needed:
echo "- - -" &gt; /sys/class/scsi_host/host1/scan

Create Logic Group and Volume:

pvcreate datavg /dev/sdb1
vgcreate datavg /dev/sdb1
lvcreate -l 100%FREE -n lvdata datavg

# vgdisplay
  --- Volume group ---
  VG Name               datavg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               59.99 GiB
  PE Size               4.00 MiB
  Total PE              15358
  Alloc PE / Size       15358 / 59.99 GiB
  Free  PE / Size       0 / 0   
  VG UUID               zWlhGf-YDZa-27xv-aH0t-F6fW-Hs1e-TY3d8t

mkfs.ext3 -m 0 /dev/datavg/lvdata
tune2fs -c0 /dev/datavg/lvdata


Expand existing root LVM partition

pvdisplay
pvcreate /dev/sdb1
vgextend root_partition_name /dev/sdb1
lvextend -l +100%FREE /dev/root_partition_name/root

resize2fs /dev/root_partition_name/root

If you need to reduce it back:

vgreduce -a root_partition_name

HTTP POST into a form from curl

I want to post some data from file.txt into a form on the website. Curl can do it from command line:

curl -X POST -d @filename.txt -u usernamehere:passwordhere https://server.company.com/session/sync.jsp

The content of filename.txt is pairs of name=value separated by &. E.g.

id=1&options=doit&field3=Submit

Code pre-formatted text to Blog

To paste code or pre-formatted text into Blog use pre tags (mind those spaces)

< pre >
< / pre >

Puppet logs on CentOS

By default Puppet sends logs to /var/log/messages Modify /etc/sysconfig/puppet file to send them to /var/log/puppet/ E.g.

# The puppetmaster server
PUPPET_SERVER=puppetmaster

# If you wish to specify the port to connect to do so here
#PUPPET_PORT=8140

# Where to log to. Specify syslog to send log messages to the system log.
PUPPET_LOG=/var/log/puppet/puppet.log

# You may specify other parameters to the puppet client here
#PUPPET_EXTRA_OPTS=--waitforcert=500

Change password without root on a list of servers

Some companies are strict on passwords and force you to to change your password every 30 days, no repeating passwords, the passwords have to be long, contain special characters, etc.  Here is a script that you can add as a cronjob assuming you have your ssh keys setup.

You'd need a file "list.txt" with the list of IPs and "oldpass.txt" file with the current password.  You can just set a cronjob to do reset password on a regular basis (every 25 days for example).

#!/bin/bash
# Generate random new password 24 characters long
newpass=`mkpasswd -l 30 -d 3 -C 5 -s 3`
# Get old password from oldpass file
oldpass=`cat oldpass.txt`
echo Old: $oldpass
echo New: $newpass

# ssh to server and reset password
for server in `cat list.txt`
  do
    echo "Changing $server"
    ssh -t $server 'passwd <<EOF
'$oldpass'
'$newpass'
'$newpass'
EOF'
  done

Sort IP addresses

cat list.txt |sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n

Enable port on Arista switch

Similar to Cisco, enabling port 35 on switch1 for example.

switch1>en
switch1#show interfaces Ethernet 35
switch1#configure
switch1(config)#interface Ethernet 35
switch1(config-if-Et35)#no shutdown
switch1(config-if-Et35)#end

Save your changes:
switch1#copy running-config startup-config

Bash - cross-reference two files

I have two files: list_to_remove and current_list.  I want to remove all servers from list_to_remove from the current list.

for i in $(cat list_to_remove); do grep -v "$i.company.com" current_list > temp && mv temp current_list; done

How to replace a section of text file bash script

Goal:
I want to replace a section of hibernate.cfg.xml with new servers.  I need to remove servers between
                <property name="hibernate.memcached.servers">
server1 server2 server3
                </property>

and replace them with mem1 mem2 mem3

cat replace.sh

#!/bin/bash awk 'BEGIN {A = 1};//{A=0; print "\t\t\n\t\t\t\ mem1.atl.company.com:11212 mem2.atl.company:11212 mem3.atl.company.com:11212 mem4.atl.company.com:11212 mem5.atl.company.com:11212 mem6.atl.company:11212 mem148.atl.company.com:11212 mem149.atl.company.com:11212 mem150.atl.company.com:11212\ \n\t\t"};/.*/ { if ( A == 1) print $0};/<\/property>/{A=1}; ' $1

Run script against any xml file

./replace.sh hibernate.cfg.xml > hibernate.cfg.xml.new

Puppet manifest for multiple servers

Puppet supports regular expressions.

E.g.
Run puppet manifest on all the web servers:
node /^web.*$/ {
  class { 'server_web': }
}

Run on a few (db0-9)
node /^db[0-9]\..*$/ {
  class { 'server_db': }
}

bash script sum

Bash script exercise.
Add variable to the result of the sum.  E.g.
1 1 2 3 5 8 13 21, etc.


#!/bin/bash
x=1
y=1
ans=0

while [ $ans -lt 100 ]
#while true
do
  ans=$(( x + y ))
  echo $x + $y = $ans
  x=$y
  y=$ans
done