Storing aws credentials in S3 bucket

Make a script  aws_init.sh on your dedicated server with admin AWS privileges

#! /bin/bash
set -e
mkdir -p ~/.ssh
aws s3 cp s3://keys/chef-client ~/.ssh/chef-client
mkdir -p ~/.chef/
aws s3 cp s3://keys/validation-prod.pem    ~/.chef
aws s3 cp s3://keys/validation-stag.pem    ~/.chef


Initialize the key by running:

eval "`aws s3 cp s3://keys/aws-access-keys.txt -`"
~/src/bin/aws_init.sh

Internal private SSL Certificate Authority

Internal SSL Certificate Authority

Intermediate CA

root@devops:~/ca# openssl genrsa -aes256 -out private/intermediate.key.pem 4096 root@devops:~/ca# openssl req -new -sha256 -key private/intermediate.key -out requests/intermediate.csr -config config.txt -subj "/C=US/ST=California/L=San\ Francisco/O=Company/OU=DevOps/CN=Company Intermediate CA/emailAddress=devops@company.com"

IT signed this request with Company Root CA

Creating server cert based on Intermediate CA

root@devops:~/ca# openssl req -new -nodes -keyout newcerts/wild.company.com.key -out requests/wild.company.com.csr -config config.txt -subj "/C=US/ST=California/L=San\ Francisco/O=Company/OU=DevOps/CN=*.company.com/emailAddress=devops@company.com"   root@devops:~/ca# openssl ca -batch -notext -config config.txt -in requests/wild.company.com.csr -cert certs/intermediate_ca.crt -keyfile private/intermediate_ca.key -out newcerts/wild.company.com.crt   root@devops:~/ca# cat certs/intermediate_ca.crt >> newcerts/wild.company.com.crt

See your cert

openssl x509 -in wild.company.com.crt -text

Revocation list

root@devops:~/ca# echo "01" > crlnumber root@devops:~/ca# openssl ca -config config.txt -gencrl -out crl/certificate.crl

CA location

In order for Ubuntu system to trust your CA certificate add it to

/usr/local/share/ca-certificates/companyca.crt update-ca-certificates

Upload new cert to AWS

aws iam upload-server-certificate --server-certificate-name $hostname --certificate-body file://$hostname.crt --private-key file://$hostname.key

Two factor authentication with ssh

Two factor definition:
1.  something you have  (ssh private key)
2.  something you know  (your account password)

On Ubuntu
/etc/ssh/sshd_config

# Require public key and password by default
PasswordAuthentication yes
AuthenticationMethods publickey,password

# Allow deploy and git groups to log in without password
Match Group deploy,git
   PasswordAuthentication no
   AuthenticationMethods publickey

Restart sshd

When you ssh to the server you'll be prompted for your passphrase and then again for your password.

Chef recipe how to start and use custom service or script

Here is an example of a custom service for ubuntu:

package 'push-jobs-client' do
  action :install
end

template '/etc/chef/push-jobs-client.rb' do
  source 'push-jobs-client.rb.erb'
  owner  'root'
  group  'root'
  mode   '644'
end

template '/etc/init/chef-push-jobs.conf' do
  source 'push-jobs-client-run.erb'
  owner  'root'
  group  'root'
  mode   '644'
end

service 'chef-push-jobs' do
  supports [:stop, :start, :restart, :status]
  action [ :enable, :start]
end


cat templates/default/push-jobs-client.rb.erb
# Generated by Chef for <%= node[:fqdn] %>
# Local modifications will be overwritten!
# Added this wrapper for community cookbook to allow_unecrypted; encryption is done over SSL

LC_ALL='en_US.UTF-8'

# Chef server connect options
chef_server_url   '<%= Chef::Config[:chef_server_url] %>'
node_name         '<%= Chef::Config[:node_name] %>'
client_key        '/etc/chef/client.pem'
trusted_certs_dir '/etc/chef/trusted_certs'
verify_api_cert   false
allow_unencrypted true

# The whitelist comes from default attributes
whitelist<%= node['base']['push-jobs']['whitelist'] %>

# We're under runit, so don't output timestamp
Mixlib::Log::Formatter.show_time = false


cat templates/default/push-jobs-client-run.erb
start on runlevel [2345]
stop on runlevel [!2345]
script

exec /opt/push-jobs-client/bin/pushy-client -l info  -c /etc/chef/push-jobs-client.rb
end script

AWS Terraform

If you ever had to manage AWS Amazon environment you probably start by clicking through AWS console - Web GUI, then you quickly realize that it won't scale, because all your clicks are not repeatable and there is no code review of some sort.

Next logical step it to use aws cli, but then you'd need to create a bunch of scripts to manage it.  There are a couple of options for scalable and systematic approach - one is Cloud Formation from Amazon, another cool solution is Terraform.

Terraform allows you to manage pretty much any AWS resource from a single configuration file that could be sitting in your GIT repo.  This is truly "infrastructure as a code" solution.

Here is an example of how to create a simple ec2 instance with your parameters.

resource "aws_instance" "my-server" {
    ami                         = "ami-93742ea3"
    availability_zone           = "us-west-2a"
    ebs_optimized               = false
    instance_type               = "t2.micro"
    monitoring                  = false
    key_name                    = "your-ssh-access-key"
    subnet_id                   = "subnet-99999999"
    security_group_ids      = ["sg-ae33333", "sg-bb33333"]
    associate_public_ip_address = true
    private_ip                  = "10.10.10.50"
    source_dest_check           = true

    root_block_device {
        volume_type           = "gp2"
        volume_size           = 10
        iops                  = 30
        delete_on_termination = true
    }

    tags {
        "Name" = "my-server"
    }
}

As you can see above, some values like subnet and security group are hardcoded since we are dealing with existing infrastructure, and those resources were created by hand.

It's easy enough to refactor those into a separate file called "variables.tf", so that you could at least use them by readable human names.

variable "access_key" {}
variable "secret_key" {}
# Default AWS region is West
variable "aws_region" {
  default = "us-west-2"
}
# Security Groups
variable "sg" {
    default = {
        internal_ping = "sg-0444444"
        internal_ssh = "sg-444444444"
   }
}

Now you can start using those variables and you terraform template will be more readable
  security_group_id      = "${var.sg.internal_ping},${var.sg.internal_ssh},${var.sg.internal_http}"

Here is another example on how Terraform can manage a single DNS entry assuming that you have all your zones in that variables.tf file
resource "aws_route53_record" "my-server-A" {
    zone_id = "${var.dns_zone.company_com}"
    name    = "my-server.company.com"
    type    = "A"
    records = ["10.10.10.50"]
    ttl     = "600"
}

Active Directory authentication for Linux

Add the following lines to your kickstart config.

# Join AD domain
yum -y install pbis-open
/usr/bin/domainjoin-cli setname ${hostname}.corp.yourdomain.com
/usr/bin/domainjoin-cli join --ou OU=LinuxServers,OU=Internal,DC=corp,DC=yourdomain,DC=com corp.yourdomain.com joinaccount joinpassword
/opt/pbis/bin/config AssumeDefaultDomain true
/opt/pbis/bin/config LoginShellTemplate /bin/bash
/opt/pbis/bin/config HomeDirTemplate %H/%D/%U


Simple shell script to do the same:
joinad.sh

#!/bin/bash
if [ -z "$1" ]
  then
    echo "Usage: joinad.sh yourservername"
    exit 1
fi

echo "Joining $1"
ssh -t $1 "domainjoin-cli join --disable hostname --ou OU=LinuxServers,OU=Internal,DC=corp,DC=yourdomain,DC=com corp.yourdomain.com joinaccount joinpassword; /opt/pbis/bin/config AssumeDefaultDomain true; /opt/pbis/bin/config LoginShellTemplate /bin/bash; /opt/pbis/bin/config HomeDirTemplate %H/%U"

Terminal/Console

Screen

Below is some customization for screen utility.
cat .screenrc

# SSH agent link
setenv SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock

# Number of lines
defscrollback 10000

# for ctrl right and left arrows
bindkey ^[[1;5D prev
bindkey ^[[1;5C next

bindkey "^[[D" prev # ctrl-left
bindkey "^[[C" next # ctrl-right

# To remove splits
bind X remove

# Window list at the bottom. hostname, centered tabs and redmarked active windows:
#hardstatus alwayslastline
#hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %d/%m %{W}%c %{g}]'
#caption     always        "%{+b rk}%H%{gk} |%c %{yk}%d.%m.%Y | %72=Load: %l %{wk}"
hardstatus alwayslastline "%?%{yk}%-Lw%?%{wb}%n*%f %t%?(%u)%?%?%{yk}%+Lw%?"

To keep your ssh-agent running in screen session add rc file:
cat .ssh/rc
if test "$SSH_AUTH_SOCK" ; then
  ln -sf $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi

VMware CLI management

I'm trying to use command line interface as much as I can.  Below are some examples of VMware CLI management.

Get a list of all VMs:

vim-cmd vmsvc/getallvms

Take a look at template:

/vmfs/volumes/541abff4-d8f5aafc-5d95-002590e90bb0 # /vmfs/volumes/datastore1/vmwa
re-ovftool/ovftool prve-loadtest.ova

Deploy VM from Windows ovf tool:

C:\Program Files (x86)\VMware\VMware OVF Tool>ovftool --disableVerification --no
SSLVerify -dm=thick --datastore=datastore1 --name=loadtest1  --net:"VM Net
work"="VM Network" c:\Users\igrinkin\loadtest.ova vi://root:root_password@hypervisor1
Opening OVA source: c:\Users\igrinkin\loadtest.ova
Opening VI target: vi://root@10.107.130.29:443/
Deploying to VI: vi://root@10.107.130.29:443/
Transfer Completed
The manifest validates
Warning:
 - No manifest entry found for: 'loadtest1-disk1.vmdk'.
Completed successfully

Copy VM to another ESX server

ssh hypervisor1
cd /vmfs/volumes/datastore1/
scp -r /vmfs/volumes/datastore1/testvm/ hypervisor2:/vmfs/volumes/datastore1/

Open hypervisor2 in vSphere

Configuration - Storage - right-click on datastore1 - Browse to "testvm" folder - click on testvm.vmx - Add to inventory

Remove VM from hypervisor1 to cleanup

You can easily script it to emulate what Vcenter does.

Add a second hard drive to VM without reboot

Add second drive to live server.

Run

echo "- - -" > /sys/class/scsi_host/host2/scan

fdisk should show you the second drive.  Create partition and format.

mkfs.ext3 -m0 /dev/sdb1
tune2fs -c0 -i0 /dev/sdb1

Using mysql module in puppet manifest

I downloaded mysql module from Puppet Labs.

Here is an example of using it in graphite module.

class graphite {

    $graphite_packages = ["graphite-web","MySQL-python","python-carbon","python-whisper"]
    package { $graphite_packages: ensure => installed }

    $graphite_services = ["httpd","carbon-cache"]
    service { $graphite_services: ensure => running, enable => true }

    file { "/etc/graphite-web/local_settings.py":
           owner => "root",
           group => "root",
           mode  => 644,
           source => "puppet:///modules/graphite/local_settings.py",
           notify  => Service["httpd"],
    }

    file { "/etc/carbon/storage-schemas.conf":
           owner => "root",
           group => "root",
           mode  => 644,
           source => "puppet:///modules/graphite/storage-schemas.conf",
           notify  => Service["httpd"],
    }

    file { "/root/graph.sql":
           owner => "root",
           group => "root",
           mode  => 644,
           source => "puppet:///modules/graphite/graph.sql",
           notify  => Service["mysqld"],
    }

    include '::mysql::server'

    mysql::db { 'mydb':
      dbname         => 'graphite',
      user           => 'graphite',
      password       => 'graphitepassword',
      host           => 'localhost',
      grant          => ['ALL'],
      sql            => '/root/graph.sql',
      import_timeout => 900,
    }

}

How to: puppet - hiera - foreman enc

Here is an example of a puppet setup with hiera and Foreman as ENC (external node classifier)

Let's say I want to manage my /etc/resolv.conf using puppet.  Since I have multiple datacenters, I want to point linux clients to the closest DNS server.

I want my puppet templates to be generic, so that I don't have to touch it again.  All hard-coded data (like IP addresses) goes into hiera.

Puppet

I separate puppet classes by modules for convenience.  Here is how puppet manifest looks like:

cat puppet/modules/dns/manifests/init.pp

class dns ( $dns_search = hiera("dns::search"),
            $dns_servers = hiera("dns::servers")) {

    file { "/etc/resolv.conf":
           owner => "root",
           group => "root",
           mode  => 644,
           content => template("dns/resolv.conf.erb"),
    }
}
Basically, I'm saying that "dns_search" and "dns_servers" variables will come from hiera data.

Here is the template that puppet will apply:
cat puppet/modules/dns/templates/resolv.conf.erb

# This file is controlled by Puppet
search <%= dns_search %>
<% @dns_servers.each do |server| -%>
nameserver <%= server %>
<% end -%>

Hiera

Hiera configuration file goes by location:
cat /etc/puppet/hiera.yaml

---
:backends:
  - yaml
:yaml:
  :datadir: /etc/puppet/hieradata
:hierarchy:
  - "%{::clientcert}"
  - "nodes/%{::fqdn}"
  - "%{::environment}"
  - "location/%{::location}"
  - defaults

Location hiera file for Redwood City will look like:
cat puppet/hieradata/location/rwc.yaml

---
dns::search: rwc.mycompany.com mycompany.com
dns::servers:
 - 192.168.0.2
 - 192.168.0.3
 - 10.10.0.2

Foreman

You don't have to use Foreman but it gives you a nice GUI, dashboard and can easily be used as ENC to create puppet host groups and configuration groups.

Foreman installation was pretty straightforward from:
http://theforeman.org/manuals/1.1/quickstart_guide.html

foreman-installer --foreman-db-type=mysql

I used mysql database and my own certificate that was signed by my own CA.
cat /etc/puppet/foreman.yaml

---
:url: "https://foreman.mycompany.com"
:ssl_ca: "/etc/pki/tls/certs/mycompanyca.crt"
:ssl_cert: "/etc/pki/tls/certs/foreman.crt"
:ssl_key: "/etc/pki/tls/private/foreman.key"
:user: ""
:password: ""
:puppetdir: "/var/lib/puppet"
:puppetuser: "puppet"
:facts: true
:timeout: 10
:threads: null

GIT

Create new GIT repo on the server:
add user git
create athorized_keys for git
mkdir /data/git
ln -s /data/git /git
cd /git
mkdir ops.git
cd ops.git
git --bare init

Puppet notes

Built-in puppet variables in templates:

# This file is controlled by Puppet
# /etc/puppet/puppet.conf
# <%= scope.lookupvar('::osfamily') %> <%= scope.lookupvar('::operatingsystemmajrelease') %>

Example of template and file:

    file { "/etc/puppet/puppet.conf":
           owner => "root",
           group => "root",
           mode  => 644,
           content => template("puppet/puppet.conf.erb"),
           notify  => Service["puppet"],
    }

    file { "/etc/sysconfig/puppet":
           owner => "root",
           group => "root",
           mode  => 644,
           source => "puppet:///modules/puppet/puppet",
           notify  => Service["puppet"],
    }

Hieradata

cat hiera.yaml
---
:backends:
  - yaml
:yaml:
  :datadir: /etc/puppet/hieradata
:hierarchy:
  - "%{::clientcert}"
  - "nodes/%{::fqdn}"
  - "%{::environment}"
  - "location/%{::location}"
  - defaults


Example of location yaml

cat location/mylocation.yaml
---
dns::search: myl.location.com location.com
dns::servers:
 - 192.168.0.100
 - 192.168.0.101
ssh::group_key: AAAAB3NzaC1yc........

How to setup E-mail relay

Gmail rejects e-mail from my server.

Solution: E-mail relay through Mandrill (MailChimp service)

1.  Create account with Mandrill (it's free)
2.  Generate some API key on Mandrill website
3.  Setup your Postfix
 yum -y install postfix cyrus-sasl-plain cyrus-sasl-md5

4.  Modify /etc/postfix/main.cf
relayhost = smtp.mandrillapp.com
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
smtp_sasl_mechanism_filter = AUTH LOGIN
smtp_sasl_security_options =

5.  Put your credentials in /etc/postfix/sasl_passwd file
smtp.mandrillapp.com yourusername@yourdomain.com:yourapikeyhere

6.  Create sasl db readable by postfix
postmap sasl_passwd
chmod 600 sasl_passwd
chown postfix:postfix sasl_passwd.db


Your maillog should look similar to this:
Mar 15 23:45:05 tm1 postfix/master[29546]: daemon started -- version 2.6.6, configuration /etc/postfix
Mar 15 23:45:41 tm1 postfix/pickup[29548]: 34B98580496: uid=500 from=<user>
Mar 15 23:45:41 tm1 postfix/cleanup[29678]: 34B98580496: message-id=<20150316064541.34B98580496@mail.tagmap.me>
Mar 15 23:45:41 tm1 postfix/qmgr[29549]: 34B98580496: from=<user@mail.tagmap.me>, size=419, nrcpt=1 (queue active)
Mar 15 23:45:42 tm1 postfix/smtp[29680]: 34B98580496: to=<user@gmail.com>, relay=smtp.mandrillapp.com[54.70.134.182]:25, delay=1.3, delays=0.02/0.01/1.1/0.24, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 5A3F038028A)
Mar 15 23:45:42 tm1 postfix/qmgr[29549]: 34B98580496: removed

LVM

Logical Volume Manager

How to expand existing root partition using LVM

Add a second physical drive.  Scan the system, no reboot needed:
echo "- - -" &gt; /sys/class/scsi_host/host1/scan

Create Logic Group and Volume:

pvcreate datavg /dev/sdb1
vgcreate datavg /dev/sdb1
lvcreate -l 100%FREE -n lvdata datavg

# vgdisplay
  --- Volume group ---
  VG Name               datavg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               59.99 GiB
  PE Size               4.00 MiB
  Total PE              15358
  Alloc PE / Size       15358 / 59.99 GiB
  Free  PE / Size       0 / 0   
  VG UUID               zWlhGf-YDZa-27xv-aH0t-F6fW-Hs1e-TY3d8t

mkfs.ext3 -m 0 /dev/datavg/lvdata
tune2fs -c0 /dev/datavg/lvdata


Expand existing root LVM partition

pvdisplay
pvcreate /dev/sdb1
vgextend root_partition_name /dev/sdb1
lvextend -l +100%FREE /dev/root_partition_name/root

resize2fs /dev/root_partition_name/root

If you need to reduce it back:

vgreduce -a root_partition_name

HTTP POST into a form from curl

I want to post some data from file.txt into a form on the website. Curl can do it from command line:

curl -X POST -d @filename.txt -u usernamehere:passwordhere https://server.company.com/session/sync.jsp

The content of filename.txt is pairs of name=value separated by &. E.g.

id=1&options=doit&field3=Submit

Code pre-formatted text to Blog

To paste code or pre-formatted text into Blog use pre tags (mind those spaces)

< pre >
< / pre >

Puppet logs on CentOS

By default Puppet sends logs to /var/log/messages Modify /etc/sysconfig/puppet file to send them to /var/log/puppet/ E.g.

# The puppetmaster server
PUPPET_SERVER=puppetmaster

# If you wish to specify the port to connect to do so here
#PUPPET_PORT=8140

# Where to log to. Specify syslog to send log messages to the system log.
PUPPET_LOG=/var/log/puppet/puppet.log

# You may specify other parameters to the puppet client here
#PUPPET_EXTRA_OPTS=--waitforcert=500

Change password without root on a list of servers

Some companies are strict on passwords and force you to to change your password every 30 days, no repeating passwords, the passwords have to be long, contain special characters, etc.  Here is a script that you can add as a cronjob assuming you have your ssh keys setup.

You'd need a file "list.txt" with the list of IPs and "oldpass.txt" file with the current password.  You can just set a cronjob to do reset password on a regular basis (every 25 days for example).

#!/bin/bash
# Generate random new password 24 characters long
newpass=`mkpasswd -l 30 -d 3 -C 5 -s 3`
# Get old password from oldpass file
oldpass=`cat oldpass.txt`
echo Old: $oldpass
echo New: $newpass

# ssh to server and reset password
for server in `cat list.txt`
  do
    echo "Changing $server"
    ssh -t $server 'passwd <<EOF
'$oldpass'
'$newpass'
'$newpass'
EOF'
  done

Sort IP addresses

cat list.txt |sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n

Enable port on Arista switch

Similar to Cisco, enabling port 35 on switch1 for example.

switch1>en
switch1#show interfaces Ethernet 35
switch1#configure
switch1(config)#interface Ethernet 35
switch1(config-if-Et35)#no shutdown
switch1(config-if-Et35)#end

Save your changes:
switch1#copy running-config startup-config