GIT

.gitconfig

[alias]
  co = checkout
  ci = commit
  st = status
  br = branch
  hist = log --pretty=format:\"%h %ad | %s%d [%an]\" --graph --date=short
  type = cat-file -t
  dump = cat-file -p
[user]
        name = Igor Grinkin
        email =

Remove file extensions linux shell

for i in `ls`; do mv $i `echo "$i" | cut -f 1 -d '.' ` ; done

Docker

Installation

https://docs.docker.com/engine/getstarted/step_one/

Examples

Run nginx container

docker run -d -p 80:80 --name webserver nginx

You should be able to access it by going to http://localhost
List all docker images

docker images

List running containers

docker ps

Copy file to a docker container

docker cp html.tar 213389b682d6:/tmp/

Connect to a running container

docker exec -it 213389b682d6 /bin/bash

Saving new container after the changes

docker commit -m "Adding ops runbook" -a "Igor" 213389b682d6 runbook

Run your new container

docker run -d -p 80:80 -it runbook /bin/bash

Login to the container shell

docker exec -it 7e7d9c35a216 bash

Exit from the container shell
Ctrl + p , Ctrl + q
Restart nginx inside your container

docker exec -it 213389b682d6 /etc/init.d/nginx restart

After docker service restart - make sure ecs-agent is running

root@ecs00-us-west-2b:~# docker start ecs-agent

Logging

It seems like docker puts logs in two places:

1.  Actual container logs

   /var/lib/docker/containers

2. docker system log

   /var/log/docker.log

The following options should take care of log rotation:
/etc/default/docker

DOCKER_OPTS="--pidfile=/var/run/docker.pid --log-level=warning --log-opt max-size=1g --storage-driver=zfs --storage-opt=zfs.fsname=zd0/containers"

log-level makes docker.log file less verbose.  log-opt makes container logs no bigger than 1GB

Terraform notes

This section serves as a guide line for some usage of terraform to make life easier. This is built from hard experience using terraform in a multi-member team and across multiple environments in production.

Modules

Modules are the reusable construct in terraform. In most cases, we want to make a module out of ANY provisioning process that we expect to reuse.
In addition to reuse, modules provide a repeatable structure and versioning mechanism.
For instance, let’s look at our VPC module. It does the following:

• Creates a VPC
• Creates two subnets in that VPC in each AZ - one public/one private
• Creates a private Route53 zone for that VPC
• Creates basic security groups for that VPC
• Creates dhcp options for nodes provisioned in that VPC
• Provisions NAT nodes in each AZ for outbound traffic from the private subnets

This is a codified and repeatable mechanism for our base AWS environment and best practices for that environment. By using this module, you get all of the basics required to create an isolated environment in AWS for free.
Additionally, as this is a git repo for this module, you can get VERSIONED environments. Consider the following module declaration in terraform code:

module "vpc" {
source = "git::ssh://git@opsgit.i.stormpath.com/tf-modules/vpc?ref=894abd9"

By specifically putting the ref in there, you can be sure that FUTURE changes to the module will not unintentionally propogate down to existing environments. You can easily test changes by spinning up the old version, bumping the rev, calling terraform get --update=true and then terraform plan.
While terraform TRIES to do the right thing here, being strict in your versions lets you know for sure that even if you unintentially issued --update=true, you would not accidentally tear down a stack.

Note

Always use versioned module sources in terraform. Always. This is required for determinism and repeatability.

Use remote state

Terraform provides another mechanism for a form of reusability - remote state.
Looking back out our pushbutton setup, we leverage remote state to help isolate independent infrastucture components. There is no reason creating a new instance needs to modify a VPC. There is no reason, creating an RDS instance needs to modify dhcp options.
You _can_ use non-remote state files but those are not currently allowed in terraform for reuse. We use artifactory as our universal remote state mechanism because it provides built in versioning for us.
In our pushbutton runs, EACH subdirectory pushes its state up to artifactory at the end of the run. That makes it usable for downstream modules. The following is an example of using the VPC remote state for an environment downstream:

resource "terraform_remote_state" "vpc" {
        backend = "artifactory"
        config {
                url = "https://artifactory.i.stormpath.com/artifactory"
                repo = "terraform-state"
                subpath = "customers/${var.orgname}/vpc"
        }
}

This allows us to reference outputs from that remote state:

module "frontend_subnet_c" {
        source = "git::ssh://git@opsgit.i.stormpath.com/tf-modules/frontend_node?ref=60e6ecd"
        orgname = "${var.orgname}"
        gateway_host = "${terraform_remote_state.vpc.output.nat_c_public_ip}"
        .......
}

Note

In terraform 0.7 (unreleased), the syntax for using values from remote state is changing slightly. Instead of terraform_remote_state.vpc.output.some_output, the extra output is being dropped:terraform_remote_state.vpc.some_output

Additionally, 0.7 is adding a new construct called data sources which are immutable, read-only sources of data than can be used in terraform runs - i.e. a json file with a list of current amis or versions of software.

Passing variables and outputs

One area that is painful in terraform is passing variables from modules and other things. This leads to a bit of code duplication however the tradeoffs for safety are worth it.
Use consistent variable input names across modules to help minimize confusion. A good example is our orgname variable. This is used in EVERY module. This provides a programatic way to build tags and names for AWS resources as well as ensures a factor of uniqueness during provisioning. When you have multiple “stacks” in a single AWS account, it’s very easy to see _WHICH_ environment a resource goes with as all names are prefixed with the orgname (i.e. nat-a.orgname.local)
Also, modules to not natively export all outputs from a given resource. For this reason and for future unknowns, you should export ALL outputs from a resource inside a module. Consider the following resource and output in a module:

resource "aws_vpc" "default" {
        cidr_block = "${var.vpc_cidr}"
        enable_dns_hostnames = true
        enable_dns_support = true
        tags { Name = "${var.orgname}-secure"}
}

output "vpc_id" { value = "${aws_vpc.default.id}" }

In this module, we’re ONLY exporting the vpc id. However, what we should be doing is exporting everything that terraform returns as an output in case we need that information in future reuses of the module (originally this resource did not support the additional outputs. They were added to terraform later. This code only exports older outputs):

output "vpc_cidr_block" { value = "${aws_vpc.default.cidr_block}" }
output "vpc_main_route_table_id" { value = "${aws_vpc.default.main_route_table_id}" }
output "vpc_default_network_acl_id" { value = "${aws_vpc.default.default_network_acl_id}" }
output "vpc_default_security_group_id" { value = "${aws_vpc.default.default_security_group_id}" }

You can find the exported attributes of a given resource at the bottom of any resource’s page on the terraform website

Note

Doing a terraform apply will update any remote state with new outputs from modules used assuming a new version of the module was pulled in. Let’s say we needed to get those new attributes for a new terraform use elsewhere. Following our best practices we would:

• Make changes to the git repo of the module and note the new hash
• Update the rev in our downstream use of the module
• call terraform get --update=true to pull in the new version of the module
• run terraform plan to ensure that we aren’t actually CHANGING anything (or maybe your module allows that - it’s situational)
• run terraform apply which will grab the new exported outputs from the module and shove them into the current terraform run’s state.

Now we have a remote state file with additional information that we require when using remote state.

Be careful with count

Using the count trick is really handy in terraform when you need multiple copies of something. Be aware, however, that counts “break” the dependency graph. For instance let’s assume you have an aws_ec2_instance resource and you use a count to provision 3 of them. Now let’s assume that you ALSO use that count inside a dependent resource (like a broker id in kafka) and that is a dependency on that instance resource.
When you increase the count, the planner recomputes EVERYTHING using that count variable.
If you must use a count, you should ensure that the count is not used in ANY other resources unless you can safely update those resources as well - this includes passing the same _count_ to a template that you use in the instance resource.

Scope creep

Terraform brings a much needed true infrastructure-as-code approach to things. It is important, however, to not try and do too much in terraform. This is the reason we wrap terraform in Rundeck. Rundeck allows us to wrap terraform with steps that don’t FIT in terraform to create a final deliverable.
Terraform is a really poor fit for application deployment and volatile lifecycle components.
My general rule is if something feels hard in terraform it probably is.

Recreating resource

taint will destroy and re-create resource

terraform taint aws_instance.nat

Terraforming

Terraforming is a discovery utility to build terraform tf and state files from a running AWS infrastructure.
https://github.com/dtan4/terraforming

Installation

gem install terraforming
gem install io-console

Usage

terraforming elb --profile fitch > elb.tf
terraforming elb --profile fitch --tfstate > terraform.tfstate

"terraform plan" should show you no changes after that.

Merge

Example of merging terraform state with existing infrastructure
Disable remote state

terraform remote config -disable

Pull routing data

terraforming rt > route.tf

Update current state file

terraforming rt --tfstate --merge=terraform.tfstate --overwrite

Upload state to remote location

terraform remote config -backend=artifactory -backend-config="repo=terraform-state" -backend-config="subpath=customers/enterpriseeu/vpn"

Terraform encryption

How to encrypt db password

1. Create KMS master key from AWS GUI console.
http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html

(be careful about the Region!  Make sure the key is created in the right region)

2.  Create a text file with your password in it
echo "mypassword" > /home/igorg/db.txt

3.  Encrypt your plain text password with aws key

aws kms encrypt --key-id 3fb2...c7f3 --plaintext fileb:///home/igorg/db.txt --output text --query CiphertextBlob

the output would be an encrypted string.

4.  Put your encrypted password in terraform definition

data "aws_kms_secret" "db" {
  secret {
    name    = "master_password"
    payload = "AQICAHg...C0rTg="
  }
}

resource "aws_db_instance" "krdb1" {
  allocated_storage    = 5
  storage_type         = "gp2"
  engine               = "mysql"
  engine_version       = "5.6.35"
  instance_class       = "db.t2.micro"
  name                 = "krdb1"
  username             = "admin"
  password             = "${data.aws_kms_secret.db.master_password}"
  db_subnet_group_name = "krdb_group"
  vpc_security_group_ids = ["${aws_security_group.db.id}"]
}

Storing aws credentials in S3 bucket

Make a script  aws_init.sh on your dedicated server with admin AWS privileges

#! /bin/bash
set -e
mkdir -p ~/.ssh
aws s3 cp s3://keys/chef-client ~/.ssh/chef-client
mkdir -p ~/.chef/
aws s3 cp s3://keys/validation-prod.pem    ~/.chef
aws s3 cp s3://keys/validation-stag.pem    ~/.chef


Initialize the key by running:

eval "`aws s3 cp s3://keys/aws-access-keys.txt -`"
~/src/bin/aws_init.sh

Internal private SSL Certificate Authority

Internal SSL Certificate Authority

Intermediate CA

root@devops:~/ca# openssl genrsa -aes256 -out private/intermediate.key.pem 4096 root@devops:~/ca# openssl req -new -sha256 -key private/intermediate.key -out requests/intermediate.csr -config config.txt -subj "/C=US/ST=California/L=San\ Francisco/O=Company/OU=DevOps/CN=Company Intermediate CA/emailAddress=devops@company.com"

IT signed this request with Company Root CA

Creating server cert based on Intermediate CA

root@devops:~/ca# openssl req -new -nodes -keyout newcerts/wild.company.com.key -out requests/wild.company.com.csr -config config.txt -subj "/C=US/ST=California/L=San\ Francisco/O=Company/OU=DevOps/CN=*.company.com/emailAddress=devops@company.com"   root@devops:~/ca# openssl ca -batch -notext -config config.txt -in requests/wild.company.com.csr -cert certs/intermediate_ca.crt -keyfile private/intermediate_ca.key -out newcerts/wild.company.com.crt   root@devops:~/ca# cat certs/intermediate_ca.crt >> newcerts/wild.company.com.crt

See your cert

openssl x509 -in wild.company.com.crt -text

Revocation list

root@devops:~/ca# echo "01" > crlnumber root@devops:~/ca# openssl ca -config config.txt -gencrl -out crl/certificate.crl

CA location

In order for Ubuntu system to trust your CA certificate add it to

/usr/local/share/ca-certificates/companyca.crt update-ca-certificates

Upload new cert to AWS

aws iam upload-server-certificate --server-certificate-name $hostname --certificate-body file://$hostname.crt --private-key file://$hostname.key

Two factor authentication with ssh

Two factor definition:
1.  something you have  (ssh private key)
2.  something you know  (your account password)

On Ubuntu
/etc/ssh/sshd_config

# Require public key and password by default
PasswordAuthentication yes
AuthenticationMethods publickey,password

# Allow deploy and git groups to log in without password
Match Group deploy,git
   PasswordAuthentication no
   AuthenticationMethods publickey

Restart sshd

When you ssh to the server you'll be prompted for your passphrase and then again for your password.

Chef recipe how to start and use custom service or script

Here is an example of a custom service for ubuntu:

package 'push-jobs-client' do
  action :install
end

template '/etc/chef/push-jobs-client.rb' do
  source 'push-jobs-client.rb.erb'
  owner  'root'
  group  'root'
  mode   '644'
end

template '/etc/init/chef-push-jobs.conf' do
  source 'push-jobs-client-run.erb'
  owner  'root'
  group  'root'
  mode   '644'
end

service 'chef-push-jobs' do
  supports [:stop, :start, :restart, :status]
  action [ :enable, :start]
end


cat templates/default/push-jobs-client.rb.erb
# Generated by Chef for <%= node[:fqdn] %>
# Local modifications will be overwritten!
# Added this wrapper for community cookbook to allow_unecrypted; encryption is done over SSL

LC_ALL='en_US.UTF-8'

# Chef server connect options
chef_server_url   '<%= Chef::Config[:chef_server_url] %>'
node_name         '<%= Chef::Config[:node_name] %>'
client_key        '/etc/chef/client.pem'
trusted_certs_dir '/etc/chef/trusted_certs'
verify_api_cert   false
allow_unencrypted true

# The whitelist comes from default attributes
whitelist<%= node['base']['push-jobs']['whitelist'] %>

# We're under runit, so don't output timestamp
Mixlib::Log::Formatter.show_time = false


cat templates/default/push-jobs-client-run.erb
start on runlevel [2345]
stop on runlevel [!2345]
script

exec /opt/push-jobs-client/bin/pushy-client -l info  -c /etc/chef/push-jobs-client.rb
end script