Thursday, June 02, 2016

Storing aws credentials in S3 bucket

Make a script  aws_init.sh on your dedicated server with admin AWS privileges

#! /bin/bash
set -e
mkdir -p ~/.ssh
aws s3 cp s3://keys/chef-client ~/.ssh/chef-client
mkdir -p ~/.chef/
aws s3 cp s3://keys/validation-prod.pem    ~/.chef
aws s3 cp s3://keys/validation-stag.pem    ~/.chef


Initialize the key by running:
eval "`aws s3 cp s3://keys/aws-access-keys.txt -`"
~/src/bin/aws_init.sh

Internal private SSL Certificate Authority

Internal SSL Certificate Authority

Intermediate CA

root@devops:~/ca# openssl genrsa -aes256 -out private/intermediate.key.pem 4096

root@devops:~/ca# openssl req -new -sha256 -key private/intermediate.key -out requests/intermediate.csr -config config.txt -subj "/C=US/ST=California/L=San\ Francisco/O=Company/OU=DevOps/CN=Company Intermediate CA/emailAddress=devops@company.com"
IT signed this request with Company Root CA

Creating server cert based on Intermediate CA

root@devops:~/ca# openssl req -new -nodes -keyout newcerts/wild.company.com.key -out requests/wild.company.com.csr -config config.txt -subj "/C=US/ST=California/L=San\ Francisco/O=Company/OU=DevOps/CN=*.company.com/emailAddress=devops@company.com"
 
root@devops:~/ca# openssl ca -batch -notext -config config.txt -in requests/wild.company.com.csr -cert certs/intermediate_ca.crt -keyfile private/intermediate_ca.key -out newcerts/wild.company.com.crt
 
root@devops:~/ca# cat certs/intermediate_ca.crt >> newcerts/wild.company.com.crt

See your cert

openssl x509 -in wild.company.com.crt -text

Revocation list

root@devops:~/ca# echo "01" > crlnumber
root@devops:~/ca# openssl ca -config config.txt -gencrl -out crl/certificate.crl

CA location

In order for Ubuntu system to trust your CA certificate add it to
/usr/local/share/ca-certificates/companyca.crt
update-ca-certificates

Upload new cert to AWS

aws iam upload-server-certificate --server-certificate-name $hostname --certificate-body file://$hostname.crt --private-key file://$hostname.key

Wednesday, June 01, 2016

Two factor authentication with ssh

Two factor definition:
1.  something you have  (ssh private key)
2.  something you know  (your account password)

On Ubuntu
/etc/ssh/sshd_config

# Require public key and password by default
PasswordAuthentication yes
AuthenticationMethods publickey,password

# Allow deploy and git groups to log in without password
Match Group deploy,git
   PasswordAuthentication no
   AuthenticationMethods publickey

Restart sshd

When you ssh to the server you'll be prompted for your passphrase and then again for your password.

Thursday, May 26, 2016

Chef recipe how to start and use custom service or script

Here is an example of a custom service for ubuntu:

package 'push-jobs-client' do
  action :install
end

template '/etc/chef/push-jobs-client.rb' do
  source 'push-jobs-client.rb.erb'
  owner  'root'
  group  'root'
  mode   '644'
end

template '/etc/init/chef-push-jobs.conf' do
  source 'push-jobs-client-run.erb'
  owner  'root'
  group  'root'
  mode   '644'
end

service 'chef-push-jobs' do
  supports [:stop, :start, :restart, :status]
  action [ :enable, :start]
end


cat templates/default/push-jobs-client.rb.erb
# Generated by Chef for <%= node[:fqdn] %>
# Local modifications will be overwritten!
# Added this wrapper for community cookbook to allow_unecrypted; encryption is done over SSL

LC_ALL='en_US.UTF-8'

# Chef server connect options
chef_server_url   '<%= Chef::Config[:chef_server_url] %>'
node_name         '<%= Chef::Config[:node_name] %>'
client_key        '/etc/chef/client.pem'
trusted_certs_dir '/etc/chef/trusted_certs'
verify_api_cert   false
allow_unencrypted true

# The whitelist comes from default attributes
whitelist<%= node['base']['push-jobs']['whitelist'] %>

# We're under runit, so don't output timestamp
Mixlib::Log::Formatter.show_time = false


cat templates/default/push-jobs-client-run.erb
start on runlevel [2345]
stop on runlevel [!2345]
script

exec /opt/push-jobs-client/bin/pushy-client -l info  -c /etc/chef/push-jobs-client.rb
end script

Tuesday, March 15, 2016

AWS Terraform

If you ever had to manage AWS Amazon environment you probably start by clicking through AWS console - Web GUI, then you quickly realize that it won't scale, because all your clicks are not repeatable and there is no code review of some sort.

Next logical step it to use aws cli, but then you'd need to create a bunch of scripts to manage it.  There are a couple of options for scalable and systematic approach - one is Cloud Formation from Amazon, another cool solution is Terraform.

Terraform allows you to manage pretty much any AWS resource from a single configuration file that could be sitting in your GIT repo.  This is truly "infrastructure as a code" solution.

Here is an example of how to create a simple ec2 instance with your parameters.

resource "aws_instance" "my-server" {
    ami                         = "ami-93742ea3"
    availability_zone           = "us-west-2a"
    ebs_optimized               = false
    instance_type               = "t2.micro"
    monitoring                  = false
    key_name                    = "your-ssh-access-key"
    subnet_id                   = "subnet-99999999"
    security_group_ids      = ["sg-ae33333", "sg-bb33333"]
    associate_public_ip_address = true
    private_ip                  = "10.10.10.50"
    source_dest_check           = true

    root_block_device {
        volume_type           = "gp2"
        volume_size           = 10
        iops                  = 30
        delete_on_termination = true
    }

    tags {
        "Name" = "my-server"
    }
}

As you can see above, some values like subnet and security group are hardcoded since we are dealing with existing infrastructure, and those resources were created by hand.

It's easy enough to refactor those into a separate file called "variables.tf", so that you could at least use them by readable human names.

variable "access_key" {}
variable "secret_key" {}
# Default AWS region is West
variable "aws_region" {
  default = "us-west-2"
}
# Security Groups
variable "sg" {
    default = {
        internal_ping = "sg-0444444"
        internal_ssh = "sg-444444444"
   }
}

Now you can start using those variables and you terraform template will be more readable
  security_group_id      = "${var.sg.internal_ping},${var.sg.internal_ssh},${var.sg.internal_http}"

Here is another example on how Terraform can manage a single DNS entry assuming that you have all your zones in that variables.tf file
resource "aws_route53_record" "my-server-A" {
    zone_id = "${var.dns_zone.company_com}"
    name    = "my-server.company.com"
    type    = "A"
    records = ["10.10.10.50"]
    ttl     = "600"
}