Bash Function to SSH into ec2 Instances

I’ve often found myself with an instance id that I want to login to look at something. It sucks looking up the IP when you don’t know the DNS name. I’m sure there are other ways to do this but here is what I came up with.


getec2ip() {
 aws ec2 describe-instances --instance-ids $1 | jq [.Reservations[0].Instances[0].PrivateIpAddress] | jq --raw-output .[]
}

assh() {
 host=$(getec2ip ${1})
 ssh user@${host}
}

This relies on the aws cli and jq to parse out the ip and has made it much easier for me to quickly hop on an instance.

Jenkins Dynamic EC2 Slaves

There is a nice plugin for Jenkins that lets you dynamically add capacity by spinning up EC2 instances on demand and then terminating them when the job queue expires. This is a great way to save money on an AWS based build infrastructure.

Unfortunately, the plugin documentation is really light and there are a few gotchas to look out for.

Security Groups

This field only accepts comma separated security group IDs, not names. This is frustrating because other fields in the plugin take a space separated list (e.g. labels)

Running in VPC

If you’re a sane person you’re going to want to run these instances in a private VPC. This is entirely possible but is hidden in the advanced settings. If you expand the advanced settings you’ll see a field to enter your desired subnet ID. Set this to the ID of the private subnet in your VPC you want the instances to run in.

Don’t Rely On the User Data/Init Scrip to Install Dependencies

This adds a lot of time to the instance coming on line and being usable by Jenkins. A better approach is to make an AMI with all the build dependencies you need. The only delay is then the instance boot time.

This is far from an exhaustive walkthrough but highlights the issues I ran into setting it up.

Speeding up RSpec Tests in Bamboo

Now that roles and profiles are in my control repo my RSpec tests are taking longer then ever. As of this writing the control repo contains 938 tests and I’m still a long way from 100% coverage. This really slows down the feedback loop when running tests. When running locally I often just run RSpec against a specific spec file rather then run the whole test suite, but I still wanted a way to speed things up in Bamboo.

I had used parallel_tests before to run things quicker on my local machine but was having issues with each test overwriting the JUnit output file and giving me an incomplete result set at the end. I stumbled across a fix for this yesterday which I’m pretty happy with. My original .rspec file had the file name of the JUnit output hard coded.

--format documentation
--color
--format RspecJunitFormatter
--out results.xml

By making the following change each parallel test writes to its own JUnit output file.

--format documentation
--color
--format RspecJunitFormatter
--out results<%= ENV['TEST_ENV_NUMBER'] %>.xml

Bamboo was already parsing the results using a wildcard so no change was needed there (see this post for details on my Bamboo setup).  The last step was to change the rake task Bamboo is running from rake spec to rake parallel_spec. This change cut the test time down from an average of 24 minutes to 8 minutes and faster feedback is always a plus!

 

Tackling Tech Debt in Puppet

I spent some time tackling technical debt in our Puppet code this week. The biggest outstanding item was implementing eyaml for protecting secrets in Hiera. I’d also been encouraging developers to contribute to the Puppet code base  for some time, but they were restricted from the control repo due to some secrets kept in Hiera. This put a big damper on collaboration as Hiera is the data engine for our roles and profiles. Separate git repos were also used for the profile and role modules due to this workflow.

Hiera-eyaml to the rescue! Props to voxpupuli as this was dead simple to implement. Once the secrets were encrypted I tidied up a few more things before collaboration could rain down !

  • created a new branch on the existing control repo
  • moved the roles and profiles modules into the site directory of the control repo
  • create an environment.conf file to add the site dir to the module path
  • tested an r10k run on the new environment
  • spent some time fighting RSpec, as you do
  • merged into production
  • created a new git repo for the control module to remove commit history containing secrets
  • opened up access to the development team

We’ve now got a control repo with encrypted secrets open to contributions from across the org. I’m also enjoying the simplified workflow with environments now that hieradata, roles, and profiles are all in a single git repo.

Update to lita-activedirectory

I updated our Active Directory lita plugin today with support for querying the members of a given group. See https://github.com/knuedge/lita-activedirectory. It still needs some work to properly present errors when a user or group doesn’t actually exist in the directory. Right now it returns nothing rather then a helpful error. It works splendidly with legit users and groups however.

ChatOps ftw!

Automated Puppet Tests with Bamboo

rnelson0 is walking through automated Puppet testing with Jenkins on his blog. I thought I’d highlight how you can use a similar workflow in Atlassian’s Bamboo application. This assumes you already have a working Bamboo setup and are familiar with the general process for testing Puppet modules with rspec.

Create a new plan

The first step is to set up a new plan to use for the testing. Click “Create” and then “Create a new plan” in the top menu bar.Pasted_Image_1_13_17__3_08_PM.png

Bamboo organizes related jobs, builds, etc. into projects. On the next screen either create a new Puppet project or select an existing project if you’ve already set that up.Pasted_Image_1_13_17__3_10_PM.png

Fill in the details and select the repository you’d like to start testing. Bamboo can read from a large number of version control systems. I happen to use Bitbucket so this is simple.  Once your happy with the selections click “Configure plan”.

Pasted_Image_1_13_17__3_15_PM.png

At the next screen we setup our tasks. The first task, Source Code Checkout, is added by default and checks out the repo configured in the previous step. I like to break things down into small script tasks so they are easier to troubleshoot and duplicate between jobs.

Click add task and select script.

Pasted_Image_1_13_17__3_17_PM.png

The first step is to setup the Ruby environment. This presumes you already have Bamboo build agents up and running and that rvm is installed. This script below is what I use. It performs some checks to ensure ruby is properly setup.


#!/bin/bash

if [ "$(ps -p "$$" -o comm=)" != "bash" ]; then
 /bin/bash "$0" "$@"
 exit "$?"
fi 
source /etc/profile.d/rvm.sh
ruby="ruby-2.1.8"
install_count=`rvm list | grep $ruby | wc -l`

if [ $install_count -lt 1 ]; then
  rvm install $ruby
fi
rvm use $ruby
rvm user gemsets

Now click “Add task” and add another script task. We will use this script step to create a new gemset and install the gems listed in the Gemfile using bundler. If you use environment variables to specify the version of puppet to install enter that in the environment variables field.Pasted_Image_1_13_17__3_55_PM.png

#!/bin/bash

if [ "$(ps -p "$$" -o comm=)" != "bash" ]; then
/bin/bash "$0" "$@"
exit "$?"
fi
source /etc/profile.d/rvm.sh
ruby="ruby-2.1.8"
gemset="puppet-4.8.0-validate"
rvm use $ruby
rvm user gemsets
rvm gemset create $gemset
rvm gemset use $gemset
gem install bundler
rm Gemfile.lock
bundle install

Now lets add another script step and use it to run the actual tests. At the end of the test this removes the Gemset to ensure a clean environment for each new run.  You can also add other options here using environment variables such as STRICT_VARIABLES=no.Pasted_Image_1_13_17__3_30_PM.png

#!/bin/bash

if [ "$(ps -p "$$" -o comm=)" != "bash" ]; then
/bin/bash "$0" "$@"
exit "$?"
fi
source /etc/profile.d/rvm.sh
ruby="ruby-2.1.8"
gemset="puppet-4.8.0-validate"
rvm use $ruby
rvm user gemsets
rvm gemset use $gemset

bundle exec rake validate
bundle exec rake spec
rvm gemset delete --force $gemset

We’ve now covered the basics needed to get started and can click create at the bottom to finalize the plan. After doing so we’ll be looking  at the job configuration page. The script tasks we just created are in the “Default Job”.

Pasted_Image_1_13_17__3_33_PM.pngThat name isn’t very clear so lets update it. Click “Default Job” and then select the “Job details” tab. Here we’ll enter a more descriptive name such as “Puppet 4.8 rspec” and click save. A descriptive name is handy because you can clone stages across plans to avoid repeating the script setup.

Repository Triggers

If you’re using the built in Bitbucket Server integration like my setup then Bamboo will automatically run the plan whenever a new commit is pushed to the repo. You can customize the type of trigger to use polling, scheduling, or other options as well. Simply select the “Triggers” tab under plan configuration and set appropriately.Pasted_Image_1_13_17__3_37_PM.png

Tracking Results

If you were to run this plan now you would get a pass or fail in Bamboo  but would have to look at the logs for the job to see the actual details of the results. We can use JUnit to get an better view of the test in Bamboo. To set this up create a .rspec file in the root of the Puppet module you’re testing with the following content

--format RspecJunitFormatter
--out results.xml

This will write the results of the test in JUnit format to the results.xml file. You’ll also want to add this file to .gitignore. Now we can add a step to our plan to parse this file. Go back to the plan configuration and select the stage we setup previously. Click add task and select “JUnit Parser.”Pasted_Image_1_13_17__3_41_PM.png

Configure the JUnit parser so it finds our results file.Pasted_Image_1_13_17__3_42_PM.png

Click save and run the plan again. You’ll now see your test results nicely formatted in the plan history. Here’s an example of what that looks like for failed tests.Pasted_Image_1_13_17__3_44_PM.pngPasted_Image_1_13_17__3_45_PM.png

You don’t need to redo this work for each repo. When setting up a plan for a new module start by removing the default stage. Then click add job and select clone an existing job. Pasted_Image_1_13_17__4_01_PM.png

You can then select which plan to clone from.Pasted_Image_1_13_17__4_02_PM.png

It takes a bit more setup then using Travis CI but its not difficult to get rspec testing up and running in Bamboo.