The Consul root CA is generated using the consul tls ca create command. If created with the original options the root CA is only valid for a few years. After running production for a while you inevitably need to extend this certificate. To do so we need to generate and sign a new certificate using the existing private key. Consul does not provide any commands for doing so but it can be done using OpenSSL. ...
I received a fun email from Lets Encrypt today letting me know that they were revoking all of my certificates on March 4. The bug is described here. All of my certificates are managed by cert-manager inside Kubernetes. This led to the fun challenge of figuring out how to force a reissue of every certificate. There were 2 approaches that came up in the Kubernetes community slack. Delete all secrets containing cert-manager issued certificates. ...
I’ve been working on helm charts a lot lately. For better or worse that has involved running helm install —debug —dry-run… a lot to ensure things render correctly. It is much easier to parse that output when there is syntax highlighting. Enter [bat](GitHub - sharkdp/bat: A cat(1) clone with wings.). I can helm install —debug —dry-run… | bat -l yaml to get full syntax highlighting. It’s a small thing but it makes a big difference. ...
Traditionally it has been a pain to manage the infrastructure necessary for running integration tests within a CI/CD pipeline. Several years ago I accomplished this with an RDS instance for the database in AWS dedicated solely to the test environment. The problem is that multiple tests running at the same time would cause conflicts as they inserted and removed data in the database. At the time I set a lock in Jenkins to only allow one service to utilize the test database at a time, but this was far from ideal. Thankfully there are a lot of good options for solving this problem. I’m particularly fond of using the Docker plugin for Maven to handle this when dealing with Java-based applications. ...
I manage several apps that write various pieces of data to the local file system and rely on Fluentd to ship them to s3. There is solid monitoring around the fluentd aggregator process, but I wanted better visibility and alerting when things aren’t written to s3 as expected. The solution I came up with was a custom Datadog check. The files I am monitoring are written into a bucket named something like example-logs/data/event-files/year/month/day. A new path is set up in the s3 bucket for the current day’s date, e.g. logs/data/example-log/2018/08/15 each day. The Datadog check sends the count of objects in the current date’s directory as a gauge. You can then monitor that objects are created each day as expected and at a normal rate. ...
You’ve spent the past hour meticulously crafting a Readme update and its time to commit. Great, but what if you don’t want that commit to trigger automated testing, deploys, and other actions? If you’re using Jenkins declarative pipelines there’s a pretty simple solution. Add the below when block to each stage you wish to skip in your Jenkinsfile. when { not { changelog '\\[skip-ci\\]' } } We can also expand upon this for other actions if desired. For example, use the following around a deploy stage to avoid deploying pull requests, anything not on the master branch, and to respect the [skip-ci] param in a commit message. ...
I’ve been using declarative pipelines in Jenkins for a while with the Slack plugin to send build notifications to Slack. The plugin does what it says on the tin but gives you a pretty boring message by default.  I used the environment variables available in the pipeline to make things a little bit better and link back to the job.  But I was still always disappointed the notifications didn’t contain more information. Thankfully version 2.3 of the plugin added support for the attachments portion of the Slack message API. I was able to leverage the attachments feature to get better message formatting. Meanwhile, I took some inspiration from this thread to incorporate test result summaries. ...
Pagerduty provides a built-in way to export your incident data but only a limited number of data fields are available on the basic plan. Rather than upgrade you can use the API to export the data to a CSV. I found this gist listed here. The python script works great but some of my incident messages contained JSON data that threw off Excel when opening the CSV. I slightly modified the script with character escaping to work around this (lines 98- 107). ...
I spent way too long troubleshooting a script step in a Jenkins declarative pipeline this week. After lots of frustration I stumbled upon this gist that was a life saver. Long story short you need \\ to escape literal quotes.
I’ve often found myself with an instance id that I want to login to look at something. It sucks looking up the IP when you don’t know the DNS name. I’m sure there are other ways to do this but here is what I came up with. getec2ip() { aws ec2 describe-instances --instance-ids $1 | jq [.Reservations[0].Instances[0].PrivateIpAddress] | jq --raw-output .[] } assh() { host=$(getec2ip ${1}) ssh user@${host} } This relies on the aws cli and jq to parse out the ip and has made it much easier for me to quickly hop on an instance. ...