Posts

I’ve been using AWS’ EKS service for many years. What began as a bare bones offering has slowly and steadily added more features over the years. Amazon recently announced “auto” mode for EKS clusters. In this new mode AWS manages a Karpenter installation, core addons such as coredns and vpc-cni, as well as the underlying ec2 nodes. On paper this sounds great, but as always the devil is in the details. In this instance, it is important to be aware of the ~ 10% price premium for this mode. In auto mode AWS will charge an hourly rate for the ec2 instances in addition to the standard instance charge. While some folks may find this a fair trade off, I would not choose this mode myself. I run clusters using Karpenter as the autoscaler and use bottlerocket-os for the underlying nodes. In this configuration the management overhead is extremely low. I even gave a short talk on this setup for AWS a while back. IMO, if you are already running an EKS cluster using Karpenter and Bottlerocket you have little to gain from auto mode relative to the price premium. ...

I’ve been using my custom domain with iCloud email for a couple weeks. Truth be told I hardly notice any difference from when I used Gmail. That’s largely due to using Apple Mail app as my primary interface. I haven’t had any reliability issues and the only real difference I’ve noticed is the email size limit is 20MB as opposed to Gmail’s 25MB. The primary benefit I’ve seen is that calendar invites end up on my iCloud calendar, which is where I prefer them. ...

I’m in the process of migrating my email from Google Workspace to iCloud+. I’ve been using Google Workspace (aka gsuite, aka whatever name Google switches to next week) for my email for a long time. For ages it was free and made perfect sense as a service. Those free accounts have gone away, and the service’s focus is clearly aimed at the enterprise and not individuals like me. The price per user is also about to increase on me, so this felt like a good time to try something else. I’m only really interested in hosting email for a few custom domains and have no needs for any features beyond that. Our family already uses iCloud for everything so I figured why not give it a try. ...

The Consul root CA is generated using the consul tls ca create command. If created with the original options the root CA is only valid for a few years. After running production for a while you inevitably need to extend this certificate. To do so we need to generate and sign a new certificate using the existing private key. Consul does not provide any commands for doing so but it can be done using OpenSSL. ...

I received a fun email from Lets Encrypt today letting me know that they were revoking all of my certificates on March 4. The bug is described here. All of my certificates are managed by cert-manager inside Kubernetes. This led to the fun challenge of figuring out how to force a reissue of every certificate. There were 2 approaches that came up in the Kubernetes community slack. Delete all secrets containing cert-manager issued certificates. ...

I’ve been working on helm charts a lot lately. For better or worse that has involved running helm install —debug —dry-run… a lot to ensure things render correctly. It is much easier to parse that output when there is syntax highlighting. Enter [bat](GitHub - sharkdp/bat: A cat(1) clone with wings.). I can helm install —debug —dry-run… | bat -l yaml to get full syntax highlighting. It’s a small thing but it makes a big difference. ...

Traditionally it has been a pain to manage the infrastructure necessary for running integration tests within a CI/CD pipeline. Several years ago I accomplished this with an RDS instance for the database in AWS dedicated solely to the test environment. The problem is that multiple tests running at the same time would cause conflicts as they inserted and removed data in the database. At the time I set a lock in Jenkins to only allow one service to utilize the test database at a time, but this was far from ideal. Thankfully there are a lot of good options for solving this problem. I’m particularly fond of using the Docker plugin for Maven to handle this when dealing with Java-based applications. ...

I manage several apps that write various pieces of data to the local file system and rely on Fluentd to ship them to s3. There is solid monitoring around the fluentd aggregator process, but I wanted better visibility and alerting when things aren’t written to s3 as expected. The solution I came up with was a custom Datadog check. The files I am monitoring are written into a bucket named something like example-logs/data/event-files/year/month/day. A new path is set up in the s3 bucket for the current day’s date, e.g. logs/data/example-log/2018/08/15 each day. The Datadog check sends the count of objects in the current date’s directory as a gauge. You can then monitor that objects are created each day as expected and at a normal rate. ...

You’ve spent the past hour meticulously crafting a Readme update and its time to commit. Great, but what if you don’t want that commit to trigger automated testing, deploys, and other actions? If you’re using Jenkins declarative pipelines there’s a pretty simple solution. Add the below when block to each stage you wish to skip in your Jenkinsfile. when { not { changelog '\\[skip-ci\\]' } } We can also expand upon this for other actions if desired. For example, use the following around a deploy stage to avoid deploying pull requests, anything not on the master branch, and to respect the [skip-ci] param in a commit message. ...

I’ve been using declarative pipelines in Jenkins for a while with the Slack plugin to send build notifications to Slack. The plugin does what it says on the tin but gives you a pretty boring message by default.  I used the environment variables available in the pipeline to make things a little bit better and link back to the job.  But I was still always disappointed the notifications didn’t contain more information. Thankfully version 2.3 of the plugin added support for the attachments portion of the Slack message API. I was able to leverage the attachments feature to get better message formatting. Meanwhile, I took some inspiration from this thread to incorporate test result summaries. ...