https://danielschaaff.com/tags/boto3/ Recent content in Boto3 on Daniel Schaaff Hugo -- 0.147.4 en Wed, 29 Aug 2018 03:45:14 +0000 https://danielschaaff.com/posts/2018/monitoring-creation-of-log-files-in-s3/ Wed, 29 Aug 2018 03:45:14 +0000 https://danielschaaff.com/posts/2018/monitoring-creation-of-log-files-in-s3/ <p>I manage several apps that write various pieces of data to the local file system and rely on Fluentd to ship them to s3. There is solid monitoring around the fluentd aggregator process, but I wanted better visibility and alerting when things aren’t written to s3 as expected.</p> <p>The solution I came up with was a <a href="https://github.com/dschaaff/datadog-checks">custom Datadog check</a>. The files I am monitoring are written into a bucket named something like <code>example-logs/data/event-files/year/month/day</code>. A new path is set up in the s3 bucket for the current day’s date, e.g. <code>logs/data/example-log/2018/08/15</code> each day. The Datadog check sends the count of objects in the current date’s directory as a gauge. You can then monitor that objects are created each day as expected and at a normal rate.</p>