Introduction

What is happening everyone! Hope you’re all having an amazing start of the summer. In this article i’m diving into something I recently spent time reading about and was fascinated about how cool & useful it is.

In systems that produce logs whether it’s a web server, database, background job, etc the logs tend - overtime - to grow in size. For example a web server that writes to a log file every request would grow linearly as the users grow.

Over time this would significantly affect the disk size which overtime could cause the whole system to be slower. (peep the next article to know why)

The disk needed a hero, enter log rotation

Log rotation basically controls the whole process and prevents files from getting bigger (makes everything more controllable)

You can fully control how you want the logs to be rotated. Either every fixed period of time or once the file reaches a certain size.

The older log files then can be compressed to reduce its size & maybe send to some archive to free up that disk space. You can also control how many log files you can have before it starts deleting them. So to summarize;

Log rotation is a process or strategy that:

Limits log file size or age
Archives old logs (optionally compressing them)
Starts fresh logs for continued logging
Enforces a retention policy (e.g. keep last 7 logs)

Now that we understand what rotation is, let’s look at one of the most famous tools used for log rotation

LogRotate

Log rotate is one of the most popular and widely used log rotation tools, especially on Linux systems. It’s the de facto standard for rotating log files created by system services, daemons, and applications.

We’ll dive into a quick demo rotating logs for a service that produces logs nonstop

To install on Mac brew install logrotate
Ubuntu: sudo apt install logrotate

Once installed. The main entry point for the binary is the logrotate.conf file. On Mac It’s located in /opt/homebrew/etc/logrotate.conf

If we take a look at it we’ll find the following

# see "man logrotate" for details

# global options do not affect preceding include directives

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /opt/homebrew/etc/logrotate.d

# system-specific logs may also be configured here.

weekly rotates log files every week (not by itself a cronjob is required to run the logrotate binary but it looks a week behind and rotates), There exists different options daily hourly & more
rotate 4 keeps the last 4 rotations and deletes the rest.
create creates new log files after rotating, be careful when using this because any process dumping logs has an open file descriptor to the log file, when a new one is created you’re going to have to manually update the process to point to the new one. copytruncate is a life saver here because it copies the logs to a new file and empties the original file. Meaning the file descriptor still points to that file.
compress uses gzip to compress the rotated files to reduce size
include /opt/homebrew/etc/logrotate.d basically is for application management where in the logrotate.d directory you can create different configurations for different applications each customized with their own set of settings. (We’ll dive into more below)

Example of a simple app I created /opt/homebrew/etc/logrotate.d/myapp

  /var/log/myapp/*.log # specify the path of the logs to operate on

  {
  size 5k # Rotate when size is 5 kilobytes, K is kilobytes, M is megabytes, etc (was testing that's why its low) 
  copytruncate # copytruncate basically instead of creating a new file copies the data from the file to another file then empties it
  compress # gzip old logs
  rotate 3 # keep last 3 rotations
  missingok # if no logs are found don't panic
  notifempty # don't rotate if the log is empty.
  }

For all the configurations you can visit the man page for log rotate here

But with the setup above I have a simple script that writes logs

#!/bin/bash

logfile="/var/log/myapp/production.log"

# Function to generate a random log record
generate_log_record() {
    local loglevel=("INFO" "WARNING" "ERROR")
    local services=("web" "database" "app" "network")
    local timestamps=$(date +"%Y-%m-%d %H:%M:%S")
    local random_level=${loglevel[$RANDOM % ${#loglevel[@]}]}
    local random_service=${services[$RANDOM % ${#services[@]}]}
    local message="This is a sample log record for ${random_service} service."

    echo "${timestamps} [${random_level}] ${message}"
}

# Main loop to write log records every second
while true; do
    log_record=$(generate_log_record)
    echo "${log_record}" >> "${logfile}"
    sleep 1
done

Now we have:

Log rotate setup for our application
Logs being created

But we need a way to actually invoke the logrotate command because it doesn’t on its own and here we have two options:

A cron job that runs every specified period
Using a file watcher that watches file sizes and acts accordingly

In my case since I added the size config i’ll use a file watcher. On Mac I used fswatch

fswatch -0 /var/log/myapp/production.log | while read -d "" event
do
  size=$(stat -f%z /var/log/myapp/production.log)
  if [ "$size" -ge 5120 ]; then
    sudo logrotate -f /opt/homebrew/etc/logrotate.conf
  fi
done

Basically watch for the size of production.log once its more than 5 KB (small for testing) we rotate.

We can watch it in action to:

Watch how production.log reaches 5KB and rotates to be production.log.3.gz and the existing production.log.3.gz gets deleted

You can do so much more with rotations, add scripts that run on every rotate maybe upload the rotated files to some cloud storage, etc. Endless ideas can be made here.