Logrotate is a Linux utility responsible for rotating, compressing and finally pruning logs in the server as per the directives given to it. It is installed by default on most of the popular Linux distributions like Ubuntu, Rhel, Centos, Rocky Linux, etc and is set up to logrotate most of the installed packages logs including rsyslog, the default system log processor.

The Linux Operating system and applications running often generate logs to files. These files are often stored in the /var/log directory or any other file specified by the app. If these log files are not rotated, compressed and periodically pruned, they will eventually consume all available disk space causing the server to freeze.

In this guide, we will take a look at how logrotate works then look at an example on how to configure logrotation for our custom app.

Prerequisites

To follow along with this guide, you need:

A modern Linux distro like Ubuntu, Debian, Centos, Rhel, Rocky Linux
Root access to the server or a user with sudo privileges

Check your logrotate version with this command:

logrotate --version

Output from my system:

# logrotate --version
logrotate 3.8.6

If Logrotate is not installed you will get an error. Please install the software using your Linux distribution’s package manager.

Logrotate Configuration

The logrotate configuration can be found in this file /etc/logrotate.conf. This is where the default settings are located. You will also find some rotation configs for logs that are not owned by any system packages.

The most important line in this config file is include /etc/logrotate.d. This instructs logrotate to also check configurations added in that directory. This means that for our applications, we do not need to add everything in that main file, we can create individual configuration files and add it to the /etc/logrotate.d directory. Most packages that you install that need help with log rotation will place their logrotate configuration files in that directory as well.

This is some content from the main /etc/logrotate.conf file:

weekly
rotate 4
create
dateext
compress

weekly configure weekly log rotations with log files owned by the root user and the syslog group
rotate 4 four log files being kept (keep 4 weeks worth of backlogs)
create new empty log files being created after the current one is rotated
dateex use date as a suffix of the rotated file
compress compress rotated files

For the individual files in the /etc/logrotate.d directory, lets take a look at one as an example:

cat /etc/logrotate.d/yum

Output:

/var/log/yum.log {
    missingok
    notifempty
    maxsize 30k
    yearly
    create 0600 root root
}

Each of those configs will inherit the default configs in the /etc/logrotate.conf directory.

Explaination:

/var/log/yum.log this first line states which file the configuration targets
missingok if the file doesn’t exist, it is still ok and logrotate should not error or log any error
notifempty don’t rotate the log file if it is empty
maxsize if the size exceed 30k, rotate
yearly run once a year
create 0600 root root create the new rotated file as root:root with permissions 0600

Some other options not listed above:

rotate 30 keep thirty files, prune the oldest when they exceed
monthly rotate once a month
compress compress the rotated files. this uses gzip by default and results in files ending in .gz. The compression command can be changed using the compresscmd option.

Logrotating Custom app log file

When we have our custom application generating and writing logs to a file, it is our task to configure how rotation is done for that file.

If we have root access, we would ideally place the config file in /etc/logrotate.d/ directory and it would work like described above
If we have no root access we would have to configure a cron to run logrotate command passing the config we defined
If you want to rotate the logs more frequently than daily – like hourly because the system’s Logrotate setup only runs once a day

Configuration in /etc/logrotate.d/

Let’s say we have out app generating logs to /var/log/our-app/app.log and it runs as centos. Let’s create our logrotate confog file /etc/logrotate/ourapp.conf with some example configs:

sudo vim /etc/logrotate.d/ourapp.conf

Config:

/var/log/our-app/*.log {
    daily
    missingok
    rotate 30
    compress
    notifempty
    create 0640 centos centos
    sharedscripts
    postrotate
        systemctl reload our-app
    endscript
}

Some new directives:

create 0640 centos centos creates an empty logfile after rotation with the specified permissions 0640 and the specified user and group centos
sharedscripts means any scripts added to the configuration are run only once per run, instead of for each file rotated. If the directive /var/log/our-app/*.log matches more than one log file, the script specified in postrotate would run twice without this option
postrotate to endscript block contains a script to run after the log file is rotated. In our case reloading the app. This is sometimes necessary to get your application to switch over to the newly created log file. Note that postrotate runs before logs are compressed. Compression could take a long time, and your software should switch to the new logfile immediately. For tasks that need to run after logs are compressed, use the lastaction block instead.

To test the above config, use this command:

sudo logrotate /etc/logrotate.conf --debug

The output will be about which log files Logrotate is handling and what it would have done to them. The standard Logrotate job will run once a day and include your new configuration.

Sample data in the output

...
rotating pattern: /var/log/our-app/*.log  after 1 days (30 rotations)
empty log files are not rotated, old logs are removed
No logs found. Rotation not needed.
...

Using `logrotate` command for custom configuration

If you do not have root access to the system and you want to logrotate your own custom files, this section gives you a way to go about that.

Let’s say an app our-app is generating log files and saving in a custom path within our home directory e.g. ~/apps/our-app/logs/. If the logs are generated frequently, we would want to rotate them hourly. But logrotate doesn’t work hourly?

First lets create a logrotate config file. I will place the file inside the app conf directory in this path ~/apps/our-app/conf/lorotate.conf with the following configuration:

Edit the file:

vim ~/apps/our-app/conf/lorotate.conf

With the following configuration:

/home/centos/apps/our-app/logs/*.log {
    rotate 30
    hourly
    compress
    sharedscripts
    delaycompress
    missingok
    notifempty
    dateext
    dateformat -%Y-%m-%d-%s
    copytruncate
}

The above config rotates the files in the directory /home/centos/apps/our-app/logs/*.log hourly, compressing and only keeping 30 old logs.

copytruncate copies content of a log file to a new file then truncates the log file.
dateformat specifies the format of the rotated file
dateext instructs logrotate to include date extension for the rotated files.

To test:

mkdir -p ~/apps/our-app/{logs,conf}
touch ~/apps/our-app/logs/app.log

Now that we have the file in place we can now use logrotate to rotate it. The only thing we need to do is to specify a state file. A state file records what logrotate saw the last time it run so it knows what to do. For the system install set up, it is already handled in /var/lib/logrotate/status but in our case we need to specify.

We can use the home directory for this:

logrotate ~/apps/our-app/config/logrotate.conf --state ~/logrotate-status --verbose

--verbose will print out detailed information about what Logrotate is doing.

You should see an output like this:

 $ logrotate ~/apps/our-app/config/logrotate.conf --state ~/logrotate-status --verbose
reading config file /home/centos/apps/our-app/config/logrotate.conf
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /home/centos/apps/our-app/logs/*.log  hourly (30 rotations)
empty log files are not rotated, old logs are removed
considering log /home/centos/apps/our-app/logs/app.log
  log does not need rotating (log has been already rotated)set default create context

Some information was recorded in the logrotate status file:

$ cat ~/logrotate-status
logrotate state -- version 2
"/home/centos/apps/our-app/logs/app.log" 2021-9-4-6:0:0

Logrotate noted the logs that it saw and when it last considered them for rotation. If we run this same command one hour later, the log will be rotated as expected.

If you want to force Logrotate to rotate the log file when it otherwise would not have, use the --force flag:

logrotate ~/apps/our-app/config/logrotate.conf --state ~/logrotate-status --verbose --force

This is useful when testing postrotate and other scripts.

The next thing is to have the above command run automatically. This can be achieved using crons. The crontab command comes in handy.

To edit crons so we can use this command:

crontab -e

This will open a up a text file. There may be some comments already in the file that explain the basic syntax expected. Move the cursor down to a new blank line at the end of the file and add the following:

3 * * * * /usr/sbin/logrotate ~/apps/our-app/config/logrotate.conf --state ~/logrotate-status

The above will run the logrotate command every 3rd minute of every hour. We use the full path /usr/sbin/logrotate so we don’t get Command not found error.

Save the file and exit. This will install the crontab and our task will run on the specified schedule.

Conclusion

We looked at logrotate in this guide. We were able to verify its version, check default configuration and set up our own custom set up. To learn more about the command line and configuration options available for Logrotate, you can read its manual page by running man logrotate in your terminal:

man logrotate