Logrotate is a Linux utility responsible for rotating, compressing and finally pruning logs in the server as per the directives given to it. It is installed by default on most of the popular Linux distributions like Ubuntu, Rhel, Centos, Rocky Linux, etc and is set up to logrotate most of the installed packages logs including rsyslog
, the default system log processor.
The Linux Operating system and applications running often generate logs to files. These files are often stored in the /var/log
directory or any other file specified by the app. If these log files are not rotated, compressed and periodically pruned, they will eventually consume all available disk space causing the server to freeze.
In this guide, we will take a look at how logrotate works then look at an example on how to configure logrotation for our custom app.
# Prerequisites
To follow along with this guide, you need:
- A modern Linux distro like Ubuntu, Debian, Centos, Rhel, Rocky Linux
- Root access to the server or a user with sudo privileges
Check your logrotate version with this command:
logrotate --version
Output from my system:
# logrotate --version
logrotate 3.8.6
If Logrotate is not installed you will get an error. Please install the software using your Linux distribution’s package manager.
# Logrotate Configuration
The logrotate configuration can be found in this file /etc/logrotate.conf
. This is where the default settings are located. You will also find some rotation configs for logs that are not owned by any system packages.
The most important line in this config file is include /etc/logrotate.d
. This instructs logrotate to also check configurations added in that directory. This means that for our applications, we do not need to add everything in that main file, we can create individual configuration files and add it to the /etc/logrotate.d
directory. Most packages that you install that need help with log rotation will place their logrotate configuration files in that directory as well.
This is some content from the main /etc/logrotate.conf
file:
weekly
rotate 4
create
dateext
compress
weekly
configure weekly log rotations with log files owned by the root user and the syslog grouprotate 4
four log files being kept (keep 4 weeks worth of backlogs)create
new empty log files being created after the current one is rotateddateex
use date as a suffix of the rotated filecompress
compress rotated files
For the individual files in the /etc/logrotate.d
directory, lets take a look at one as an example:
cat /etc/logrotate.d/yum
Output:
/var/log/yum.log {
missingok
notifempty
maxsize 30k
yearly
create 0600 root root
}
Each of those configs will inherit the default configs in the /etc/logrotate.conf
directory.
Explaination:
/var/log/yum.log
this first line states which file the configuration targetsmissingok
if the file doesn’t exist, it is still ok and logrotate should not error or log any errornotifempty
don’t rotate the log file if it is emptymaxsize
if the size exceed 30k, rotateyearly
run once a yearcreate 0600 root root
create the new rotated file asroot
:root
with permissions0600
Some other options not listed above:
rotate 30
keep thirty files, prune the oldest when they exceedmonthly
rotate once a monthcompress
compress the rotated files. this usesgzip
by default and results in files ending in.gz
. The compression command can be changed using thecompresscmd
option.
# Logrotating Custom app log file
When we have our custom application generating and writing logs to a file, it is our task to configure how rotation is done for that file.
- If we have root access, we would ideally place the config file in
/etc/logrotate.d/
directory and it would work like described above - If we have no root access we would have to configure a cron to run
logrotate
command passing the config we defined - If you want to rotate the logs more frequently than daily – like hourly because the system’s Logrotate setup only runs once a day
# Configuration in /etc/logrotate.d/
Let’s say we have out app generating logs to /var/log/our-app/app.log
and it runs as centos
. Let’s create our logrotate confog file /etc/logrotate/ourapp.conf
with some example configs:
sudo vim /etc/logrotate.d/ourapp.conf
Config:
/var/log/our-app/*.log {
daily
missingok
rotate 30
compress
notifempty
create 0640 centos centos
sharedscripts
postrotate
systemctl reload our-app
endscript
}
Some new directives:
create 0640 centos centos
creates an empty logfile after rotation with the specified permissions0640
and the specified user and groupcentos
sharedscripts
means any scripts added to the configuration are run only once per run, instead of for each file rotated. If the directive/var/log/our-app/*.log
matches more than one log file, the script specified inpostrotate
would run twice without this optionpostrotate
toendscript
block contains a script to run after the log file is rotated. In our case reloading the app. This is sometimes necessary to get your application to switch over to the newly created log file. Note thatpostrotate
runs before logs are compressed. Compression could take a long time, and your software should switch to the new logfile immediately. For tasks that need to run after logs are compressed, use thelastaction
block instead.
To test the above config, use this command:
sudo logrotate /etc/logrotate.conf --debug
The output will be about which log files Logrotate is handling and what it would have done to them. The standard Logrotate job will run once a day and include your new configuration.
Sample data in the output
...
rotating pattern: /var/log/our-app/*.log after 1 days (30 rotations)
empty log files are not rotated, old logs are removed
No logs found. Rotation not needed.
...
#
Using logrotate
command for custom configuration
If you do not have root access to the system and you want to logrotate your own custom files, this section gives you a way to go about that.
Let’s say an app our-app
is generating log files and saving in a custom path within our home directory e.g. ~/apps/our-app/logs/
. If the logs are generated frequently, we would want to rotate them hourly
. But logrotate doesn’t work hourly?
First lets create a logrotate config file. I will place the file inside the app conf directory in this path ~/apps/our-app/conf/lorotate.conf
with the following configuration:
Edit the file:
vim ~/apps/our-app/conf/lorotate.conf
With the following configuration:
/home/centos/apps/our-app/logs/*.log {
rotate 30
hourly
compress
sharedscripts
delaycompress
missingok
notifempty
dateext
dateformat -%Y-%m-%d-%s
copytruncate
}
The above config rotates the files in the directory /home/centos/apps/our-app/logs/*.log
hourly, compressing and only keeping 30 old logs.
copytruncate
copies content of a log file to a new file then truncates the log file.dateformat
specifies the format of the rotated filedateext
instructs logrotate to include date extension for the rotated files.
To test:
mkdir -p ~/apps/our-app/{logs,conf}
touch ~/apps/our-app/logs/app.log
Now that we have the file in place we can now use logrotate
to rotate it. The only thing we need to do is to specify a state
file. A state
file records what logrotate
saw the last time it run so it knows what to do. For the system install set up, it is already handled in /var/lib/logrotate/status
but in our case we need to specify.
We can use the home directory for this:
logrotate ~/apps/our-app/config/logrotate.conf --state ~/logrotate-status --verbose
--verbose
will print out detailed information about what Logrotate is doing.
You should see an output like this:
$ logrotate ~/apps/our-app/config/logrotate.conf --state ~/logrotate-status --verbose
reading config file /home/centos/apps/our-app/config/logrotate.conf
Allocating hash table for state file, size 15360 B
Handling 1 logs
rotating pattern: /home/centos/apps/our-app/logs/*.log hourly (30 rotations)
empty log files are not rotated, old logs are removed
considering log /home/centos/apps/our-app/logs/app.log
log does not need rotating (log has been already rotated)set default create context
Some information was recorded in the logrotate status file:
$ cat ~/logrotate-status
logrotate state -- version 2
"/home/centos/apps/our-app/logs/app.log" 2021-9-4-6:0:0
Logrotate noted the logs that it saw and when it last considered them for rotation. If we run this same command one hour later, the log will be rotated as expected.
If you want to force Logrotate to rotate the log file when it otherwise would not have, use the --force
flag:
logrotate ~/apps/our-app/config/logrotate.conf --state ~/logrotate-status --verbose --force
This is useful when testing postrotate
and other scripts.
The next thing is to have the above command run automatically. This can be achieved using crons. The crontab
command comes in handy.
To edit crons so we can use this command:
crontab -e
This will open a up a text file. There may be some comments already in the file that explain the basic syntax expected. Move the cursor down to a new blank line at the end of the file and add the following:
3 * * * * /usr/sbin/logrotate ~/apps/our-app/config/logrotate.conf --state ~/logrotate-status
The above will run the logrotate command every 3rd minute of every hour. We use the full path /usr/sbin/logrotate
so we don’t get Command not found error.
Save the file and exit. This will install the crontab and our task will run on the specified schedule.
# Conclusion
We looked at logrotate in this guide. We were able to verify its version, check default configuration and set up our own custom set up. To learn more about the command line and configuration options available for Logrotate, you can read its manual page by running man logrotate
in your terminal:
man logrotate