How to Setup Central Logging Server with Rsyslog in Ubuntu 20.04

Rsyslog is an open-source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IP network. It is an open-source utility for log processing. It permits the logging of data from different types of systems in a central repository. Rsyslog is a Syslog protocol with more extensions, features, and benefits.

Rsyslog can deliver over one million messages per second to local destinations when limited processing is applied making it so powerful. By default, Linux uses the syslog daemon to record logs about how the system is running and then putting these logs into a central repository in: /var/log where we can examine and investigate them.
In simple terms of meaning everything from kernel events to user actions is logged by Linux, allowing you to see al most any action performed on your pc or servers.

Logging is really beneficial. The log information can really help you to see what is exactly happened on your system, or what exactly is going on under the hood. For example if you have an application that crashes unexpectedly, the messages in the logs can help identify the cause.

In this tutorial we learn how to install and configure rsyslog on Ubuntu 20.04.

Directory where different sorts of logs stored are:

/var/log/syslog – Stores all startup messages, application startup messages etc. Practically stores all global system logs.
/var/log/cron – The Cron jobs are basically kind of scheduled and automated task created in the system, that runs periodically and repeatedly. You can see what this logs directory would store.
/var/log/kern.log – it stores kernel logs. No matter what logs they are. Event logs, errors, or warning logs.
/var/log/auth.log – Authentication logs
/var/log.boot.log – System boot logs
/var/log/mysql.d – Mysql logs
/var/log/httpd – Apache logs directory
/var/log/maillog – Mail server logs

How to Setup Central Logging Server with Rsyslog in Ubuntu 22.04

Prerequisites

Two Ubuntu servers one to be set up as a rsyslog server (10.2.11.10 in my case) and a client (10.2.11.11).

Table of Content

Ensure that the server is up to date
Ensure that the Rsyslog is installed
Configuring Rsyslog
Configure firewall rules for Rsyslog
Configure the rsyslog client system
View Client’s log files

1. Ensure that the server is installed

Before proceeding ensure that the system has updated packages. Use this command:

sudo apt update
sudo apt upgrade -y

2. Ensure that Rsyslog is installed

Rsyslog package comes pre installed in Ubuntu. Use this command to ensure it is present:

sudo apt install rsyslog

Confirm the version with this command:

$ rsyslogd -v
rsyslogd  8.2001.0 (aka 2020.01) compiled with:
	PLATFORM:				x86_64-pc-linux-gnu
	PLATFORM (lsb_release -d):
	FEATURE_REGEXP:				Yes
	GSSAPI Kerberos 5 support:		Yes
	FEATURE_DEBUG (debug build, slow code):	No
	32bit Atomic operations supported:	Yes
	64bit Atomic operations supported:	Yes
	memory allocator:			system default
	Runtime Instrumentation (slow code):	No
	uuid support:				Yes
	systemd support:			Yes
	Config file:				/etc/rsyslog.conf
	PID file:				/run/rsyslogd.pid
	Number of Bits in RainerScript integers: 64

See https://www.rsyslog.com for more information.

Please check that the service is up and running using this command:

$ sudo systemctl status rsyslog
● rsyslog.service - System Logging Service
     Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-01-19 13:03:07 UTC; 1 day 6h ago
TriggeredBy: ● syslog.socket
       Docs: man:rsyslogd(8)
             https://www.rsyslog.com/doc/
   Main PID: 70335 (rsyslogd)
      Tasks: 4 (limit: 4624)
     Memory: 3.3M
     CGroup: /system.slice/rsyslog.service
             └─70335 /usr/sbin/rsyslogd -n -iNONE

Jan 19 13:03:07 dev-ubuntusrv.inv.re systemd[1]: Starting System Logging Service...
Jan 19 13:03:07 dev-ubuntusrv.inv.re rsyslogd[70335]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2001.0]
Jan 19 13:03:07 dev-ubuntusrv.inv.re systemd[1]: Started System Logging Service.
Jan 19 13:03:07 dev-ubuntusrv.inv.re rsyslogd[70335]: rsyslogd's groupid changed to 110
Jan 19 13:03:07 dev-ubuntusrv.inv.re rsyslogd[70335]: rsyslogd's userid changed to 104
Jan 19 13:03:07 dev-ubuntusrv.inv.re rsyslogd[70335]: [origin software="rsyslogd" swVersion="8.2001.0" x-pid="70335" x-info="https://www.rsyslog.com"] start
Jan 20 00:00:01 adoring-shannon.15-237-143-120.plesk.page rsyslogd[70335]: [origin software="rsyslogd" swVersion="8.2001.0" x-pid="70335" x-info="https://www.rsyslog.com"]

3. Configuring Rsyslog

Let us configure rsyslog to run in server mode. Rsyslog config file is located in this file path /etc/rsyslog.conf . Open it with your prefered text editor:

sudo vim /etc/rsyslog.conf

Uncomment the following lines to allow for UDP and TCP syslog reception from remote clients.

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

Then add the following likes to define the template that the Rsyslog daemon will use to store incoming logs from client systems.

$template remote-incoming-logs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?remote-incoming-logs

The log files will use the following naming convention:

/%HOSTNAME%/ – This is the hostname of the client system.
/%PROGRAMNAME%/ – This identifies the client program that created the log file.

To apply the changes, restart the rsyslog daemon.

sudo systemctl restart rsyslog

By default, rsyslog listens to port 514. You can confirm that this is the port that the rsyslog daemon is listening to by executing the ss command.

$ sudo ss -tunlp | grep 514
udp    UNCONN  0       0                                         0.0.0.0:514                                         0.0.0.0:*                                   users:(("rsyslogd",pid=263425,fd=7))
udp    UNCONN  0       0                                            [::]:514                                            [::]:*                                   users:(("rsyslogd",pid=263425,fd=8))
tcp    LISTEN  0       25                                        0.0.0.0:514                                         0.0.0.0:*                                   users:(("rsyslogd",pid=263425,fd=9))
tcp    LISTEN  0       25                                           [::]:514                                            [::]:*                                   users:(("rsyslogd",pid=263425,fd=10))

4. Configure firewall rules for rsyslog

If you are running a UFW firewall, be sure to allow port 514 so as to allow incoming incoming log messages.

$ sudo ufw allow 514/tcp
$ sudo ufw allow 514/udp

Then reload the firewall to apply the firewall rule as follows.

$ sudo ufw reload

5. Configure the rsyslog client system

The final step is to configure the client system to send log files to the rsyslog server. Login to the client and once again, ensure that the rsyslog daemon is installed and running.

Next, edit the rsyslog configuration file.

sudo vim /etc/rsyslog.conf

And allow preservation of FQDN: Alternatively, to have rsyslog send with the fully-qualified domain name (FQDN, such as system1.example.com) instead of simply the hostname (system1), use the directive:

$PreserveFQDN on

Navigate to the very end of the file and add these lines.

#Enable sending system logs over UDP to rsyslog server
*.* @rsyslog-ip-address:514

#Enable sending system logs over TCP to rsyslog server
*.* @@rsyslog-ip-address:514

These lines instruct the client to send the log files over both UDP and TCP protocols to the rsyslog server. Note that the first line has a single @ symbol which signifies UDP and the second line has two @@ symbols to signify TCP protocol.

Should the remote server experience any downtime and you want to preserve your logs, you can set the disk queue buffer by appending the lines shown.

Set disk queue when rsyslog server will be down:

$ActionQueueFileName queue
$ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionResumeRetryCount -1

Save and exit the file. Now restart the rsyslog service for the changes to take effect.

sudo systemctl restart rsyslog

At this point, the client system should be sending the log files over to the rsyslog server. Let’s head back to the server and find out.

6. View Client’s log files

All the log files (including the servers) are stored in the /var/log/ directory. To view the client’s logs, run the following command:

$ ls /var/log/

The client’s log files will be saved in a directory that corresponds to the hostname of the client system. For me there is a directory with client hostname

dev-client-ubuntusrv.

You can further view the log files contained in the main log directory.

$ sudo ls -l /var/log/dev-client-ubuntusrv/
total 20
-rw-r--r-- 1 syslog syslog   96 Jan 20 19:58 dhclient.log
-rw-r--r-- 1 syslog syslog 2871 Jan 20 19:59 loki.log
-rw-r--r-- 1 syslog syslog  465 Jan 20 19:58 rsyslogd.log
-rw-r--r-- 1 syslog syslog   99 Jan 20 19:58 sudo.log
-rw-r--r-- 1 syslog syslog  422 Jan 20 19:58 systemd.log

You can tail follow any of the log files to see the content in real time.

Conclusion

We have successfully installed rsyslog and configured it to allow the sending of log files to the rsyslog server from the client system. You can now configure as many client systems as you wish to send logs to rsyslog server in order to achieve a centralized log management structure.