Rsyslog is an open-source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IP network. It is an open-source utility for log processing. It permits the logging of data from different types of systems in a central repository. Rsyslog is a Syslog protocol with more extensions, features, and benefits.
Rsyslog can deliver over one million messages per second to local destinations when limited processing is applied making it so powerful. By default, Linux uses the syslog daemon to record logs about how the system is running and then putting these logs into a central repository in: /var/log where we can examine and investigate them.
In simple terms of meaning everything from kernel events to user actions is logged by Linux, allowing you to see al most any action performed on your pc or servers.
Logging is really beneficial. The log information can really help you to see what is exactly happened on your system, or what exactly is going on under the hood. For example if you have an application that crashes unexpectedly, the messages in the logs can help identify the cause.
In this tutorial we learn how to install and configure rsyslog on Ubuntu 20.04.
Directory where different sorts of logs stored are:
- /var/log/syslog – Stores all startup messages, application startup messages etc. Practically stores all global system logs.
- /var/log/cron – The Cron jobs are basically kind of scheduled and automated task created in the system, that runs periodically and repeatedly. You can see what this logs directory would store.
- /var/log/kern.log – it stores kernel logs. No matter what logs they are. Event logs, errors, or warning logs.
- /var/log/auth.log – Authentication logs
- /var/log.boot.log – System boot logs
- /var/log/mysql.d – Mysql logs
- /var/log/httpd – Apache logs directory
- /var/log/maillog – Mail server logs
- Two Ubuntu servers one to be set up as a rsyslog server (10.2.11.10 in my case) and a client (10.2.11.11).
Table of Content
- Ensure that the server is up to date
- Ensure that the Rsyslog is installed
- Configuring Rsyslog
- Configure firewall rules for Rsyslog
- Configure the rsyslog client system
- View Client’s log files
1. Ensure that the server is installed
Before proceeding ensure that the system has updated packages. Use this command:
sudo apt update sudo apt upgrade -y
2. Ensure that Rsyslog is installed
Rsyslog package comes pre installed in Ubuntu. Use this command to ensure it is present:
sudo apt install rsyslog
Confirm the version with this command:
$ rsyslogd -v rsyslogd 8.2001.0 (aka 2020.01) compiled with: PLATFORM: x86_64-pc-linux-gnu PLATFORM (lsb_release -d): FEATURE_REGEXP: Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No 32bit Atomic operations supported: Yes 64bit Atomic operations supported: Yes memory allocator: system default Runtime Instrumentation (slow code): No uuid support: Yes systemd support: Yes Config file: /etc/rsyslog.conf PID file: /run/rsyslogd.pid Number of Bits in RainerScript integers: 64 See https://www.rsyslog.com for more information.
Please check that the service is up and running using this command:
$ sudo systemctl status rsyslog ● rsyslog.service - System Logging Service Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2022-01-19 13:03:07 UTC; 1 day 6h ago TriggeredBy: ● syslog.socket Docs: man:rsyslogd(8) https://www.rsyslog.com/doc/ Main PID: 70335 (rsyslogd) Tasks: 4 (limit: 4624) Memory: 3.3M CGroup: /system.slice/rsyslog.service └─70335 /usr/sbin/rsyslogd -n -iNONE Jan 19 13:03:07 dev-ubuntusrv.inv.re systemd: Starting System Logging Service... Jan 19 13:03:07 dev-ubuntusrv.inv.re rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2001.0] Jan 19 13:03:07 dev-ubuntusrv.inv.re systemd: Started System Logging Service. Jan 19 13:03:07 dev-ubuntusrv.inv.re rsyslogd: rsyslogd's groupid changed to 110 Jan 19 13:03:07 dev-ubuntusrv.inv.re rsyslogd: rsyslogd's userid changed to 104 Jan 19 13:03:07 dev-ubuntusrv.inv.re rsyslogd: [origin software="rsyslogd" swVersion="8.2001.0" x-pid="70335" x-info="https://www.rsyslog.com"] start Jan 20 00:00:01 adoring-shannon.15-237-143-120.plesk.page rsyslogd: [origin software="rsyslogd" swVersion="8.2001.0" x-pid="70335" x-info="https://www.rsyslog.com"]
3. Configuring Rsyslog
Let us configure rsyslog to run in server mode. Rsyslog config file is located in this file path
/etc/rsyslog.conf . Open it with your prefered text editor:
sudo vim /etc/rsyslog.conf
Uncomment the following lines to allow for UDP and TCP syslog reception from remote clients.
# provides UDP syslog reception module(load="imudp") input(type="imudp" port="514") # provides TCP syslog reception module(load="imtcp") input(type="imtcp" port="514")
Then add the following likes to define the template that the Rsyslog daemon will use to store incoming logs from client systems.
$template remote-incoming-logs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" *.* ?remote-incoming-logs
The log files will use the following naming convention:
- /%HOSTNAME%/ – This is the hostname of the client system.
- /%PROGRAMNAME%/ – This identifies the client program that created the log file.
To apply the changes, restart the rsyslog daemon.
sudo systemctl restart rsyslog
By default, rsyslog listens to port 514. You can confirm that this is the port that the rsyslog daemon is listening to by executing the ss command.
$ sudo ss -tunlp | grep 514 udp UNCONN 0 0 0.0.0.0:514 0.0.0.0:* users:(("rsyslogd",pid=263425,fd=7)) udp UNCONN 0 0 [::]:514 [::]:* users:(("rsyslogd",pid=263425,fd=8)) tcp LISTEN 0 25 0.0.0.0:514 0.0.0.0:* users:(("rsyslogd",pid=263425,fd=9)) tcp LISTEN 0 25 [::]:514 [::]:* users:(("rsyslogd",pid=263425,fd=10))
4. Configure firewall rules for rsyslog
If you are running a UFW firewall, be sure to allow port 514 so as to allow incoming incoming log messages.
$ sudo ufw allow 514/tcp $ sudo ufw allow 514/udp
Then reload the firewall to apply the firewall rule as follows.
$ sudo ufw reload
5. Configure the rsyslog client system
The final step is to configure the client system to send log files to the rsyslog server. Login to the client and once again, ensure that the rsyslog daemon is installed and running.
Next, edit the rsyslog configuration file.
sudo vim /etc/rsyslog.conf
And allow preservation of FQDN: Alternatively, to have rsyslog send with the fully-qualified domain name (FQDN, such as system1.example.com) instead of simply the hostname (system1), use the directive:
Navigate to the very end of the file and add these lines.
#Enable sending system logs over UDP to rsyslog server *.* @rsyslog-ip-address:514 #Enable sending system logs over TCP to rsyslog server *.* @@rsyslog-ip-address:514
These lines instruct the client to send the log files over both UDP and TCP protocols to the rsyslog server. Note that the first line has a single @ symbol which signifies UDP and the second line has two @@ symbols to signify TCP protocol.
Should the remote server experience any downtime and you want to preserve your logs, you can set the disk queue buffer by appending the lines shown.
Set disk queue when rsyslog server will be down:
$ActionQueueFileName queue $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionQueueType LinkedList $ActionResumeRetryCount -1
Save and exit the file. Now restart the rsyslog service for the changes to take effect.
sudo systemctl restart rsyslog
At this point, the client system should be sending the log files over to the rsyslog server. Let’s head back to the server and find out.
6. View Client’s log files
All the log files (including the servers) are stored in the /var/log/ directory. To view the client’s logs, run the following command:
$ ls /var/log/
The client’s log files will be saved in a directory that corresponds to the hostname of the client system. For me there is a directory with client hostname
You can further view the log files contained in the main log directory.
$ sudo ls -l /var/log/dev-client-ubuntusrv/ total 20 -rw-r--r-- 1 syslog syslog 96 Jan 20 19:58 dhclient.log -rw-r--r-- 1 syslog syslog 2871 Jan 20 19:59 loki.log -rw-r--r-- 1 syslog syslog 465 Jan 20 19:58 rsyslogd.log -rw-r--r-- 1 syslog syslog 99 Jan 20 19:58 sudo.log -rw-r--r-- 1 syslog syslog 422 Jan 20 19:58 systemd.log
You can tail follow any of the log files to see the content in real time.
We have successfully installed rsyslog and configured it to allow the sending of log files to the rsyslog server from the client system. You can now configure as many client systems as you wish to send logs to rsyslog server in order to achieve a centralized log management structure.