Table of contents

This is for internal use by the PaaS team. Public-facing documentation is located at docs.cloud.service.gov.uk.

Investigating rsyslog issues

rsyslog is “the rocket-fast system for log processing”. It’s an application that runs on all of our VMs that receives log messages, performs some processing, and forwards the messages to our logging system (currently logit).

We have occasionally suspected rsyslog (or other parts of our logging pipeline) of causing reliability issues. This documentation is intended to help operators diagnose issues with rsyslog more quickly.

Suspected memory usage issues

On 2018-07-31 the production router virtual machines exhausted their available memory and swap. Initial investigation suggested that the rsyslogd process on these machines was using far more than the expected amount of memory.

We investigated rsyslog’s behaviour in situations where it can’t ship logs to its destination in #159559834, but weren’t able to reproduce the issue. There’s extensive documentation of our investigation on that ticket.

Investigating memory usage issues

If there are suspected issues with rsyslog’s memory usage in future there are a few things you should look into:

In Kibana, search for @source.component: "rsyslogd-pstats" - this should show if rsyslog has full queues / is dropping messages. (See #159559834 for more details)

Use the bosh-cli to ssh onto a router, then:

# May show rsyslog-pstats messages that didn't make it to Kibana
tail -20000 /var/log/syslog | grep rsyslog-pstats

# How much CPU / memory are the rsyslog processes using?
ps uax | awk '/rsyslo[g]/ || NR==1'

# How much overall disk usage is there?
df -h | grep ^/

# How much overall memory usage is there?
free -o -m

It may also be useful to capture the traffic that rsyslog is sending to logit to investigate possible packet loss etc.

sudo tcpdump host "$(grep -o '[^@]*logit.io' /etc/rsyslog.d/35-syslog-release-forwarding-rules.conf)" -G 120 -W 1 -i eth0 -w logit-traffic.pcap