Centralize and Monitor Application and System Logs

No comments

Centralize and Monitor Application and System Logs

It becomes a new challenge for organization to make sense of the millions of log lines. In addition, log management became more critical to monitor system and application performance and security risk. On the other hand, log management may be very time consuming with traditional methods.

If you need Enterprise solution, there are too many options that you can integrate to your platform. Please check this link for more information. Open-Source software Rsyslog, Elasticsearch, Logstash and Kibana provide same functions that you can transport, transmit, store and visualize systems and application logs.

In this post we will perform these tasks:

  • Install and configure Syslog Server
  • Use Logstash to format logs
  • Use Elasticsearch to manage logs
  • Use Kibana to visualize logs

Prerequisites:

  • Linux Operating System (Ubuntu, Fedora, Centos, RHEL etc.)
  • Client server which forward application or system logs
  • Syslog Server to collect client forwarded logs
  • Download ELK binaries from this link

We will perform ELK installation on Syslog Server. But you should check your configuration if you plan to deploy installation on production. You may separate syslog server and elk server on production.

Step 1: Download and Install ELK Binaries

Before start installation, you need to download binaries from this link. When you click download button then you need to select your Linux base binary file. For Debian base Linux server choose DEB file. Centos, Fedora, RHEL use RPM base package.

a.Elasticsearch Installation

Check this link for “how to install and configure elasticsearch?

b.Logstash and Kibana Installation

You can perform default installation steps for Kibana and logstash. Download binaries from this link then use “rpm” for fedora base and “dpkg” for debian base command to install application.

Fedora Base Linux Server:

Debian Base Linux Server:

c. Configure Logstash and Kibana

I added simple base configuration for Kibana and logstash. You should check manual page to find out which attributes you need and how to use.

Kibana:

Server Port: 5601,  we will connect Kibana dashboard from this port.You can change it as you wish

Server Host: Define host information to open sockets. You should define your ELK server ip address.

Elastichsearch URL: Define your elastichsearch server’s ip address and port.

Logstash:

I did not change any configuration on logstash YML file. We only need to define our logstash log format to manage apache and syslog logs.

I added an example of logstash configuration for Apache logs and syslogs. After define configuration you should restart logstash. Logstash will be open TCP 6000 port and capture incoming logs. Then it will forward the formatted logs to elastichsarch.

Step 2: Configure Syslog server and client

First, I will explain how to configure syslog server to redirect incoming logs to the logstash, which we explained above (Port: 6000)

This package must be installed on your server. Please check if you have already installed rsyslog package to your system both client and server.

After package installation then define a module which redirect all incoming logs to the logstash deamon.

The facilities local0 to local7 are “custom” unused facilities that syslog provides for the user. If a developer create an application and wants to make it log to syslog, or if you want to redirect the output of anything to syslog (for example, Apache logs), you can choose to send it to any of the local# facilities. Then, you can use /etc/syslog.conf (or /etc/rsyslog.conf) to save the logs being sent to that local# to a file, or to send it to a remote server. More

At this example, we used local6 to redirect the output of application logs on client site. We perform installation ELK and rsyslog server at the same machine. Therefore, we defined “127.0.0.1”.

Rsyslog Server Side Configuration:

Rsyslog Client Side Configuration:

First, create rsyslogd modules to collect application logs.

Last step, define rsyslog server to send application logs.

After follow installation steps, open up Kibana in browser with “http://<kibana_Ip_address>:5601” . You will be presented with the Kibana home page. From management tab you will be able to manage your Elasticsearch indices.

kibana_dash2

Follow me

Abdurrahim

I'm a System Engineer with extensive experience and administration skills and works for Interbank Card Center Of Turkey.I provide hardware and software support for the following Unix/Linux and Windows platforms.(Oracle Solaris,HP-UX, Linux, IBM-AIX, Windows Servers)
Follow me

Latest posts by Abdurrahim (see all)

facebooktwittergoogle_pluslinkedinby feather

No comments yet.

You must be logged in to post a comment.