Cdtjkyj

rsyslog server template consideration for multiple remote hosts
---> link to previously answered question

@ meuh, I find this post very useful as am currently working on this configuration.

I have done the steps which is mentioned above and its working fine. thanks for that

My case is, I have ELK setup were rsyslog forwards the logs to it.

My templates are,

$template
templmesg,"/data01/RemoteLogs/DLF/%$YEAR%/%$MONTH%/%HOSTNAME%/%HOSTNAME%-%$DAY%-%$MONTH%-%$YEAR%.log"

$template mylogsec,"/data01/RemoteLogs/Logserver/%$YEAR%/%$MONTH%/%HOSTNAME%/%HOSTNAME%-%$DAY%-%$MONTH%-%$YEAR%.log"

if $fromhost startswith "10.100.10" then ?templmesg

& stop

if $fromhost startswith "10.100.112" then ?mylogsec

& stop

so I have two locations were logs are stored

Because of multiple locations of log storages like DLF and Logserver. Kibana from (ELK) does not show logs which is received from rsyslog. It only reads from one location of logs that is from DLF/ dir and not from Logserver

Now am stuck and dont know "How to forward rsyslog logs from multiple locations to ELK and make it show in kibana?" or "is there any specific configuration in rsyslog that i need to work out?"

below is the rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html

# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html



#### MODULES ####



# The imjournal module bellow is now used as a message source instead of imuxsock.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)

$ModLoad imjournal # provides access to the systemd journal

$ModLoad imklog # reads kernel messages (the same are read from journald)

#$ModLoad immark  # provides --MARK-- message capability



# Provides UDP syslog reception

$ModLoad imudp

$UDPServerRun 514



# Provides TCP syslog reception

$ModLoad imtcp

$InputTCPServerRun 514





#### GLOBAL DIRECTIVES ####



# Where to place auxiliary files

$WorkDirectory /var/lib/rsyslog



# Use default timestamp format

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat



# File syncing capability is disabled by default. This feature is usually not required,

# not useful and an extreme performance hit

#$ActionFileEnableSync on



# Include all config files in /etc/rsyslog.d/

$IncludeConfig /etc/rsyslog.d/*.conf



# Turn off message reception via local log socket;

# local messages are retrieved through imjournal now.

$OmitLocalLogging on



# File to store the position in the journal

$IMJournalStateFile imjournal.state





#### RULES ####



# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.*                                                 /dev/console



$template templmesg,"/data01/RemoteLogs/DLF/%$YEAR%/%$MONTH%/%HOSTNAME%/%HOSTNAME%-%$DAY%-%$MONTH%-%$YEAR%.log"

$template mylogsec,"/data01/RemoteLogs/DLF/Logserver/%$YEAR%/%$MONTH%/%HOSTNAME%/%HOSTNAME%-%$DAY%-%$MONTH%-%$YEAR%.log"



#if $fromhost startswith "10.100.10" then ?templmesg 

#& stop

if $fromhost startswith "10.100.112" then ?mylogsec 

& stop



local0.*                                                        ?templmesg

local1.*                                                        ?templmesg

local2.*                                                        ?templmesg

local3.*                                                        ?templmesg

local4.*                                                        ?templmesg

local5.*                                                        ?templmesg

local6.*                                                        ?templmesg





template(name="json-template"

  type="list") {

   constant(value="{")

      constant(value=""@timestamp":"")     property(name="timereported" dateFormat="rfc3339")

      constant(value="","@version":"1")

      constant(value="","message":"")     property(name="msg" format="json")

      constant(value="","sysloghost":"")  property(name="hostname")

      constant(value="","severity":"")    property(name="syslogseverity-text")

      constant(value="","facility":"")    property(name="syslogfacility-text")

      constant(value="","programname":"") property(name="programname")

      constant(value="","procid":"")      property(name="procid")

    constant(value=""}n")

}





# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

#

#$createDirs on



*.info;mail.none;authpriv.none;cron.none;local0.none;local1.none;local2.none;local3.none;local4.none;local5.none;local6.none              ?templmesg



# The authpriv file has restricted access.

authpriv.*                                              ?templmesg



# Log all the mail messages in one place.

mail.*                                                  ?templmesg





# Log cron stuff

cron.*                                                  ?templmesg



# Everybody gets emergency messages

#*.emerg                                                 :omusrmsg:*



# Save news errors of level crit and higher in a special file.

uucp,news.crit                                          /var/log/spooler



# Save boot messages also to boot.log

local7.*                                                ?templmesg





# ### begin forwarding rule ###

# The statement between the begin ... end define a SINGLE forwarding

# rule. They belong together, do NOT split them. If you create multiple

# forwarding rules, duplicate the whole block!

# Remote Logging (we use TCP for reliable delivery)

#

# An on-disk queue is created for this action. If the remote host is

# down, messages are spooled to disk and sent when it is up again.

#$ActionQueueFileName fwdRule1 # unique name prefix for spool files

#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)

#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown

#$ActionQueueType LinkedList   # run asynchronously

#$ActionResumeRetryCount -1    # infinite retries if host is down

# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional

*.* @10.100.10.30:10514;json-template

# ### end of the forwarding rule ###