syslog-ng for collecting remote logs


Results 1 to 10 of 10

Thread: syslog-ng for collecting remote logs

Threaded View

  1. #1
    Join Date
    Jan 2004
    Location
    boston, mass USA
    Posts
    1,878

    syslog-ng for collecting remote logs

    Syslog-ng Configuration

    This is a breakdown of what the syslog-ng.conf file looks like and what it does. I never found a guide like this, so I thought I would post one.

    Syslog-ng takes system logging to the next level. The “normal” syslog daemon leaves a lot to be desired especially in collecting logs from remote servers.

    Syslogd (the original one) does not separate logs into different files, just collects the logs and dumps them all together, which is good sometimes, but mostly bad.

    So here we go:
    As with all good config files this one begins with comments (starting with #) describing the files purpose:

    (NOTE: the actual conf file is all bold in this doc for easy reference. Also in code tags, so the indentation gets preserved -- bwkaz )

    Code:
    # Begin /etc/syslog-ng/syslog-ng.conf
    # Taken from Syslog-ng configuration for Linux from Scratch
    # Re-authored by MLeo
    # Date:  February 2006
    # Version 2 (my own version number)
    
    Now, as with most linux apps, there are global options to define:

    Code:
    options { 	sync (0);
    		time_reopen (10);
    		log_fifo_size (1000);
    		long_hostnames(off);
    # this next one is important as it will build folders based on the remote servers name:
    		use_dns (yes);	
    		use_fqdn (no);
    # this will automatically build the folder structure for incoming logs:
    		create_dirs (yes);
    		keep_hostname (yes);
    };
    There are 4 major sections (besides the above) that make up this file:

    Source : this basically tells syslog-ng what to listen too.
    Destination : this defines the files to send the logs too
    filter : defines how to break the incoming logs into individual reporting services
    log : the log section brings the source, destination and filters together to actually begin the logging.



    the first source is defined as “src” but can be named anything from what I have found

    Code:
    # the unix-stream seems to be just the regular mechanism
    # linux uses to log from services running locally 
    source src {	unix-stream("/dev/log");
    		internal();
    		pipe("/proc/kmsg");
        };
    
    # here we define 2 additional source's (remotetcp and remoteudp)
    # to receive logs from remote machines.  By default, syslog sends logs via UDP
    # and syslog-ng uses TCP, so we define both so we can receive log's from
    # either type of clients.  Again, the name's are whatever you define
    # them as:
    
    source remotetcp { tcp(ip(0.0.0.0) port(514)); };
    source remoteudp { udp(); };
    Now that we have determined what to listen too, lets target where we want the logs to go.

    As you will see, syslog-ng can use system variables ($HOST, $DATE, etc) which makes it incredibly customizable.

    One thing to consider, however, is that if you are troubleshooting an issue across many systems, you would have to look at several different logs if you were to parse all the logs into their own directories. If you dump all logs into one giant file, it would be easy to sort by time, then see a common thread across all your systems. However, that file could become huge and unwieldy after a while.

    The format should be obvious, and you can make the destinations where ever you want.
    In our example we are only separating the auth and authpriv logs into their own directories
    in /var/log/, but all other logs just dump into their respective paths.
    You do not have to create these folders before hand as the global setting will allow this to
    happen on the fly.

    Code:
    # The name (authpriv, auth, syslog, etc) is up to you.  These do not have to match the
    # facility you are trying to log from (or too), but it just makes sense to name them the
    # same.
    
    destination authpriv { file("/var/log/$HOST/authorize.log"); };
    destination auth     { file("/var/log/$HOST/authorize.log"); };
    destination syslog { file("/var/log/syslog.log"); };
    destination cron { file("/var/log/cron.log"); };
    destination daemon { file("/var/log/daemon.log"); };
    destination kernel { file("/var/log/kernel.log"); };
    destination lpr { file("/var/log/lpr.log"); };
    destination user { file("/var/log/user.log"); };
    destination uucp { file("/var/log/uucp.log"); };
    destination mail { file("/var/log/mail.log"); };
    destination news { file("/var/log/news.log"); };
    destination debug { file("/var/log/debug.log"); };
    destination messages { file("/var/log/$HOST/messages.log"); };
    destination everything { file("/var/log/everything.log"); };
    destination console { usertty("root"); };
    destination console_all { file("/dev/tty12"); };
    The filter section is where we define what daemon/process/facility we want to log from.
    The names (f_auth, f_cron, f_kernel) can be named what ever you like. The facility, however, must be the actual process from where the logs are coming from.

    Code:
    # Nothing special here, just like source and destination, name them what you want.
    filter f_auth { facility(auth); };
    filter f_authpriv { facility(auth, authpriv); };
    filter f_syslog { not facility(authpriv, mail); };
    filter f_cron { facility(cron); };
    filter f_daemon { facility(daemon); };
    filter f_kernel { facility(kern); };
    filter f_lpr { facility(lpr); };
    filter f_mail { facility(mail); };
    filter f_news { facility(news); };
    filter f_user { facility(user); };
    filter f_uucp { facility(cron); };
    filter f_news { facility(news); };
    filter f_debug { not facility(auth, authpriv, news, mail); };
    filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); };
    filter f_everything { level(debug..emerg) and not facility(auth, authpriv); };
    filter f_emergency { level(emerg); };
    filter f_info { level(info); };
    filter f_notice { level(notice); };
    filter f_warn { level(warn); };
    filter f_crit { level(crit); };
    filter f_err { level(err); };
    Now, onto the actual execution of the syslog-ng process.

    In the above sections, we have simply defined many variables, and in this section we will execute on them:


    The format is simple:

    log { source(yoursourcehere); filter(yourfilterhere); destination(yourdestinationhere);

    Simple, huh?

    The log process then parses out and piece's together the different variables and starts logging the information.

    As clients connect, syslog-ng will build the directory structure on the fly for you if you have defined $HOST above.

    Code:
    log { source(src); filter(f_authpriv); destination(authpriv); };
    log { source(remotetcp); filter(f_authpriv); destination(authpriv); };
    log { source(remotetcp); filter(f_auth); destination(auth); };
    log { source(remoteudp); filter(f_authpriv); destination(authpriv); };
    log { source(remoteudp); filter(f_auth); destination(auth); };
    log { source(remoteudp); filter(f_messages); destination(messages); };
    log { source(src); filter(f_syslog); destination(syslog); };
    log { source(src); filter(f_cron); destination(cron); };
    log { source(src); filter(f_daemon); destination(daemon); };
    log { source(src); filter(f_kernel); destination(kernel); };
    log { source(src); filter(f_lpr); destination(lpr); };
    log { source(src); filter(f_mail); destination(mail); };
    log { source(src); filter(f_news); destination(news); };
    log { source(src); filter(f_user); destination(user); };
    log { source(src); filter(f_uucp); destination(uucp); };
    log { source(src); filter(f_debug); destination(debug); };
    log { source(src); filter(f_messages); destination(messages); };
    log { source(src); filter(f_emergency); destination(console); };
    log { source(src); filter(f_everything); destination(everything); };
    log { source(src); destination(console_all); };
    
    # END /etc/syslog-ng/syslog-ng.conf
    A few things to remember:

    Its a good idea to shutoff syslogd permanently since we don't want to get confused later on.

    Also, make sure you set syslog-ng to start on boot for you particular distro.

    Finally, lets look at a simple client config to send its log files to our central log server:
    To add the ability to any syslog client to send its logs to a remote server, simply add a line including what to log and instead of listing a local file, list @remotelogserver.

    Of course, it is a good idea to also log files locally, but you can have your syslog log the same message to multiple places.

    This is just the modified piece of the client config file:

    Code:
    authpriv.*                                                  /var/log/secure
    auth.*;authpriv.*                                       @yourremote.log.server
    As you see, the authpriv logs to /var/log/secure, but also sends it's info to yourremote.log.server as well.

    However, when it arrives at yourremote.log.server, it adheres to its path for logging. In our case:

    log { source(remoteudp); filter(f_authpriv); destination(authpriv); };

    which will show up in /var/log/$HOST/authorize.log on our log server.

    get it?


    Disclaimer: I am no expert on this. I just happened to get this to work after a few hours of tweaks. Some things might work if configured differently. For instance:

    source remotetcp { tcp(ip() port(514)); };
    Might work just as well as:
    source remotetcp { tcp(ip(0.0.0.0) port(514)); };

    I just haven't gone back to test each line of changes I had to make to get this to work.

    I was just at a loss for a break down of the file so I thought I'd share what I had found.
    Last edited by bwkaz; 02-10-2006 at 11:41 AM. Reason: Added code tags to preserve indentation

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •