LogZillaインストールと設定

オープンソースとしてのログビューワーってあんまりいいものが内が、ようやくまともなsyslogビューワーが出てきたので早速いれてみた。

複数のサーバーからsyslogサーバーにsyslog-ngを用いて集中管理し、それらをビューワーで見るという使い方。

URLに変数などを仕込むことで、任意の期間の任意のホストのログだけを見るってこともできるので、障害時などのアラートとしてURLを送るなどの使い方もできる。

logzillaのインストールと設定系の日本語のドキュメントは全くないので、ここに書いておく。
syslog-ngがインストールされている前提での手順。

ソースダウンロード

# cd /usr/local/src
# wget http://php-syslog-ng.googlecode.com/files/logzilla_3.0.85.tgz

ソース展開

# tar zxvfp logzilla_3.0.85.tgz

必要パッケージのインストール（rpmforgeレポジトリの登録の必要有り）

# yum install httpd mysql mysql-server php.i386 php-devel.i386 php-pear.noarch php-mysql.i386 gcc httpd-devel

# yum –enablerepo=rpmforge install perl-Date-Calc.i386 perl-Text-LevenshteinXS.i386

phpモジュールのインストール

# pecl install json
# pecl install apc

php.iniに設定追加

# vi /etc/php.ini
# 一番下くらいに下記追加
extension=apc.so
extension=json.so

my.cnf設定

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
old_passwords=1
skip-character-set-client-handshake
default-character-set = utf8

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
default-character-set = utf8

[mysqldump]
default-character-set = utf8

[mysql]
default-character-set = utf8

[client]
default-character-set = utf8

MySQL起動

# /etc/init.d/mysqld start

MySQLのrootパス変更

mysql> SET PASSWORD FOR root@localhost=PASSWORD(‘hogehoge’);

logzillaインストールの準備

# cd /usr/local/src
# cp -pr logzilla /usr/local/.
# cd /usr/local/logzilla/scripts

yumでMySQLをいれるとバージョンが古い場合があるので、install.pl内のmysqlのチェックを外す

121 =pod
122 # Check version of MySQL
123 my $sth = $dbh->prepare(“SELECT version()”) or die “Could not create the $dbname database: $DBI::errstr”;
124 $sth->execute;
125 while (my @data = $sth->fetchrow_array()) {
126 my $ver = $data[0];
127 if ($ver !~ /5¥.1/) {
128 print(“¥n¥033[1m¥tERROR!¥n¥033[0m”);
129 print “LogZilla requires MySQL v5.1 or better.¥n”;
130 print “Your version is $ver¥n”;
131 print “Please upgrade MySQL to v5.1 or better and re-run this installation.¥n”;
132 exit;
133 }
134 }
135 =cut

インストールスクリプト実行（標準出力をはりつけとく）

# ./install.pl

========================================

LogZilla Installation

========================================

Enter the MySQL root username [root]: root
Note: Mysql passwords with a ‘ in them may not work
Enter the password for root [mysql]: “hogehoge”
Database to install to [syslog]:
Database table to install to [logs]:
Enter the name of the MySQL server [127.0.0.1]:
Enter the port of the MySQL server [3306]:
Enter the name to create as the owner of the logs database [syslogadmin]:
Note that a password containing ‘ may not work.
Enter the password for the syslogadmin user [syslogadmin]:
Enter the name to create as the WEBSITE owner [admin]:
Enter the password for admin [admin]:
Enter your email address [cdukes@cdukes.com]: matsumoto_r@hoge.com
Enter a name for your website [The home of LogZilla]: Syslog Monitoring Site
Enter the base url for your site (include trailing slash) [/logs/]:
Where should log files be stored? [/var/log/logzilla]:
How long should I keep old logs? (in days) [30]:

========================================

Path Updates

========================================

Getting ready to replace paths in all files with “/usr/local/logzilla”
Ok to continue? [y]: y
Updating file paths
Modifying ../scripts/db_insert.pl
Modifying ../scripts/contrib/daily_text_report/mail_daily_counts.pl
Modifying ../scripts/contrib/genreport/genreport.pl
Modifying ../scripts/contrib/system_configs/logzilla.crontab
Modifying ../scripts/contrib/system_configs/logzilla.apache
Modifying ../scripts/contrib/system_configs/syslog-ng.conf
Modifying ../sphinx/indexer.sh
Modifying ../sphinx/sphinx.conf
Updating log paths
Modifying ../scripts/contrib/system_configs/logzilla.crontab
Modifying ../scripts/contrib/system_configs/logzilla.logrotate

========================================

Database Installation

========================================

All data will be installed into the syslog database
Ok to continue? [y]:
DBD::mysql::st execute failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘PARTITION BY RANGE( TO_DAYS( lo ) ) (
PARTITION p20100810 VALUES LESS TH’ at line 1 at ./install.pl line 322, <STDIN> line 17.
DBD::mysql::st execute failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘EVENT logs_add_partition ON SCHEDULE EVERY 1 DAY STARTS ‘2010-08-10 00:00:00′ ON’ at line 1 at ./install.pl line 331, <STDIN> line 17.
DBD::mysql::st execute failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘EVENT logs_add_archive ON SCHEDULE EVERY 1 DAY STARTS ‘2010-08-10 00:10:00′ ON C’ at line 1 at ./install.pl line 339, <STDIN> line 17.
DBD::mysql::st execute failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘EVENT logs_del_partition ON SCHEDULE EVERY 1 DAY STARTS ‘2010-08-10 00:15:00′ ON’ at line 1 at ./install.pl line 347, <STDIN> line 17.
DBD::mysql::st execute failed: Unknown table ‘partitions’ in information_schema at ./install.pl line 407, <STDIN> line 17.
DBD::mysql::st execute failed: Unknown system variable ‘event_scheduler’ at ./install.pl line 413, <STDIN> line 17.

========================================

Config.php generation

========================================

Generating /usr/local/logzilla/html/config/config.php
Ok to continue? [y]:

========================================

System files

========================================

Adding LogZilla logrotate.d file to /etc/logrotate.d
Ok to continue? [y]:
Where is your syslog-ng.conf file located? [/etc/syslog-ng/syslog-ng.conf]: /tmp/syslog-ng.conf
Unable to locate your syslog-ng.conf file
You will need to manually merge contrib/system_configs/syslog-ng.conf with yours.
LogZilla installation complete…
Note: you may need to enable the MySQL Event Scheduler in your /etc/my.cnf file.
Please visit http://forum.logzilla.info/index.php/topic,71.0.html for more information.
Also, please visit http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.0#UDP_Buffers to learn how to increase your UDP buffer size (otherwise you may drop messages).
Please run /etc/init.d/syslog-ng restart

syslog-ng設定（他のサーバから受け取ったsyslogを、/var/log/DEVICE/以下にホスト名毎にファイル単位で分けつつ、全てのsyslogをDBにつっこむ設定）

@version: 3.0
#Default configuration file for syslog-ng.
#
# For a description of syslog-ng configuration file directives, please read
# the syslog-ng Administrator’s guide at:
#
# http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html
#

options {
dir_perm(0755);
perm(0644);
chain_hostnames(no);
stats_freq(0);
flush_lines(1);
mark_freq(0);
long_hostnames(off);
log_msg_size(8192);
log_fifo_size(16384);
time_reopen(10);
use_dns(yes);
dns_cache(yes);
keep_hostname(yes);
};

# sources
source s_local {
internal();
unix-stream(“/dev/log”);
file(“/proc/kmsg” program_override(“kernel: “));
};

source s_remote { tcp(ip(0.0.0.0) port(514) max-connections(1024)); };
source s_remote_udp { udp(ip(0.0.0.0) port(514)); };

# filter
filter f_secure { facility(authpriv); };
filter f_maillog { facility(mail); };
filter f_cron { facility(cron); };
filter f_emerg { level(emerg); };
filter f_spooler { level(crit..emerg) and facility(uucp, news); };
filter f_local7 { facility(local7); };
filter f_messages { level(info..emerg); };
filter f_syslog { level(debug..emerg); };

# destinations from local
destination d_console { usertty(“root”); };

# destinations from remote
destination d_separate_syslog { file(“/var/log/DEVICE/${HOST}/syslog” owner(“root”) group(“root”) perm(0644) dir_perm(0755) create_dirs(yes)); };
destination d_separate_maillog { file(“/var/log/DEVICE/${HOST}/maillog” owner(“root”) group(“root”) perm(0644) dir_perm(0755) create_dirs(yes)); };
destination d_separate_sercure { file(“/var/log/DEVICE/${HOST}/secure” owner(“root”) group(“root”) perm(0644) dir_perm(0755) create_dirs(yes)); };
destination d_separate_messages { file(“/var/log/DEVICE/${HOST}/messages” owner(“root”) group(“root”) perm(0644) dir_perm(0755) create_dirs(yes)); };
destination d_separate_cron { file(“/var/log/DEVICE/${HOST}/cron” owner(“root”) group(“root”) perm(0644) dir_perm(0755) create_dirs(yes)); };
destination d_separate_spooler { file(“/var/log/DEVICE/${HOST}/spooler” owner(“root”) group(“root”) perm(0644) dir_perm(0755) create_dirs(yes)); };
destination d_separate_bootlog { file(“/var/log/DEVICE/${HOST}/boot.log” owner(“root”) group(“root”) perm(0644) dir_perm(0755) create_dirs(yes)); };
destination d_local { file(“/var/log/syslog”); };

# db insert syslog
destination d_syslog_db {
program(“/usr/local/logzilla/scripts/db_insert.pl”
template(“$HOST¥t$PRI¥t$PROGRAM¥t$MSGONLY¥n”)
template_escape(yes)
);
};

# logging
# for syslog-ng
log { source(s_local); filter(f_emerg); destination(d_console); };
log { source(s_remote); filter(f_maillog); destination(d_separate_maillog); };
log { source(s_local); filter(f_maillog); destination(d_separate_maillog); };
log { source(s_remote); filter(f_syslog); destination(d_separate_syslog); };
log { source(s_local); filter(f_syslog); destination(d_separate_syslog); };
log { source(s_remote); filter(f_secure); destination(d_separate_sercure); };
log { source(s_local); filter(f_secure); destination(d_separate_sercure); };
log { source(s_remote); filter(f_cron); destination(d_separate_cron); };
log { source(s_local); filter(f_cron); destination(d_separate_cron); };
log { source(s_remote); filter(f_spooler); destination(d_separate_spooler); };
log { source(s_local); filter(f_spooler); destination(d_separate_spooler); };
log { source(s_remote); filter(f_local7); destination(d_separate_bootlog); };
log { source(s_local); filter(f_local7); destination(d_separate_bootlog); };
log { source(s_remote); filter(f_messages); destination(d_separate_messages); };
log { source(s_local); filter(f_messages); destination(d_separate_messages); };
log { source(s_local); destination(d_local); };

# for syslogd or switch
log { source(s_remote_udp); filter(f_maillog); destination(d_separate_maillog); };
log { source(s_remote_udp); filter(f_syslog); destination(d_separate_syslog); };
log { source(s_remote_udp); filter(f_secure); destination(d_separate_sercure); };
log { source(s_remote_udp); filter(f_cron); destination(d_separate_cron); };
log { source(s_remote_udp); filter(f_spooler); destination(d_separate_spooler); };
log { source(s_remote_udp); filter(f_local7); destination(d_separate_bootlog); };
log { source(s_remote_udp); filter(f_messages); destination(d_separate_messages); };

# db insert
log { source(s_remote); source(s_local); source(s_remote_udp); destination(d_syslog_db); };

apacheの設定（/etc/httpd/conf.d/以下くらいにlogzilla.confを作っておく）

RewriteEngine on
RewriteRule ^/$ /logs/ [R=301,L]
Alias /logs /usr/local/logzilla/html

<Directory “/usr/local/logzilla/html”>
Options FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
php_value max_execution_time 300
php_value date.timezone Asia/Tokyo
php_value memory_limit 128M
php_value post_max_size 16M
php_value upload_max_filesize 2M
php_value max_input_time 300
php_value mbstring.func_overload 6
</Directory>

ブラウザにアクセスして指示通り設定

32bit選択

local install選択

指示されるままにioncubeインストール（以下のioncube設定参照）

ioncube設定

# wget http://downloads2.ioncube.com/loader_downloads/ioncube_loaders_lin_x86.tar.gz
# tar zxvfp ioncube_loaders_lin_x86.tar.gz
# cp -pr ioncube /usr/local/.
# vi /etc/php.ini
一番下に
zend_extension = /usr/local/ioncube/ioncube_loader_lin_5.1.so

sphinx設定（MySQLインデックスエンジン）

# yum install mysql-devel gcc-c++
# cd /usr/local/logzilla/sphinx/src/
# tar zxvfp sphinx-0.9.9.tar.gz
# cd sphinx-0.9.9
# ./configure –prefix `pwd`/../..
# make && make install
# cd /usr/local/logzilla/sphinx
# ./indexer.sh full

Starting Sphinx Indexer: 2010-08-09 11:30:13

No previous index files found
Creating NEW indexes, this may take a while, so be patient…
Running command: /usr/local/logzilla/sphinx/bin/indexer –config /usr/local/logzilla/sphinx/sphinx.conf idx_logs idx_delta_logs
Sphinx 0.9.9-release (r2117)
Copyright (c) 2001-2009, Andrew Aksyonoff
using config file ‘/usr/local/logzilla/sphinx/sphinx.conf’…
indexing index ‘idx_logs’…
collected 14673 docs, 1.3 MB
sorted 0.2 Mhits, 100.0% done
total 14673 docs, 1340756 bytes
total 0.135 sec, 9891228 bytes/sec, 108247.87 docs/sec
indexing index ‘idx_delta_logs’…
collected 1 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 1 docs, 38 bytes
total 0.003 sec, 12578 bytes/sec, 331.01 docs/sec
total 23 reads, 0.002 sec, 101.6 kb/call avg, 0.0 msec/call avg
total 24 writes, 0.005 sec, 160.3 kb/call avg, 0.2 msec/call avg

Finished Sphinx Indexer: 2010-08-09 11:30:14

sphinx起動

# /usr/local/logzilla/sphinx/bin/searchd

Sphinx 0.9.9-release (r2117)
Copyright (c) 2001-2009, Andrew Aksyonoff
using config file ‘./sphinx.conf’…
listening on 127.0.0.1:9312
listening on all interfaces, port=3312

インデックス解析処理をcronに登録

# crontab -e
30 0 1 * * /usr/local/logzilla/sphinx/indexer.sh full >> /var/log/logzilla/sphinx_indexer.log 2>&1
*/5 * * * * /usr/local/logzilla/sphinx/indexer.sh delta >> /var/log/logzilla/sphinx_indexer.log 2>&1
0 0 * * * /usr/local/logzilla/sphinx/indexer.sh merge >> /var/log/logzilla/sphinx_indexer.log 2>&1

起動時に起動するように設定

# vi /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don’t
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# 下記追加
/usr/local/logzilla/sphinx/bin/searchd -c /usr/local/logzilla/sphinx/sphinx.conf