タイトルまんまですね。
fluentdを導入する目的で、一番欲しかった物がリリースされて本当に
id:tagomorisさん++ です。
プラグインの詳細はこちら。
fluent-plugin-sampling-filter and fluent-plugin-datacounter released! #fluentd
で、実際に試してみた結果がこちら。(ほとんどの設定はサンプルまんまですよ。)
apacheのログフォーマットはこれ。
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D" combined
apacheログを読む為のfluentd configはこれ。
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<source> | |
type tail | |
format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<status>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<response_time>[^ ]*))?$/ | |
time_format %d/%b/%Y:%H:%M:%S %z | |
path /var/log/httpd/access_log_sym | |
tag apache.access | |
pos_file /var/tmp/access.log.pos | |
</source> | |
<source> | |
type tail | |
format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<status>[^1-3][0-9]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<response_time>[^ ]*))?$/ | |
time_format %d/%b/%Y:%H:%M:%S %z | |
path /var/log/httpd/access_log_sym | |
tag apache.error | |
pos_file /var/tmp/error.log.pos | |
</source> | |
<match apache.access> | |
flush_interval 1 | |
type tcp | |
host localhost | |
</match> | |
<match apache.error> | |
flush_interval 1 | |
type tcp | |
host localhost | |
</match> |
受け取ったログでfluent-plugin-datacounterでゴニョゴニョした結果をログに吐く
fluentd configはこれ。
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<source> | |
type tcp | |
</source> | |
#<match apache.access> | |
# type file | |
# path /var/log/td-agent/access_log | |
#</match> | |
#<match apache.error> | |
# type file | |
# path /var/log/td-agent/error_log | |
#</match> | |
<match apache.*> | |
type datacounter | |
unit minute | |
count_key status | |
tag result | |
# patternX: X(1-9) | |
pattern1 2xx ^2\d\d$ | |
pattern2 3xx ^3\d\d$ | |
pattern3 404 ^404$ # we want only 404 counts... | |
pattern4 4xx ^4\d\d$ # pattern4 doesn't matches messages matches pattern[123] | |
pattern5 5xx ^5\d\d$ | |
</match> | |
<match result> | |
type file | |
path /var/log/td-agent/counter.log | |
</match> |
で、テキトーにabで200と404返ってくるように、ばばーっと打ってみた結果がこれ。
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2012-02-23T14:25:41+09:00 result { | |
"apache.error_unmatched_count":0, | |
"apache.error_unmatched_rate":0.0, | |
"apache.error_unmatched_percentage":0.0, | |
"apache.error_2xx_count":0, | |
"apache.error_2xx_rate":0.0, | |
"apache.error_2xx_percentage":0.0, | |
"apache.error_3xx_count":0, | |
"apache.error_3xx_rate":0.0, | |
"apache.error_3xx_percentage":0.0, | |
"apache.error_404_count":1000, | |
"apache.error_404_rate":16.66, | |
"apache.error_404_percentage":100.0, | |
"apache.error_4xx_count":0, | |
"apache.error_4xx_rate":0.0, | |
"apache.error_4xx_percentage":0.0, | |
"apache.error_5xx_count":0, | |
"apache.error_5xx_rate":0.0, | |
"apache.error_5xx_percentage":0.0, | |
"apache.access_unmatched_count":0, | |
"apache.access_unmatched_rate":0.0, | |
"apache.access_unmatched_percentage":0.0, | |
"apache.access_2xx_count":4006, | |
"apache.access_2xx_rate":66.76, | |
"apache.access_2xx_percentage":80.02397123451858, | |
"apache.access_3xx_count":0, | |
"apache.access_3xx_rate":0.0, | |
"apache.access_3xx_percentage":0.0, | |
"apache.access_404_count":1000, | |
"apache.access_404_rate":16.66, | |
"apache.access_404_percentage":19.976028765481423, | |
"apache.access_4xx_count":0, | |
"apache.access_4xx_rate":0.0, | |
"apache.access_4xx_percentage":0.0, | |
"apache.access_5xx_count":0, | |
"apache.access_5xx_rate":0.0, | |
"apache.access_5xx_percentage":0.0 | |
} |
とても分かりやすいですね!
これで、Nagiosやらグラフやらも簡単に書けますし、apache以外にもcountするタグを変更すれば
欲しいログのcountとかがお手軽ですね!!たぶん!まだ試していないけど!
おまけに、本日リリースされたtd-agentの最新版(1.1.2.2)が
dev系のRPMパッケージ依存問題が解決されているので、インストールがお手軽になりましたね!
※まだrubyパッケージは求められますが・・・。
でもrubyだけならrpmをレポジトリから直接取ってきて、
rpm -ivh --nodeps td-agentで無理やり入れても動いたっぽいので、
なんとかなるかも。