linuxea:ELK5.5-nginx访问日志grok切割(filebeat)


监控nginx访问日志
filebeat+redis+logstash
filebeat收集日志后传给redis,logstash读取redis后grok后存储

I. 安装filebeat

[root@linuxea.com-Node117 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.5.1-x86_64.rpm
[root@linuxea.com-Node117 ~]# yum install filebeat-5.5.1-x86_64.rpm -y

II. 传递给redis

配置文件如下

[root@linuxea.com-Node117 /etc/filebeat]# cat filebeat.yml 
filebeat.prospectors:
 - input_type: log
   paths:
    - /data/logs/access_nginx.log
   document_type: nginx-access117
output.redis:
  hosts: ["10.10.0.98"]
  password: "OTdmOWI4ZTM4NTY1M2M4OTZh"
  key: "default_list"
  db: 1
  timeout: 5
  keys:
    - key: "%{[type]}"
      mapping:
      "nginx-access117": "nginx-access117"

启动程序

[root@linuxea.com-Node117 /etc/filebeat]# systemctl restart filebeat
[root@linuxea.com-Node117 /etc/filebeat]# tail -f /var/log/filebeat/filebeat
2017-08-25T20:53:09+08:00 INFO States Loaded from registrar: 11
2017-08-25T20:53:09+08:00 INFO Loading Prospectors: 1
2017-08-25T20:53:09+08:00 INFO Prospector with previous states loaded: 1
2017-08-25T20:53:09+08:00 WARN DEPRECATED: document_type is deprecated. Use fields instead.
2017-08-25T20:53:09+08:00 INFO Starting prospector of type: log; id: 12123466383741208858 
2017-08-25T20:53:09+08:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-08-25T20:53:09+08:00 INFO Metrics logging every 30s
2017-08-25T20:53:09+08:00 INFO Starting Registrar
2017-08-25T20:53:09+08:00 INFO Start sending events to output
2017-08-25T20:53:09+08:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-08-25T20:53:29+08:00 INFO Harvester started for file: /data/logs/access_nginx.log
2017-08-25T20:53:39+08:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.publisher.published_events=243 libbeat.redis.publish.read_bytes=1367 libbeat.redis.publish.write_bytes=126046 publish.events=245 registrar.states.current=11 registrar.states.update=245 registrar.writes=2
2017-08-25T20:54:09+08:00 INFO No non-zero metrics in the last 30s

III. redis查看

但启动后,写入access_nginx.log日志后就会写到redis,这个时候如果没有被拿走是可以看到的,如下

[root@linuxea.com-Node98 ~]# redis-cli -h 10.10.0.98 -a OTdmOWI4ZTM4NTY1M2M4OTZh
10.10.0.98:6379> select 1
OK
10.10.0.98:6379[1]> keys *
1) "nginx-access117"
10.10.0.98:6379[1]> type "nginx-access117"
list
10.10.0.98:6379[1]> lrange nginx-access117 0 -1
  1) "{\"@timestamp\":\"2017-08-25T12:53:29.279Z\",\"beat\":{\"hostname\":\"linuxea.com-Node117.cluster.com\",\"name\":\"linuxea.com-Node117.cluster.com\",\"version\":\"5.5.1\"},\"input_type\":\"log\",\"message\":\"10.10.0.96 - - [25/Aug/2017:12:53:21 +0000] GET / HTTP/1.1 - 304 0 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36 -\",\"offset\":48321607,\"source\":\"/data/logs/access_nginx.log\",\"type\":\"nginx-access117\"}"

IV. 创建patterns目录和文件

现看下具体格式有哪些:

nginx日志格式如下:

log_format upstream2  '$proxy_add_x_forwarded_for $remote_user [$time_local] "$request" $http_host'
        '[$body_bytes_sent] $request_body "$http_referer" "$http_user_agent" [$ssl_protocol] [$ssl_cipher]'
        '[$request_time] [$status] [$upstream_status] [$upstream_response_time] [$upstream_addr]';

这个是在logstash机器上创建patterns.d目录存放grok格式

[root@linuxea.com-Node49 /etc/logstash/conf.d]# mkdir /etc/logstash/patterns.d/ -p

把patterns写到文件

[root@linuxea.com-Node49 /etc/logstash/conf.d]# cat /etc/logstash/patterns.d/nginx 
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IP:clent_ip} (?:-|%{USER:ident}) \[%{HTTPDATE:log_date}\] \"%{WORD:http_verb} (?:%{PATH:baseurl}\?%{NOTSPACE:params}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" (%{IPORHOST:url_domain}|%{URIHOST:ur_domain}|-)\[(%{BASE16FLOAT:request_time}|-)\] %{NOTSPACE:request_body} %{QS:referrer_rul} %{GREEDYDATA:User_Agent} \[%{GREEDYDATA:ssl_protocol}\] \[(?:%{GREEDYDATA:ssl_cipher}|-)\]\[%{NUMBER:time_duration}\] \[%{NUMBER:http_status_code}\] \[(%{BASE10NUM:upstream_status}|-)\] \[(%{NUMBER:upstream_response_time}|-)\] \[(%{URIHOST:upstream_addr}|-)\]

但是在安装完成kibana后,在dev tools中有grok debugger,如果日志格式不同,增减后直接simulate测试即可,如下图:

写进elasticsearch配置如下
其中用了GeoLite2-City.mmdb,但是发现没有什么卵用
下载地址:https://dev.maxmind.com/zh-hans/geoip/geoip2/geolite2-%E5%BC%80%E6%BA%90%E6%95%B0%E6%8D%AE%E5%BA%93/
或者就用自带的,注释database即可

[root@linuxea-Node49 /etc/logstash/conf.d]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
[root@linuxea.com-Node49 /etc/logstash]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100%   
-> Installed ingest-user-agent
[root@linuxea.com-Node49 /etc/logstash]# 

V. input从redis取数据发送给elasticsearch

[root@linuxea.com-Node49 /etc/logstash/conf.d]# cat /etc/logstash/conf.d/redis_input.conf 
input { 
    redis {
        host => "10.10.0.98"
        port => "6379"
        key => "nginx-access117"
        data_type => "list"
        password => "OTdmOWI4ZTM4NTY1M2M4OTZh"
        threads => 10
        db => "1"
    }
}
filter {
   if [type] == "nginx-access-117" {
    grok {
        patterns_dir => [ "/etc/logstash/patterns.d" ]
        match => { "message" => "%{NGINXACCESS}" }
        overwrite => [ "message" ]
        }
    geoip {
        source => "clent_ip"
        target => "geoip"
#        database => "/etc/logstash/GeoLiteCity.dat"
        database => "/etc/logstash/GeoLite2-City.mmdb"
         }
    useragent {
        source => "User_Agent"
        target => "userAgent"
        }
    urldecode {
        all_fields => true
        }
     mutate {
            gsub => ["User_Agent","[\"]",""]        #将user_agent中的 " 换成空
            convert => [ "response","integer" ]
            convert => [ "body_bytes_sent","integer" ]
            convert => [ "bytes_sent","integer" ]
            convert => [ "upstream_response_time","float" ]
            convert => [ "upstream_status","integer" ]
            convert => [ "request_time","float" ]
            convert => [ "port","integer" ]
       }
    date {
    match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
        }
        }
    }
output {
    if [type] == "nginx-access117" {
    elasticsearch {
        hosts => ["10.0.1.49:9200"]
        index => "logstash-nginx-access-117-%{+YYYY.MM.dd}"
        user => "elastic"
        password => "linuxea"
    }
    }
    stdout {codec => rubydebug}
}

VI. 最后几步

在启动logstash时候可以观察下日志:

打开kibana,在management-->create即可,输入logstash-nginx-access-117-*,如下图:

当日志写入,字段会grok,在kibana上表现这样

ok,基本上日志切割完成

2 分享

您可以选择一种方式赞助本站

支付宝扫码赞助

支付宝扫码赞助

日期: 2017-09-06分类: ELK Stack

标签: elk

发表评论