logstash6.5.4 解析 nginx 日志格式报错

[sakurazensen]

日志格式如下：
log_format elk '{"time_local":"$time_iso8601",'
'"remote_addr":"$remote_addr",'
'"referer":"$http_referer",'
'"request":"$request",'
'"status":$status,'
'"bytes":$body_bytes_sent,'
'"agent":"$http_user_agent",'
'"x_forwarded":"$http_x_forwarded_for",'
'"up_addr":"$upstream_addr",'
'"up_host":"$upstream_http_host",'
'"reqeust_time":"$request_time"'

日志如下：
{"time_local":"2020-05-16T12:43:48+08:00","remote_addr":"192.168.5.148","referer":"-","request":"GET / HTTP/1.1","status":304,"bytes":0,"agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36","x_forwarded":"-","up_addr":"-","up_host":"-","reqeust_time":"0.000"}

如果只是单纯解析时间，是没问题
input {
file {
path => "/var/log/nginx/access.elk.log"
}
}
filter {
grok {
match => [ "message","%{TIMESTAMP_ISO8601:locals}" ]
}
}
output {
stdout { codec => rubydebug }
}
结果：
"@version" => "1",
"host" => "localhost.localdomain",
"path" => "/var/log/nginx/access.elk.log",
"message" => "{\"time_local\":\"2020-05-16T12:43:48+08:00\",\"remote_addr\":\"192.168.5.148\",\"referer\":\"-\",\"request\":\"GET / HTTP/1.1\",\"status\":304,\"bytes\":0,\"agent\":\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36\",\"x_forwarded\":\"-\",\"up_addr\":\"-\",\"up_host\":\"-\",\"reqeust_time\":\"0.000\"}",
"locals" => "2020-05-16T12:43:48+08:00",
"@timestamp" => 2020-05-16T04:43:49.638Z
}

如果和解析 IP 一起使用，就报错：
input {
file {
path => "/var/log/nginx/access.elk.log"
}
}
filter {
grok {
match => [ "message","%{IP:client} %{TIMESTAMP_ISO8601:locals}" ]
}
}
output {
stdout { codec => rubydebug }
}
结果：
{
"@version" => "1",
"host" => "localhost.localdomain",
"path" => "/var/log/nginx/access.elk.log",
"message" => "{\"time_local\":\"2020-05-16T12:50:00+08:00\",\"remote_addr\":\"192.168.5.148\",\"referer\":\"-\",\"request\":\"GET / HTTP/1.1\",\"status\":304,\"bytes\":0,\"agent\":\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36\",\"x_forwarded\":\"-\",\"up_addr\":\"-\",\"up_host\":\"-\",\"reqeust_time\":\"0.000\"}",
"tags" => [
[0] "_grokparsefailure"
],
"@timestamp" => 2020-05-16T04:50:01.476Z
}

有大神了解这是为什么吗，只要是和解析时间的表达式一起用，就报错。%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"一起用就没问题

[37Y37]

直接记录 json 的应该不用 grok 解析，可以参考下这个 https://blog.ops-coffee.cn/s/cyuls7uczvwgzwptzox0dg

[polaa]

emmm 写 logstash 的时候经常出现奇怪的问题


这种直接 codec= json 解析就行 不用 grok