Shipping your Nginx logs to Elasticsearch using Logstash


First of all create a new log format in /etc/nginx/nginx.conf and add/replace an access log directive:

log_format logstash '$http_host '
                    '$remote_addr [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    '$request_time '
                    '$upstream_response_time';
access_log /var/log/nginx/access.log logstash;

Install Elasticsearch like described here.
Install Logstash like described here.

Add the following pattern to logstash in /opt/logstash/paggerns/nginx:

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} %{NUMBER:upstream_time:float}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float}

Update your /etc/logstash/logstash.conf like so:

input {
  file {
    path => "/var/log/nginx/*access*"
  }
}
filter {
  mutate { replace => { "type" => "nginx_access" } }
  grok {
    match => { "message" => "%{NGINXACCESS}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
  }
  geoip {
    source => "clientip"
  }
}
output {
  elasticsearch {
    host => "localhost"
    protocol => "http"
  }
  stdout { codec => rubydebug }
}

Finally restart all services:

service nginx restart
service elasticsearch restart
service logstash restart