IIS-Logstash conf 的設定說明

Polin Chen

Published in

elkplus

11 min readMay 29, 2017

原圖來源 http://www.thedevlog.com/wp/en/2015/09/elasticsearch-elk-stack-helping-software-quality/

IIS log 的logstash grok 範例說明：

IIS log 欄位說明：

date: 2010–07–30
time: 01:06:43
c-ip: 192.168.0.102
cs-username: -
s-ip: 192.168.0.102
s-port: 80
cs-method: GET
cs-uri-stem: /css/rss.xslt
cs-uri-query: -
sc-status: 304
sc-win32-status : 0
sc-bytes: 140
cs-bytes: 358
time-taken: 0
cs-version: HTTP/1.1
cs-host: www.microsoft.com
cs(User-Agent): Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+Trident/4.0;+InfoPath.2;+360SE)
cs(Referer): -

grok 的解析說明：

利用grok debug 解析log 的格式和內容

#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status sc-win32-status sc-bytes cs-bytes time-taken cs-version cs-host cs(User-Agent) cs(Referer)
2010-07-30 01:06:43 192.168.0.102 - 192.168.0.102 80 GET /css/rss.xslt - 304 0 140 358 0 HTTP/1.1 www.mvpboss1004.com Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+Trident/4.0;+InfoPath.2;+360SE) -%{TIMESTAMP_ISO8601:@timestamp} %{IP:c_ip} %{NOTSPACE:cs_username} %{IP:s_ip} %{NUMBER:s_port} %{WORD:cs_method} %{URIPATH:cs_uri_stem} %{NOTSPACE:cs_uri_query} %{NUMBER:sc_status} %{NUMBER:sc_winstatus} %{NUMBER:sc_bytes} %{NUMBER:cs_bytes} %{NUMBER:time_taken} %{NOTSPACE:cs_version} %{NOTSPACE:cs_host} %{NOTSPACE:cs_useragent} %{NOTSPACE:cs_referer}

IIS log 的logstash 的config 設定方式：

IIS 預設的log 解析方式：

#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version   sc-status sc-substatus sc-win32-status   time-taken
#2017-04-12 00:00:01 W3SVC1 Web07-IIS1 10.140.45.187 GET / - 80 - 10.140.45.162 HTTP/0.9 200 0 0 0%{TIMESTAMP_ISO8601:@timestamp} %{NOTSPACE:s_sitename} %{NOTSPACE:s_computername} %{IP:s_ip} %{WORD:cs_method} %{URIPATH:cs_uri_stem} %{NOTSPACE:cs_uri_query} %{WORD:s_port} %{NOTSPACE:cs_username} %{IP:c_ip} %{NOTSPACE:cs_version} %{NOTSPACE:cs_status} %{NOTSPACE:cs_substatus} %{NOTSPACE:sc_win32_status} %{NUMBER:time_taken}

解析執行步驟

在grok debugger 中測試成功

修改/etc/logstash/conf.d/logstash-elkplus-web07.conf

input {
    file {
        path => "/data/iislog/elkplus/web07/u_ex*"
        start_position => beginning
        type =>"iislog-web07"
    }
}
filter {
 #ignore log comments
  if [message] =~ "^#" {
    drop {}
  }  grok {
break_on_match => false#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version   sc-status sc-substatus sc-win32-status 
time-taken
#2017-04-12 00:00:01 W3SVC1 Web07-IIS1 10.140.45.187 GET / - 80 - 10.140.45.162 HTTP/0.9   200 0 0   0    match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{NOTSPACE:s_sitename} %{NOTSPACE:s_computername} %{IP:s_ip} %{WORD:cs_method} %{URIPATH:cs_uri_stem} %{NOTSPACE:cs_uri_query} %{WORD:s_port} %{NOTSPACE:cs_username} %{IP:c_ip} %{NOTSPACE:cs_version} %{NOTSPACE:cs_status} %{NOTSPACE:cs_substatus} %{NOTSPACE:sc_win32_status} %{NUMBER:time_taken}"]
  }  #Set the Event Timesteamp from the log
        date {
      # +8 for next day
          timezone => "Etc/UTC"
      match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
          #timezone => "Asia/Taipei"
  }
}
output {
#  elasticsearch {
    #template_overwrite => true
  #hosts => ["10.1.1.96:9200", "10.1.1.97:9200"]
    #index => "logstash-iis-web07-%{+YYYY.MM.dd}"
  #}
  stdout { codec => rubydebug }
}

測試conf 文件是否正確

# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-elkplus-web07.conf --configtest
Configuration OK

測試是否解析成功，在螢幕上顯示，不寫到ES 上

#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-elkplus-web07.conf -w 1Settings: User set pipeline workers: 1, Default pipeline workers: 4
Pipeline main started

畫面上顯示的結果範本

{
            "message" => "2017-04-12 00:18:46 W3SVC1 Web07-IIS1 10.140.45.187 GET / - 80 - 10.140.45.162 HTTP/0.9 200 0 0 0\r",
           "@version" => "1",
         "@timestamp" => "2017-04-12T00:18:46.000Z",
               "path" => "/data/iislog/elkplus/web07/u_ex170411.log",
               "host" => "ELKServer03",
               "type" => "iislog-web07",
      "log_timestamp" => "2017-04-12 00:18:46",
         "s_sitename" => "W3SVC1",
     "s_computername" => "Web07-IIS1",
               "s_ip" => "10.140.45.187",
          "cs_method" => "GET",
        "cs_uri_stem" => "/",
       "cs_uri_query" => "-",
             "s_port" => "80",
        "cs_username" => "-",
               "c_ip" => "10.140.45.162",
         "cs_version" => "HTTP/0.9",
          "cs_status" => "200",
       "cs_substatus" => "0",
    "sc_win32_status" => "0",
         "time_taken" => "0"
}

正式寫到ES 中，同時顯示在畫面中debug

第一次的測試過程，建議只要執行測試log，如日誌的前100 行，看是否正確

output {
  elasticsearch {
    template_overwrite => true
    hosts => ["10.1.1.96:9200", "10.1.1.97:9200"]
    index => "logstash-elkplus-web07-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

檢查ES 的index 是否產生

# curl '10.1.1.96:9200/_cat/indices?v' | grep logstash-elkplus-web07  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54057  100 54057   green  open   logstash-elkplus-web07-2017.04.12       5   1        496            0    587.7kb        323.6kb
 0     0  12290      0  0:00:04  0:00:04 --:--:-- 12294

刪除測試過程產生的index

#curl -XDELETE 10.1.1.96:9200/logstash-elkplus-web07-2017.04.12
{"acknowledged":true}

常見問題

時間差的設定
時間格式
geoip 的設定
number 轉換為integer
將bytes 轉為KBytes
增加欄位
刪除欄位

時間差的設定問題

在grok 增加timezone 的設定

date {
        match => ["timestamp", "yyyy-MM-dd HH:mm:ss"]
        remove_field => [ "timestamp" ]
        timezone => "Etc/UTC"
    }

將number 轉換為integer

將grok 中設定為number ，利用mutate 更改為integer，以便在kibana 中可以計算

mutate {
    convert => { "time_taken" => "integer" }
    convert => { "sc_bytes" => "integer" }
    convert => { "cs_bytes" => "integer" }
  }

將IP 轉換為geoip

將grok 中設定為IP，轉換為geoIP ，以便分析IP 的相關資訊

geoip {
    source => "c_ip"
    target => "geoip"
    add_tag => [ "nginx-geoip" ]
    }