ELK部署-logstash配置（四）

一、logstash简介

简单来说logstash就是一根具备实时数据传输能力的管道，负责将数据信息从管道的输入端传输到管道的输出端；与此同时这根管道还可以让你根据自己的需求在中间加上滤网，Logstash提供里很多功能强大的滤网以满足你的各种应用场景。

二、logstash安装

1、下载logstash

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.16.2-linux-x86_64.tar.gz

2、解压logstash

tar xf logstash-7.16.2-linux-x86_64.tar.gz -C /data/env/ln -s /data/env/logstash-7.16.2 /data/soft/logsta

3、配置文件

mkdir -p /data/soft/logstash/config/conf.d
vim /data/soft/logstash/config/conf.d/nginx.conf

配置文件内容（读取redis数据，写入到es里面）：

input {
    redis {
        host => "127.0.0.1"
        port => "6379"
        db => "0"
        data_type => "list"
        key => "FliebeatTest-nginx-log"
        type => "FliebeatTest-nginx-log"
        #password => "hmh5xc4pJ"
    }
}


output {
    elasticsearch {
        hosts => ["192.168.30.132:9200"]
        index => "fliebeattest-nginx-log-%{+YYYY.MM.dd}"
    }
}

4、配置权限

chown -R www:www data/env/logstash-7.16.2

5、配置启动文件

vim /usr/lib/systemd/system/logstash.service

[Unit]
Description=logstash

[Service]
Type=simple
User=www
Group=www
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
#EnvironmentFile=-/etc/default/logstash
#EnvironmentFile=-/etc/sysconfig/logstash
JAVACMD="/usr/bin/java"
LS_HOME="/data/soft/logstash"
LS_SETTINGS_DIR="/data/soft/logstash"
LS_PIDFILE="/var/run/logstash.pid"
LS_USER="www"
LS_GROUP="www"
LS_GC_LOG_FILE="/data/soft/logstash/gc.log"
LS_OPEN_FILES="16384"
LS_NICE="19"
SERVICE_NAME="logstash"
SERVICE_DESCRIPTION="logstash"
ExecStart=/data/soft/logstash/bin/logstash -f /data/soft/logstash/config/conf.d
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

6、启动

systemctl start logstash

检查日志文件，看是否有报错

tail -f /data/soft/logstash/logs/logstash-plain.log

[root@bogon logstash-7.16.2]# tail -f /data/soft/logstash/logs/logstash-plain.log 
[2022-01-10T17:23:14,076][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.30.132:9200/]}}
[2022-01-10T17:23:14,541][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://192.168.30.132:9200/"}
[2022-01-10T17:23:14,556][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.16.2) {:es_version=>7}
[2022-01-10T17:23:14,570][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-01-10T17:23:15,018][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/data/soft/logstash/config/conf.d/nginx.conf"], :thread=>"#<Thread:0x5bb5ca1d run>"}
[2022-01-10T17:23:15,044][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-01-10T17:23:16,115][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.09}
[2022-01-10T17:23:16,135][INFO ][logstash.inputs.redis    ][main] Registering Redis {:identity=>"redis://@127.0.0.1:6379/0 list:FliebeatTest-nginx-log"}
[2022-01-10T17:23:16,148][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-01-10T17:23:16,280][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

检查无报错之后

访问ES查看是否有索引：http://192.168.30.132:9200/_cat/indices?v

显示如下，fliebeattest-nginx-log就是：

health status index                                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .geoip_databases                          vlsUnSRgRBO6o1GHhIFZZw   1   0         43           49     44.4mb         44.4mb
green  open   .kibana_task_manager_7.16.2_001           cq1SZWLbTHqBURERjYFxfQ   1   0         17         5308    759.7kb        759.7kb
yellow open   fliebeattest-nginx-log-2022.01.10         mFxVqEQ5QC-JjNG6V89GOA   1   1          8            0     20.7kb         20.7kb
green  open   .apm-custom-link                          -zEvghFmTUmLTbrf5-5aaQ   1   0          0            0       226b           226b
green  open   .kibana_7.16.2_001                        ofkk1mpUQ0KWoMfYVdX50A   1   0        285            6      2.4mb          2.4mb
green  open   .apm-agent-configuration                  oDJEVlm-RMaJrwC7L48TBA   1   0          0            0       226b           226b
yellow open   logstash-messages-pakepuser202-2022.01.10 -7R26PASSTOzioMJmr8jrg   1   1         14            0     60.5kb         60.5kb