使用logstash提交日志到Elasticsearch,logstash和Elasticsearch都没有报错,为什么日志提交不到Elasticsearch,是什么原因,如何解决?以下是logstash的配置文件和需要提交的日志。
input {
file {
path => ["/data/www/adDataSync/es/es*.log"]
codec => "json"
type => "es"
max_open_files => 65535
#start_position => "beginning"
}
}
#时区跨8小时问题#
filter {
ruby {
code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*3600)"
}
ruby {
code => "event.set('@timestamp', event.get('timestamp'))"
}
mutate {
# remove_field => ["host"]
remove_field => ["timestamp"]
}
}
output {
if [type] == "es" {
elasticsearch {
hosts => ["172.18.10.10:9200"]
index => "lele-adDataSync-%{+yyyy.MM}"
codec => rubydebug
#document_type => "logs"
}
}
}
{"level":"INFO","ts":"2023-07-27 19:26:00","file":"huds/dayo:53","msg":"华始","type":"huawei","param":{},"result":{},"status":"success"}
{"level":"INFO","ts":"2023-07-27 19:26:00","file":"te/dayRe161","msg":"腾账号","type":"ten","param":{"date":"2023-07-27"},"result":{"number":235},"status":"success"}
{"level":"INFO","ts":"2023-07-27 19:26:00","file":"to/dayRe135","msg":"头账户","type":"tou","param":{"date":"2023-07-27"},"result":{"number":88},"status":"success"}
{"level":"INFO","ts":"2023-07-27 19:26:00","file":"huds/dayo:64","msg":"华账号","type":"hua","param":{"date":"2023-07-27"},"result":{"number":3},"status":"success"}
{"level":"INFO","ts":"2023-07-27 19:26:07","file":"huds/dayo:91","msg":"华成","type":"华为","par":{"date":"2023-07-27"},"result":{"number":3},"status":"success"}
hosts路径带上http://试下
不知道你这个问题是否已经解决, 如果还没有解决的话:在之前的配置中,我们配置了Logstash输出到控制台,现在我们让它输出到Elasticsearch集群。
编辑pipeline.conf文件,替换output区域为:
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "lo-1212"
}
}
在这段配置中,Logstash用http协议连接到Elasticsearch,而且假设Logstash和Elasticsearch允许在同一台机器上。你也可以指定一个远程的Elasticsearch实例,比如host=>[“es-machine:9092”]
说明:
hosts: Elasticsearch地址
index: Elasticsearch索引名
重启后就可以在Elasticsearch中看到对应的索引数据