ELK6.5.0+Filebeat ๆฅๅฟ็ณป็ป้จ็ฝฒ
ๅๅฎขไฝ่
๏ผ่็ณป่ฏท็นๅป๏ผๆฌ่ฟไธๆ๏ผๅธๆ่ฏทไฝ่
ๅๅๅก๏ผๅฏไปฅ็นๅป่็ณปๅๅฎขไฝ่
1. ๅผ่จ
Elasticsearchๆฏๅฎๆถๅ จๆๆ็ดขๅๅๆๅผๆ๏ผๆไพๆ้ใๅๆใๅญๅจๆฐๆฎไธๅคงๅ่ฝ๏ผๆฏไธๅฅๅผๆพRESTๅJAVA API็ญ็ปๆๆไพ้ซๆๆ็ดขๅ่ฝ๏ผๅฏๆฉๅฑ็ๅๅธๅผ็ณป็ปใๅฎๆๅปบไบApache Luceneๆ็ดขๅผๆๅบไนไธใ
Logstashๆฏไธไธช็จๆฅๆ้ใๅๆใ่ฟๆปคๆฅๅฟ็ๅทฅๅ ทใๅฎๆฏๆๅ ไนไปปไฝ็ฑปๅ็ๆฅๅฟ๏ผๅ ๆฌ็ณป็ปๆฅๅฟใ้่ฏฏๆฅๅฟๅ่ชๅฎไนๅบ็จ็จๅบๆฅๅฟใๅฎๅฏไปฅไป่ฎธๅคๆฅๆบๆฅๆถๆฅๅฟ๏ผ่ฟไบๆฅๆบๅ ๆฌ syslogใๆถๆฏไผ ้๏ผไพๅฆ RabbitMQ๏ผๅJMX๏ผๅฎ่ฝๅคไปฅๅค็งๆนๅผ่พๅบๆฐๆฎ๏ผๅ ๆฌ็ตๅญ้ฎไปถใwebsocketsๅElasticsearchใ
KibanaๆฏไธไธชๅบไบWeb็ๅพๅฝข็้ข๏ผ็จไบๆ็ดขใๅๆๅๅฏ่งๅๅญๅจๅจ Elasticsearchๆๆ ไธญ็ๆฅๅฟๆฐๆฎใๅฎๅฉ็จElasticsearch็RESTๆฅๅฃๆฅๆฃ็ดขๆฐๆฎ๏ผไธไป ๅ ่ฎธ็จๆทๅๅปบไปไปฌ่ชๅทฑ็ๆฐๆฎ็ๅฎๅถไปช่กจๆฟ่งๅพ๏ผ่ฟๅ ่ฎธไปไปฌไปฅ็นๆฎ็ๆนๅผๆฅ่ฏขๅ่ฟๆปคๆฐๆฎใ
Filebeatๆฏๆฌๅฐๆไปถ็ๆฅๅฟๆฐๆฎ้้ๅจใ ไฝไธบๆๅกๅจไธ็ไปฃ็ๅฎ่ฃ ๏ผFilebeat็่งๆฅๅฟ็ฎๅฝๆ็นๅฎๆฅๅฟๆไปถ๏ผtail file๏ผๅนถๅฐๅฎไปฌ่ฝฌๅ็ปElasticsearchๆLogstash่ฟ่ก็ดขๅผใkafka ็ญใ
1.1 ็ผๅ็ฎ็
ELK ไธๆฏไธๆฌพ่ฝฏไปถ๏ผ่ๆฏ ElasticsearchใLogstash ๅ Kibana ไธ็ง่ฝฏไปถไบงๅ็้ฆๅญๆฏ็ผฉๅใ่ฟไธ่ ้ฝๆฏๅผๆบ่ฝฏไปถ๏ผ้ๅธธ้ ๅไฝฟ็จ๏ผ่ไธๅๅ ๅๅฝไบ Elastic.co ๅ ฌๅธๅไธ๏ผๆไปฅ่ขซ็ฎ็งฐไธบ ELK Stackใๆ นๆฎ Google Trend ็ไฟกๆฏๆพ็คบ๏ผELK Stack ๅทฒ็ปๆไธบ็ฎๅๆๆต่ก็้ไธญๅผๆฅๅฟ่งฃๅณๆนๆกใ
2. ้จ็ฝฒๅฏน่ฑก่ฏดๆ
2.1 ็จๅบๅ็งฐ
Elasticsearch๏ผๅๅธๅผๆ็ดขๅๅๆๅผๆ๏ผๅ ทๆ้ซๅฏไผธ็ผฉใ้ซๅฏ้ ๅๆ็ฎก็็ญ็น็นใๅบไบ Apache Lucene ๆๅปบ๏ผ่ฝๅฏนๅคงๅฎน้็ๆฐๆฎ่ฟ่กๆฅ่ฟๅฎๆถ็ๅญๅจใๆ็ดขๅๅๆๆไฝใ้ๅธธ่ขซ็จไฝๆไบๅบ็จ็ๅบ็กๆ็ดขๅผๆ๏ผไฝฟๅ ถๅ ทๆๅคๆ็ๆ็ดขๅ่ฝ๏ผ
Logstash๏ผๆฐๆฎๆถ้ๅผๆใๅฎๆฏๆๅจๆ็ไปๅ็งๆฐๆฎๆบๆ้ๆฐๆฎ๏ผๅนถๅฏนๆฐๆฎ่ฟ่ก่ฟๆปคใๅๆใไธฐๅฏใ็ปไธๆ ผๅผ็ญๆไฝ๏ผ็ถๅๅญๅจๅฐ็จๆทๆๅฎ็ไฝ็ฝฎ๏ผ
Kibana๏ผๆฐๆฎๅๆๅๅฏ่งๅๅนณๅฐใ้ๅธธไธ Elasticsearch ้ ๅไฝฟ็จ๏ผๅฏนๅ ถไธญๆฐๆฎ่ฟ่กๆ็ดขใๅๆๅไปฅ็ป่ฎกๅพ่กจ็ๆนๅผๅฑ็คบ๏ผ
Filebeat๏ผELK ๅ่ฎฎๆ ็ๆฐๆๅ๏ผไธไธช่ฝป้็บงๅผๆบๆฅๅฟๆไปถๆฐๆฎๆ้ๅจ๏ผๅบไบ Logstash-Forwarder ๆบไปฃ็ ๅผๅ๏ผๆฏๅฏนๅฎ็ๆฟไปฃใๅจ้่ฆ้้ๆฅๅฟๆฐๆฎ็ server ไธๅฎ่ฃ Filebeat๏ผๅนถๆๅฎๆฅๅฟ็ฎๅฝๆๆฅๅฟๆไปถๅ๏ผFilebeat ๅฐฑ่ฝ่ฏปๅๆฐๆฎ๏ผ่ฟ ้5.Logstash ่ฟ่ก่งฃๆ๏ผไบฆๆ็ดๆฅๅ้ๅฐ Elasticsearch ่ฟ่ก้ไธญๅผๅญๅจๅๅๆใ
ๅฆๆๆจๅฏน ELK Stack ่ฟๅฐไธไบ่งฃ๏ผๆๆฏๆณไบ่งฃๆดๅค๏ผ่ฏท็นๅป้ไธญๅผๆฅๅฟ็ณป็ป ELK ๅ่ฎฎๆ ่ฏฆ่งฃ๏ผๆฅ็ๅ ทไฝไป็ปใ
2.2 ็จๅบๅ่ฝ
Elasticsearch๏ผๅๅธๅผ็ๆ็ดขๅผๆๅๆฐๆฎๅๆๅผๆ
Logstash๏ผๆฐๆฎๆถ้ๅผๆ
Kibana๏ผๆฐๆฎๅๆๅๅฏ่งๅๅนณๅฐ
Filebeat๏ผ่ฝป้ๅๆฅๅฟ้้ๅจ
3. ้จ็ฝฒๆไฝ
3.1 ็ฏๅขๅๅค
ELK5.3ไปฅๅไพ่ตJDK๏ผๆไปฅๆไปฌๅ ๅจๆๅกๅจๅฎ่ฃ ไพ่ต ๏ผJDK็ๆฌ1.8.0ๅไปฅไธ๏ผ
OpenJDKไธ่ฝฝ็ฝ็ซ๏ผ
OpenJDKๅฎ่ฃ ๆๆกฃ็ฝ็ซ๏ผ
yum -y install java-1.8.0-openjdk wget lrzsz vim net-tools coreutils npm
yum groupinstall 'Development Tools'ๆฃๆฅไธไธJAVA็ฏๅขๅ็ๆฌ
java -versionๅ
ณ้ญ้ฒ็ซๅขๅ้
็ฝฎSELinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
setenforce 0 #ๅฏไปฅ่ฎพ็ฝฎ้
็ฝฎๆไปถๆฐธไน
ๅ
ณ้ญ
systemctl stop firewalld.serviceELKๅฎๆนไธ่ฝฝ็ฝ็ซ๏ผ
Elasticsearchไธ่ฝฝ๏ผ
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.0.tar.gzLogstashไธ่ฝฝ๏ผ
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.0.zipKibanaไธ่ฝฝ๏ผ
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.5.0-linux-x86_64.tar.gzFilebeatไธ่ฝฝ๏ผ
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.0-linux-x86_64.tar.gz3.2 ๅฎ่ฃ
ๆญฅ้ชค
ๅฎ่ฃ
elasticsearch
tar -zvxf elasticsearch-6.5.0.tar.gz -C /usr/local/
vim /usr/local/elasticsearch-6.5.0/config/elasticsearch.yml
17 cluster.name: elk-cluster #้็พคๅ็งฐ
23 node.name: node-1 #่็นๅ็งฐ
33 path.data: /usr/local/elasticsearch-6.5.0/data #่ฎพ็ฝฎ็ดขๅผๆฐๆฎ็ๅญๅจ่ทฏๅพ๏ผ้ป่ฎคๆฏesๆ น็ฎๅฝไธ็dataๆไปถๅคน๏ผๅฏไปฅ่ฎพ็ฝฎๅคไธชๅญๅจ่ทฏๅพ๏ผ็จ้ๅท้ๅผ
37 path.logs: /usr/local/elasticsearch-6.5.0/logs #่ฎพ็ฝฎๆฅๅฟๆไปถ็ๅญๅจ่ทฏๅพ๏ผ้ป่ฎคๆฏesๆ น็ฎๅฝไธ็logsๆไปถๅคน
55 network.host: 10.88.216.26 #ๅฝๅ่็นIPๅฐๅ
59 http.port: 9200 #่ฎพ็ฝฎๅฏนๅคๆๅก็http็ซฏๅฃ๏ผ้ป่ฎคไธบ9200ใ
68 discovery.seed_hosts: ["10.88.216.17","10.88.216.19", "10.88.216.22"] #้็พคๆฏไธช่็นIPๅฐๅ๏ผไนๅฏไปฅไฝฟ็จelsใels.shuaiguoxia.com็ญๅ็งฐ๏ผ้่ฆๅ่็น่ฝๅค่งฃๆ
72 cluster.initial_master_nodes: ["node-1", "node-2","node-3"] #ๅๅงไธป่็นๅบ็ฑๅ
ถๆ ่ฏ ่็นๅ็งฐ๏ผ้ป่ฎคไธบๅ
ถไธปๆบๅใ็กฎไฟๅผcluster.initial_master_nodesไธ่็นๅ็งฐ็กฎๅๅน้
ใ
80 gateway.recover_after_nodes: 3 #่ฎพ็ฝฎ้็พคไธญNไธช่็นๅฏๅจๆถ่ฟ่กๆฐๆฎๆขๅค๏ผ้ป่ฎคไธบ3ใๅ
ทไฝ่ฏฆ็ป้
็ฝฎ้
่ฏปๅฎๆนๆๆกฃไบ่งฃๆดๅค๏ผ
mkdir -p /usr/local/elasticsearch-6.5.0/dataไฟฎๆนsysctlๅ
ๆ ธๅๆฐ
echo "vm.max_map_count=262144" >> /etc/sysctl.conf && sysctl -p
vim /etc/security/limits.conf ๏ผๆทปๅ ไธ้ขๅๆฐ๏ผ
* soft nofile 65536
* hard nofile 655363.3 ็จๅบ่ฟ่กๆๅ็ถๆๆฃๆฅ
ESๆไปถ็ไฝฟ็จ
headๆไปถ(ๆฅ็,ๅบ็บงๅซ็ๆไฝ)
ๅฎ่ฃ
node
็ฑไบheadๆไปถๆฌ่ดจไธ่ฟๆฏไธไธชnode.js็ๅทฅ็จ๏ผๅ ๆญค้่ฆๅฎ่ฃ node๏ผไฝฟ็จnpmๆฅๅฎ่ฃ ไพ่ต็ๅ ใ๏ผnpmๅฏไปฅ็่งฃไธบmaven๏ผ
ๅฎ็ฝไธ่ฝฝๅฏนๅบ็ๆฌ็node:
wget http://nodejs.org/dist/v0.10.24/node-v0.10.24.tar.gzๅฎ่ฃ ็ผ่ฏ่ฝฏไปถ
yum gcc gcc-c++ kernel-devel make็ถๅ่งฃๅnode็ๅฎ่ฃ ๅ
tar -xvf node-v0.10.24.tar.gz -C /usr/local/
cd /usr/local/node-v0.10.24/ๆบ็ ็ผ่ฏๅฎ่ฃ
./configure && make && make installๆฅ็ๆฏๅฆๅฎ่ฃ ๆๅ๏ผ-vๆฅ็็ๆฌ
node -v
ไฝฟ็จไธ้ข็ๅฝไปคๅๆถssl้ช่ฏ
npm config set strict-ssl falseๅฆ้้่ฏฏerror-code-ELIFECYCLE๏ผๆง่กๅฆไธๅฝไปค
npm cache clear --force
npm install -g npmไธ่ฝฝelasticsearch-headๆไปถ
headไธ่ฝฝๅฐๅ๏ผhttps://github.com/mobz/elasticsearch-head
git clone git://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head
npm install #ๅฎ่ฃ
ไพ่ตไฟฎๆนๅฏๅจๆไปถ
ๆๆไพ่ตๅ ๅฎ่ฃ ๆๅๅ๏ผไฟฎๆน elasticsearch-head ็ฎๅฝไธ็ Gruntfile.js ๆไปถ๏ผๅจ options ๅฑๆงๅ ๅขๅ hostname๏ผ่ฎพ็ฝฎไธบ 0.0.0.0ใ
vim Gruntfile.js
ไฟฎๆน Elasticsearch ้ ็ฝฎๆไปถ config/elasticsearch.ymlๆฏๆ่ทจๅ่ฎฟ้ฎ
ๅ ทไฝ้ ็ฝฎๅ่ๅฎๆนๆๆกฃ
vim /usr/local/elasticsearch-6.5.0/config/elasticsearch.yml ๏ผๅจๆซ่กๆทปๅ ๅฆไธไธคๆก้
็ฝฎ๏ผ
89 http.cors.enabled: true
90 http.cors.allow-origin: '*'
Elasticsearch่ช5็ๆฌไนๅ๏ผๅคไบๅฎๅ จ่่๏ผไธๅ ่ฎธไฝฟ็จroot็จๆท่ฟ่กใ
่งฃๅณ๏ผๅๅปบไธไธชๆฎ้็จๆท๏ผๅฐelasticsearch ๅฎ่ฃ ็ฎๅฝๆ้ไฟฎๆนไธไธ๏ผๅๆข่ณๆฎ้็จๆท่ฟ่กelasticsearchๅฐฑๅฏไปฅไบ
useradd elasticsearch
mv elasticsearch-head /usr/local/
chown -R elasticsearch:elasticsearch /usr/local/elasticsearch-head
chown -R elasticsearch:elasticsearch /usr/local/elasticsearch-6.5.0ๅๆข็จๆทๅฏๅจๆๅก
su โ elasticsearch
nohup /usr/local/elasticsearch-6.5.0/bin/elasticsearch & #ๅฟฝ็ฅ่พๅ
ฅๅนถๆ่พๅบ่ฟฝๅ ๅฐ"nohup.outโ
cd /usr/local/elasticsearch-head/
nohup npm run start &ๆฅ็ๆๅกๅฏๅจ็ซฏๅฃ
ss -ntulp | grep -E "9100|9200|9300"
่ฎฟ้ฎ http://10.88.216.26:9100 ๅฐๅ๏ผๅฐฑๅฏไปฅ็ๅฐๅฝๅ Elasticsearch ้็พคไฟกๆฏ
Cereboๆฏkopfๅจes5ไธ็ๆฟไปฃ่
(้็พคๆไฝ)
Cerebroไธ่ฝฝ็ฝ็ซ๏ผ
wget https://github.com/lmenezes/cerebro/releases/download/v0.8.4/cerebro-0.8.4.tgz
tar -zvxf cerebro-0.8.4.tgz -C /usr/local/
cd /usr/local/cerebro-0.8.4/
vim /usr/local/cerebro-0.8.4/conf/application.conf
nohup ./bin/cerebro -Dhttp.port=[ๅฎไนๅฏๅจ็็ซฏๅฃ] -Dhttp.address=[ๅฎไนๅฏๅจIP] &
ๆฅ็ๅฏๅจๅนถ่ฎฟ้ฎ่็น
Bigdeskๆไปถ(้็พค็ถๆๆฅ็)
wget https://github.com/hlstudio/bigdesk/archive/master.zip
unzip master.zip -d /usr/local/
cd /usr/local/bigdesk-master/_site/ๅฏๅจwebๆๅกๅจ
python -m SimpleHTTPServer #้ป่ฎค็ๅฌ็ซฏๅฃๅทๆฏ 8000
nohup python -m SimpleHTTPServer 8900 & #ๆๅฎๅฏๅจ็ซฏๅฃ,ๅนถๅๅฐๅฏๅจๆฅ็ๅฏๅจๅนถ่ฎฟ้ฎ่็น
่ฎฟ้ฎๅนถ่ฟๆฅ่็นๆฅ็
่ฎฟ้ฎhttp://10.88.216.26:8900ๅณๅฏ่ฟๅ ฅ็ๆง้กต้ข
Beatsๅฏไปฅ็ดๆฅ๏ผๆ่ ้่ฟLogstash๏ผๅฐๆฐๆฎๅ้ๅฐElasticsearch๏ผๅจ้ฃ้ไฝ ๅฏไปฅ่ฟไธๆญฅๅค็ๅๅขๅผบๆฐๆฎ๏ผ็ถๅๅจKibanaไธญๅฐๅ ถๅฏ่งๅใ
Filebeatๅฎ่ฃ ้จ็ฝฒ
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.0-linux-x86_64.tar.gz
tar -zvxf filebeat-6.5.0-linux-x86_64.tar.gz -C /usr/local/
cd /usr/local/filebeat-6.5.0-linux-x86_64/้ ็ฝฎFilebeat
ๅฎๆนๅ่ๆๆกฃ๏ผ
vim /usr/local/filebeat-6.5.0-linux-x86_64/filebeat.ymlLogstashๅฎ่ฃ
้จ็ฝฒ
Logstash็ๆฐๆฎๅค็่ฟ็จไธป่ฆๅ ๆฌ๏ผ่พๅ ฅ๏ผ่ฟๆปคๅจ๏ผ่พๅบไธ้จๅ๏ผๅฆๅคๅจ่พๅ ฅๅ่พๅบไธญๅฏไปฅไฝฟ็จCodecsๅฏนๆฐๆฎๆ ผๅผ่ฟ่กๅค็ใ่ฟๅไธช้จๅๅไปฅๆไปถๅฝขๅผๅญๅจ๏ผ็จๆท้่ฟๅฎไน็ฎก้้ ็ฝฎๆไปถ๏ผ่ฎพ็ฝฎ้่ฆไฝฟ็จ็่พๅ ฅ๏ผ่ฟๆปคๅจ๏ผ่พๅบ๏ผ็ผ่งฃ็ ๅจๆไปถ๏ผไปฅๅฎ็ฐ็นๅฎ็ๆฐๆฎ้้๏ผๆฐๆฎๅค็๏ผๆฐๆฎ่พๅบ็ญๅ่ฝ
๏ผ1๏ผ่พๅ ฅ๏ผ็จไบไปๆฐๆฎๆบ่ทๅๆฐๆฎ๏ผๅธธ่ง็ๆไปถๅฆfile๏ผsyslog๏ผredis๏ผbeats็ญ[ ่ฏฆ็ปๅ่ ]
๏ผ2๏ผ่ฟๆปคๅจ๏ผ็จไบๅค็ๆฐๆฎๅฆๆ ผๅผ่ฝฌๆข๏ผๆฐๆฎๆดพ็็ญ๏ผๅธธ่ง็ๆไปถๅฆgrok๏ผmutate๏ผdrop๏ผclone๏ผgeoip็ญ[ ่ฏฆ็ปๅ่ ]
๏ผ3๏ผ่พๅบ๏ผ็จไบๆฐๆฎ่พๅบ๏ผๅธธ่ง็ๆไปถๅฆelastcisearch๏ผfile๏ผgraphite๏ผstatsd็ญ[ ่ฏฆ็ปๅ่ ]
๏ผ4๏ผ็ผ่งฃ็ ๅจ๏ผ็ผ่งฃ็ ๅจไธๆฏไธไธชๅ็ฌ็ๆต็จ๏ผ่ๆฏๅจ่พๅ ฅๅ่พๅบ็ญๆไปถไธญ็จไบๆฐๆฎ่ฝฌๆข็ๆจกๅ๏ผ็จไบๅฏนๆฐๆฎ่ฟ่ก็ผ็ ๅค็๏ผๅธธ่ง็ๆไปถๅฆjson๏ผmultiline [ ่ฏฆ็ปๅ่ ]
Logstashไธ่ฝฝ๏ผ๏ผ้่ฆ้ขๅ
ๅฎ่ฃ
JDK8๏ผ
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.0.zip
unzip logstash-6.5.0.zip -d /usr/local/
cd /usr/local/logstash-6.5.0/
#็ฎๅๆต่ฏ
# bin/logstash -e 'input { stdin { } } output { stdout {} }'logstash้ ็ฝฎ่ฏญๅฅ่ฏฆ่งฃ
logstash้ ็ฝฎๆไปถๅ ๅซไธไธช้ ็ฝฎ้จๅ๏ผๅๅซไธบ๏ผinput{}ใfilter{}ใoutput{}ใ
{} ๅฎไนๅบๅ๏ผๅบๅๅ ๅฏไปฅๅฎไนไธไธชๆๅคไธชๆไปถ๏ผ้่ฟๆไปถๅฏนๆฐๆฎ่ฟ่กๆถ้๏ผๅ ๅทฅๅค็๏ผ่พๅบใ
logstash.yml
ๅ ๅซLogstash้ ็ฝฎๆ ๅฟ๏ผไฝ ๅฏไปฅๅจ่ฟไธชๆไปถไธญ่ฎพ็ฝฎๆ ๅฟ๏ผ่ไธๆฏๅจๅฝไปค่กไธญไผ ้ๆ ๅฟ๏ผๅจๅฝไปค่กไธญ่ฎพ็ฝฎ็ไปปไฝๆ ๅฟ้ฝ่ฆ็logstash.ymlๆไปถไธญ็็ธๅบ่ฎพ็ฝฎ๏ผๆดๅคไฟกๆฏ่งlogstash.ymlๅฎๆนๅ่ๆๆกฃใ
Logstashๆฒกๆ้ป่ฎค็้ ็ฝฎๆไปถ,้่ฆๆๅจ้
ๆฐๆฎ็ฑปๅ๏ผ
ๅธๅฐๅผ็ฑปๅ: ssl_enable => true
ๅญ่็ฑปๅ: bytes => "1MiB"
ๅญ็ฌฆไธฒ็ฑปๅ: name => "xkops"
ๆฐๅผ็ฑปๅ: port => 22
ๆฐ็ป: match => ["datetime","UNIX"]
ๅๅธ: options => {key1 => "value1",key2 => "value2"}
็ผ็ ่งฃ็ : codec => "json"
่ทฏๅพ: file_path => "/tmp/filename"
ๆณจ้: #
ๆกไปถๅคๆญ๏ผ
็ญไบ: ==
ไธ็ญไบ: !=
ๅฐไบ: <
ๅคงไบ: >
ๅฐไบ็ญไบ: <=
ๅคงไบ็ญไบ: >=
ๅน้ ๆญฃๅ: =~
ไธๅน้ ๆญฃๅ: !~
ๅ ๅซ: in
ไธๅ ๅซ: not in
ไธ: and
ๆ: or
้ไธ: nand
้ๆ: xor
ๅคๅ่กจ่พพๅผ: ()
ๅๅ็ฌฆๅ: !()
ๅๅปบlogstash.conf้
็ฝฎๆไปถ๏ผ
mkdir /usr/local/logstash-6.5.0/config.d
vim /usr/local/logstash-6.5.0/config.d/logstash.conf
input {
beats {
port => 5044 #ๅฎไนๆๅก็ๅฌ็็ซฏๅฃ
}
}
filter {
if [fields][logtype] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
if [fields][logtype] == "spring-boot-log4j2" {
json {
source => "message"
target => "data"
}
}
}
output {
if [fields][logtype] == "spring-boot-log4j2"{
elasticsearch {
hosts => ["10.88.216.26:9200"]
index => "spring-boot-log4j2-%{+YYYY.MM.dd}"
}
}
if [fields][logtype] == "syslog"{
elasticsearch {
hosts => ["10.88.216.26:9200"]
index => "filebeat-%{+YYYY.MM.dd}"
}
}
}ๆฃๆฅ้ ็ฝฎๅนถๅฏๅจLogstash
cd /usr/local/logstash-6.5.0/
bin/logstash -f config.d/logstash.conf --config.test_and_exit #--config.test_and_exit้้กน็ๆๆๆฏ่งฃๆ้
็ฝฎๆไปถๅนถๆฅๅไปปไฝ้่ฏฏ
bin/logstash -f config.d/logstash.conf --config.reload.automatic #config.reload.automatic้้กน็ๆๆๆฏๅฏ็จ่ชๅจ้
็ฝฎๅ ่ฝฝ๏ผไปฅ่ณไบๆฏๆฌกไฝ ไฟฎๆนๅฎ้
็ฝฎๆไปถไปฅๅๆ ้ๅๆญข็ถๅ้ๅฏLogstash
logstashไธ็งๅฏๅจๆนๅผ๏ผ
-e #sting็ฑปๅๅฏๅจ
-f #ๆๅฎ้
็ฝฎๆไปถๅฏๅจ
service logstash start #ๆๅกๅฏๅจ logstashๅธธ็จๆไปถ
ๆฅ็ๆไปถ็ๆนๅผ
cd /usr/local/logstash-6.5.0/bin
./logstash-plugin listinputๆไปถ๏ผๅธธ็จ็ๆไปถ๏ผfileใtcpใudpใsyslog๏ผbeats
filterๆไปถ๏ผๅธธ็จ็ๆไปถ๏ผjsonใgrok
outputๆไปถ๏ผๅธธ็จ็ๆไปถ๏ผfile; tcp/udp; redis/kfaka; elasticsearch
codec็ฑปๆไปถ๏ผๅธธ็จ็ๆไปถ๏ผplainใjsonใjson_linesใrubydebugใmultiline็ญ
ๅฏๅจfilebeatๅlogstash
cd /usr/local/filebeat-6.5.0-linux-x86_64/
nohup ./filebeat -e -c filebeat.yml -d "publish" &
cd /usr/local/logstash-6.5.0/
nohup bin/logstash -f config.d/logstash.conf --config.reload.automatic &ๅฎ่ฃ
kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.5.0-linux-x86_64.tar.gz
tar -zvxf kibana-6.5.0-linux-x86_64.tar.gz -C /usr/local/
cd /usr/local/kibana-6.5.0-linux-x86_64/ไฟฎๆน้
็ฝฎๆไปถ
vim /usr/local/kibana-6.5.0-linux-x86_64/config/kibana.yml
ๆฑๅkibanaๆไปถไธ่ฝฝๅฐๅ๏ผ
git clone https://github.com/anbai-inc/Kibana_Hanization.git
cd Kibana_Hanization/old/
python main.py /usr/local/kibana-6.5.0-linux-x86_64ๆฑๅๆๅๅๅฏๅจkibana
cd /usr/local/kibana-6.5.0-linux-x86_64/bin/
nohup ./kibana &
ss -ntulp | grep 5601
ๆต่งๅจ่ฎฟ้ฎ๏ผhttp://ๆๅกๅจIP:5601/
ๆๅๆดๆฐไบ
่ฟๆๅธฎๅฉๅ๏ผ