Contact Us

Address: 1st Floor,Building 4, 1088th, Huyi highway, Shanghai
TEL:021-31080981
Email:soline@soline.com.cn
P.C.:201802

Utilize the ELK+Kafka solution

ELK is a combination of three software. It is a complete set of solutions. There are three softwares: Logstash (collection + analysis), ElasticSearch (search + storage), and Kibana (visual display). ELK is mainly to implement distributed log data centralized management and query in the massive log system, which is convenient for monitoring and troubleshooting.


Elasticsearch deployment and installation


ElasticSearch is a search server based on Lucene. It provides a full-text search engine with distributed multi-user capabilities and a web interface based on RESTful API. Elasticsearch is developed in Java and released as an open source under the terms of the Apache license. It is a popular enterprise search engine. Designed for use in cloud computing, it can achieve real-time search, stable, reliable, fast, and easy to install and use.

[root@seichung ] wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.2.rpm    # 官网下载软件包[root@seichung ] yum -y install java-1.8.0     # 相关依赖包[root@seichung ] yum -y install epel-release[root@seichung ] yum localinstall -y elasticsearch-6.4.2.rpm

After the installation is complete, modify the ElasticSearch configuration file, and then start the ElasticSearch service.。

[root@seichung ] vim /etc/elasticsearch/elasticsearch.yml            # 将如下的内容的注释取消,并将相关信息改成你在配置时的实际参数
            ***注意:配置文件的参数,在冒号后都有一个空格,如没有空格,服务会启动失败,并且无任何提示
              cluster.name: ES-cluster   # 集群名称,可更改
              node.name: es1             # 为本机的主机名
              network.host: 192.168.4.1  # 为本机的IP
              http.port: 9200            # elasticsearch服务端口,可更改
              discovery.zen.ping.unicast.hosts: ["es1", "es2","es3"]    # 集群内的主机(主机名,不一定全部都要写)
              node.master: true                                # 节点是否被选举为 master
              node.data: true                                  # 节点是否存储数据
              index.number_of_shards: 4                        # 索引分片的个数
              index.number_of_replicas: 1                      # 分片的副本个数
              path.conf: /etc/elasticsearch/                   # 配置文件的路径
              path.data: /data/es/data                         # 数据目录路径
              path.work: /data/es/elasticsearch                # 工作目录路径
              path.logs:  /data/es/logs/elasticsearch/logs/    # 日志文件路径
              path.plugins:  /data/es/plugins                  # 插件路径
              bootstrap.mlockall: true                         # 内存不向 swap 交换
              http.enabled: true                               # 启用http
              # 如果目录路径不存在的,需要先创建[root@seichung ] /etc/init.d/elasticsearch start    # 启动服务[root@seichung ] systemctl enable elasticsearch     # 开机自启

A simple ElasticSearch is built. If you want to deploy an ES cluster, you only need to deploy the Java environment on all hosts, and the /etc/hosts resolution host on all hosts, as follows:


This operation is mandatory on the ES cluster host

[root@seichung ] yum -y install java-1.8.0[root@seichung ] vim /etc/hosts            192.168.4.1 es1            192.168.4.2 es2            192.168.4.3 es3

Copy the hosts file and elasticsearch.yml configuration file to the cluster host on the host where the elasticsearch service has been installed, just modify node.name: to be the host name of the machine, and then start the service


Finally, let’s check whether the Es cluster has been deployed. If it is unsuccessful, number_of_nodes will always be 1. If it succeeds, there will be the following information:

[root@seichung ] curl -i http://192.168.4.1:9200/_cluster/health?pretty
    # 返回的信息包括集群名称、集群数量等,如果 number_of_nodes 显示的是实际得集群数量,则说明集群部署成功
    {
      "cluster_name" : "ES-cluster",
      "status" : "green",
      "timed_out" : false,
      "number_of_nodes" : 3,
      "number_of_data_nodes" : 3,
      "active_primary_shards" : 26,
      "active_shards" : 52,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 0,
      "delayed_unassigned_shards" : 0,
      "number_of_pending_tasks" : 0,
      "number_of_in_flight_fetch" : 0,
      "task_max_waiting_in_queue_millis" : 0,
      "active_shards_percent_as_number" : 100.0
    }

ElasticSearch plugin installation

# Use local installation or remote installation[root@seichung ] /usr/share/elasticsearch/bin/plugin install file:///data/es/plugins/elasticsearch-head-master.zip    [root@seichung ] /usr/share/elasticsearch/bin/plugin install file:///data/es/plugins/elasticsearch-kopf-master.zip[root@seichung ] /usr/share/elasticsearch/bin/plugin install file:///data/es/plugins/bigdesk-master.zip[root@seichung ] /usr/share/elasticsearch/bin/plugin list     # 查看已安装的插件

After the plug-in installation is complete, access to the URL of the corresponding plug-in is successful