多集群互联

当你的 ES 集群发展到一定规模，单集群不足以应对庞大的在线索引量级，或者由于业务隔离需求，都有可能划分成多个集群。这时候，另一个问题就出来了：可能其中有一部分数据，被分割在两个集群里，但是还是需要一起使用的。如果是自己写程序，当然可以初始化两个对象，分别连接两个集群，得到结果集后再自行合并。但是如果用 Elastic Stack 的，Kibana 可不支持同时连接两个集群地址，这时候，就要用到 ES 中一个特殊的角色：tribe 节点。

tribe 节点只需要提供集群自动发现方面的配置，连接上多个集群后，对外提供只读功能。elasticsearch.yml 配置示例如下：

tribe:
    1002:
        cluster.name: es1002
        discovery.zen.ping.timeout: 100s
        discovery.zen.ping.multicast.enabled: false
        discovery.zen.ping.unicast.hosts: ["10.19.0.22","10.19.0.24",10.19.0.21"]
    1003:
        cluster.name: es1003
        discovery.zen.ping.timeout: 100s
        discovery.zen.ping.multicast.enabled: false
        discovery.zen.ping.unicast.hosts: ["10.19.0.97","10.19.0.98","10.19.0.99","10.19.0.100"]
    blocks:
        write:    true
        metadata: true
    on_conflict: prefer_1003

注意这里的 on_conflict 设置，当多个集群内，索引名称有冲突的时候，tribe 节点默认会把请求轮询转发到各个集群上，这显然是不可以的。所以可以设置一个优先级，在索引名冲突的时候，偏向于转发给某一个集群。

以 tribe 配置启动的 Elasticsearch 服务，其日志输入如下：

[2015-06-18 18:05:51,983][INFO ][node                     ] [Manslaughter] version[1.5.1], pid[12846], build[5e38401/2015-04-09T13:41:35Z]
[2015-06-18 18:05:51,984][INFO ][node                     ] [Manslaughter] initializing ...
[2015-06-18 18:05:51,990][INFO ][plugins                  ] [Manslaughter] loaded [], sites []
[2015-06-18 18:05:54,891][INFO ][node                     ] [Manslaughter/1003] version[1.5.1], pid[12846], build[5e38401/2015-04-09T13:41:35Z]
[2015-06-18 18:05:54,891][INFO ][node                     ] [Manslaughter/1003] initializing ...
[2015-06-18 18:05:54,891][INFO ][plugins                  ] [Manslaughter/1003] loaded [], sites []
[2015-06-18 18:05:55,654][INFO ][node                     ] [Manslaughter/1003] initialized
[2015-06-18 18:05:55,655][INFO ][node                     ] [Manslaughter/1002] version[1.5.1], pid[12846], build[5e38401/2015-04-09T13:41:35Z]
[2015-06-18 18:05:55,655][INFO ][node                     ] [Manslaughter/1002] initializing ...
[2015-06-18 18:05:55,656][INFO ][plugins                  ] [Manslaughter/1002] loaded [], sites []
[2015-06-18 18:05:56,275][INFO ][node                     ] [Manslaughter/1002] initialized
[2015-06-18 18:05:56,285][INFO ][node                     ] [Manslaughter] initialized
[2015-06-18 18:05:56,286][INFO ][node                     ] [Manslaughter] starting ...
[2015-06-18 18:05:56,486][INFO ][transport                ] [Manslaughter] bound_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish_address {inet[/10.19.0.100:9301]}
[2015-06-18 18:05:56,499][INFO ][discovery                ] [Manslaughter] elasticsearch/Oewo-L2fR3y2xsgpsoI4Og
[2015-06-18 18:05:56,499][WARN ][discovery                ] [Manslaughter] waited for 0s and no initial state was set by the discovery
[2015-06-18 18:05:56,529][INFO ][http                     ] [Manslaughter] bound_address {inet[/0:0:0:0:0:0:0:0:9201]}, publish_address {inet[/10.19.0.100:9201]}
[2015-06-18 18:05:56,530][INFO ][node                     ] [Manslaughter/1003] starting ...
[2015-06-18 18:05:56,603][INFO ][transport                ] [Manslaughter/1003] bound_address {inet[/0:0:0:0:0:0:0:0:9302]}, publish_address {inet[/10.19.0.100:9302]}
[2015-06-18 18:05:56,609][INFO ][discovery                ] [Manslaughter/1003] es1003/m1-cDaFTSoqqyC2iiQhECA
[2015-06-18 18:06:26,610][WARN ][discovery                ] [Manslaughter/1003] waited for 30s and no initial state was set by the discovery
[2015-06-18 18:06:26,610][INFO ][node                     ] [Manslaughter/1003] started
[2015-06-18 18:06:26,611][INFO ][node                     ] [Manslaughter/1002] starting ...
[2015-06-18 18:06:26,674][INFO ][transport                ] [Manslaughter/1002] bound_address {inet[/0:0:0:0:0:0:0:0:9303]}, publish_address {inet[/10.19.0.100:9303]}
[2015-06-18 18:06:26,676][INFO ][discovery                ] [Manslaughter/1002] es1002/4FPiRPh7TFyBk-BaPc_TLg
[2015-06-18 18:06:56,676][WARN ][discovery                ] [Manslaughter/1002] waited for 30s and no initial state was set by the discovery
[2015-06-18 18:06:56,677][INFO ][node                     ] [Manslaughter/1002] started
[2015-06-18 18:06:56,677][INFO ][node                     ] [Manslaughter] started
[2015-06-18 18:07:37,266][INFO ][cluster.service          ] [Manslaughter/1003] detected_master [10.19.0.97][jnA-rt2fS_22Mz9nYl5Ueg][localhost.localdomain][inet[/10.19.0.97:9300]]{max_local_storage_nodes=1, data=false, master=true}, added {[10.19.0.73][_S8ylz1OTv6Nyp1YoMRNGQ][esnode073.mweibo.bx.sinanode.com][inet[/10.19.0.73:9300]]{max_local_storage_nodes=1, master=false},}, reason: zen-disco-receive(from master [[10.19.0.97][jnA-rt2fS_22Mz9nYl5Ueg][localhost.localdomain][inet[/10.19.0.97:9300]]{max_local_storage_nodes=1, data=false, master=true}])
[2015-06-18 18:07:37,382][INFO ][tribe                    ] [Manslaughter] [1003] adding node [[10.19.0.73][_S8ylz1OTv6Nyp1YoMRNGQ][esnode073.mweibo.bx.sinanode.com][inet[/10.19.0.73:9300]]{max_local_storage_nodes=1, tribe.name=1003, master=false}]
[2015-06-18 18:07:37,391][INFO ][tribe                    ] [Manslaughter] [1003] adding node [[Manslaughter/1003][m1-cDaFTSoqqyC2iiQhECA][localhost.localdomain][inet[/10.19.0.100:9302]]{data=false, tribe.name=1003, client=true}]
[2015-06-18 18:07:37,393][INFO ][tribe                    ] [Manslaughter] [1003] adding node [[10.19.0.97][_mIrWKzZTYifp1xshngBew][esnode054.mweibo.bx.sinanode.com][inet[/10.19.0.54:9300]]{max_local_storage_nodes=1, tribe.name=1003, master=false}]
[2015-06-18 18:07:37,393][INFO ][tribe                    ] [Manslaughter] [1003] adding index [logstash-mweibo-vip-2015.06.15]
[2015-06-18 18:07:37,394][INFO ][tribe                    ] [Manslaughter] [1003] adding index [logstash-php-2015.06.08]
[2015-06-18 18:07:37,394][INFO ][tribe                    ] [Manslaughter] [1003] adding index [logstash-mweibo-vip-2015.06.16]
[2015-06-18 18:07:37,395][INFO ][tribe                    ] [Manslaughter] [1003] adding index [.kibana]
[2015-06-18 18:07:37,398][INFO ][tribe                    ] [Manslaughter] [1003] adding index [logstash-php-2015.06.14]
[2015-06-18 18:07:37,403][INFO ][tribe                    ] [Manslaughter] [1003] adding index [logstash-mweibo-vip-2015.06.10]
[2015-06-18 18:07:37,403][INFO ][tribe                    ] [Manslaughter] [1003] adding index [kibana-int]
[2015-06-18 18:07:37,404][INFO ][tribe                    ] [Manslaughter] [1003] adding index [logstash-mweibo-2015.06.13]
[2015-06-18 18:07:37,411][INFO ][cluster.service          ] [Manslaughter] added {[10.19.0.73][_S8ylz1OTv6Nyp1YoMRNGQ][esnode073.mweibo.bx.sinanode.com][inet[/10.19.0.73:9300]]{max_local_storage_nodes=1, tribe.name=1003, master=false},[10.19.0.97][jnA-rt2fS_22Mz9nYl5Ueg][localhost.localdomain][inet[/10.19.0.97:9300]]{max_local_storage_nodes=1, tribe.name=1003, data=false, master=true},}, reason: cluster event from 1003, zen-disco-receive(from master [[10.19.0.97][jnA-rt2fS_22Mz9nYl5Ueg][localhost.localdomain][inet[/10.19.0.97:9300]]{max_local_storage_nodes=1, data=false, master=true}])
[2015-06-18 18:08:07,316][INFO ][cluster.service          ] [Manslaughter/1002] detected_master [10.19.0.22][6qyQh9EURUyO7RBC_dXDow][localhost.localdomain][inet[/10.19.0.22:9300]]{max_local_storage_nodes=1, master=true}, added {[10.19.0.93][qAklY08iSsSfIf2vvu6Iyw][localhost.localdomain][inet[/10.19.0.93:9300]]{max_local_storage_nodes=1, master=false}])
[2015-06-18 18:08:07,350][INFO ][indices.breaker          ] [Manslaughter/1002] Updating settings parent: [PARENT,type=PARENT,limit=259489792/247.4mb,overhead=1.0], fielddata: [FIELDDATA,type=MEMORY,limit=155693875/148.4mb,overhead=1.03], request: [REQUEST,type=MEMORY,limit=103795916/98.9mb,overhead=1.0]
[2015-06-18 18:08:07,353][INFO ][tribe                    ] [Manslaughter] [1002] adding node [[10.19.0.93][qAklY08iSsSfIf2vvu6Iyw][localhost.localdomain][inet[/10.19.0.93:9300]]{max_local_storage_nodes=1, tribe.name=1002, master=false}]
[2015-06-18 18:08:07,357][INFO ][tribe                    ] [Manslaughter] [1002] adding node [[Manslaughter/1002][4FPiRPh7TFyBk-BaPc_TLg][localhost.localdomain][inet[/10.19.0.100:9303]]{data=false, tribe.name=1002, client=true}]
[2015-06-18 18:08:07,358][INFO ][tribe                    ] [Manslaughter] [1002] adding node [[10.19.0.22][tkrBsbnLTry0zzZEdbQR0A][localhost.localdomain][inet[/10.19.0.27:9300]]{max_local_storage_nodes=1, tribe.name=1002, master=false}]
[2015-06-18 18:08:07,358][INFO ][tribe                    ] [Manslaughter] [1002] adding index [test.yingju1-mweibo_client_downstream_success-2015.06.07]
[2015-06-18 18:08:07,363][INFO ][tribe                    ] [Manslaughter] [1002] adding index [logstash-mweibo_client_downstream_error-2015.06.02]
[2015-06-18 18:08:07,366][INFO ][tribe                    ] [Manslaughter] [1002] adding index [.kibana_5601]
[2015-06-18 18:08:07,377][INFO ][cluster.service          ] [Manslaughter] added {[10.19.0.22][6qyQh9EURUyO7RBC_dXDow][localhost.localdomain][inet[/10.19.0.22:9300]]{max_local_storage_nodes=1, tribe.name=1002, master=false},[10.19.0.93][l7nkk-H7S6GvMzWwGe0_CA][localhost.localdomain][inet[/10.19.0.93:9300]]{max_local_storage_nodes=1, tribe.name=1002, master=false},}, reason: cluster event from 1002, zen-disco-receive(from master [[10.19.0.22][6qyQh9EURUyO7RBC_dXDow][localhost.localdomain][inet[/10.19.0.22:9300]]{max_local_storage_nodes=1, master=true}])
[2015-06-18 18:08:13,208][DEBUG][discovery.zen.publish    ] [Manslaughter/1003] received cluster state version 782404
[2015-06-18 18:08:21,803][DEBUG][discovery.zen.publish    ] [Manslaughter/1003] received cluster state version 782405
[2015-06-18 18:08:33,229][DEBUG][discovery.zen.publish    ] [Manslaughter/1003] received cluster state version 782406

日志中可以明显看到，节点是如何分别连接上两个集群的。

最后，我们可以使用标准的 RESTful 接口来验证一下：

# curl 10.19.0.100:9201/_cat/indices?v
health status index                                                    pri rep docs.count docs.deleted store.size pri.store.size
green  open   test.yingju1-mweibo_client_downstream_success-2015.06.07  20   1   40692459            0    154.1gb           77gb
green  open   weibo-client-video-2015.06.19                              5   1          0            0       970b           575b
green  open   dpool-pc-weibo-2015.06.19                                 20   1          0            0      3.7kb          2.2kb
green  open   logstash-video-2015.06.16                                 27   0  149015413            0     13.4gb         13.4gb

不同集群的索引，都可以通过 tribe node 访问到了。