ZooKeeper cluster migration
Here is a plan for ZK 3.4.9 (no dynamic reconfiguration):
- Add the 3 new ZK nodes to the old cluster. No changes needed for the 3 old ZK nodes at this time.
- Configure one of the new ZK nodes as a cluster of 4 nodes (3 old + 1 new), start it.
- Configure the other two new ZK nodes as a cluster of 6 nodes (3 old + 3 new), start them.
- Make sure the 3 new ZK nodes connected to the old ZK cluster as followers (run
echo stat | nc localhost 2181on the 3 new ZK nodes)
- Confirm that the leader has 5 synced followers (run
echo mntr | nc localhost 2181on the leader, look for
- Stop data ingestion in CH (this is to minimize errors when CH loses ZK).
- Change the zookeeper section in the configs on the CH nodes (remove the 3 old ZK servers, add the 3 new ZK servers)
- Make sure that there are no connections from CH to the 3 old ZK nodes (run
echo stat | nc localhost 2181on the 3 old nodes, check their
Clientssection). Restart all CH nodes if necessary (In some cases CH can reconnect to different ZK servers without a restart).
- Remove the 3 old ZK nodes from
zoo.cfgon the 3 new ZK nodes.
- Restart the 3 new ZK nodes. They should form a cluster of 3 nodes.
- When CH reconnects to ZK, start data loading.
- Turn off the 3 old ZK nodes.
This plan works, but it is not the only way to do this, it can be changed if needed.
Last modified 2021.08.12: General corrections and updates. (876b7b6)