李广良

zookeeper集群环境部署

前置说明：

  • 日志文件：dataLogDir=/opt/infra/apache-zookeeper-3.7.1-bin/logs
  • 数据文件：dataDir=/tmp/zookeeper
  • 安装路径：/opt/infra/apache-zookeeper-3.7.1-bin

1. 下载zookeeper包

注意生产环境请使用最新稳定版本。

1
The Apache ZooKeeper community supports two release branches at a time: stable and current. The stable version of ZooKeeper is 3.7.x and the current version is 3.8.x.

2. zookeeper上传至配置服务器（ansible）

安装文件上传至ansible主机如下路径

  • /etc/ansible/script/zookeeper/infra_pkgs/apache-zookeeper-3.7.1-bin.tar.gz

3. 编辑playbook脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@localhost etc]# tree ansible/
ansible/
├── ansible.cfg
├── hosts(主机列表)
├── roles(空文件夹)
└── script
    └── zookeeper
        ├── infra_pkgs
        │   ├── apache-zookeeper-3.7.1-bin.tar.gz(zookeeper安装包)
        │   ├── myid.j2(配置文件)
        │   ├── zk.sh(配置文件)
        │   └── zoo.cfg.j2(配置文件)
        └── zookeeper_install_playbook.yml(playbook)
4 directories, 7 files
  • hosts文件
    在主机IP后添加zk_myid=[number]定义zookeeper id，后续myid.j2模版中会取这个值。
1
2
3
4
[kubernetes]
192.168.132.129 ansible_ssh_user=root ansible_ssh_pass='***' zk_myid=1
192.168.132.130 ansible_ssh_user=root ansible_ssh_pass='***' zk_myid=2
192.168.132.131 ansible_ssh_user=root ansible_ssh_pass='***' zk_myid=3
  • myid.j2模板
    根据ip发送到对应主机，如不明白，需先了解手动配置集群。看参考链接2。
1
{{ zk_myid }}
  • zoo.cfg.j2模板
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
tickTime=2000
initLimit=10
syncLimit=5
dataDir={{zk_data_dir}}
dataLogDir={{zk_datalog_dir}}
clientPort=2181

autopurge.snapRetainCount=5
autopurge.purgeInterval=1
maxClientCnxns=2000

#master,node1,node2均为主机名
server.1=master:2888:3888
server.2=node1:2888:3888
server.3=node2:2888:3888
  • playbook文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
- hosts: kubernetes
  vars:
      cfg_files:
        - src: "./infra_pkgs/zoo.cfg.j2"
          dest: "/opt/infra/apache-zookeeper-3.7.1-bin/conf/zoo.cfg"
        - src: "./infra_pkgs/myid.j2"
          dest: "/tmp/zookeeper/myid"
        - src: "./infra_pkgs/zk.sh"
          dest: "/etc/profile.d/zk.sh"
  gather_facts: no
  user: root
  tasks:
    - name: 删除原文件
      shell:
        cmd: rm -rf /opt/infra/apache-zookeeper-3.7.1-bin/ /tmp/zookeeper/
    - name: Create zookeeper installation directory
      file:
        path: "{{item}}"
        state: directory
      notify: Restart zookeeper service
      with_items:
          - /opt/infra
          - /tmp/zookeeper
    - name: Upload zookeeper package
      unarchive:
        src: ./infra_pkgs/apache-zookeeper-3.7.1-bin.tar.gz
        dest: /opt/infra
      notify: Restart zookeeper service
    - name: Init zookeeper config
      template:
        src: "{{item.src}}"
        dest: "{{item.dest}}"
      with_items: "{{cfg_files}}"
      notify: Restart zookeeper service
  handlers:
    - name: Restart zookeeper service
      shell:
        cmd: /opt/infra/apache-zookeeper-3.7.1-bin/bin/zkServer.sh restart

4. 验证zookeeper运行情况

  • 确认/opt/infra/apache-zookeeper-3.7.1-bin/conf/zoo.cfg文件
  • 确认/tmp/zookeeper/myid文件
1
2
cat /opt/infra/apache-zookeeper-3.7.1-bin/conf/zoo.cfg
cat /tmp/zookeeper/myid
  • 执行playbook
1
2
3
cd /etc/ansible/script/zookeeper/

ansible-playbook zookeeper_install_playbook.yml  -e '{ "zk_servers": [ {"ip": "127.0.0.1", "myid": 1 } ], "zk_election_port": 3888, "zk_communication_port": 2888, "zk_data_dir": "/tmp/zookeeper", "zk_datalog_dir": "/opt/infra/apache-zookeeper-3.7.1-bin/logs" }'

5. 单独维护

1
2
3
4
5
6
7
8
9
10
# 查看状态
/opt/infra/apache-zookeeper-3.7.1-bin/bin/zkServer.sh status
# 停止
/opt/infra/apache-zookeeper-3.7.1-bin/bin/zkServer.sh stop
# 重启
/opt/infra/apache-zookeeper-3.7.1-bin/bin/zkServer.sh restart
# 启动
/opt/infra/apache-zookeeper-3.7.1-bin/bin/zkServer.sh start
# 启动失败时，带输出启动查看
zkServer.sh start-foreground

6.异常处理

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
2023-05-27 22:34:50,874 [myid:1] - ERROR [main:QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:461)
        at sun.nio.ch.Net.bind(Net.java:453)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:662)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:169)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
2023-05-27 22:34:50,875 [myid:1] - INFO  [main:ZKAuditProvider@42] - ZooKeeper audit is disabled.
2023-05-27 22:34:50,876 [myid:1] - ERROR [main:ServiceUtils@48] - Exiting JVM with code 1
[root@node71 ~]# netstat -nltp | grep 2181
tcp6       0      0 :::2181                 :::*                    LISTEN      8305/java
[root@node71 ~]# kill -9 8305

空空
2024-07-18
EXPAND TOP BOTTOM