linuxea: kafka单机版SCRAM认证 仍然使用旧版的2.5.0配置单节点的zookeeper和kafka，mechanism使用SCRAM-SHA-256进行认证不使用docker，就需要安装jdk,如下：jdkcurl -Lk https://repo.huaweicloud.com/java/jdk/8u202-b08/jdk-8u202-linux-x64.tar.gz |tar xz -C /usr/local cd /usr/local && ln -s jdk1.8.0_202 java cat > /etc/profile.d/java.sh <<EOF export JAVA_HOME=/usr/local/java export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar EOF source /etc/profile.d/java.sh java -versionkafka-2.5.0安装配置kafka目录DPATH=/data mkdir ${DPATH}/logging/zookeeper/logs ${DPATH}/logging/kafka/ ${DPATH}/logs/ -p groupadd -r -g 699 kafka useradd -u 699 -s /sbin/nologin -c 'kafka server' -g kafka kafka -M chown -R kafka.kafka ${DPATH}/loggingdownload kafkawget https://archive.apache.org/dist/kafka/2.5.0/kafka_2.12-2.5.0.tgz tar xf kafka_2.12-2.5.0.tgz -C /usr/local cd /usr/local ln -s kafka_2.12-2.5.0 kafka而后配置必要的认证文件cat > /usr/local/kafka/config/kafka_client_jaas.conf << EOF KafkaClient { org.apache.kafka.common.security.scram.ScramLoginModule required username="marksugar" password="linuxea.com"; }; EOF cat > /usr/local/kafka/config/kafka_client_jaas.conf << EOF KafkaClient { org.apache.kafka.common.security.scram.ScramLoginModule required username="marksugar" password="linuolxloADMINXP[QP[1]]"; }; EOF配置文件zookeeper和kafka在一起的，不需要重新安装zookeepermv /usr/local/kafka/config/zookeeper.properties /usr/local/kafka/config/zookeeper.properties.bak cat > /usr/local/kafka/config/zookeeper.properties << EOF tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/logging/zookeeper dataLogDir=/data/logging/zookeeper/logs clientPort=2181 EOFecho "0" > /data/logging/zookeeper/myid启动脚本cat > /etc/systemd/system/zookeeper.service << EOF [Unit] Description=ZooKeeper Service After=network.target After=network-online.target Wants=network-online.target [Service] Environment=ZOO_LOG_DIR=/data/logging/zookeeper/logs PIDFile=/data/logging/zookeeper/zookeeper_server.pid User=kafka Group=kafka ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties #RestartSec=15 #LimitNOFILE=65536 #OOMScoreAdjust=-999 Type=simple Restart=on-failure [Install] WantedBy=default.target EOFkafkabroker.id=1 listeners=SASL_PLAINTEXT://:9092 advertised.listeners=SASL_PLAINTEXT://172.16.100.151:9092 log.dirs=/data/logging/kafka/ zookeeper.connect=172.16.100.151:2181 sasl.enabled.mechanisms=SCRAM-SHA-256 sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 security.inter.broker.protocol=SASL_PLAINTEXT authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer super.users=User:markadmin;User:marksugar num.network.threads=9 num.io.threads=16 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 num.partitions=1 #auto.create.topics.enable=true default.replication.factor=1启动脚本cat > /etc/systemd/system/kafka.service << EOF [Unit] Description=kafka Service After=network.target syslog.target [Service] Environment=ZOO_LOG_DIR=/data/logging/kafka/logs SyslogIdentifier=kafka # 添加limit参数 LimitFSIZE=infinity LimitCPU=infinity LimitAS=infinity LimitMEMLOCK=infinity LimitNOFILE=64000 LimitNPROC=64000 User=kafka Group=kafka Type=simple Restart=on-failure Environment="KAFKA_OPTS=-Djava.security.auth.login.config=/usr/local/kafka/config/kafka_server_jaas.conf" Environment="PATH=${PATH}:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server-scram.properties ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh [Install] WantedBy=default.target EOF修改内存配置，修改java配置，bin下的kafka-server-start.sh，配置内存大小，并且配置9999端口eagleif [ "x$KAFKA_HEAP_OPTS" = "x" ]; then #export KAFKA_HEAP_OPTS="-server -Xms4G -Xmx4G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParalupancyPercent=70" export KAFKA_HEAP_OPTS="-server -Xms4G -Xmx4G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200" export JMX_PORT="9999" fi而后添加环境变量到JAVA_HOME=/usr/local/java到 /usr/local/kafka/bin/kafka-run-class.shes开始启动chown -R kafka.kafka /usr/local/kafka_2.13-2.8.2/ systemctl start zookeeper systemctl enable zookeeper systemctl status zookeeper创建用户/usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.151:2181 --alter \ --add-config 'SCRAM-SHA-256=[iterations=8192,password=linuxea.com],SCRAM-SHA-256=[password=linuxea.com]' \ --entity-type users --entity-name marksugar /usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.151:2181 --alter \ --add-config 'SCRAM-SHA-256=[iterations=8192,password=MwMzA0MGIwZjMwMjg3MjY4NWE2ZGFmOG],SCRAM-SHA-256=[password=MwMzA0MGIwZjMwMjg3MjY4NWE2ZGFmOG]' \ --entity-type users --entity-name markadmin而后使用sudo -u kafka启动kafka进行调试chown -R kafka.kafka /usr/local/kafka* sudo -u kafka KAFKA_OPTS=-Djava.security.auth.login.config=/usr/local/kafka/config/kafka_server_jaas.conf /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server-scram.properties没有问题后，配置开机启动systemctl start kafka systemctl enable kafka systemctl status kafka一旦使用了kafka SCRAM，那么kafka-eagle才是一个好选择参考https://linuxea.com/2678.html

linuxea: docker-compose构建postgresql12主从 postgresql的同步流复制，我们就简单叫主库和备库来表示两个不同的角色，我将分享主从的搭建过程为了减少在安装上的繁琐流程，我将使用docker镜像，使用docker-compose编排Docker-Compose version： v2.16.0 image: registry.cn-zhangjiakou.aliyuncs.com/marksugar-k8s/postgres:12.14-alpine3.17因此，需要提前安装docker和docker-compose，镜像使用的是官方镜像，只是被搬到阿里而已IDIP角色配置1172.168.204.41master4c8g2172.168.204.42slave4c8g启动的部分命令如下wal_level = replica # 这个是设置主为wal的主机 max_wal_senders = 5 # 这个设置了可以最多有几个流复制连接，差不多有几个从，就设置几个 wal_keep_segments = 128 # 设置流复制保留的最多的xlog数目 wal_sender_timeout = 60s # 设置流复制主机发送数据的超时时间 max_connections = 200 # 一般查多于写的应用从库的最大连接数要比较大 hot_standby = on # 说明这台机器不仅仅是用于数据归档，也用于数据查询 max_standby_streaming_delay = 30s # 数据流备份的最大延迟时间 wal_receiver_status_interval = 10s # 多久向主报告一次从的状态，当然从每次数据复制都会向主报告状态，这里只是设置最长的间隔时间 hot_standby_feedback = on # 如果有错误的数据复制，是否向主进行反馈 wal_log_hints = on # also do full page writes of non-critical updates这段摘自其他网页master使用docker-compose后，就需要在Command种使用-c指定即可，如下version: '3.3' services: postgresql12-m: container_name: postgresql12-m image: registry.cn-zhangjiakou.aliyuncs.com/marksugar-k8s/postgres:12.14-alpine3.17 restart: always environment: - POSTGRES_USER=postgres - POSTGRES_PASSWORD=mysecretpassword - PGDATA=/var/lib/postgresql/data/pgdata command: "postgres -c wal_level=replica -c max_wal_senders=15 -c wal_keep_segments=128 -c wal_sender_timeout=60s -c max_connections=200 -c hot_standby=on -c max_standby_streaming_delay=30s -c wal_receiver_status_interval=10s -c hot_standby_feedback=on -c wal_log_hints=on" volumes: - /etc/localtime:/etc/localtime:ro # 时区2 - /data/postgresql12:/var/lib/postgresql/data - /data/postgresql12/pg_hba.conf:/var/lib/postgresql/data/pgdata/pg_hba.conf logging: driver: "json-file" options: max-size: "50M" ports: - 8080:8080 - 5432:5432而后启动，命令如下docker-compose -f docker-compose.yaml up -d进入容器，创建用户create role replica with replication login password '123456'; alter user replica with password '123456'; 或者 CREATE USER replica WITH REPLICATION LOGIN CONNECTION LIMIT 10 ENCRYPTED PASSWORD '123456';如下docker exec --user=postgres -it postgresql12-m bash 2860b7926327:/$ psql psql (12.14) Type "help" for help. postgres=# create role replica with replication login password '123456'; CREATE ROLE postgres=# alter user replica with password '123456'; ALTER ROLE启动完成后，我们需要删除原有自动生成的配置文件在替换后启动才能生效postgres的配置文件是自动生成的，只有在容器生成在替换才可以被替换因此，修改pg_hba.conf 的内容如下docker rm -f postgresql12-m rm -f /data/postgresql12/pgdata/pg_hba.conf cat > /data/postgresql12/pgdata/pg_hba.conf << EOFFF local all all trust host all all 0.0.0.0/0 md5 host all all ::1/128 trust host replication replica 0.0.0.0/0 md5 EOFFF除此之外，我们还需要配置流复制的必要配置，直接添加到/data/postgresql12/pgdata/postgresql.conf文件内PG_FILE=/data/postgresql12/pgdata/postgresql.conf echo "synchronous_standby_names = 'standbydb1' #同步流复制才配置该值" >> $PG_FILE echo "synchronous_commit = 'remote_write'" >> $PG_FILE最后启动在重新启动docker-compose -f docker-compose.yaml up -d如果有必要，需要关闭防火墙或者配置放行5432端口slave从节点的command命令和master一致即可，修改名称添加-s以便于区分version: '3.3' services: postgresql12-s: container_name: postgresql12-s image: registry.cn-zhangjiakou.aliyuncs.com/marksugar-k8s/postgres:12.14-alpine3.17 restart: always environment: - POSTGRES_USER=postgres - POSTGRES_PASSWORD=mysecretpassword - PGDATA=/var/lib/postgresql/data/pgdata command: "postgres -c wal_level=replica -c max_wal_senders=15 -c wal_keep_segments=128 -c wal_sender_timeout=60s -c max_connections=200 -c hot_standby=on -c max_standby_streaming_delay=30s -c wal_receiver_status_interval=10s -c hot_standby_feedback=on -c wal_log_hints=on" volumes: - /etc/localtime:/etc/localtime:ro # 时区2 - /data/postgresql12:/var/lib/postgresql/data logging: driver: "json-file" options: max-size: "50M" ports: - 8080:8080 - 5432:5432启动数据库docker-compose -f docker-compose.yaml up -d备份数据备份数据，可以通过pg_start_backup和pg_basebackup，这里我们使用pg_basebackup1,pg_basebackup镜像内本身带有pg_basebackup，因此，我们在从节点，进入容器内，执行如下命令pg_basebackup -h 172.168.204.41 -p 5432 -U replica -Fp -Xs -Pv -R -D /var/lib/postgresql/data/pgdata-latest上述命令远程备份到当前的/var/lib/postgresql/data/pgdata-latest目录下也是在容器的挂载路径内，先存放在pgdata-latest，而后在切换目录即可[root@node2 pgdata-m]# docker exec -it postgresql12-s bash 142531a6f29e:/# pg_basebackup -h 172.168.204.41 -p 5432 -U replica -Fp -Xs -Pv -R -D /var/lib/postgresql/data/pgdata-latest Password: pg_basebackup: initiating base backup, waiting for checkpoint to complete pg_basebackup: checkpoint completed pg_basebackup: write-ahead log start point: 0/6000028 on timeline 1 pg_basebackup: starting background WAL receiver pg_basebackup: created temporary replication slot "pg_basebackup_76" 24656/24656 kB (100%), 1/1 tablespace pg_basebackup: write-ahead log end point: 0/6000100 pg_basebackup: waiting for background process to finish streaming ... pg_basebackup: syncing data to disk ... pg_basebackup: base backup completed此时备份的数据目录是在/var/lib/postgresql/data/pgdata-latest跳过pg_start_backup如果使用了pg_backup，现在可以跳过次方式备份。如果不能使用pg_backup，就在主节点使用pg_start_backup后进行复制目录docker exec -it --user=postgres postgresql12-m bash2 8e481427e025:/$ psql -U postgres psql (12.14) Type "help" for help. postgres=# select pg_start_backup('$DATE',true); pg_start_backup ----------------- 0/2000028 (1 row)使用pg_start_backup这个方法后，所有请求在写日志之后不会再刷新到磁盘。直到执行pg_stop_backup()这个函数。拷贝一份data目录，并通过scp复制到子数据库中cp -r /data/postgresql/pgdata ./pgdata-m tar -zcf pgdata-m.tar.gz pgdata-m scp -r ./pgdata-m.tar.gz 172.168.204.42:~/复制完成停止postgres=# select pg_stop_backup(); NOTICE: WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup pg_stop_backup ---------------- 0/2000138 (1 row)回到从节点，删除容器，解压从主拿来的数据[root@node2 postgres]# docker rm -f postgresql12-s postgresql12-s [root@node2 ~]# tar xf pgdata-m.tar.gz -C /data/postgresql12/ [root@node2 ~]# ll /data/postgresql12/ total 8 drwx------ 2 70 root 35 Mar 6 17:31 pgdata drwx------ 19 root root 4096 Mar 6 17:33 pgdata-m -rw-r--r-- 1 70 root 113 Mar 6 17:25 pg_hba.conf2,修改数据目录此时备份的数据目录是在/var/lib/postgresql/data/pgdata-latest，修改docker-compose映射的环境变量关系，指向备份好的目录- PGDATA=/var/lib/postgresql/data/pgdata-latest在posgress12种，需要创建文件standby.signal来声明自己是从，并且standby.signal本身也优先于其他创建文件即可，你也可以写点其他信息。这里我们为了怀念老版本，追加standby_mode = 'on'echo "standby_mode = 'on'" > /data/postgresql12/pgdata-latest/standby.signal此时我们还需要检查从节点的配置。从节点的postgresql.auto.conf文件内的属性是否和预期一致，之所以是postgresql.auto.conf，只是因为postgresql.auto.conf优先于postgresql.conf被读取pg_basebackup -R会修改postgresql.auto.conf的授权的权限信息，一旦使用了pg_basebackup ，就需要重新修改此时我的postgresql.auto.conf和postgresql.conf都添加如下配置：其中包含了授权的账号信息hot_standby = 'on' primary_conninfo = 'application_name=standbydb1 user=replica password=123456 host=172.168.204.41 port=5432 sslmode=disable sslcompression=0 gssencmode=disable krbsrvname=postgres target_session_attrs=any'如果修改的是文件，而这样的修改需要重启启动，也可以通过命令行进行配置，如下：show primary_conninfo # 查看 alter system set primary_conninfo = 'application_name=standbydb1 user=replica password=123456 host=172.168.204.41 port=5432 sslmode=disable sslcompression=0 gssencmode=disable krbsrvname=postgres target_session_attrs=any';而后启动从库docker-compose -f docker-compose.yaml up -d验证主从回到主库查看linuxea=# select * from pg_stat_replication; pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | backend_xmin | state | sent_lsn | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state | reply_time -----+----------+---------+------------------+----------------+-----------------+-------------+------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------+-----------+------------+---------------+------------+------------------------------- 144 | 16384 | replica | standbydb1 | 172.168.204.42 | | 47492 | 2023-03-07 10:18:45.56999+08 | 502 | streaming | 0/F3449D8 | 0/F3449D8 | 0/F3449D8 | 0/F3449D8 | | | | 1 | sync | 2023-03-07 10:25:38.354315+08 (1 row)linuxea=# select pid,state,client_addr,sync_priority,sync_state from pg_stat_replication; pid | state | client_addr | sync_priority | sync_state -----+-----------+----------------+---------------+------------ 207 | streaming | 172.168.204.42 | 1 | sync (1 row)linuxea=# select pg_is_in_recovery(); pg_is_in_recovery ------------------- f (1 row)在主库创建数据库，并插入数据CREATE DATABASE linuxea OWNER postgres; \c linuxea CREATE TABLE test( id integer, test integer)WITH (OIDS=FALSE); ALTER TABLE test OWNER TO postgres;如下postgres=# \c linuxea You are now connected to database "linuxea" as user "postgres". linuxea=# CREATE TABLE test( id integer, test integer)WITH (OIDS=FALSE); CREATE TABLE linuxea=# ALTER TABLE test OWNER TO postgres; ALTER TABLE linuxea=# insert into test SELECT generate_series(1,1000000) as key, (random()*(10^3))::integer; INSERT 0 1000000来到从库[root@node2 pgdata-m]# docker exec -it --user=postgres postgresql12-s bash da91ac9e2a19:/$ psql -U postgres psql (12.14) Type "help" for help. postgres=# select pg_is_in_recovery(); pg_is_in_recovery ------------------- t (1 row) postgres=# \c linuxea You are now connected to database "linuxea" as user "postgres". linuxea=# SELECT * FROM test; id | test ---------+------ 1 | 935 2 | 652 3 | 204 4 | 367 5 | 100 6 | 743 --More--从切主我们假设主挂掉了，一时半会好不了，就简单的将从切换到主，提供服务即可开始之前，我们直接关闭主库模拟主库不可用，而后进入从库的容器，使用 pg_ctl promote -Dc683e39637ea:/$ pg_ctl promote -D /var/lib/postgresql/data/pgdata-latest waiting for server to promote.... done server promoted将数据写入42，测试数据写入是否正常CREATE DATABASE linuxea2 OWNER postgres; \c linuxea2 CREATE TABLE test( id integer, test integer)WITH (OIDS=FALSE); ALTER TABLE test OWNER TO postgres; insert into test SELECT generate_series(1,1000000) as key, (random()*(10^3))::integer; SELECT * FROM test;现在从库已经可以写入数据了配置从库此时主库起来了，如果代理已经将请求改到42上了，我们直接在41的主节点上，同步42数据，将41配置为从库1.备份[root@node1 pgdata]# docker-compose -f ~/postgresql/docker-compose.yaml up -d [+] Running 1/1 ⠿ Container postgresql12-m Started 0.4s [root@node1 pgdata]# docker exec -it postgresql12-m bash d7108d41f908:/# pg_basebackup -h 172.168.204.42 -p 5432 -U replica -Fp -Xs -Pv -R -D /var/lib/postgresql/data/pgdata-latest Password: pg_basebackup: initiating base backup, waiting for checkpoint to complete pg_basebackup: checkpoint completed pg_basebackup: write-ahead log start point: 0/14000028 on timeline 2 pg_basebackup: starting background WAL receiver pg_basebackup: created temporary replication slot "pg_basebackup_100" 147294/147294 kB (100%), 1/1 tablespace pg_basebackup: write-ahead log end point: 0/14000100 pg_basebackup: waiting for background process to finish streaming ... pg_basebackup: syncing data to disk ... pg_basebackup: base backup completed修改数据目录- PGDATA=/var/lib/postgresql/data/pgdata-latest创建文件echo "standby_mode = 'on'" > /data/postgresql12/pgdata-latest/standby.signalpostgresql.auto.conf和postgresql.conf都添加如下配置：hot_standby = 'on' primary_conninfo = 'application_name=standbydb1 user=replica password=123456 host=172.168.204.42 port=5432 sslmode=disable sslcompression=0 gssencmode=disable krbsrvname=postgres target_session_attrs=any'删除容器后，重新启动docker-compose -f ~/postgresql/docker-compose.yaml down docker-compose -f ~/postgresql/docker-compose.yaml up -d6.验证回到主库42CREATE DATABASE linuxea23 OWNER postgres; \c linuxea23 CREATE TABLE test( id integer, test integer)WITH (OIDS=FALSE); ALTER TABLE test OWNER TO postgres; insert into test SELECT generate_series(1,1000000) as key, (random()*(10^3))::integer;而后到从库41查看\c linuxea23 SELECT * FROM test;如下[root@node1 postgresql12]# docker exec -it --user=postgres postgresql12-m bash 06a1e5ecc51b:/$ psql -U postgres psql (12.14) Type "help" for help. postgres=# \c linuxea23 You are now connected to database "linuxea23" as user "postgres". linuxea23=# SELECT * FROM test; id | test ---------+------ 1 | 785 2 | 654 3 | 881 4 | 19 5 | 37 6 | 482 7 | 938 8 | 25 9 | 209 10 | 820 11 | 445 12 | 238 13 | 772 14 | 233 15 | 158 16 | 964 17 | 815 18 | 890 19 | 977 20 | 437 21 | 56 22 | 241 23 | 266 24 | 123 25 | 139 26 | 207 27 | 90 28 | 4 29 | 95 30 | 896 31 | 698 32 | 752 33 | 972 --More--参考PostgreSQL12恢复配置总结

linuxea:mongodb 4.4.6副本集配置笔记 version : 4.4.6先决条件1，防火墙互相放行270202, 节点配置应该一致副本集解决不了写入瓶颈，如果副本集规模越大，写入性能越会下降，副本集可以应对读多的场景序号ip配置1172.16.100.108*16 / hdd（推荐ssd）2172.16.100.118*16 / hdd（推荐ssd）3172.16.100.128*16 / hdd（推荐ssd）echo never > /sys/kernel/mm/transparent_hugepage/enabledecho never > /sys/kernel/mm/transparent_hugepage/defrag3, deploy.sh#!/bin/bash \# auther: mark \# descriptions: the shell scripts install mongodb 4.4.6 version and start script and firewalld if ! grep ntp.aliyun.com /var/spool/cron/root >/dev/null 2>&1;then (crontab -l; echo -e "10 * * * * ntpdate ntp.aliyun.com") | crontab -;fi timedatectl set-timezone Asia/Shanghai hostnamectl set-hostname mongodb1 tar xf mongodb-linux-x86_64-rhel70-4.4.6.tgz -C /usr/local cd /usr/local/ ln -s mongodb-linux-x86_64-rhel70-4.4.6 mongodb mkdir /data/mongodb/{data,logs,pid,conf} -p groupadd mongodb useradd -g mongodb mongodb chown -R mongodb.mongodb /data/mongodb ln -s /usr/local/mongodb/bin/mongo /usr/local/bin/ cp /etc/firewalld/zones/public.xml /etc/firewalld/zones/public.xml.oldone cat > /etc/firewalld/zones/public.xml << EOF <?xml version="1.0" encoding="utf-8"?> <zone> <short>Public</short> <service name="ssh"/> <service name="dhcpv6-client"/> <port protocol="tcp" port="27020"/> </zone> EOF systemctl restart firewalld cat > /etc/systemd/system/mongodb.service << EPF [Unit] Description=mongodb Service After=network.target syslog.target [Service] Environment=ZOO_LOG_DIR=/data/mongodb/logs SyslogIdentifier=mongodb User=mongodb Group=mongodb Type=forking LimitFSIZE=infinity LimitCPU=infinity LimitAS=infinity LimitMEMLOCK=infinity LimitNOFILE=64000 LimitNPROC=64000 PIDfile=/mydata/data/mongodb/pid/mongod.pid ExecReload=/bin/kill -s HUP $MAINPID ExecStart=/usr/local/mongodb/bin/mongod -f /data/mongodb/conf/mongodb.conf ExecStop=/usr/local/mongodb/bin/mongod --shutdown -f /data/mongodb/conf/mongodb.conf Restart=on-failure PrivateTmp=true [Install] WantedBy=default.target EPF systemctl daemon-reload chmod +x /etc/systemd/system/mongodb.service systemctl enable mongodb示例: 172.16.100.10/11/12echo "mongodb soft nofile 64000" >> /etc/security/limits.conf echo "mongodb hard nofile 64000" >> /etc/security/limits.conf echo "mongodb soft nproc 32000" >> /etc/security/limits.conf echo "mongodb hard nproc 32000" >> /etc/security/limits.conf echo "never" > /sys/kernel/mm/transparent_hugepage/enabled echo "never" > /sys/kernel/mm/transparent_hugepage/defragLimitFSIZE=infinity LimitCPU=infinity LimitAS=infinity LimitMEMLOCK=infinity LimitNOFILE=64000 LimitNPROC=64000配置文件10配置systemLog: destination: file logAppend: true path: /data/mongodb/logs/mongod.log storage: dbPath: /data/mongodb/data journal: enabled: true directoryPerDB: true wiredTiger: engineConfig: cacheSizeGB: 8 directoryForIndexes: true processManagement: fork: true pidFilePath: /data/mongodb/pid/mongod.pid net: port: 27020 bindIp: 0.0.0.0 #bindIp: 0.0.0.0,mongodb1,localhost # ip and hostname maxIncomingConnections: 5000 #security: # keyFile: /data/mongodb/conf/keyfile # authorization: enabled replication: # oplogSizeMB: 1024 replSetName: rs0先注释security字段，没有密码登陆。三台节点防火墙开启27020互通初始化集群直接使用mongo连入admin/usr/local/mongodb/bin/mongo 172.16.100.10:27020 --authenticationDatabase admin配置仲裁priority。如果节点配置不一样，根据配置大小调整权重比例config = { _id:"rs0", members:[ {_id:0,host:"172.16.100.10:27020",priority:90}, {_id:1,host:"172.16.100.11:27020",priority:90}, {_id:2,host:"172.16.100.12:27020",arbiterOnly:true} ] };或者不配置。节点少,不配置仲裁config = { _id:"rs0", members:[ {_id:0,host:"172.16.100.10:27020",priority:90}, {_id:1,host:"172.16.100.11:27020",priority:90}, {_id:2,host:"172.16.100.12:27020",priority:90} ] };配置集群要进入admin> use admin switched to db admin > config = { _id:"rs0", ... members:[ ... {_id:0,host:"172.16.100.10:27020",priority:90}, ... {_id:1,host:"172.16.100.11:27020",priority:90}, ... {_id:2,host:"172.16.100.12:27020",priority:90} ... ] ... } { "_id" : "rs0", "members" : [ { "_id" : 0, "host" : "172.16.100.10:27020", "priority" : 90 }, { "_id" : 1, "host" : "172.16.100.11:27020", "priority" : 90 }, { "_id" : 2, "host" : "172.16.100.12:27020", "priority" : true } ] }使用rs.initiate(config);进行初始化> rs.initiate(config); { "ok" : 1 }使用rs.status()查看状态rs0:SECONDARY> rs.status() { "set" : "rs0", "date" : ISODate("2021-05-18T02:37:24.202Z"), "myState" : 1, "term" : NumberLong(1), "syncSourceHost" : "", "syncSourceId" : -1, "heartbeatIntervalMillis" : NumberLong(2000), "majorityVoteCount" : 2, "writeMajorityCount" : 2, "votingMembersCount" : 3, "writableVotingMembersCount" : 2, "optimes" : { "lastCommittedOpTime" : { "ts" : Timestamp(1621305433, 1), "t" : NumberLong(1) }, "lastCommittedWallTime" : ISODate("2021-05-18T02:37:13.348Z"), "readConcernMajorityOpTime" : { "ts" : Timestamp(1621305433, 1), "t" : NumberLong(1) }, "readConcernMajorityWallTime" : ISODate("2021-05-18T02:37:13.348Z"), "appliedOpTime" : { "ts" : Timestamp(1621305433, 1), "t" : NumberLong(1) }, "durableOpTime" : { "ts" : Timestamp(1621305433, 1), "t" : NumberLong(1) }, "lastAppliedWallTime" : ISODate("2021-05-18T02:37:13.348Z"), "lastDurableWallTime" : ISODate("2021-05-18T02:37:13.348Z") }, "lastStableRecoveryTimestamp" : Timestamp(1621305431, 3), "electionCandidateMetrics" : { "lastElectionReason" : "electionTimeout", "lastElectionDate" : ISODate("2021-05-18T02:37:11.853Z"), "electionTerm" : NumberLong(1), "lastCommittedOpTimeAtElection" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) }, "lastSeenOpTimeAtElection" : { "ts" : Timestamp(1621305421, 1), "t" : NumberLong(-1) }, "numVotesNeeded" : 2, "priorityAtElection" : 90, "electionTimeoutMillis" : NumberLong(10000), "numCatchUpOps" : NumberLong(0), "newTermStartDate" : ISODate("2021-05-18T02:37:11.879Z"), "wMajorityWriteAvailabilityDate" : ISODate("2021-05-18T02:37:13.328Z") }, "members" : [ { "_id" : 0, "name" : "172.16.100.10:27020", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 733, "optime" : { "ts" : Timestamp(1621305433, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2021-05-18T02:37:13Z"), "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1621305431, 1), "electionDate" : ISODate("2021-05-18T02:37:11Z"), "configVersion" : 1, "configTerm" : 1, "self" : true, "lastHeartbeatMessage" : "" }, { "_id" : 1, "name" : "172.16.100.11:27020", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 22, "optime" : { "ts" : Timestamp(1621305433, 1), "t" : NumberLong(1) }, "optimeDurable" : { "ts" : Timestamp(1621305433, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2021-05-18T02:37:13Z"), "optimeDurableDate" : ISODate("2021-05-18T02:37:13Z"), "lastHeartbeat" : ISODate("2021-05-18T02:37:23.871Z"), "lastHeartbeatRecv" : ISODate("2021-05-18T02:37:23.378Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncSourceHost" : "172.16.100.10:27020", "syncSourceId" : 0, "infoMessage" : "", "configVersion" : 1, "configTerm" : 1 }, { "_id" : 2, "name" : "172.16.100.12:27020", "health" : 1, "state" : 7, "stateStr" : "SECONDARY", "uptime" : 22, "lastHeartbeat" : ISODate("2021-05-18T02:37:23.872Z"), "lastHeartbeatRecv" : ISODate("2021-05-18T02:37:23.876Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "configVersion" : 1, "configTerm" : 1 } ], "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1621305433, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1621305433, 1) }授权root用户创建super man用户db.createUser({user: "root", pwd: "linuxea.com", roles: [{role: "root", db: "admin"}]}); db.createUser({user: "root", pwd: "linuxea.com", roles: [{role: "dbAdminAnyDatabase", db: "admin"}]}); db.createUser({user: "marksugar", pwd: "TdmMzIyNjRmMjViOTc1MGIwZGU", roles: [{role: "dbAdminAnyDatabase", db: "admin"}]});创建普通用户https://www.jianshu.com/p/0a7452d8843ddb.createUser( { user: "marksugar2",pwd: "TdmMzIyNjRmMjViOTc1MGIwZGU",roles: [ { role:"readWrite", db: "marksugar" }] }) db.createUser( { user: "marksugar",pwd: "TdmMzIyNjRmMjViOTc1MGIwZGU",roles: [ { role:"dbAdmin", db: "marksugar" }] }) db.createUser({ user: "marksugar",pwd: "TdmMzIyNjRmMjViOTc1MGIwZGU",roles: [ { role: "dbAdmin", db: "marksugar" }]}) db.createUser({ user: "marksugar",pwd: "TdmMzIyNjRmMjViOTc1MGIwZGU",roles: [ { role: "dbOwner", db: "marksugar" }]})如下：修改密码 db.createUser({user: "root", pwd: "linuxea.com", roles: [{role: "root", db: "admin"}]}) 执行结果 Successfully added user: { "user" : "root", "roles" : [ { "role" : "root", "db" : "admin" } ] }创建完成可以使用如下命令验证授权mongo -u root -p "linuxea.com" 17.168.0.175:27020/admin rs0:PRIMARY> db.get admin.get mongo -u marksugar -p "TdmMzIyNjRmMjViOTc1MGIwZGU" 17.168.0.175:27020/marksugar rs0:PRIMARY> db.get marksugar.get验证集群登入：mongo 172.16.100.10:27020/admin[root@localhost ~]# mongo 172.16.100.10:27020/admin MongoDB shell version v4.4.6 connecting to: mongodb://172.16.100.10:27020/admin?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("e73613cb-6004-46b0-b229-8356c6ed8cf5") } MongoDB server version: 4.4.6 --- The server generated these startup warnings when booting: 2021-05-18T10:25:11.991+08:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See https://dochub.mongodb.org/core/prodnotes-filesystem 2021-05-18T10:25:13.535+08:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted 2021-05-18T10:25:13.535+08:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never' 2021-05-18T10:25:13.535+08:00: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. We suggest setting it to 'never' 2021-05-18T10:25:13.535+08:00: Soft rlimits too low 2021-05-18T10:25:13.535+08:00: currentValue: 1024 2021-05-18T10:25:13.535+08:00: recommendedMinimum: 64000 --- --- Enable MongoDB's free cloud-based monitoring service, which will then receive and display metrics about your deployment (disk utilization, CPU, operation statistics, etc). The monitoring data will be available on a MongoDB website with a unique URL accessible to you and anyone you share the URL with. MongoDB may use this information to make product improvements and to suggest MongoDB products and deployment options to you. To enable free monitoring, run the following command: db.enableFreeMonitoring() To permanently disable this reminder, run the following command: db.disableFreeMonitoring() ---查看库show dbsrs0:PRIMARY> show dbs admin 0.000GB config 0.000GB local 0.000GB创建库use yxtops-testrs0:PRIMARY> use yxtops-test switched to db yxtops-test查看db.getName()rs0:PRIMARY> db.getName() yxtops-test插入：db.student.insert([{'name':'mark'},{}])rs0:PRIMARY> db.student.insert([{'name':'mark'},{}]) BulkWriteResult({ "writeErrors" : [ ], "writeConcernErrors" : [ ], "nInserted" : 2, "nUpserted" : 0, "nMatched" : 0, "nModified" : 0, "nRemoved" : 0, "upserted" : [ ] })查询rs0:PRIMARY> show tables; student rs0:PRIMARY> db.student.find() { "_id" : ObjectId("60a329d15d32fd9c982ccde1"), "name" : "mark" } { "_id" : ObjectId("60a329d15d32fd9c982ccde2") }验证登陆其他节点： mongo 172.16.100.11:27020/admin/usr/local/mongodb/bin/mongo -u root -p 'linuxea.com' 172.16.100.11:27020 --authenticationDatabase admin打开读rs0:SECONDARY> rs.slaveOk()进入其他库验证rs0:SECONDARY> use yxtops-test switched to db yxtops-test rs0:SECONDARY> show tables; student rs0:SECONDARY> db.student.find() { "_id" : ObjectId("60a329d15d32fd9c982ccde1"), "name" : "mark" } { "_id" : ObjectId("60a329d15d32fd9c982ccde2") }用户认证集群与集群之间要使用Keyfile。首先创建keyfile。而后复制到其他节点[root@mongodb1 ~]# cd /data/mongodb/conf/ [root@mongodb1 conf]# openssl rand -base64 756 > keyfile [root@mongodb1 conf]# chmod 400 keyfile [root@mongodb1 conf]# pwd /mydata/data/mongodb/conf chown mongodb.mongodb keyfile将内容复制到其他两个节点：cd /mydata/data/mongodb/conf for i in 11 12;do scp keyfile 172.16.100.$i:/mydata/data/mongodb/conf/;done for i in 11 12;do ssh 172.16.100.$i chmod 400 /mydata/data/mongodb/conf/keyfile;done for i in 11 12;do ssh 172.16.100.$i chown mongodb.mongodb /mydata/data/mongodb/conf/keyfile;done或者直接把keyfil文件内容粘贴出来复制到其他两个节点cat > keyfile << EOF kSb+lVdnA9K9/fJnldG65Vvem3r0dKHX/4ag0qcAIZ1Bl87VxmqVho5vKtDMZLN1 uQE9aZ4A3gZ0LFgdPpIXkAHazP9ZqNqBfIJO6JRpBDClFuN8KU1FIJI1ghyVTTAh sjPgdGr8NB7bnloPfm92LwT5kArNMuAd6aqa5EggmpnUZmxkpc81msx39S2mLsxC YEerP0Rj0j8ZA36hsgOsHy+PGvADhjcscqIFAbcqKSpP0CJGZC2WQLUo6EfL0/u9 ChLcUJp0FNgW2vguVXyOjwIhv+x9Rn7fHGokidjnVl54sylPz4AYBtgdmLXsmxi8 5JFXl56kT5W/SD5tvG1IJtteGJfyrWmVqOjbAqhQd6JSQ5zOvRVCUqpndxVy/IRP fw1thjQdBkpcLzf92rUqTUSyANsPV+M1/3L6VEU0KmarkL5yxQvKHlYR1/MtSr7+ bI1UvPsXAMdXWoyZUuZDAeh+J1mMlebeNZ7lYJagskd4SVk9sQoQfmAEnjTuJvSz xpvW4cbJIjzGx58rrfZ/DX+JowhDcsom3Icvl/PlV2JK3CEs4Rna0C70z4KZ0W7B uEY34aTi3IKS3dXQ06qNu3Zja3Zeff9CK2tW0CUs+MiO7Jqp1pVVZRHT6mrG3HWF VbcmvOiHBoEt0/YoMSU6fQsJMgk16Zv53GF0UvY3ZF3x1w6OKT5u7kp9OzHWnPr0 oK6fogfEgi94VBpdSHOXKUyeEI+0LgX8xSb4rmTn4Ud3tBCxXbnqlgVzEuBG+r55 chO4Xmk53FDW5UGtBC9QMWCtGT2I3vymEmKDlnIpzqQXJoKNOQi8UcIw/fluZeaG a1tcaaNn7Vrt9eNfgrBy1w7jL/6FGWF+lhbRmlTAMOMU6k4nGWinaZZA/MAmQmr8 55Y99wT18feHlt+FDH0eSewk6HnTW5QxZYgudP+V89/2vMQHH5vXEBD0JeDHSrpY rkTWULfr9s8nXxoDr5tbLxP1oc+CULt1pZOwXH+Exx5fQuT8 EOF chown mongodb.mongodb keyfile chmod 400 keyfile在配置文件中取消注释security: keyFile: /mydata/data/mongodb/conf/keyfile authorization: enabled而后关闭三台mongodb节点，关闭完成后再依次启动删除用户rs0:PRIMARY> db.dropUser("marksugar") true登陆开启认证后登陆：创建操作等，只能在PRIMARY节点操作才可以。通过rs.status()查看/usr/local/mongodb/bin/mongo -u root -p 'linuxea.com' 172.16.100.10:27020 --authenticationDatabase admin创建其他库和授权库用户use marksugar db.createUser({user: "marksugar",pwd: "TdmMzIyNjRmMjViOTc1MGIwZGU",roles: [ { role: "dbAdmin", db: "marksugar" }]}); db.createUser({user: "marksugar", pwd: "TdmMzIyNjRmMjViOTc1MGIwZGU", roles: [{role: "dbOwner", db: "marksugar"}]}); db.createUser({ user: "marksugar",pwd: "TdmMzIyNjRmMjViOTc1MGIwZGU",roles: [ { role: "dbAdmin", db: "marksugar" }] }) db.createUser({ user: "marksugar",pwd: "TdmMzIyNjRmMjViOTc1MGIwZGU",roles: [ { role: "dbAdmin", db: "marksugar" }]}) db.createUser({user: "root", pwd: "linuxea.com", roles: [{role: "root", db: "admin"}]}); db.createUser({user: "root", pwd: "linuxea.com", roles: [{role: "dbAdminAnyDatabase", db: "admin"}]});插入rs0:PRIMARY> use marksugar switched to db marksugar rs0:PRIMARY> db marksugar rs0:PRIMARY> db.marksugar.insert({"name":"mark"}) WriteResult({ "nInserted" : 1 }) rs0:PRIMARY> show dbs admin 0.000GB config 0.000GB marksugar 0.000GB local 0.001GB rs0:PRIMARY>登陆marksugar/usr/local/mongodb/bin/mongo -u marksugar -p "TdmMzIyNjRmMjViOTc1MGIwZGU" 172.16.100.10:27020/marksugar监控授权db.grantRolesToUser("root", [{role:"__system", db:"admin"}]) db.grantRolesToUser("root", [{role:"dbAdminAnyDatabase", db:"admin"}]);如下：mongo -u root -p "linuxea.com" 172.16.100.10:27020/admin rs0:PRIMARY> use admin switched to db admin rs0:PRIMARY> db.grantRolesToUser("root", [{role:"__system", db:"admin"}]) rs0:PRIMARY> db.grantRolesToUser("root", [{role:"dbAdminAnyDatabase", db:"admin"}]);副本读rs.slaveOk()或者rs.secondaryOk()日志清理logpath日志中存储了日志过程。将会保留7天#!/bin/bash IPADDRES=172.16.100.10:27020 DBNAME=admin USERNAME="USER_NAME" PASSWORDS="PASSWORD" LOGPATHS=/var/log/mongodb LOGDYA=7 # mongodb logpath logfile roll mongo ${IPADDRES}/${DBNAME} --authenticationDatabase ${DBNAME} -u ${USERNAME} -p "${PASSWORDS}" --eval "db.runCommand({logRotate:1});" sleep 3s # logfile delete last 7 days find ${LOGPATHS}/mongod.log.20* -type f -mtime +${LOGDYA} -delete # 1 12 * * * /bin/bash /data/script/mongomore.sh其他参考mongodb4.4.8复制(副本)集简单配置mongodb

linuxea:kafka在私有云DNAT环境中集群的典型应用 今天配置的是一个2.5.0的一个kafka集群，新的版本将废弃zookeeper，今天不讨论新版本有一个私有的云环境，业务需求希望通过公网向kafka发送数据，使用SCRAM-SHA-256加密，内网仍然需要能够正常访问， 而其中，需要通过DNAT的方式来映射内网端口暴漏给互联网。而在做映射的时候，必然是一个端口对一个端口的，于是，大致的示意拓扑如下如果你不是这种方式，可以尝试Kafka 实现内外网访问流量分离来解决问题而在实际的生产中，你会发现，内网采用内网IP进行访问的时候，kafka是可以正常协商进行处理请求而在公网通过6.78.5.32的9092,9093,9094端口访问的时候会出现出现一个问题，客户端当请求A通过6.78.5.32:9092发送，经过防火墙DNAT层后，发给后端kafka，而此时kafka收到消息后回复给发送者，而回复的时候是使用的172.16.100.7:9092端口，你的客户端根本就不认识172.16.100.7，因此发送失败而这个现象在你只是向kafka发送消息，而不在乎他是否返回的时候，代码层面显示是成功的，但是数据并未成功插入。于是，就有了另外一种方式消息发送后需要返回，服务端和客户端都分别写ip和hostname，通过域名和本地hosts的方式解析出ip，分别发送到代理服务器和客户端，而不是某一个固定的ip。无论来自公网的访问还是内网的访问，最终在本地的hosts各自指向一个可以被访问到的一个ip，从而完成响应。这种形式在官网的某些字段中被解读为“防止中间人攻击”如下version: kafka_2.12-2.5.0jdk: 1.8.0_211先决条件：同步时间10 * * * * ntpdate ntp.aliyun.com修改hosts并本地hosts#172.16.100.7 hostnamectl set-hostname kafka1 #172.16.100.8 hostnamectl set-hostname kafka2 #172.16.100.9 hostnamectl set-hostname kafka3172.16.100.7 kafka1 172.16.100.8 kafka2 172.16.100.9 kafka3准备工作二进制安装java，或者rpm安装即可tar xf jdk-8u211-linux-x64.tar.gz -C /usr/local/ cd /usr/local && ln -s jdk1.8.0_211 java cat > /etc/profile.d/java.sh <<EOF export JAVA_HOME=/usr/local/java export PATH=\$JAVA_HOME/bin:\$PATH export CLASSPATH=.:\$JAVA_HOME/lib/dt.jar:\$JAVA_HOME/lib/tools.jar EOF source /etc/profile.d/java.sh准备工作，所有节点创建目录和用户DPATH=/data mkdir ${DPATH}/zookeeper/logs ${DPATH}/kafka/ ${DPATH}/logs/ -p groupadd -r -g 699 kafka useradd -u 699 -s /sbin/nologin -c 'kafka server' -g kafka kafka -M chown -R kafka.kafka ${DPATH}下载kafka_2.12-2.5.0，解压到/usr/local/下，创建软连接到当前的kafkatar xf kafka_2.12-2.5.1.gz -C /usr/local/ cd /usr/local/ ln -s kafka_2.12-2.5.1 kafka tar xf kafka_2.12-2.5.0.tgz -C /usr/local/ cd /usr/local/ ln -s kafka_2.12-2.5.0 kafka/usr/local/kafka/config准备两个认证文件作为kafka认证kafka_client_jaas.conf#kafka客户端连接方式及生产者、消费者连接集群的用户密码 cat > /usr/local/kafka/config/kafka_client_jaas.conf << EOF KafkaClient { org.apache.kafka.common.security.scram.ScramLoginModule required username="marksugar" password="linuxea.com"; }; EOF#kafka客户端连接方式及生产者、消费者连接集群的用户密码 cat > /usr/local/kafka/config/kafka_client_jaas.conf << EOF KafkaClient { org.apache.kafka.common.security.scram.ScramLoginModule required username="marksugar" password="linuolxloADMINXP[QP[1]]"; }; EOFkafka_server_jaas.confmarkadmin用户作为超级管理员，这里的用户和密码文件是为了后面启动使用cat > /usr/local/kafka/config/kafka_server_jaas.conf << EOF KafkaServer { org.apache.kafka.common.security.scram.ScramLoginModule required username="markadmin" password="MwMzA0MGZGFmOG" user_markadmin="markadmin"; }; EOFzookeeper备份mv /usr/local/kafka/config/zookeeper.properties /usr/local/kafka/config/zookeeper.properties.bak1, 在10.100.63.7修改配置文件cat > /usr/local/kafka/config/zookeeper.properties << EOF tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper dataLogDir=/data/zookeeper/logs clientPort=2181 server.0=172.16.100.7:2888:3888 server.1=172.16.100.8:2888:3888 server.2=172.16.100.9:2888:3888 EOF2, 创建id。每台节点不一样echo "0" > /data/zookeeper/myid3, 启动脚本cat > /etc/systemd/system/zookeeper.service << EOF [Unit] Description=ZooKeeper Service After=network.target After=network-online.target Wants=network-online.target [Service] Environment=ZOO_LOG_DIR=/u01/data/zookeeper/logs PIDFile=/data/zookeeper/zookeeper_server.pid User=kafka Group=kafka ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties #RestartSec=15 #LimitNOFILE=65536 #OOMScoreAdjust=-999 Type=simple Restart=on-failure [Install] WantedBy=default.target EOFkafkadefault.replication.factor=2 1不备份，2备份 num.network.threads=3 大于CPU+1 num.io.threads=8 cpu的两倍1, 在10.100.63.7修改配置文件我们新创建一个文件，不用原来的文件我们直接配置一个advertised.listeners=SASL_PLAINTEXT://kakfa.linuxea.com:9092，kakfa.linuxea.com在本地hosts写入，写入的ip是代理的ip地址假设不需要代理，而只是集群访问，则配置为当前的IP 即可cat > /usr/local/kafka/config/server-scram.properties << EOF broker.id=1 ##### Socket Server Settings 监听协议和端口####### listeners=SASL_PLAINTEXT://172.16.100.7:9092 advertised.listeners=SASL_PLAINTEXT://kafka.linuxea.com:9092 ######### Log Basics ########## #日志路径 log.dirs=/data/kafka/ #num.partitions=16 ######## Zookeeper 集群信息 ########## zookeeper.connect=172.16.100.7:2181,172.16.100.8:2181,172.16.100.9:2181 ###### SCRAM Settings 认证部分######## sasl.enabled.mechanisms=SCRAM-SHA-256 sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 security.inter.broker.protocol=SASL_PLAINTEXT authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer super.users=User:markadmin;User:marksugar num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 # 创建三个副本和分区 num.partitions=3 #auto.create.topics.enable=true default.replication.factor=2 EOF执行cat > /usr/local/kafka/config/server-scram.properties << EOF broker.id=0 listeners=SASL_PLAINTEXT://172.16.100.7:9092 advertised.listeners=SASL_PLAINTEXT://kafka.linuxea.com:9092 log.dirs=/data/kafka/ zookeeper.connect=172.16.100.7:2181,172.16.100.8:2181,172.16.100.9:2181 sasl.enabled.mechanisms=SCRAM-SHA-256 sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 security.inter.broker.protocol=SASL_PLAINTEXT authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer super.users=User:markadmin;User:marksugar num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 num.partitions=3 #auto.create.topics.enable=true default.replication.factor=2 EOF启动脚本cat > /etc/systemd/system/kafka.service << EOF [Unit] Description=kafka Service After=network.target syslog.target [Service] Environment=ZOO_LOG_DIR=/data/kafka/logs SyslogIdentifier=kafka # 添加limit参数 LimitFSIZE=infinity LimitCPU=infinity LimitAS=infinity LimitMEMLOCK=infinity LimitNOFILE=64000 LimitNPROC=64000 User=kafka Group=kafka Type=simple Restart=on-failure Environment="KAFKA_OPTS=-Djava.security.auth.login.config=/usr/local/kafka/config/kafka_server_jaas.conf" Environment="PATH=${PATH}:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server-scram.properties ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh [Install] WantedBy=default.target EOF修改java配置，bin下的kafka-server-start.sh，配置内存大小，并且配置9999端口eagleif [ "x$KAFKA_HEAP_OPTS" = "x" ]; then #export KAFKA_HEAP_OPTS="-server -Xms4G -Xmx4G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParalupancyPercent=70" export KAFKA_HEAP_OPTS="-server -Xms4G -Xmx4G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200" export JMX_PORT="9999" fi配置----- 将kafka程序目录复制和启动脚本到172.16.100.8和172.16.100.9的/usr/local下，修改如下：scp -r kafka_2.12-2.5.1 172.16.100.8:/usr/local/ scp -r kafka_2.12-2.5.1 172.16.100.9:/usr/local/ scp /etc/systemd/system/zookeeper.service 172.16.100.8:/etc/systemd/system/ scp /etc/systemd/system/zookeeper.service 172.16.100.9:/etc/systemd/system/ scp /etc/systemd/system/kafka.service 172.16.100.8:/etc/systemd/system/ scp /etc/systemd/system/kafka.service 172.16.100.9:/etc/systemd/system/登录到172.16.100.8，172.16.100.9创建目录，做软连接cd /usr/local && ln -s kafka_2.12-2.5.1 kafka mkdir /u01/data/zookeeper/logs -p mkdir -p /u01/data/kafka/ groupadd -r -g 699 kafka useradd -u 699 -s /sbin/nologin -c 'kafka server' -g kafka kafka -M mkdir /u01/data/logs/ -p chown -R kafka.kafka /u01/data/ chown -R /usr/local/kafka_2.12-2.5.1/ kafka.kafka*根据server.0=172.16.100.7:2888:3888 server.1=172.16.100.8:2888:3888 server.2=172.16.100.9:2888:3888对应修改172.16.100.8echo "1" > /u01/data/zookeeper/myid172.16.100.9echo "2" > /u01/data/zookeeper/myidkafka修改server-scram.properties文件内容，这四项修改172.16.100.8broker.id=2 ##### Socket Server Settings 监听协议和端口####### listeners=SASL_PLAINTEXT://172.16.100.8:9093 advertised.listeners=SASL_PLAINTEXT://kafka.linuxea.com:9093172.16.100.9broker.id=3 ##### Socket Server Settings 监听协议和端口####### listeners=SASL_PLAINTEXT://172.16.100.9:9094 advertised.listeners=SASL_PLAINTEXT://kafka.linuxea.com:9094172.16.100.8 kafka配置修改后如下broker.id=2 ##### Socket Server Settings 监听协议和端口####### listeners=SASL_PLAINTEXT://172.16.100.8:9093 advertised.listeners=SASL_PLAINTEXT://kafka.linuxea.com:9093 ######### Log Basics ########## #日志路径 log.dirs=/u01/data/kafka/ #num.partitions=16 ######## Zookeeper 集群信息 ########## zookeeper.connect=172.16.100.7:2181,172.16.100.8:2181,172.16.100.9:2181 ###### SCRAM Settings 认证部分######## sasl.enabled.mechanisms=SCRAM-SHA-256 sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 security.inter.broker.protocol=SASL_PLAINTEXT authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer super.users=User:markadmin;User:marksugar num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 num.partitions=3 #auto.create.topics.enable=true default.replication.factor=2172.16.100.9 kafka配置修改后如下broker.id=3 ##### Socket Server Settings 监听协议和端口####### listeners=SASL_PLAINTEXT://172.16.100.9:9094 advertised.listeners=SASL_PLAINTEXT://kafka.linuxea.com:9094 ######### Log Basics ########## #日志路径 log.dirs=/u01/data/kafka/ #num.partitions=16 ######## Zookeeper 集群信息 ########## zookeeper.connect=172.16.100.7:2181,172.16.100.8:2181,172.16.100.9:2181 ###### SCRAM Settings 认证部分######## sasl.enabled.mechanisms=SCRAM-SHA-256 sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 security.inter.broker.protocol=SASL_PLAINTEXT authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer super.users=User:markadmin;User:marksugar num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 num.partitions=3 #auto.create.topics.enable=true default.replication.factor=2zookeeper授权先启动整个zookeeper集群，分别授权两个用户如果环境变量有问题可以在脚本/usr/local/kafka/bin/kafka-run-class.sh里面添加JAVA_HOME=/usr/local/javasystemctl start zookeeper systemctl enable zookeeper systemctl status zookeeper开始创建用户创建语句/usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.9:2181 --alter \ --add-config 'SCRAM-SHA-256=[iterations=8192,password=linuxea.com],SCRAM-SHA-256=[password=linuxea.com]' \ --entity-type users --entity-name marksugar /usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.9:2181 --alter \ --add-config 'SCRAM-SHA-256=[iterations=8192,password=MwMzA0MGIwZjMwMjg3MjY4NWE2ZGFmOG],SCRAM-SHA-256=[password=MwMzA0MGIwZjMwMjg3MjY4NWE2ZGFmOG]' \ --entity-type users --entity-name markadmin# 开始创建markadmin [root@kafka1 config]# /usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.9:2181 --alter \ --add-config 'SCRAM-SHA-256=[iterations=8192,password=MwMzA0MGIwZjMwMjg3MjY4NWE2ZGFmOG],SCRAM-SHA-256=[password=MwMzA0MGIwZjMwMjg3MjY4NWE2ZGFmOG]' \ --entity-type users --entity-name markadmin Warning: --zookeeper is deprecated and will be removed in a future version of Kafka. Use --bootstrap-server instead to specify a broker to connect to. Completed updating config for entity: user-principal 'markadmin' # 开始创建marksugar [root@kafka1 config]# /usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.9:2181 --alter \ --add-config 'SCRAM-SHA-256=[iterations=8192,password=linuxea.com],SCRAM-SHA-256=[password=linuxea.com]' \ --entity-type users --entity-name marksugar Warning: --zookeeper is deprecated and will be removed in a future version of Kafka. Use --bootstrap-server instead to specify a broker to connect to. Completed updating config for entity: user-principal 'marksugar'.查看所有SCRAM证书/usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.9:2181 --describe --entity-type users如下[root@kafka1 config]# /usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.9:2181 --describe --entity-type users Warning: --zookeeper is deprecated and will be removed in a future version of Kafka. Use --bootstrap-server instead to specify a broker to connect to. Configs for user-principal 'markadmin' are SCRAM-SHA-256=salt=MWtxOG56cHNybGhoank1Nmg2M3dsa2JwZGw=,stored_key=G6nlglpSF0uQDskBmV3uOrpuGwEcFKfeTOcaIpuqINY=,server_key=pBEXAihvOLqAGzns2fbu2p96LqLVLM78clUAyftpMjg=,iterations=4096 Configs for user-principal 'marksugar' are SCRAM-SHA-256=salt=MTJnbGxpMWRzajZoMXRvcnBxcXF3b241MDY=,stored_key=mCocSbPBI0yPp12Kr9131nFDA6GIP11p++FQwp0+Ri4=,server_key=vzMqKkT+ZwaVWOryD2owVlMk5gMEaSW2wZI+s1x9Fd8=,iterations=4096查看单个用户的证书/usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.9:2181 --describe --entity-type users --entity-name marksugar /usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.9:2181 --describe --entity-type users --entity-name markadmin查看[root@kafka1 config]# /usr/local/kafka/bin/kafka-configs.sh --zookeeper 172.16.100.9:2181 --describe --entity-type users --entity-name marksugar Warning: --zookeeper is deprecated and will be removed in a future version of Kafka. Use --bootstrap-server instead to specify a broker to connect to. Configs for user-principal 'marksugar' are SCRAM-SHA-256=salt=MTJnbGxpMWRzajZoMXRvcnBxcXF3b241MDY=,stored_key=mCocSbPBI0yPp12Kr9131nFDA6GIP11p++FQwp0+Ri4=,server_key=vzMqKkT+ZwaVWOryD2owVlMk5gMEaSW2wZI+s1x9Fd8=,iterations=4096 [root@kafka1 config]# /usr/local/kafka/bin/kafka-configs.sh --zookeeper 10.100.63.9:2181 --describe --entity-type users --entity-name markadmin Warning: --zookeeper is deprecated and will be removed in a future version of Kafka. Use --bootstrap-server instead to specify a broker to connect to. Configs for user-principal 'markadmin' are SCRAM-SHA-256=salt=MWtxOG56cHNybGhoank1Nmg2M3dsa2JwZGw=,stored_key=G6nlglpSF0uQDskBmV3uOrpuGwEcFKfeTOcaIpuqINY=,server_key=pBEXAihvOLqAGzns2fbu2p96LqLVLM78clUAyftpMjg=,iterations=4096启动kafka授权完成，启动第一台kafka。对目录进行授权chown -R kafka.kafka /usr/local/kafka*先手动启动测试是否正常sudo -u kafka KAFKA_OPTS=-Djava.security.auth.login.config=/usr/local/kafka/config/kafka_server_jaas.conf /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server-scram.properties观察日志/usr/local/kafka/logs/server.log，正常情况下能够看到如下提示已经启动[2021-05-21 17:12:03,524] INFO [KafkaServer id=3] started (kafka.server.KafkaServer)此时kafka需要配置hosts，hosts包含所有的主机名和代理主机名172.16.100.7 kafka1 172.16.100.8 kafka2 172.16.100.9 kafka3 172.16.100.10 kafka.linuxea.com如果没用问题配置开启启动systemctl enable kafka systemctl start kafka systemctl status kafka并且以此启动其他两台验证用户创建主题/usr/local/kafka/bin/kafka-topics.sh --zookeeper 10.100.63.7:2181 --create --topic test --partitions 12 --replication-factor 3发送消息export KAFKA_OPTS="-Djava.security.auth.login.config=/usr/local/kafka/config/kafka_client_jaas.conf"cat producer.confsecurity.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="xyt#*admin.com&!k4";内网访问/usr/local/kafka/bin/kafka-console-producer.sh --broker-list 172.16.100.7:9092,172.16.100.8:9093,172.16.100.9:9094 --topic test --producer.config producer.conf > hello远程/usr/local/kafka/bin/kafka-console-producer.sh --broker-list kafka1:9092,kafka2:9093,kafka3:9094 --topic test --producer.config producer.conf消费消息cat consumer.confsecurity.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="admin.com&!k4";内网访问/usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server 172.16.100.7:9092,172.16.100.8:9093,172.16.100.9:9094 --topic test --from-beginning --consumer.config consumer.conf hello远程/usr/local/kafka/bin/kafka-console-consumer.sh --broker-list kafka1:9092,kafka2:9093,kafka3:9094 --topic test --from-beginning --consumer.config consumer.conf构建代理层nginx stream配置stream { log_format proxy '$remote_addr [$time_local]' '$protocol $status $bytes_sent $bytes_received' '$session_time "$upstream_addr" ' '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"'; upstream kafka1 { server kafka1:9092 weight=1; } server { listen 9092; proxy_pass kafka1; access_log /data/logs/9092.log proxy ; } upstream kafka2 { server kafka2:9093 weight=1; } server { listen 9093; proxy_pass kafka2; access_log /data/logs/9093.log proxy ; } upstream kafka3 { server kafka3:9094 weight=1; } server { listen 9094; proxy_pass kafka3; access_log /data/logs/9094.log proxy ; } }添加hosts，并且kafka节点也要如下配置172.16.100.7 kafka1 172.16.100.8 kafka2 172.16.100.9 kafka3 172.16.100.10 kafka.linuxea.com测试kafka连通性测试节点也需要配置hosts指向proxy172.16.100.10 kafka.linuxea.com安装 python 3.8 ，并且安装confluent_kafkapip install -i https://pypi.tuna.tsinghua.edu.cn/simple confluent_kafka Andpip install -i https://mirrors.aliyun.com/pypi/simple confluent_kafka脚本如下# !/usr/bin/python # #encoding=utf-8 from confluent_kafka import Producer import json from datetime import datetime """ def producer_demo(): # 假设生产的消息为键值对（不是一定要键值对），且序列化方式为json producer = KafkaProducer(bootstrap_servers=['IP:9092'], security_protocol='SASL_PLAINTEXT', #sasl_mechanism="SCRAM-SHA-256", sasl_mechanism='PLAIN', #sasl_kerberos_service_name='admin', #sasl_kerberos_domain_name='hadoop.hadoop.com', sasl_plain_username='admin', sasl_plain_password="*admin.com", #key_serializer=lambda k: json.dumps(k).encode('utf-8'), value_serializer=lambda v: json.dumps(v).encode('utf-8') # ,api_version=(0, 10) ) # 连接kafka msg_dict = "Hello World".encode('utf-8') # 发送内容,必须是bytes类型 for i in range(0, 3): #msg = json.dumps(msg_dict) future = producer.send('test', msg_dict, partition=0) try: future.get(timeout=10) # 监控是否发送成功 except kafka_errors: # 发送失败抛出kafka_errors traceback.format_exc() producer.close() """ def confluentKafkaDemo(): topic_name = 'test' ## count = 100 start = 0 conf = { 'bootstrap.servers': 'kafka.linuxea.com:9092,kafka.linuxea.com:9093,kafka.linuxea.com:9094', 'security.protocol': 'SASL_PLAINTEXT', 'sasl.mechanisms': 'SCRAM-SHA-256', 'sasl.username': 'linuxea', ## 用户名 'sasl.password': 'MwMzA0MGFmOG' ## 密码 } producer = Producer(**conf) data = { 'name': 'test1 is ok', 'time': str(datetime.now()) } try: while start < count: producer.produce(topic_name, (json.dumps(data)).encode(), callback=delivery_report) producer.flush() start = start+1 except Exception as e: print(e) def delivery_report(err, msg): if err is not None: print('Message delivery failed: {}'.format(err)) else: print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition())) if __name__ == '__main__': #producer_demo() confluentKafkaDemo()执行脚本查看是否插入成功kafka-eaglekafka-eagle在被使用了用户验证的集群将能不能够正常使用，总会有一些瑕疵kafka-eagle上仍然需要做hosts解析172.16.100.7 kafka1 172.16.100.8 kafka2 172.16.100.9 kafka3下载2.0.5tar xf kafka-eagle-bin-2.0.5.tar.gz -C /usr/local/ cd /usr/local/kafka-eagle-bin-2.0.5 tar xf kafka-eagle-web-2.0.5-bin.tar.gz ln -s /usr/local/kafka-eagle-bin-2.0.5/ /usr/local/kafka-eagle cp /usr/local/kafka-eagle/kafka-eagle-web-2.0.5/conf/system-config.properties /usr/local/kafka-eagle/kafka-eagle-web-2.0.5/conf/system-config.properties.bak mkdir /data/kafka-eagle/db/ -p为了方便，kafka-eagle必须修改hostname为ip地址hostnamectl set-hostname 192.168.3.6配置环境变量cat > /etc/profile.d/kafka-eagle.sh <<EOF export KE_HOME=/usr/local/kafka-eagle/kafka-eagle-web-2.0.5 export PATH=\$PATH:\$JAVA_HOME/bin:\$KE_HOME/bin EOF source /etc/profile.d/kafka-eagle.shjavaexport JAVA_HOME=/usr/local/jdk1.8.0_211 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH创建ke.dbcd /data/kafka-eagle/db/ && sqlite3 ke.db配置文件中删掉cluster2的配置。修改zk地址，sasl开启验证###################################### # multi zookeeper & kafka cluster list ###################################### kafka.eagle.zk.cluster.alias=cluster1 cluster1.zk.list=172.16.100.7:2181,172.16.100.8:2181,172.16.100.9:2181 ###################################### # zookeeper enable acl ###################################### cluster1.zk.acl.enable=false cluster1.zk.acl.schema=digest cluster1.zk.acl.username=test cluster1.zk.acl.password=test123 ###################################### # broker size online list ###################################### cluster1.kafka.eagle.broker.size=20 ###################################### # zk client thread limit ###################################### kafka.zk.limit.size=32 ###################################### # kafka eagle webui port ###################################### kafka.eagle.webui.port=8048 ###################################### # kafka jmx acl and ssl authenticate ###################################### cluster1.kafka.eagle.jmx.acl=false cluster1.kafka.eagle.jmx.user=keadmin cluster1.kafka.eagle.jmx.password=keadmin123 cluster1.kafka.eagle.jmx.ssl=false cluster1.kafka.eagle.jmx.truststore.location=/Users/dengjie/workspace/ssl/certificates/kafka.truststore cluster1.kafka.eagle.jmx.truststore.password=ke123456 ###################################### # kafka offset storage ###################################### cluster1.kafka.eagle.offset.storage=kafka ###################################### # kafka jmx uri ###################################### cluster1.kafka.eagle.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/jmxrmi ###################################### # kafka metrics, 15 days by default ###################################### kafka.eagle.metrics.charts=true kafka.eagle.metrics.retain=15 ###################################### # kafka sql topic records max ###################################### kafka.eagle.sql.topic.records.max=5000 ###################################### # delete kafka topic token ###################################### kafka.eagle.topic.token=keadmin ###################################### # kafka sasl authenticate ###################################### cluster1.kafka.eagle.sasl.enable=true cluster1.kafka.eagle.sasl.protocol=SASL_PLAINTEXT cluster1.kafka.eagle.sasl.mechanism=SCRAM-SHA-256 cluster1.kafka.eagle.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="markadmin" password="markadmin.com"; cluster1.kafka.eagle.sasl.client.id= cluster1.kafka.eagle.blacklist.topics= cluster1.kafka.eagle.sasl.cgroup.enable=false cluster1.kafka.eagle.sasl.cgroup.topics= ###################################### # kafka sqlite jdbc driver address ###################################### kafka.eagle.driver=org.sqlite.JDBC kafka.eagle.url=jdbc:sqlite:/data/kafka-eagle/db2.0.5/ke.db kafka.eagle.username=root kafka.eagle.password=www.kafka-eagle.org启动/usr/local/kafka-eagle/kafka-eagle-web-2.0.5/bin/ke.sh startkafka增加副本{ "topics": [ {"topic": "linuxea_position_shaanxi_1"} ], "version": 1 }[root@linuxea06 bin]# ./kafka-reassign-partitions.sh --zookeeper 172.16.100.9:2181 --topics-to-move-json-file linuxea_position_shaanxi_1.json --broker-list "0,1,2,3,4,5" --generate Current partition replica assignment {"version":1,"partitions":[{"topic":"linuxea_position_shaanxi_1","partition":3,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"linuxea_position_shaanxi_1","partition":0,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"linuxea_position_shaanxi_1","partition":5,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"linuxea_position_shaanxi_1","partition":4,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"linuxea_position_shaanxi_1","partition":1,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"linuxea_position_shaanxi_1","partition":2,"replicas":[2,1,3],"log_dirs":["any","any","any"]}]} Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"linuxea_position_shaanxi_1","partition":3,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"linuxea_position_shaanxi_1","partition":0,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"linuxea_position_shaanxi_1","partition":5,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"linuxea_position_shaanxi_1","partition":2,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"linuxea_position_shaanxi_1","partition":1,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"linuxea_position_shaanxi_1","partition":4,"replicas":[1,3,2],"log_dirs":["any","any","any"]}]}复制{"version":1,"partitions":[ {"topic":"linuxea_position_shaanxi_1","partition":3,"replicas":[3,2,1],"log_dirs":["any","any","any"]}, {"topic":"linuxea_position_shaanxi_1","partition":0,"replicas":[3,1,2],"log_dirs":["any","any","any"]}, {"topic":"linuxea_position_shaanxi_1","partition":5,"replicas":[2,1,3],"log_dirs":["any","any","any"]}, {"topic":"linuxea_position_shaanxi_1","partition":2,"replicas":[2,3,1],"log_dirs":["any","any","any"]}, {"topic":"linuxea_position_shaanxi_1","partition":1,"replicas":[1,2,3],"log_dirs":["any","any","any"]}, {"topic":"linuxea_position_shaanxi_1","partition":4,"replicas":[1,3,2],"log_dirs":["any","any","any"]}]}执行./kafka-reassign-partitions.sh --zookeeper 172.16.100.9:2181 --reassignment-json-file linuxea_position_shaanxi_1.json -execute查看[root@linuxea06 bin]# ./kafka-reassign-partitions.sh --zookeeper 172.16.100.9:2181 --reassignment-json-file linuxea_position_shaanxi_1.json --verify Status of partition reassignment: Reassignment of partition linuxea_position_shaanxi_1-3 completed successfully Reassignment of partition linuxea_position_shaanxi_1-0 completed successfully Reassignment of partition linuxea_position_shaanxi_1-5 completed successfully Reassignment of partition linuxea_position_shaanxi_1-4 completed successfully Reassignment of partition linuxea_position_shaanxi_1-1 completed successfully Reassignment of partition linuxea_position_shaanxi_1-2 completed successfully [root@linuxea06 bin]#kafka删除topicsleder列出/usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181删除/usr/local/kafka/bin/kafka-topics.sh --delete --zookeeper 17.168.0.174:2181 --topic test1 Topic test1 is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true.创建节点数：--replication-factor 3 分区： --partitions 18/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 18 --topic topic1删除group我们使用kafka-consumer-groups.sh列出consumer[root@linuxea06 bin]# ./kafka-consumer-groups.sh --bootstrap-server 172.16.100.9:9092 --list --command-config ../config/admin.conf linuxea-python-consumer-group-position_3 spring-boot-group-position-agg-linuxea-pre spring-boot-group-position-agg-linuxea-1 linuxea-consumer-third-party spring-boot-alarm-outage-shaanxi-group1 spring-boot-alarm-offlinenotask-linuxea-group1 spring-boot-group-position-linuxea-2 spring-boot-alarm-outage-linuxea-group1开始删除./kafka-consumer-groups.sh \ --bootstrap-server <bootstrap-server-url> \ --delete-offsets \ --group linuxea_position_shaanxi_1 \ --topic spring-boot-group-position-agg-linuxea-1报错解决Error: Executing consumer group command failed due to org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups java.util.concurrent.ExecutionException: org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.listGroups(ConsumerGroupCommand.scala:180) at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:61) at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala) Caused by: org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups at org.apache.kafka.clients.admin.KafkaAdminClient$23.handleFailure(KafkaAdminClient.java:2773) at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:641) at org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:757) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutPendingCalls(KafkaAdminClient.java:825) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1119) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.添加认证信息 --command-config ../config/admin.conf参考kafka-consumer-group-script-to-see-all-consumer-group-not-workinghow-to-remove-a-kafka-consumer-group-from-a-specific-topickafka-consumer-groups.sh消费者组管理Kafka 实现内外网访问流量分离

linuxea:mongodb性能诊断工具 下载mongodb-database-tools后就自带了mongostat命令使用mongostat使用超级管理员查看admin库 1 是间隔1秒的意思 ./mongostat -h 127.0.0.1:27020 -u superadmin --password=password --authenticationDatabase=admin 1insert query update delete getmore command ：在最近的一秒钟发生的操作。getmore：是针对一些游标操作，第一次拿了100条，而游标返回了10000w条，需要一次次抓回来，这里表示有多少个getmore在执行command ：表示除了增删改查，包含增删改查，包括index等状态的命令都是通过command 来完成的dirty ：mongodb是通过内存来缓存和增加数据，mongodb通过缓存来管理一些数据，默认是百分之60的内存使用，在这些内存之间有多少是被更新的，修改的，或者插入的。这些操作且没有刷盘的，就是dirty。这里的百分比表示没有刷盘的占用百分比。如果没有超过百分之5，后台程序会5分钟刷一次数据到盘上。如果百分比超过5，后台会启动程序自动把这些刷到盘上。从而尽量的保证在百分之5以下。而一旦超过百分之20，mongodb可能会停止接受前面的请求也要将数据刷到盘上，也就是超过百分之20时阻塞新请求。这种情况，要么是刷盘跟不上，磁盘IO跟不上。要么是写入压力太大。used ：默认是百分之60的内存使用，百分之60的内存已经有百分之多少被使用了。低于百分之80的时候，mongdb不会做什么事情，超过百分之80开始清理动作，基于一些很久没有被使用的数据剔除，缓存清理动作。超过百分之95时阻塞新请求，并专注于清理缓存。vsize ：使用的内存大小res: 实际使用的内存大小qrw /arw ： 超过10存在等待net_in /net_out ： 进出conn ： 连接数使用mongotop./mongotop -h 127.0.0.1:27020 -u superadmin --password=password --authenticationDatabase=admin 2间隔2秒抓一次ns : 来源库total：总时间消耗read：读时间消耗write：写时间消耗使用日志配置db.getProfilingStatus()查看是否开启rs02:PRIMARY> db.getProfilingStatus() { "was" : 1, "slowms" : 10, "sampleRate" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1628512885, 1), "signature" : { "hash" : BinData(0,"RXoRmbr9npCqd6IxClxVYWeIjzg="), "keyId" : NumberLong("6993996578134425604") } }, "operationTime" : Timestamp(1628512885, 1) }设置：db.setProfilingLevel(level, slowms)查看：db.system.profile.find()evel：0，关闭；1，记录慢速操作；2，记录所有操作。slowms：时间阈值，单位ms。表示10毫秒的：db.setProfilingLevel( 1 , 10);默认100ms示例2019-10-25T11:20:30.775+0800 I COMMAND [conn12] command test.test appName: "MongoDB Shell" command: find { find: "test", filter: { i: 10000.0 }, $db: "test" } planSummary: COLLSCAN keysExamined:0 docsExamined:27108 cursorExhausted:1 numYields:211 nreturned:1 reslen:243 locks:{ Global: { acquireCount: { r: 424 } }, Database: { acquireCount: { r: 212 } }, Collection: { acquireCount: { r: 212 } } } protocol:op_command 19msdocsExamined ： 扫描文档数locks： 锁信息numYields: 执行计划durationMillis: 执行时间mtoolshttps://github.com/rueckstiess/mtools/releaseshttps://github.91chifun.workers.dev//https://github.com/rueckstiess/mtools/archive/refs/tags/v1.6.4.tar.gzmtools可以分析mongo日志yum install python3 -ytar xf v1.6.4.tar.gz && cd mtools-1.6.4 python3 setup.py install常用指令：• mplotqueries 日志文件：将所有慢查询通过图表形式展现；• mloginfo --queries 日志文件：总结出所有慢查询的模式和出现次数、消耗时间等；