Redis集群模式部署文档（redis集群的三种模式）

创建和使用Redis群集

参考链接：http://www.redis.cn/topics/cluster-tutorial.html

节点备注

192.168.0.59:7000

192.168.0.59:7001

192.168.0.59:7002

192.168.0.60:7003

192.168.0.61:7004

192.168.0.62:7005

192.168.0.59操作

1.下载编译安装

cd /opt/
wget http://download.redis.io/releases/redis-3.2.4.tar.gz
tar -zxvf redis-3.2.4.tar.gz
cd redis-3.2.4
make && make install

2.配置redis

初始化目录

mkdir -p /opt/redis-cluster/{etc,bin,data,etc,log,var,var/run}
cp -r src/{redis-benchmark,redis-check-aof,redis-check-rdb,redis-cli,redis-sentinel,redis-server,redis-trib.rb} /opt/redis-cluster/bin/
cp -r redis.conf /opt/redis-cluster/etc/redis-7000.conf
cp -r redis.conf /opt/redis-cluster/etc/redis-7001.conf
cp -r redis.conf /opt/redis-cluster/etc/redis-7002.conf
cp -r redis.conf /opt/redis-cluster/etc/redis-common.conf

配置文件

redis-common.conf通用配置，redis-7000.conf节点配置，其他节点参考

redis-common.conf

#GENERAL 
daemonize yes 
tcp-backlog 511 
timeout 15
tcp-keepalive 0 
loglevel notice 
databases 16 
dir /opt/redis-cluster/data/
slave-serve-stale-data yes 
#slave只读 
slave-read-only yes 
#not use default 
repl-disable-tcp-nodelay yes 
slave-priority 100 
#打开aof持久化 
appendonly yes 
#每秒一次aof写 
appendfsync everysec 
#关闭在aof rewrite的时候对新的写操作进行fsync 
no-appendfsync-on-rewrite yes 
auto-aof-rewrite-min-size 500mb 
lua-time-limit 5000 
#打开redis集群 
cluster-enabled yes 
#节点互连超时的阀值 
cluster-node-timeout 5000 
cluster-migration-barrier 1 
slowlog-log-slower-than 10000 
slowlog-max-len 128 
notify-keyspace-events "" 
hash-max-ziplist-entries 512 
hash-max-ziplist-value 64 
list-max-ziplist-entries 512 
list-max-ziplist-value 64 
set-max-intset-entries 512 
zset-max-ziplist-entries 128 
zset-max-ziplist-value 64 
activerehashing yes 
client-output-buffer-limit normal 0 0 0 
client-output-buffer-limit slave 256mb 64mb 60 
client-output-buffer-limit pubsub 32mb 8mb 60 
hz 10 
aof-rewrite-incremental-fsync yes
cluster-require-full-coverage no
maxclients 65535

redis-7000.conf

bind {{本机ip}}
#包含通用配置 
include /opt/redis-cluster/etc/redis-common.conf 
#监听tcp端口 
port 7000
#最大可用内存，根据实际情况调整 
maxmemory 12000m
#内存耗尽时采用的淘汰策略: 
# volatile-lru -> remove the key with an expire set using an LRU algorithm 
# allkeys-lru -> remove any key accordingly to the LRU algorithm 
# volatile-random -> remove a random key with an expire set 
# volatile-lru -> remove a random key, any key 
# volatile-ttl -> remove the key with the nearest expire time (minor TTL) 
# noeviction -> don't expire at all, just return an error on write operations 
maxmemory-policy allkeys-lru 
#aof存储文件 
appendfilename "appendonly-7000.aof" 
#rdb文件,只用于动态添加slave过程 
#dbfilename dump-7000.rdb 
#cluster配置文件(启动自动生成) 
cluster-config-file nodes-7000.conf 
#部署在同一机器的redis实例，把<span style="font-size: 1em; line-height: 1.5;">auto-aof-rewrite搓开，防止瞬间fork所有redis进程做rewrite,占用大量内存</span> 
auto-aof-rewrite-percentage 80-100
pidfile /opt/redis-cluster/var/run/redis_7000.pid
logfile "/opt/redis-cluster/log/redis-7000.log"

安装上面方法，依次修改redis-7001.conf redis-7002.conf

192.168.0.60操作（同上面1、2步骤一样，注意配置文件名称）

3.启动redis实例（两台机器操作）

192.168.0.59

/opt/redis-cluster/bin/redis-server /opt/redis-cluster/etc/redis-7000.conf
/opt/redis-cluster/bin/redis-server /opt/redis-cluster/etc/redis-7001.conf
/opt/redis-cluster/bin/redis-server /opt/redis-cluster/etc/redis-7002.conf

192.168.0.60

/opt/redis-cluster/bin/redis-server /opt/redis-cluster/etc/redis-7003.conf
/opt/redis-cluster/bin/redis-server /opt/redis-cluster/etc/redis-7004.conf
/opt/redis-cluster/bin/redis-server /opt/redis-cluster/etc/redis-7005.conf

查看启动情况

ps -ef | grep redis
redis-cli -h 192.168.0.60 -p 7003 info

4.创建集群

Redis 官方提供了 redis-trib.rb 这个工具，可以管理集群

redis-trib.rb create --replicas 1 192.168.0.59:7000 192.168.0.59:7001 192.168.0.59:7002 192.168.0.60:7003 192.168.0.60:7004 192.168.0.60:7005

这个命令在这里用于创建一个新的集群, 选项–replicas 1 表示我们希望为集群中的每个主节点创建一个从节点。 最后可以得到如下信息：

[OK] All 16384 slots covered
这表示集群中的 16384 个槽都有至少一个主节点在处理， 集群运作正常。

如果创建集群报错“/usr/bin/env: ruby: 没有那个文件或目录”，执行以下命令

或者报这个错/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in require': cannot load such file -- redis (LoadError) from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:inrequire' from /opt/redis-cluster/bin/redis-trib.rb:25:in `

执行以下命令

yum install centos-release-scl-rh
yum install rh-ruby23 -y
scl enable rh-ruby23 bash
gem install redis

5.查看集群状态

集群节点分配情况

$ redis-cli -p 7000 -h 192.168.0.59 -c cluster nodes
18b35126c0f506d3b8337e1f503138e14840d28f 192.168.0.60:7005 master - 0 1564646879573 8 connected 10923-16383
ca9f969706955afadd1e409754a539b1787dfb53 192.168.0.59:7000 myself,master - 0 0 1 connected 0-5460
3285ba0e2d549e132f898411cef9d83f03f9b3f6 192.168.0.60:7003 slave d214086f4368309af899c757f9a5793e5a73e99d 0 1564646878070 7 connected
2f2f40deb9283ee1b3d2eaa74fa9e35086c6b857 192.168.0.60:7004 slave ca9f969706955afadd1e409754a539b1787dfb53 0 1564646879072 5 connected
d214086f4368309af899c757f9a5793e5a73e99d 192.168.0.59:7002 master - 0 1564646879072 7 connected 5461-10922
69fbc2ef97cfc2c94ceb3e649d2aa38fc5890e1a 192.168.0.59:7001 slave 18b35126c0f506d3b8337e1f503138e14840d28f 0 1564646877568 8 connected

集群状态

$ redis-cli -p 7000 -h 192.168.0.59 -c cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:8
cluster_my_epoch:1
cluster_stats_messages_sent:9925
cluster_stats_messages_received:6496

自动故障转移

查看下目前集群节点分布情况,7005、7000、7002是master

$ redis-cli -p 7000 -h 192.168.0.59 -c cluster nodes
18b35126c0f506d3b8337e1f503138e14840d28f 192.168.0.60:7005 master - 0 1564646879573 8 connected 10923-16383
ca9f969706955afadd1e409754a539b1787dfb53 192.168.0.59:7000 myself,master - 0 0 1 connected 0-5460
3285ba0e2d549e132f898411cef9d83f03f9b3f6 192.168.0.60:7003 slave d214086f4368309af899c757f9a5793e5a73e99d 0 1564646878070 7 connected
2f2f40deb9283ee1b3d2eaa74fa9e35086c6b857 192.168.0.60:7004 slave ca9f969706955afadd1e409754a539b1787dfb53 0 1564646879072 5 connected
d214086f4368309af899c757f9a5793e5a73e99d 192.168.0.59:7002 master - 0 1564646879072 7 connected 5461-10922
69fbc2ef97cfc2c94ceb3e649d2aa38fc5890e1a 192.168.0.59:7001 slave 18b35126c0f506d3b8337e1f503138e14840d28f 0 1564646877568 8 connected

查入几条数据，在看下分别在哪个节点,test1在7000、test2在7002、test3在7005

$ redis-cli -p 7000 -h 192.168.0.59 -c
192.168.0.59:7000> set test1 1
OK
192.168.0.59:7000> set test2 2
-> Redirected to slot [8899] located at 192.168.0.59:7002
OK
192.168.0.59:7002> set test3 3
-> Redirected to slot [13026] located at 192.168.0.60:7005
OK
192.168.0.60:7005> set test4 4
-> Redirected to slot [517] located at 192.168.0.59:7000
OK
192.168.0.59:7000>

模拟7005故障，可以看到其他从节点切换为master节点，7005节点上面的key test3也自动迁移了

192.168.0.59:7000> cluster nodes
18b35126c0f506d3b8337e1f503138e14840d28f 192.168.0.60:7005 master,fail - 1564648759168 1564648756963 8 disconnected
ca9f969706955afadd1e409754a539b1787dfb53 192.168.0.59:7000 myself,master - 0 0 1 connected 0-5460
3285ba0e2d549e132f898411cef9d83f03f9b3f6 192.168.0.60:7003 slave d214086f4368309af899c757f9a5793e5a73e99d 0 1564648765504 7 connected
2f2f40deb9283ee1b3d2eaa74fa9e35086c6b857 192.168.0.60:7004 slave ca9f969706955afadd1e409754a539b1787dfb53 0 1564648765003 5 connected
d214086f4368309af899c757f9a5793e5a73e99d 192.168.0.59:7002 master - 0 1564648766006 7 connected 5461-10922
69fbc2ef97cfc2c94ceb3e649d2aa38fc5890e1a 192.168.0.59:7001 master - 0 1564648763997 9 connected 10923-16383
192.168.0.59:7000> get test3
-> Redirected to slot [13026] located at 192.168.0.59:7001
"3"

可以从上面看到7001从slave变成了master，key test3也从7005迁移到了7001，正常读写也没问题

手动故障迁移

Redis集群使用 CLUSTER FAILOVER命令来进行故障转移，不过要被转移的主节点的从节点上执行该命令 手动故障转移比主节点失败自动故障转移更加安全

人为failover支持三种模式的failover：缺省、force、takeover。具体意思自行官网看，

特别说明CLUSTER FAILOVER命令只能在slave节点执行

查看目前集群节点情况

192.168.0.59:7001> cluster nodes
3285ba0e2d549e132f898411cef9d83f03f9b3f6 192.168.0.60:7003 slave d214086f4368309af899c757f9a5793e5a73e99d 0 1564649673499 7 connected
69fbc2ef97cfc2c94ceb3e649d2aa38fc5890e1a 192.168.0.59:7001 myself,master - 0 0 9 connected 10923-16383
2f2f40deb9283ee1b3d2eaa74fa9e35086c6b857 192.168.0.60:7004 slave ca9f969706955afadd1e409754a539b1787dfb53 0 1564649671995 5 connected
d214086f4368309af899c757f9a5793e5a73e99d 192.168.0.59:7002 master - 0 1564649673498 7 connected 5461-10922
18b35126c0f506d3b8337e1f503138e14840d28f 192.168.0.60:7005 slave 69fbc2ef97cfc2c94ceb3e649d2aa38fc5890e1a 0 1564649672998 9 connected
ca9f969706955afadd1e409754a539b1787dfb53 192.168.0.59:7000 master - 0 1564649674000 1 connected 0-5460

在slave节点192.168.0.69:7005执行,可以看到7005变成master了，7001变成slave了

$ redis-cli -h 192.168.0.60 -c -p 7005 
192.168.0.60:7005> CLUSTER FAILOVER
OK
192.168.0.60:7005> cluster nodes
18b35126c0f506d3b8337e1f503138e14840d28f 192.168.0.60:7005 myself,master - 0 0 10 connected 10923-16383
3285ba0e2d549e132f898411cef9d83f03f9b3f6 192.168.0.60:7003 slave d214086f4368309af899c757f9a5793e5a73e99d 0 1564649957272 7 connected
69fbc2ef97cfc2c94ceb3e649d2aa38fc5890e1a 192.168.0.59:7001 slave 18b35126c0f506d3b8337e1f503138e14840d28f 0 1564649958275 10 connected
ca9f969706955afadd1e409754a539b1787dfb53 192.168.0.59:7000 master - 0 1564649957773 1 connected 0-5460
2f2f40deb9283ee1b3d2eaa74fa9e35086c6b857 192.168.0.60:7004 slave ca9f969706955afadd1e409754a539b1787dfb53 0 1564649956771 5 connected
d214086f4368309af899c757f9a5793e5a73e99d 192.168.0.59:7002 master - 0 1564649956270 7 connected 5461-10922
192.168.0.60:7005>

添加一个新节点

1.部署新的节点

...过程忽略
192.168.0.60:7006

2.使用redis-trib命令添加集群节点

$ /opt/redis-cluster/bin/redis-trib.rb add-node 192.168.0.60:7006 192.168.0.59:7000
>>> Adding node 192.168.0.60:7006 to cluster 192.168.0.59:7000
>>> Performing Cluster Check (using node 192.168.0.59:7000)
...
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.0.60:7006 to make it join the cluster.
[OK] New node added correctly.

使用add-node参数来添加节点，第一个参数是新节点的地址，第二个参数是任意一个已经存在的节点的IP和端口. 我们可以看到新的节点已经添加到集群中:

$ redis-cli -p 7000 -h 192.168.0.59 -c 
192.168.0.59:7000> cluster nodes
81fefa031d4f738f87f739a3f8d715c436f35478 192.168.0.60:7006 master - 0 1564650457080 0 connected
18b35126c0f506d3b8337e1f503138e14840d28f 192.168.0.60:7005 master - 0 1564650456035 10 connected 10923-16383
ca9f969706955afadd1e409754a539b1787dfb53 192.168.0.59:7000 myself,master - 0 0 1 connected 0-5460
3285ba0e2d549e132f898411cef9d83f03f9b3f6 192.168.0.60:7003 slave d214086f4368309af899c757f9a5793e5a73e99d 0 1564650455534 7 connected
2f2f40deb9283ee1b3d2eaa74fa9e35086c6b857 192.168.0.60:7004 slave ca9f969706955afadd1e409754a539b1787dfb53 0 1564650457581 5 connected
d214086f4368309af899c757f9a5793e5a73e99d 192.168.0.59:7002 master - 0 1564650456538 7 connected 5461-10922
69fbc2ef97cfc2c94ceb3e649d2aa38fc5890e1a 192.168.0.59:7001 slave 18b35126c0f506d3b8337e1f503138e14840d28f 0 1564650457080 10 connected
192.168.0.59:7000>

新节点现在已经连接上了集群， 成为集群的一份子， 并且可以对客户端的命令请求进行转向了， 但是和其他主节点相比， 新节点还有两点区别：

• 新节点没有包含任何数据， 因为它没有包含任何哈希槽.
• 尽管新节点没有包含任何哈希槽， 但它仍然是一个主节点， 所以在集群需要将某个从节点升级为新的主节点时， 这个新节点不会被选中。
• 接下来， 只要使用 redis-trib 程序， 将集群中的某些哈希桶移动到新节点里面， 新节点就会成为真正的主节点了。

3.下面来从集群中迁移槽到新的节点

$ ./redis-trib.rb reshard 192.168.0.59:7000
>>> Performing Cluster Check (using node 192.168.0.59:7000)
...
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
//上面过程会check集群是否正常
How many slots do you want to move (from 1 to 16384)? 4000
//输入4000表示要移动4000个哈希槽
What is the receiving node ID? 9b156fbf170e9dfadb761f30f11efc65f3b00014
//把这4000个哈希槽给7006节点对应的id
Source node #1: ca9f969706955afadd1e409754a539b1787dfb53
//输入all表示从所有的哈希槽拥有者中抽取4000个给7006节点
//输入某个节点的id表示 从该节点上拿4000个槽给7006这个节点
Source node #1: done
//输入done继续后面迁移槽

现象：
开始gem安装redis默认版本，gem install redis，部署集群完毕后，测试代码写入集群数据，然后进行分片，发现一只报错，错误如下，
[ERR] Calling MIGRATE ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)
fix 也无法修复。
解决方法：
ruby gem安装的redis库，版本不能使用最新的4.0，否则redis-trib.rb reshard 127.0.0.1:7000 重新分片时会报语法错误。
1、卸载最新redis库，gem uninstall redis
2、安装3.x版本，gem install redis -v 3.3.5 测试3.2.1到3.3.5都可以，4.x以上的分片报错。

4.查看迁移后集群槽点情况

192.168.0.59:7001> cluster nodes
0610124f6798e78d36001ee09b654c6a1ab218d2 192.168.0.59:7007 slave 9b156fbf170e9dfadb761f30f11efc65f3b00014 0 1564727723285 12 connected
3285ba0e2d549e132f898411cef9d83f03f9b3f6 192.168.0.60:7003 slave d214086f4368309af899c757f9a5793e5a73e99d 0 1564727723787 7 connected
69fbc2ef97cfc2c94ceb3e649d2aa38fc5890e1a 192.168.0.59:7001 myself,slave 18b35126c0f506d3b8337e1f503138e14840d28f 0 0 9 connected
2f2f40deb9283ee1b3d2eaa74fa9e35086c6b857 192.168.0.60:7004 slave ca9f969706955afadd1e409754a539b1787dfb53 0 1564727723788 5 connected
9b156fbf170e9dfadb761f30f11efc65f3b00014 192.168.0.60:7006 master - 0 1564727724289 12 connected 0-4517
d214086f4368309af899c757f9a5793e5a73e99d 192.168.0.59:7002 master - 0 1564727725291 7 connected 5461-10922
18b35126c0f506d3b8337e1f503138e14840d28f 192.168.0.60:7005 master - 0 1564727725291 10 connected 10923-16383
ca9f969706955afadd1e409754a539b1787dfb53 192.168.0.59:7000 master - 0 1564727724288 1 connected 4518-5460

7000节点槽点迁移到了7006新节点

添加一个从节点

1.部署新的节点

...过程忽略
192.168.0.59:7007

2.使用redis-trib命令添加

第一种方法会在集中的主节点中随机选取一个作为这个从节点的主节点。

$ /opt/redis-cluster/bin/redis-trib.rb add-node --slave 192.168.0.59:7007 192.168.0.59:7000

第二种方法会指定一个主节点作为换个从节点的主节点

$ ./redis-trib.rb add-node --slave --master-id 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 192.168.0.59:7007 192.168.0.59:7000

删除一个从节点

还以192.168.0.60:7007为例子

$ /opt/redis-cluster/bin/redis-trib.rb del-node 192.168.0.59:7000 81fefa031d4f738f87f739a3f8d715c436f35478
>>> Removing node 81fefa031d4f738f87f739a3f8d715c436f35478 from cluster 192.168.0.59:7000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

• 第一个参数是任意一个节点的地址,第二个节点是你想要移除的节点地址。
• 使用同样的方法移除主节点,不过在移除主节点前，需要确保这个主节点是空的. 如果不是空的,需要将这个节点的数据重新分片到其他主节点上.

删除一个主节点

如果主节点有从节点，将从节点转移到其他主节点；如果主节点有slot，去掉分配的slot，然后在删除主节点

1.迁移主节点的槽点到其他节点

./redis-trib.rb reshard 192.168.0.59:7000
>>> Performing Cluster Check (using node 192.168.0.59:7000)
...
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
//上面过程会check集群是否正常
How many slots do you want to move (from 1 to 16384)? 4000
//要删除主节点的哈希槽数量
What is the receiving node ID? 9b156fbf170e9dfadb761f30f11efc65f3b00014
//接收这4000个哈希槽的节点id
Source node #1: ca9f969706955afadd1e409754a539b1787dfb53
//输入要删除节点的id表示从该节点上拿走4000个槽
Source node #1: done
//输入done继续后面迁移槽

2.查看迁移后集群节点槽点

192.168.0.59:7001> cluster nodes
0610124f6798e78d36001ee09b654c6a1ab218d2 192.168.0.59:7007 slave ca9f969706955afadd1e409754a539b1787dfb53 0 1564728483184 13 connected
3285ba0e2d549e132f898411cef9d83f03f9b3f6 192.168.0.60:7003 slave d214086f4368309af899c757f9a5793e5a73e99d 0 1564728483685 7 connected
69fbc2ef97cfc2c94ceb3e649d2aa38fc5890e1a 192.168.0.59:7001 myself,slave 18b35126c0f506d3b8337e1f503138e14840d28f 0 0 9 connected
2f2f40deb9283ee1b3d2eaa74fa9e35086c6b857 192.168.0.60:7004 slave ca9f969706955afadd1e409754a539b1787dfb53 0 1564728485186 13 connected
9b156fbf170e9dfadb761f30f11efc65f3b00014 192.168.0.60:7006 master - 0 1564728484184 12 connected
d214086f4368309af899c757f9a5793e5a73e99d 192.168.0.59:7002 master - 0 1564728485185 7 connected 5461-10922
18b35126c0f506d3b8337e1f503138e14840d28f 192.168.0.60:7005 master - 0 1564728484685 10 connected 10923-16383
ca9f969706955afadd1e409754a539b1787dfb53 192.168.0.59:7000 master - 0 1564728484184 13 connected 0-5460

3.删除空槽点的主节点

$ ./redis-trib.rb del-node 192.168.0.59:7000 9b156fbf170e9dfadb761f30f11efc65f3b00014

4.最后重新平衡下每个节点上面的槽点

$ ./redis-trib.rb rebalance --use-empty-masters 192.168.6.64:6379 
>>> Performing Cluster Check (using node 192.168.6.64:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 16 nodes. Total weight = 16
Moving 342 slots from 192.168.6.61:6382 to 192.168.6.62:6381
######################################################################################################################################################################################################################################################################################################################################################
Moving 342 slots from 192.168.6.61:6382 to 192.168.6.62:6380
######################################################################################################################################################################################################################################################################################################################################################