Redis: Replication, Part 2 – Master-Slave Replication, and Redis Sentinel

setevoy4

0.00/5 (No votes)

30 Mar 2019CPOL7 min read

7.2K

Redis replication setup example with a basic Master-Slave replication and Redis Sentinel

The first part – Redis: replication, part 1 – an overview. Replication vs Sharding. Sentinel vs Cluster. Redis topology.

The whole story was started when we decided to get rid of memcached.

Currently, we have memcahced and Redis running on our backend servers.

And memcached, and Redis instances are working as standalone applications, i.e., they are not connected in any kind of replication and this leads to a problem:

We have three backend hosts which are hosted behind an AWS Application Load Balancer
An ALB has Sticky Sessions enabled but it’s working with cookies which are ignored by our mobile applications (iOS/Android)
Respectively, when a client makes a request to the backend – sometimes, it can get cached data which already was removed/updated on another backend host in Redis or memcached

We have this scheme since we migrated our backend application from an old infrastructure where the only one host was used and still had no time to update it, although it was in our planes a long time.

Currently, to solve these issues, we have a bunch of “hacks” on the backend which makes additional checks to ensure data is up to date, and now to get rid of them we decided to:

Get rid of memcached at all as Redis can be used for the functions where memcached is used now
Configure Redis replication over all hosts

Such a setup will be described in the post below.

The first example – with a basic Master-Slave replication and the second example – the Sentinel set up and configuration.

AWS EC2 instances with Debian 9 will be used here.

To work with Redis hosts, three domain names will be used – redis-0.setevoy.org.ua for a master, redis-1.setevoy.org.ua and redis-2.setevoy.org.ua for its two slaves.

Slave in a minimal setup can be only one but as the second example here will be with the Sentinel – let’s have three from the beginning.

The Basic Master-Slave Replication

In this way, slaves will be a master’s read-only replicas keeping the same data which will be added to the master.

Master will send all data updates to its slaves – new keys expire, etc.

In case the link between master and slave will be broken – a slave will try to reconnect to the master and make a partial synchronization to update data from a place where the previous sync was interrupted.

In case such a partial sync is not possible – the slave will ask the master for a full synchronization and master will perform its data full snapshot which will be sent to this slave, after this, a usual sync will be restored.

A couple of notes here to keep in mind:

One master can have multitype slaves
Slaves can accept connections from other slaves, making kind of “cascade” of a replicated nodes – a master on the top, a slave(s) in the middle and a slave(s) at the bottom
It’s strongly recommended to enable data persistence on the master to avoid data loss – see the Safety of replication when master has persistence turned off
Slave will work in the read-only mode by default, see the Read-only slave

Redis Master Configuration

Install Redis:

root@redis-0:/home/admin# apt -y install redis-server

Edit a /etc/redis/redis.conf and in the bind set interfaces to listen on:

...
bind 0.0.0.0
...

You can specify multitype IPs here separated by spaces:

...
bind 127.0.0.1 18.194.229.23
...

Other valuable options here:

port 6379 – clear enough but keep it in mind
slave-read-only yes – slaves will be working in the read-only mode, doesn’t affect a master node
requirepass foobared – password for master authorization
appendonly yes and appendfilename "appendonly.aof" – decrease data loss chance, see the Redis Persistence

Restart the service:

root@redis-0:/home/admin# systemctl restart redis

Check it using -a for the password:

root@redis-0:/home/admin# redis-cli -a foobared ping

PONG

Check data replication status:

root@redis-0:/home/admin# redis-cli -a foobared info replication

Replication

role:master

connected_slaves:0

master_repl_offset:0

repl_backlog_active:0

repl_backlog_size:1048576

repl_backlog_first_byte_offset:0

repl_backlog_histlen:0

Add a new data:

root@redis-0:/home/admin# redis-cli -a foobared set test 'test'

OK

Get it back:

root@redis-0:/home/admin# redis-cli -a foobared get test

"test"

Okay – everything works here.

Redis Slave Configuration

On the two hosts left, make a slaves configuration.

It will be the same for both – just repeat it.

Install Redis:

root@redis-1:/home/admin# apt -y install redis-server

Edit the /etc/redis/redis.conf:

...
slaveof redis-0.setevoy.org.ua 6379
...
masterauth foobared
...
requirepass foobared
...

Here:

slaveof – set the master’s host and port
masterauth – master’s auth
requirepass – auth on this replica

Restart the service:

root@redis-1:/home/admin# systemctl restart redis

Check its status:

root@redis-1:/home/admin# redis-cli -a foobared info replication

Replication

role:slave

master_host:redis-0.setevoy.org.ua

master_port:6379

master_link_status:up

master_last_io_seconds_ago:5

master_sync_in_progress:0

...

Check log:

root@redis-1:/home/admin# tail -f /var/log/redis/redis-server.log

16961:S 29 Mar 10:54:36.263 * Connecting to MASTER redis-0.setevoy.org.ua:6379

16961:S 29 Mar 10:54:36.308 * MASTER <-> SLAVE sync started

16961:S 29 Mar 10:54:36.309 * Non blocking connect for SYNC fired the event.

16961:S 29 Mar 10:54:36.309 * Master replied to PING, replication can continue...

16961:S 29 Mar 10:54:36.310 * Partial resynchronization not possible (no cached master)

16961:S 29 Mar 10:54:36.311 * Full resync from master: 93585eeb7e32c0550c35f8d4935c9a18c4177ab9:1

16961:S 29 Mar 10:54:36.383 * MASTER <-> SLAVE sync: receiving 92 bytes from master

16961:S 29 Mar 10:54:36.383 * MASTER <-> SLAVE sync: Flushing old data

16961:S 29 Mar 10:54:36.383 * MASTER <-> SLAVE sync: Loading DB in memory

16961:S 29 Mar 10:54:36.383 * MASTER <-> SLAVE sync: Finished with success

Connection to the master established, syncronization is done – okay, check the data:

root@redis-1:/home/admin# redis-cli -a foobared get test

"test"

Data present – all works here as well.

Changing Slave => Master Roles

In case the master will go down – you have to switch one of the slaves to become a new master.

If you’ll try to add any data on a current slave – Redis will raise an error as the slaves are in the read-only mode:

...
slave-read-only yes
...

Try to add something:

root@redis-1:/home/admin# redis-cli -a foobared set test2 'test2'

(error) READONLY You can't write against a read only slave.

Now connect to the slave:

root@redis-1:/home/admin# redis-cli

Authorize:

127.0.0.1:6379> auth foobared

OK

Disable the slave-role:

127.0.0.1:6379> slaveof no one

OK

Check its status now:

127.0.0.1:6379> info replication

Replication

role:master

connected_slaves:0

master_repl_offset:1989

repl_backlog_active:0

repl_backlog_size:1048576

Add a new key one more time:

127.0.0.1:6379> set test2 'test2'

OK

And get it back:

127.0.0.1:6379> get test2

"test2"

Keep in mind that as we did those changes in Redis node directly – after its restart, it will become a slave again as it still is set in its /etc/redis/redis.conf file with the slaveof parameter.

Redis Sentinel

Now let’s add the Sentinel to our replication which will monitor Redis nodes and will do roles switches automatically.

The overall scheme will be next:

Here:

M1 = Master
R1 = Replica 1 / Slave 1
R2 = Replica 2 / Slave 2
S1 = Sentinel 1
S2 = Sentinel 2
S3 = Sentinel 3

M1 and S1 – will be on the redis-0, R1 and S2 – on the redis-1, R2 and S3 – on the redis-2.

Running Sentinel

To run a Sentinel daemon, the redis-server can be used just with a separate config – /etc/redis/sentinel.conf.

First, let’s create such config file on the Redis Master host:

sentinel monitor redis-test redis-0.setevoy.org.ua 6379 2
sentinel down-after-milliseconds redis-test 6001
sentinel failover-timeout redis-test 60000
sentinel parallel-syncs redis-test 1
bind 0.0.0.0
sentinel auth-pass redis-test foobared

Here:

monitor – the master-node address to be monitored, and the 2 is the Sentinel’s instances number to make a decision
down-after-milliseconds – time after which master will be considered as out of order
failover-timeout – time to wait after changing slave=>master roles
parallel-syncs – number of simultaneous slaves synchronization after the master changed

Run it:

root@redis-0:/home/admin# redis-server /etc/redis/sentinel.conf --sentinel

...

10447:X 29 Mar 14:15:53.193 # WARNING: The TCP backlog setting of 511 cannot be enforced 
because /proc/sys/net/core/somaxconn is set to the lower value of 128.

10447:X 29 Mar 14:15:53.195 # Sentinel ID is e9fb72c8edb8ec2028e6ce820b9e72e56e07cf1e

10447:X 29 Mar 14:15:53.195 # +monitor master redis-test 35.158.154.25 6379 quorum 2

10447:X 29 Mar 14:15:53.196 * +slave slave 3.121.223.95:6379 3.121.223.95 6379 
@ redis-test 35.158.154.25 6379

10447:X 29 Mar 14:16:43.402 * +slave slave 18.194.45.17:6379 18.194.45.17 6379 
@ redis-test 35.158.154.25 6379

Check Sentinel’s status using the 26379 port:

root@redis-0:/home/admin# redis-cli -p 26379 info sentinel

Sentinel

sentinel_masters:1

sentinel_tilt:0

sentinel_running_scripts:0

sentinel_scripts_queue_length:0

sentinel_simulate_failure_flags:0

master0:name=redis-test,status=ok,address=35.158.154.25:6379,slaves=2,sentinels=1

Here:

master0:name=redis-test,status=ok – master is UP
slaves=2 – it has two slaves
sentinels=1 – only one Sentinel instance is running for now

You can get some basic information here, for example – the master’s IP:

root@redis-0:/home/admin# redis-cli -p 26379 sentinel get-master-addr-by-name redis-test

1) "35.158.154.25"

2) "6379"

Now repeat Sentinel start on both slaves nodes using the same config as we did on the master and in the Sentinel’s log, you must see new instances connected:

...

10447:X 29 Mar 14:18:40.437 * +sentinel sentinel fdc750c7d6388a6142d9e27b68172f5846e75d8c 
172.31.36.239 26379 @ redis-test 35.158.154.25 6379

10447:X 29 Mar 14:18:42.725 * +sentinel sentinel ecddb26cd27c9a17c4251078c977761faa7a3250 
172.31.35.218 26379 @ redis-test 35.158.154.25 6379

...

Check status again:

root@redis-0:/home/admin# redis-cli -p 26379 info sentinel

Sentinel

sentinel_masters:1

sentinel_tilt:0

sentinel_running_scripts:0

sentinel_scripts_queue_length:0

sentinel_simulate_failure_flags:0

master0:name=redis-test,status=ok,address=18.194.229.23:6379,slaves=2,sentinels=3

sentinels=3 – okay.

Also, Sentinel will perform its own settings updates when needed:

root@redis-1:/home/admin# cat /etc/redis/sentinel.conf

sentinel myid fdc750c7d6388a6142d9e27b68172f5846e75d8c

sentinel monitor redis-test 35.158.154.25 6379 2

sentinel down-after-milliseconds redis-test 6001

bind 0.0.0.0

sentinel failover-timeout redis-test 60000

Generated by CONFIG REWRITE

port 26379

dir "/home/admin"

sentinel auth-pass redis-test foobared

sentinel config-epoch redis-test 0

sentinel leader-epoch redis-test 0

sentinel known-slave redis-test 18.194.45.17 6379

sentinel known-slave redis-test 3.121.223.95 6379

sentinel known-sentinel redis-test 172.31.35.218 26379 ecddb26cd27c9a17c4251078c977761faa7a3250

sentinel known-sentinel redis-test 172.31.47.184 26379 e9fb72c8edb8ec2028e6ce820b9e72e56e07cf1e

sentinel current-epoch 0

Here is the sentinel myid fdc750c7d6388a6142d9e27b68172f5846e75d8c line added and the whole block after the #Generated by CONFIG REWRITE.

Redis Sentinel Automatic Failover

Now let’s check what will happen if the master will go down.

You can do it manually just by calling kill -9 or by using the redis-cli and sending the DEBUG command with a time in seconds to make a master “down” or by sending a signal to kill the master with.

root@redis-0:/home/admin# redis-cli -a foobared DEBUG sleep 30

The Sentinel’s log on the master:

...

10447:X 29 Mar 14:24:56.549 # +sdown master redis-test 35.158.154.25 6379

10447:X 29 Mar 14:24:56.614 # +new-epoch 1

10447:X 29 Mar 14:24:56.615 # +vote-for-leader ecddb26cd27c9a17c4251078c977761faa7a3250 1

10447:X 29 Mar 14:24:56.649 # +odown master redis-test 35.158.154.25 6379 #quorum 3/2

10447:X 29 Mar 14:24:56.649 # Next failover delay: 
I will not start a failover before Fri Mar 29 14:26:57 2019

10447:X 29 Mar 14:24:57.686 # +config-update-from sentinel 
ecddb26cd27c9a17c4251078c977761faa7a3250 172.31.35.218 26379 @ redis-test 35.158.154.25 6379

10447:X 29 Mar 14:24:57.686 # +switch-master redis-test 35.158.154.25 6379 3.121.223.95 6379

10447:X 29 Mar 14:24:57.686 * +slave slave 18.194.45.17:6379 18.194.45.17 6379 
@ redis-test 3.121.223.95 6379

10447:X 29 Mar 14:24:57.686 * +slave slave 35.158.154.25:6379 35.158.154.25 6379 
@ redis-test 3.121.223.95 6379

10447:X 29 Mar 14:25:03.724 # +sdown slave 35.158.154.25:6379 35.158.154.25 6379 
@ redis-test 3.121.223.95 6379

...

Currently, we are interested in those two lines here:

...

10384:X 29 Mar 14:24:57.686 # +config-update-from sentinel 
ecddb26cd27c9a17c4251078c977761faa7a3250 172.31.35.218 26379 @ redis-test 35.158.154.25 6379

10384:X 29 Mar 14:24:57.686 # +switch-master redis-test 35.158.154.25 6379 3.121.223.95 6379

...

Sentinel performed the slave-to-master reconfiguration.

35.158.154.25 – is the old master which is dead now, and 3.121.223.95 is a new master, elected from the slaves – it’s running on the redis-1 host.

Try adding data here:

root@redis-1:/home/admin# redis-cli -a foobared set test3 'test3'

OK

While a similar attempt on the old master which becomes a slave now will lead to an error:

root@redis-0:/home/admin# redis-cli -a foobared set test4 'test4'

(error) READONLY You can't write against a read only slave.

And let’s kill a node at all and see what Sentinel will do now:

root@redis-0:/home/admin# redis-cli -a foobared DEBUG SEGFAULT

Error: Server closed the connection

Log:

...

10447:X 29 Mar 14:26:21.897 * +reboot slave 35.158.154.25:6379 35.158.154.25 6379 
@ redis-test 3.121.223.95 6379

Well – Sentinel just restarted that node

Sentinel Commands

Command	Description
`sentinel masters`	list all masters and their statuses
`sentinel master`	one master’s status
`sentinel slaves`	list all slaves and their statuses
`sentinel sentinels`	list all Sentinel instances and their statuses
`sentinel failover`	run failover manually
`sentinel flushconfig`	force Sentinel to rewrite its configuration on disk
`sentinel monitor`	add a new master
`sentinel remove`	remove master from being watched

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Redis: Replication, Part 2 – Master-Slave Replication, and Redis Sentinel

The Basic Master-Slave Replication

Redis Master Configuration

Redis Slave Configuration

Changing Slave => Master Roles

Redis Sentinel

Running Sentinel

Redis Sentinel Automatic Failover

Sentinel Commands

Related Links

Similar Posts

License