Overview
I was recently introduced to Redis by a colleague. For those of you who don't know what it is, it is essentially an incredibly fast key-value store.
We had decided to adopt this technology at work, so we began to think about Performance, Redundancy, etc., and after a day or two of playing around, discovered that there isn't really an established solution for what we wanted to do.
As a result, I ended up combining a few different technologies to get the setup we required, to give us redundancy, automatic failover, and horizontal scaling.
Prerequisites
The whole solution utilizes the following technologies:
Redis Configuration
The purpose of this article isn't to tell you how to set each of these technologies up individually, it is to show you how I link them together, and more specifically, the Redis-TwemProxy Agent that does the linking of Redis-Sentinel and TwemProxy.
This diagram best explains it:
Redis slaves in each shard are read-only copies of their master.
Redis-Sentinel constantly monitors your master nodes in each shard, should a master node go down, a slave node is promoted to the master, and the other slaves are reconfigured to be slaves of this new master.
The Redis-TwemProxy agent script is subscribed to Redis-Sentinel, and detects the master-change event, when this happens, the TwemProxy config is updated and the TwemProxy service is restarted.
Redis-TwemProxy AGent
This was the missing link for me. Redis Sentinel worked perfectly, and promoted my slaves to masters when the master went down, however TwemProxy was still directing traffic to the old, now dead, slave.
The script I have written over on GitHub, is this glue. It is a simple nodejs script that is run permanently on boot on the TwemProxy server, and keeps the TwemProxy config up to date.
The Result
If you tail your TwemProxy Sentinel log, you should see something like this:
[11:31:04] Loading TwemProxy config
[11:31:04] Redis Sentinel TwemProxy Agent Started on: Fri Sep 20 2013 11:31:04 GMT+0000 (UTC)
[11:31:04] Subscribing to sentinel.
[11:31:04] Connection to Redis Sentinel established.
[11:31:04] Getting latest list of masters...
[11:31:04] Master received: redis1 172.19.111.21:6379
[11:31:04] Updating Master redis1 to 172.19.111.21:6379
[11:31:04] Master received: redis2 172.19.111.22:6379
[11:31:04] Updating Master redis2 to 172.19.111.22:6379
[11:31:04] Saving TwemProxy config
[11:31:04] TwemProxy restarted with output:
[11:31:04] Stopping ...
TwemProxy stopped
Starting TwemProxy server...
[11:09:40] Received switch-master: { details:
{ 'master-name': 'redis2',
'old-ip': '172.19.111.22',
'old-port': '6379',
'new-ip': '172.19.111.21',
'new-port': '6380' } }
[11:09:40] Nutcracker config updated with new content:
[11:09:40] twem1:
listen: 172.19.111.15:22121
hash: fnv1a_64
distribution: modula
auto_eject_hosts: false
redis: true
server_retry_timeout: 100
server_failure_limit: 500
servers:
- 172.19.111.21:6379:1 redis1
- 172.19.111.21:6380:1 redis2
[11:09:40] Nutcracker restarted with output:
[11:09:40] Stopping ...
TwemProxy stopped
Starting TwemProxy server...
Script Configuration
The script configuration is simple really, edit the /lib/cli.js script and set the rather self explanatory configuration settings.
I hope this helps those of you who are tasked with setting up and configuring Redis!
