Redis is a popular key value cache store. This solution is for a high availability enterprise cluster with N+1 redundancy allowing complete failure of 1 component or host.
Solution
This walk through describes the steps required to deploy the redis service to 3 servers resulting in redis and sentinel running for 2+1 high availability specification. This solution is written for CentOS / RHEL, update paths and packer manager for other distrubitions. Ive used 3 KVM with 2xCPU 2GB RAM and 10GB storage. There is a final step showing how to collect metrics for prometheus monitoring.
Pre requisites
Configure 3 linux hosts on the same subnet. Disable SELinux and iptables for speed on this walk through deployment.
setenforce 0
iptables -F
Install Redis packages
For CentOS 7 we can get redis from EPEL. Redis Sentinel is a part of the standard redis package. Install to all 3 servers.
At time of writing redis 3.2 was installed.
Prerequisites configuration files
Edit your /etc/redis.conf file.
By default redis will only listen on localhost. As we need Connectivity from 2 other hosts we need to comment out the bind option to listen on all interfaces.
#bind 127.0.0.1
Protected mode is enabled by default, disable to allow connections from other hosts.
With the master redis instance online on host redis-1 we can now configure slave instances 2 and 3 and join the cluster. They will intitally be masters until replication is setup. Official redis replication documentation
Redis-2
Start redis daemon.
Get replication info showing this is a new master. We will then configure as slave.
Issue command to make redis-2 slave of redis-1, you will need the IP address of your redis-1 host.
Confirm this is now a slave of redis-1. Notice the master host IP address and status is up.
Redis-3
Start redis daemon.
Get replication info showing this is a new master. We will then configure as slave.
Issue command to make redis-3 slave of redis-1.
Confirm replication status with redis-1.
Configure Redis Sentinels
Here we will remove the default redis master to monitor ‘mymaster’. We then issue command to start monitoring our master on its IP address. We give the cluster a new group name ‘app-cache’ and configure with a quorum of 2.
The quorum is the number of seninels that must be in agreement to trigger a failover and there by promoting a slave to new master. The full command specification is as follows:
Remove the default master monitor ‘mymaster’. We will issue new monitoring command for our cluster using the master IP address and giving it a service name specific to our use case.
Issue command to start monitoring new redis master. Here we name the redis cache as app-cache with a quorum of 2.
Confirm we have new cluster monitored, notice the number of slaves and sentinels.
Redis-2
Start the sentinel daemon.
Review initial configuration.
Remove the default master monitor ‘mymaster’.
Issue command to start monitoring new redis master on redis-1 host.
Confirm we have new sentinel monitoring cluster, notice the number of slaves and now extra sentinel (2).
Redis-3
Start the sentinel daemon.
Review initial configuration.
Remove the default master monitor ‘mymaster’.
Issue command to start monitoring new redis master on redis-1 host.
Confirm we have new sentinel monitoring cluster, notice the number of slaves and now extra sentinel (3).
Testing
Confirmation of slaves connected on master
Writing a key to master confirmation of replication to slaves
Force failover with sentinel
View master status.
Force failover.
Confirm new master IP address has changed in sentinel monitor.
Confirm new master redis instance is master and has slaves attached.
Monitoring with grafana and prometheus.
I use prometheus for metrics. In order for prometheus to collect metrics it needs to scrape the agent exporter available HERE.
Setup redis exporter
Download the release package.
Extract the contents.
Copy the binary to bin folder.
Setup systemd file.
Reload systemd and start the exporter.
Configure prometheus
I assume you already have a prometheus server installed and configured for your network. Adding config for the redis exporter is straightforward. The config for prometheus /etc/prometheus/prometheus.yml would be as follows for redis-1 2 and 3.
Add dashboard to grafana
Again it is assumed you are using grafana to visualise prometheus metrics. The dashboard to use is available for import here:
The redis comes with a benchmarking tool. This tool has a number of use cases here i create 1 million keys in the master instance.
Results of the metric collection visualised while running benchmarking test to input 1 million keys.
Conclusion
Redis is the most popular in memory database. Its mature, maintained and the sentinel daemon provides reliable resilience to the service.
Slave replication allows balancing reads from masters as well as hot standbys in the event of master failure.
This post demostrates the key features of a redis and sentinel setup in N+1 configuration. N being 2 as we need 2 sentinels for quorum in the event of master failure.
There are many ways to get metrics for redis, prometheus solution is upto date and paired with grafana provides excellent dashboards that can be customised.