Redis Token Store in Hybris
Redis one of the most popular in-memory key value data store. At E-bebek we decided to switch the storage at Hybris Oauth access token and refresh token to Redis. Token is one of the most frequently accessed data in API lifecycle. In this way we aimed important time savings during start mobile application. As you might have guessed, it is now much faster. Besides from token store we can also manage some other data in Hybris.
In this article, I will explain the high availability solution of Redis and give a quick demo about how to configure Sentinel and use it in Redis Java client Jedis.
The Big Picture
Redis Setup on Centos7
sudo yum update
sudo yum install epel-release
sudo yum install redis
Master redis.conf
This file contains all config for master or slave nodes. I don’t prefer explain any config at here, only I will explain that our added configs.
protected-mode no
port 6379
dir /var/lib/redis
logfile /var/log/redis/redis-server.log
syslog-enabled yes
syslog-ident redis
#authentication
masterauth your_password
requirepass your_password
maxmemory 64gbIt is useful to explain some of the configs that I see basically.
protected-mode noRedis is executed with the default configuration (binding all the interfaces) and without any password in order to access it, it enters a special mode called protected mode. We set password for master-slave replication and access external service.
dir /var/lib/redisdump.rdb file location. This file is redis database file. It creates snapshots of your dataset at specified intervals.
logfile /var/log/redis/redis-server.logThis config specified log file location. redis-server.log file create during redis setup. Server restart, stop, replication, nodes down and warning state logging. I don’t advice rename this file because old logs will remove.
syslog-enabled yesTo enable logging to the system logger, just set ‘syslog-enabled’ to yes,
and optionally update the other syslog parameters to suit your needs.
syslog-ident redisThis string will be prepended to every Syslog record.
maxmemory 4gbYou see memory info in bytes if you run following command on redis-cli.
config get maxmemory
masterauth your_passwordIf you set password in redis.conf file, redis-cli want password during each master and slave replication.
Redis Server Start, Stop, Restart
Best healthy way for start redis server :
redis-server /etc/redis.conf
restart
/etc/init.d/redis-server restart
or stop/start it:
/etc/init.d/redis-server stop
/etc/init.d/redis-server start
stop/kill:
ps -ef | grep redis
root {pidID} 1 0 Dec20 ? 00:42:31 redis-server *:6379
kill -9 pidID
Firewall Permission
Firstly, I suggest check firewall on terminal with follow command.
firewall-cmd --get-active-zonesIf not exists redis port, enable 6379 port for all machine that setup master and slave.
firewall-cmd --zone=dmz --add-port=6379/tcp --permanentSlave nodes
Redis cluster structure has one master, N slave node. We topology has 1 master and 2 slave nodes. All write transactions run on master. Slave nodes read-only and slaves failovers to data on master by replica. My two slave configs as follows redis.conf.
redis.conf
protected-mode no
port 6379
dir /var/lib/redis
supervised systemd
slaveof master-ip master-port
syslog-enabled yes
syslog-ident redis#authentication
masterauth master_pass
requirepass slave_pass
supervised systemd : To run redis under systemd, you need to set supervised systemd.
slaveof config provides replication between master and slave. Slave node connect if master up.
Run command as follows for 2 slave node:
redis-server /etc/redis.conf
Run “info replication” command on master to check the replication :
[root@testserver ~]# redis-cli
127.0.0.1:6379> auth master_pass
OK
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=10.34.79.55,port=6379,state=online,offset=427832224,lag=1
slave1:ip=10.34.79.56,port=6379,state=online,offset=427832224,lag=1
master_replid:9060ed0a03e87a1f790a1c68fd94619115c6ab7f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:427832224
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:426783649
repl_backlog_histlen:1048576Redis Sentinel
What does work Sentinel?
It is a separate structure responsible for monitoring and reporting the applications on the cluster and for the cluster to runnable in a possible error and exception situation.
We assumed master has a problem, cannot write a new data to redis. After that, system need a critic decision. Who will be new master?
Sentinel get involved here , realize that there is a problem in master, If necessary configurations are made, it communicates with other sentinels and ensures assignment of the new master. Or, regardless of role, it detects the problematic application and excludes it from the cluster; when it detects that it is in a healthy state after a while, takes it back in and performs role assignment.
Configs
Master, Slave-1 and Slave-2 redis-sentinel.conf
protected-mode no
port 16379
dir /var/lib/redis-sentinel
logfile /var/log/redis/redis-sentinel.log
sentinel monitor mymaster my_master_ip 2
sentinel down-after-milliseconds mymaster 30000
sentinel failover-timeout mymaster 60000
sentinel auth-pass master_passsentinel monitor <master-group-name> <ip> <port> <quorum>
The quorum is the number of Sentinels that need to agree about the fact the master is not reachable, in order to really mark the master as failing, and eventually start a failover procedure if possible. However the quorum is only used to detect the failure.
sentinel monitor mymaster my_master_ip 2sentinel <option_name> <master_name> <option_value>
sentinel down-after-milliseconds mymaster 30000
sentinel failover-timeout mymaster 60000HAProxy Setup and Configurations
We build ha-proxy for direct and demands request where front 3 node cluster.
HAProxy is a free and open source software that provides a high availability load balancer and proxy server for TCP and HTTP-based applications that spreads requests across multiple servers.
We can explain reason of using haproxy as follows:
1-For health-check to nodes and info us when occurred a problem.
2- Accessing N redis-server replicas from a single point.
haproxy.cfg
Centos location : /etc/haproxy/haproxy.cfg
# Specifies TCP timeout on connect for use by the frontend ft_redis
# Set the max time to wait for a connection attempt to a server to succeed
# The server and client side expected to acknowledge or send data.
defaults REDIS
mode tcp
timeout connect 4s
timeout server 30s
timeout client 30s
# Specifies listening socket for accepting client connections using # the default
# REDIS TCP timeout and backend bk_redis TCP health check.
frontend front_redis
bind 127.0.0.1:6379 name redis
default_backend bk_redis
# Specifies the backend Redis proxy server TCP health settings
# Ensure it only forward incoming connections to reach a master.
backend bk_redis
option tcp-check
tcp-check connect
tcp-check send PING\r\n
tcp-check expect string +PONG
tcp-check send info\ replication\r\n
tcp-check expect string role:master
tcp-check send QUIT\r\n
tcp-check expect string +OK
server redis-a master_ip:6379 check inter 1s
server redis-b slave_1_ip:6379 check inter 1s
server redis-c slave_2_ip:6379 check inter 1sToken Store
In Hybris, we will use the RedisTokenStore class under the spring-security-oauth2 jar that comes ready to us in the oauth2 extension to keep and read the Token in Redis.
bin/platform/ext/oauth2/lib/spring-security-oauth2–2.3.5.RELEASE.jar!/org/springframework/security/oauth2/provider/token/store/redis/RedisTokenStore.class
Follow jar must required under lib in webservices extension for Spring version 4.3.14.
> spring-data-redis-1.5.0.RELEASE.jar
> spring-data-keyvalue-1.2.23.RELEASE.jar
> spring-data-commons-1.13.23.RELEASE.jar
> jedis-2.6.2.jar
Firstly, createAccessToken method in HybrisOpenIDTokenServices initiates token creation. In the service layer, it works with DefaultTokenServices that comes with spring. We set service’s token store reference by crushing the bean. As a reference, we set the class coded ourselves. Here, by looking at the status of the parameter named “enableRedis”, which we can turn on and off via base store, if it is turned on, Redis; if it is closed, we decide to go to DB.
We extend BaseStore type with additional ‘enableRedis’ attribute.
This attribute is for managing on-off on DB without deployment while mobile app is open.
But it makes more sense to keep this flag in Redis instead of going to DB.
<attribute qualifier="enableRedis" type="java.lang.Boolean" autocreate="true"
generate="true">
<persistence type="property"/>
<modifiers optional="false"/>
<defaultvalue>java.lang.Boolean.FALSE</defaultvalue>
</attribute>security-config.xml
<!-- Redis Hybris Token Store -->
<alias name="defaultRedisHybrisTokenStore" alias="redisHybrisTokenStore"/>
<bean id="defaultRedisHybrisTokenStore" class="com.mydemo.webservices.oauth2.token.provider.RedisHybrisOAuthTokenStore">
<property name="redisTokenStore" ref="defaultRedisTokenStore"/>
<property name="hybrisOAuthTokenStore" ref="defaultOauthTokenStore"/>
<property name="baseStoreService" ref="baseStoreService"/>
</bean><!-- Redis Token Store -->
<alias name="defaultRedisTokenStore" alias="redisTokenStore"/>
<bean id="defaultRedisTokenStore"
class="org.springframework.security.oauth2.provider.token.store.redis.RedisTokenStore">
<constructor-arg ref="redisConnectionFactory"/>
</bean><!-- Redis Connection Factory -->
<bean id="redisConnectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory"
p:host-name="${redis.server}"
p:port="${redis.server.port}"
p:password="${redis.server.password}"
p:usePool="true"
p:poolConfig-ref="jedisPoolConfig">
<constructor-arg ref="redisSentinelConfiguration"></constructor-arg>
</bean><!-- Jedis Connection Pool configuration -->
<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig"
p:maxTotal="${jedis.pool.maxTotal}"
p:maxIdle="${jedis.pool.maxIdle}"
p:minIdle="${jedis.pool.minIdle}"
/><!-- Redis Sentinel Configuration -->
<bean id="redisSentinelConfiguration"
class="org.springframework.data.redis.connection.RedisSentinelConfiguration">
<property name="master">
<bean class="org.springframework.data.redis.connection.RedisNode">
<property name="name" value="mymaster">
</property>
</bean>
</property>
<property name="sentinels">
<set>
<bean class="org.springframework.data.redis.connection.RedisNode">
<constructor-arg name="host" value="${redis.sentinel1.host}" />
<constructor-arg name="port" value="${redis.sentinel.port}" />
</bean>
<bean class="org.springframework.data.redis.connection.RedisNode">
<constructor-arg name="host" value="${redis.sentinel2.host}" />
<constructor-arg name="port" value="${redis.sentinel.port}" />
</bean>
<bean class="org.springframework.data.redis.connection.RedisNode ">
<constructor-arg name="host" value="${redis.sentinel3.host}" />
<constructor-arg name="port" value="${redis.sentinel.port}" />
</bean>
</set>
</property>
</bean><alias name="defaultHybrisOauthTokenServices" alias="defaultOauthTokenServices"/>
<bean id="defaultHybrisOauthTokenServices"
class="de.hybris.platform.webservicescommons.oauth2.token.provider.HybrisOAuthTokenServices"
p:reuseRefreshToken="false"
p:clientDetailsService-ref="oauthClientDetails"
p:tokenStore-ref="redisHybrisTokenStore"
p:supportRefreshToken="true"
p:refreshTokenValiditySeconds="259200"
p:accessTokenValiditySeconds="43200"
/><alias name="defaultOpenIDTokenServices" alias="oauthTokenServices" />
<bean id="defaultOpenIDTokenServices"
class="de.hybris.platform.webservicescommons.oauth2.token.provider.HybrisOpenIDTokenServices"
parent="defaultOauthTokenServices"
p:externalScopesStrategy-ref="externalScopesStrategy"
p:keyStoreHelper-ref="keyStoreHelper"
p:clientDetailsDao-ref="oauthClientDetailsDao" />
BEFORE REDIS — ON
REDIS — ON
REDIS — OFF
Load Test Result
In the mobile application test, we observed a 50% gain in response time with a less error percentage in the Login service when we performed a redis open-closed test with 2000 users for the login service with Jmeter.
Redis-Off
Error Rate : 3.86%
Avarage Time : 1626.86
Redis-On
Error Rate : 1.96%
Avarage Time : 817.20
Conclusion
Thank you for reading this article. I explained the Redis high availability solution and use it in Hybris.
After this, we will think session storage with redis. When I did a little research, I came across a few studies that put this into practice in Hybris.
https://hybrismart.com/2016/08/02/hybris-cluster-redis-session-failover/
Resources that I have used :
https://logtail.com/tutorials/how-to-start-logging-with-redis/
https://github.com/marcel-dempers/docker-development-youtube-series/tree/master/storage/redis
