I have 1000 to 10,000 keys stored in Redis, their value type is list. When a new item is added to any one of the existing lists, I need my golang program to be notified. Once notification is received I need to spawn a new goroutine and perform a small operation.
I am using redigo for redis connection pool.
What is the best approach to solve this problem, without overloading the Redis instance?
You can enable Redis' keyspace notifications and subscribe to the relevant events on the keys/patterns that interest you.
More details can be found in the documentation: http://redis.io/topics/notifications
I haven't tried this, but as speculation, I would use Redis's Lua to implement 2 new commands - MSR_PUSH and MSR_POP - which would do the following respectively:
-- MSR_PUSH
redis.call("PUSH", KEYS[1], ARGV[1])
redis.call("PUBLISH", "notify", KEYS[1])
and:
-- MSR_POP
local v = redis.call("POP", KEYS[1])
if v then
redis.call("PUBLISH", "notify", KEYS[1])
end
return v
So, these Lua scripts update the lists as you normally do, but then also publish the keyname that was updated to the notify pub/sub, which will then allow a watching script (golang) to do something. You could also just push to another queue, and long poll that.
This link has more information on Lua with Redis: https://www.redisgreen.net/blog/intro-to-lua-for-redis-programmers/