Let's imagine that, we have 300K keys with simple counters in our storage Counter key names, for example: counter1_2014-03-25_00:01 counter2_2014-03-25_00:01 counter3_2014-03-25_00:01
Each counter is collecting data for 1 minute, thus one key per one minute: counter1_2014-03-25_00:01, counter1_2014-03-25_00:02, counter1_2014-03-25_00:03 etc.
Every next minute I have to dump all counters for previous minute and remove them from storage. My current implementation is really really simple and using Redis hashes (HINCR, HGETALL, DEL).
//incrementing counter $Redis->hincr('counters_2014-03-25_00:02', 'counter1');
//dumping and removing $result = $Redis->multi() ->hgetall('counters_2014-03-25_00:02') //dumping ->del('counters_2014-03-25_00:02') //removing hash ->exec()
Everything is fine with Redis, but dumping with HGETALL is getting very and very slow with large number of counters because HGETALL have O(N) complexity.
Right now I'm looking for approach that will allow:
UPDATE: Fast dumping is required because I need to transfer all locally collected data from backend servers to master server.
One solution can be to create simple keys, increment using INCR and each time you create a key set an expiry time with it using EXPIRE.