I have a microservice architecture application with multiple services polling an external API. The external API has a rate limiter of 600 requests per minute. How can I have all of my instances together stay below the shared 600 rate limit?
Google only brought me 3 solutions, the most promising being:
Every function in the rate.NewLimiter func seems to be a different import / dependency which i cannot figure out which it isCurrently I have an amateur solution. The code below allows to me set a limit per minute and it sleeps in between requests to spread the requests over the minute. This client rate limit is per instance, thus I would have to hardcode divide the 600 requests by the amount of instances.
var semaphore = make(chan struct{}, 5)
var rate = make(chan struct{}, 10)
func init(){
// leaky bucket
go func() {
ticker := time.NewTicker(100 * time.Millisecond)
defer ticker.Stop()
for range ticker.C {
_, ok := <-rate
// if this isn't going to run indefinitely, signal
// this to return by closing the rate channel.
if !ok {
return
}
}
}()
And inside the function that makes the http API requests.
rate <- struct{}{}
// check the concurrency semaphore
semaphore <- struct{}{}
defer func() {
<-semaphore
}()
How can I have all of my instances together stay below the shared 600 rate limit?
Preferences: - Rate limit counter based on a key, so multiple counters can be set. - Spread the requests over the set duration, so that 600 requests are not sent in the first 30 seconds but rather the full minute duration.
if you want a global rate limiter, you need a place to maintain distributed state, such as zookeeper. Usually, we don't want to pay the overhead. Alternatively, you can set up a forward proxy (https://golang.org/pkg/net/http/httputil/#ReverseProxy), do rate limit in it.
I can't speak to the libraries you found, but a leaky bucket rate limiter is quite simple. You need some kind of shared transactional storage. Each bucket (or rate limiter) is then simply an integer and a time value. The integer is the number of drops in the bucket at the particular time. Every time you have to apply the rate limit, subtract the number of drops that leaked since the last update, then add one, then check if the number of drops is within the capacity of the bucket.
We are using Redis for this sort of thing. To make this transactional in Redis a script is required (see SCRIPT LOAD and EVALSHA). In an SQL database a SELECT FOR UPDATE followed by an UPDATE statement would achieve the same thing, for instance. This is our Redis script:
-- replicate_commands allows us to use the TIME command. We depend on accurate
-- (and reasonably consistent) timestamps. Multiple clients may have
-- inacceptable clock drift.
redis.replicate_commands()
local rate = tonumber(ARGV[1]) -- how many drops leak away in one second
local cap = tonumber(ARGV[2]) -- how many drops fit in the bucket
local now, _ = unpack(redis.call('TIME'))
-- A bucket is represented by a hash with two keys, n and t. n is the number of
-- drops in the bucket at time t (seconds since epoch).
local xs = redis.call('HMGET', KEYS[1], 'n', 't')
local n = tonumber(xs[1])
local t = tonumber(xs[2])
if type(n) ~= "number" or type(t) ~= "number" then
-- The bucket doesn't exist yet (n and t are false), or someone messed with
-- our hash values. Either way, pretend the bucket is empty.
n, t = 0, now
end
-- remove drops that leaked since t
n = n - (now-t)*rate
if n < 0 then
n = 0
end
-- add one drop if it fits
if n < cap then
n = n + 1
else
n = cap
end
redis.call('HMSET', KEYS[1], 'n', n, 't', now)
redis.call('EXPIRE', KEYS[1], math.floor(n/rate) + 1)
return n
Example call for 10 drops per second with a capacity of 10 drops:
EVALSHA <SHA_IN_HEX> 1 rate-limit:my-bucket 10 10
The script returns the number of drops in the bucket. If that number is equal to the capacity, you can either sleep for a short duration and try again, or reject the request outright, depending on your requirements.
Note the the script never returns a value larger than the capacity, resulting in a recovery time of no more than a tenth of a second in your case. That may not be what you need since you are trying to match a third-party rate limiter. I.e. you may be okay with overflowing the bucket, resulting in a longer recovery time after a burst of requests.