I need to build a data-structure like this:
map[string]SomeType
But it must store values for about 10 minutes and then clear it from memory. Second condition is records amount - it must be huge. This data-structure must add at least 2-5K records per second.
So, what is the most correct way in Go to make it?
I'm trying to make goroutine with timeout for each new elemnt. And one(or more) garbage-collector goroutine with channel to receive timeouts and clear elements. But I'm not sure it's the most clear way. Is it Ok to have millions of waiting goroutines with timeouts?
Thanks.
You will have to create a struct to hold your map and provide custom get/put/delete funcs to access it.
Note that 2-5k accesses per second is not really that much at all, so you don't have to worry about that.
Here's a simple implementation:
type item struct {
value string
lastAccess int64
}
type TTLMap struct {
m map[string]*item
l sync.Mutex
}
func New(ln int, maxTTL int) (m *TTLMap) {
m = &TTLMap{m: make(map[string]*item, ln)}
go func() {
for now := range time.Tick(time.Second) {
m.l.Lock()
for k, v := range m.m {
if now.Unix() - v.lastAccess > int64(maxTTL) {
delete(m.m, k)
}
}
m.l.Unlock()
}
}()
return
}
func (m *TTLMap) Len() int {
return len(m.m)
}
func (m *TTLMap) Put(k, v string) {
m.l.Lock()
it, ok := m.m[k]
if !ok {
it = &item{value: v}
m.m[k] = it
}
it.lastAccess = time.Now().Unix()
m.l.Unlock()
}
func (m *TTLMap) Get(k string) (v string) {
m.l.Lock()
if it, ok := m.m[k]; ok {
v = it.value
it.lastAccess = time.Now().Unix()
}
m.l.Unlock()
return
}
Just for the record, I had the same problem and wrote tinykv package which uses a map internally.
time.Time for timeouts, so it does not ranges over the whole map.time.Duration greater than zero and less than max, based on the last item that timed out.CAS and Take functionality.