I'm running the benchmark tests for a caching project I contribute to, and the benchmark tests keep running failing as Virtual Memory cannot be allocated when a new cache shard is initialised for parallel benchmarking.
github.com\mxplusb\bigcache\caches_bench> go test -v -bench "." -benchtime "10s" . -timeout 30m
-benchmem
BenchmarkMapSet-8 20000000 917 ns/op 296 B/op 3 allocs/op
BenchmarkFreeCacheSet-8 30000000 861 ns/op 360 B/op 3 allocs/op
BenchmarkBigCacheSet-8 30000000 615 ns/op 311 B/op 2 allocs/op
BenchmarkMapGet-8 50000000 410 ns/op 24 B/op 2 allocs/op
BenchmarkFreeCacheGet-8 30000000 770 ns/op 152 B/op 4 allocs/op
BenchmarkBigCacheGet-8 30000000 548 ns/op 152 B/op 4 allocs/op
BenchmarkBigCacheSetParallel-8 runtime: VirtualAlloc of 7340032 bytes failed with errno=1455
fatal error: runtime: cannot map pages in arena address space
runtime stack:
runtime.throw(0x55ca82, 0x30)
C:/Go/src/runtime/panic.go:596 +0x9c
runtime.sysMap(0xc4fc330000, 0x700000, 0x428e01, 0x6152f8)
C:/Go/src/runtime/mem_windows.go:116 +0x129
runtime.(*mheap).sysAlloc(0x5fb5c0, 0x700000, 0x419fd70)
C:/Go/src/runtime/malloc.go:440 +0x37b
runtime.(*mheap).grow(0x5fb5c0, 0x380, 0x0)
C:/Go/src/runtime/mheap.go:774 +0x69
runtime.(*mheap).allocSpanLocked(0x5fb5c0, 0x380, 0x874ca20)
C:/Go/src/runtime/mheap.go:678 +0x456
runtime.(*mheap).alloc_m(0x5fb5c0, 0x380, 0x100000000, 0x0)
C:/Go/src/runtime/mheap.go:562 +0xf0
runtime.(*mheap).alloc.func1()
C:/Go/src/runtime/mheap.go:627 +0x52
runtime.systemstack(0x419fe68)
C:/Go/src/runtime/asm_amd64.s:343 +0xb5
runtime.(*mheap).alloc(0x5fb5c0, 0x380, 0x10100000000, 0x0)
C:/Go/src/runtime/mheap.go:628 +0xa7
runtime.largeAlloc(0x700000, 0x44fd01, 0xc4f61be000)
C:/Go/src/runtime/malloc.go:807 +0x9a
runtime.mallocgc.func1()
C:/Go/src/runtime/malloc.go:702 +0x45
runtime.systemstack(0xc0420278a0)
C:/Go/src/runtime/asm_amd64.s:327 +0x7e
runtime.mstart()
C:/Go/src/runtime/proc.go:1132
goroutine 31 [running]:
runtime.systemstack_switch()
C:/Go/src/runtime/asm_amd64.s:281 fp=0xc042555980 sp=0xc042555978
runtime.mallocgc(0x700000, 0x53c400, 0x1, 0xc042031800)
C:/Go/src/runtime/malloc.go:703 +0x96f fp=0xc042555a20 sp=0xc042555980
runtime.newarray(0x53c400, 0x10000, 0x522b20)
C:/Go/src/runtime/malloc.go:833 +0x6b fp=0xc042555a60 sp=0xc042555a20
runtime.makemap(0x52d8a0, 0x5f5e1, 0x0, 0x0, 0x112)
C:/Go/src/runtime/hashmap.go:281 +0x30d fp=0xc042555ab8 sp=0xc042555a60
github.com/allegro/bigcache.initNewShard(0x100, 0x8bb2c97000, 0x5f5e100, 0x100, 0x1, 0x5e1820, 0x612938, 0x0, 0x0, 0xc0423fa000, ...)
E:/Programming/Go/src/github.com/allegro/bigcache/shard.go:45 +0xc5 fp=0xc042555c80 sp=0xc042555ab8
github.com/allegro/bigcache.newBigCache(0x100, 0x8bb2c97000, 0x5f5e100, 0x100, 0x1, 0x5e1820, 0x612938, 0x0, 0x0, 0x5e1560, ...)
E:/Programming/Go/src/github.com/allegro/bigcache/bigcache.go:58 +0x36e fp=0xc042555dc8 sp=0xc042555c80
github.com/allegro/bigcache.NewBigCache(0x100, 0x8bb2c97000, 0x5f5e100, 0x100, 0x1, 0x0, 0x0, 0x0, 0x0, 0xbc02a, ...)
E:/Programming/Go/src/github.com/allegro/bigcache/bigcache.go:27 +0x7f fp=0xc042555e48 sp=0xc042555dc8
github.com/mxplusb/bigcache/caches_bench.initBigCache(0x5f5e100, 0x4b6af8)
E:/Programming/Go/src/github.com/mxplusb/bigcache/caches_bench/caches_bench_test.go:159 +0x99 fp=0xc042555f00 sp=0xc042555e48
github.com/mxplusb/bigcache/caches_bench.BenchmarkBigCacheSetParallel(0xc042088420)
E:/Programming/Go/src/github.com/mxplusb/bigcache/caches_bench/caches_bench_test.go:79 +0x3d fp=0xc042555f38 sp=0xc042555f00
testing.(*B).runN(0xc042088420, 0x5f5e100)
C:/Go/src/testing/benchmark.go:140 +0xb9 fp=0xc042555f78 sp=0xc042555f38
testing.(*B).launch(0xc042088420)
C:/Go/src/testing/benchmark.go:281 +0x129 fp=0xc042555fd8 sp=0xc042555f78
runtime.goexit()
C:/Go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc042555fe0 sp=0xc042555fd8
created by testing.(*B).doBench
C:/Go/src/testing/benchmark.go:250 +0x77
goroutine 1 [chan receive]:
testing.(*B).doBench(0xc042088420, 0x0, 0x0, 0x0, 0x0, 0x0)
C:/Go/src/testing/benchmark.go:251 +0xa1
testing.(*benchContext).processBench(0xc0420030e0, 0xc042088420)
C:/Go/src/testing/benchmark.go:433 +0x1fe
testing.(*B).run(0xc042088420, 0x0, 0x0, 0x0, 0x0, 0x0)
C:/Go/src/testing/benchmark.go:241 +0x74
testing.(*B).Run(0xc0420882c0, 0x5593df, 0x1c, 0x55dc98, 0x4b6a00)
C:/Go/src/testing/benchmark.go:494 +0x254
testing.runBenchmarks.func1(0xc0420882c0)
C:/Go/src/testing/benchmark.go:403 +0x6e
testing.(*B).runN(0xc0420882c0, 0x1)
C:/Go/src/testing/benchmark.go:140 +0xb9
testing.runBenchmarks(0xc0420030c0, 0x5f6240, 0xa, 0xa, 0xc042040100)
C:/Go/src/testing/benchmark.go:409 +0x494
testing.(*M).Run(0xc042097f20, 0xc042039f20)
C:/Go/src/testing/testing.go:828 +0x30c
main.main()
github.com/mxplusb/bigcache/caches_bench/_test/_testmain.go:60 +0xfe
exit status 2
FAIL github.com/mxplusb/bigcache/caches_bench 223.732s
This is a 64-bit Windows 10 machine with 32GB of available memory. The current pagefile is stored on an M.2 NVMe and is around 5GB (system managed). The specific amount of memory allocated from the shard for the benchmark varies, but is generally between 7-10MB. Resource usage on the machine is low, so there are plenty of available resources. I know it's a page file issue as VirtualAlloc reserves memory on the page.
A new cache shard reserves memory like this:
func initNewShard(config Config, callback onRemoveCallback) *cacheShard {
return &cacheShard{
hashmap: make(map[uint64]uint32, config.initialShardSize()),
entries: *queue.NewBytesQueue(config.initialShardSize()*config.MaxEntrySize, config.maximumShardSize(), config.Verbose),
entryBuffer: make([]byte, config.MaxEntrySize+headersSizeInBytes),
onRemove: callback,
}
}
It's specifically failing on the hashmap allocation. config.initialShardSize() sets up the shard size:
func (c Config) initialShardSize() int {
return max(c.MaxEntriesInWindow/c.Shards, minimumEntriesInShard)
}
When the test goes through the setup steps, it creates a new shard like so:
func initNewShard(config Config, callback onRemoveCallback) *cacheShard {
return &cacheShard{
hashmap: make(map[uint64]uint32, config.initialShardSize()),
entries: *queue.NewBytesQueue(config.initialShardSize()*config.MaxEntrySize, config.maximumShardSize(), config.Verbose),
entryBuffer: make([]byte, config.MaxEntrySize+headersSizeInBytes),
onRemove: callback,
}
}
I've diagnosed the initial problem to the point where I know it is an issue with the VirtualAlloc memory reservations, but I'm not sure how to troubleshoot and triage the issue further.
What are other configurations/settings/tools that I can utilise to troubleshoot this issue further?