Old-timers will be unsurprised that it's hard to get 100% memory efficiency, but I was interested to see how each allocator behaved differently over time, and that jemalloc is apparently doing something that mimalloc is not.
There are workarounds (e.g. ensuring every thread either allocates once in a while or has a mi_collect call on a heartbeat), but it's pretty unfriendly to work-stealing schedulers, and if you're not aware of the issue it's going to be surprising. It's easier to diagnose and work around than glibc's behaviour, but not amazing.
I vaguely recall that I implemented the trim API in snmalloc, but apparently I never pushed it, and it was probably on my work machine so is gone. It's a bit tricky in snmalloc because the design has a couple of tiers:
There's a global range allocator, which will call notify_not_using in the PAL when a large chunk of memory is unused. On Linux this calls madvise with MADV_FREE if you have a recentish kernel or MADV_DONTNEED on older ones. It took Linux a long time to get MADV_FREE, unfortunately (XNU has a nicer variant, I never got around to upstreaming the FreeBSD version that an intern implemented for me).
But there's also a layer of caching. Each thread owns an allocator, which manages its own free lists (if you free from another thread, it's a remote free and is batched and sent back). A thread-local malloc_trim implementation would return free slabs to the range allocator, but ideally you want malloc_trim to operate globally and that would require some synchronisation that (for performance) is intentionally not there in snmalloc.
I'm not quite sure what MADV_FREE does to RSS accounting on Linux. The semantics are that reads will either return zero or the old contents: the VM subsystem can steal the page at any point and replace it with a CoW zero page. Writes will either do the copy and give you a new zeroed page, or will make the old page no-longer available to steal. I'd expect pages to still count to RSS until they're actually stolen, which means that you won't see RSS drop unless there's memory pressure.
The way that MADV_FREE is normally implemented is that you set a bit in the page metadata indicating that it's in the MADV_FREE state and clear the dirty bit. At some point later, when you experience memory pressure, you look for pages that are in the MADV_FREE state. If their dirty bit is clear, you first make the page read-only, then you check the dirty bit (in case it was concurrently modified), then you update the page to point to the canonical CoW zero page. You hold a lock while doing this that means any racing write will be serialised (when it enters the page-fault handler). This makes putting a page in the MADV_FREE state very fast, but means you don't get precise RSS accounting when there are pages in that state. The XNU version provides an explicit operation to take pages out of the MADV_FREE state. If the page hasn't been recycled, this is very cheap (just clear that bit from the vm_page), if it has then it's a new page allocation. This gives you precise RSS accounting, at a cost of a small amount of performance.
Modern operating systems are optimised for the case where the available memory is large. Running out of memory is an infrequent operation that hits slow-path machinery. Until you hit that case, memory should be easy to use and quick to allocate.
TL;DR: Don't expect RSS to actually be meaningful with a modern allocator on Linux or FreeBSD. RSS is the amount of physical memory that your process can use cheaply. If other processes need more memory, there is some amount between 0 and 100% of a process' RSS that will be reclaimed without swapping.
I did have an implementation on Windows that used the low-memory notification to return pages when memory was low. Unfortunately, Windows doesn't trigger this until physical memory is exhausted. If commit is exhausted but there's a load of physical memory free, you don't get this notification and so it wasn't actually useful.
EDIT: For a while, we didn't use MADV_FREE on Linux because using it came with a pretty noticeable performance impact. No idea why Linux is so much worse than FreeBSD or XNU here.
[OP] robey | a day ago
Old-timers will be unsurprised that it's hard to get 100% memory efficiency, but I was interested to see how each allocator behaved differently over time, and that jemalloc is apparently doing something that mimalloc is not.
masklinn | 22 hours ago
Yeah the behaviour of mimalloc is pretty (in)famous
There are workarounds (e.g. ensuring every thread either allocates once in a while or has a
mi_collectcall on a heartbeat), but it's pretty unfriendly to work-stealing schedulers, and if you're not aware of the issue it's going to be surprising. It's easier to diagnose and work around than glibc's behaviour, but not amazing.lonjil | 20 hours ago
I wonder how snmalloc compares. I might test it later this week when I have time.
david_chisnall | 3 hours ago
I vaguely recall that I implemented the trim API in snmalloc, but apparently I never pushed it, and it was probably on my work machine so is gone. It's a bit tricky in snmalloc because the design has a couple of tiers:
There's a global range allocator, which will call
notify_not_usingin the PAL when a large chunk of memory is unused. On Linux this callsmadvisewithMADV_FREEif you have a recentish kernel orMADV_DONTNEEDon older ones. It took Linux a long time to getMADV_FREE, unfortunately (XNU has a nicer variant, I never got around to upstreaming the FreeBSD version that an intern implemented for me).But there's also a layer of caching. Each thread owns an allocator, which manages its own free lists (if you free from another thread, it's a remote free and is batched and sent back). A thread-local
malloc_trimimplementation would return free slabs to the range allocator, but ideally you wantmalloc_trimto operate globally and that would require some synchronisation that (for performance) is intentionally not there in snmalloc.I'm not quite sure what
MADV_FREEdoes to RSS accounting on Linux. The semantics are that reads will either return zero or the old contents: the VM subsystem can steal the page at any point and replace it with a CoW zero page. Writes will either do the copy and give you a new zeroed page, or will make the old page no-longer available to steal. I'd expect pages to still count to RSS until they're actually stolen, which means that you won't see RSS drop unless there's memory pressure.The way that
MADV_FREEis normally implemented is that you set a bit in the page metadata indicating that it's in theMADV_FREEstate and clear the dirty bit. At some point later, when you experience memory pressure, you look for pages that are in theMADV_FREEstate. If their dirty bit is clear, you first make the page read-only, then you check the dirty bit (in case it was concurrently modified), then you update the page to point to the canonical CoW zero page. You hold a lock while doing this that means any racing write will be serialised (when it enters the page-fault handler). This makes putting a page in theMADV_FREEstate very fast, but means you don't get precise RSS accounting when there are pages in that state. The XNU version provides an explicit operation to take pages out of theMADV_FREEstate. If the page hasn't been recycled, this is very cheap (just clear that bit from thevm_page), if it has then it's a new page allocation. This gives you precise RSS accounting, at a cost of a small amount of performance.Modern operating systems are optimised for the case where the available memory is large. Running out of memory is an infrequent operation that hits slow-path machinery. Until you hit that case, memory should be easy to use and quick to allocate.
TL;DR: Don't expect RSS to actually be meaningful with a modern allocator on Linux or FreeBSD. RSS is the amount of physical memory that your process can use cheaply. If other processes need more memory, there is some amount between 0 and 100% of a process' RSS that will be reclaimed without swapping.
I did have an implementation on Windows that used the low-memory notification to return pages when memory was low. Unfortunately, Windows doesn't trigger this until physical memory is exhausted. If commit is exhausted but there's a load of physical memory free, you don't get this notification and so it wasn't actually useful.
EDIT: For a while, we didn't use
MADV_FREEon Linux because using it came with a pretty noticeable performance impact. No idea why Linux is so much worse than FreeBSD or XNU here.thbar | 6 hours ago
I had a production Rust app (GTFS validator on https://transport.data.gouv.fr) doing OOMs on a daily basis, and changed just the allocator.
Charts at https://bsky.app/profile/thibaut-barrere.bsky.social/post/3moapjqs4r224, as you’ll see the consumption is now stable! And average consumption is also lower.
A small change well worth it! https://github.com/etalab/transport-validator/pull/241/files