kiper220/mimalloc
1
0
Fork 0
mirror of https://github.com/microsoft/mimalloc.git synced 2025-05-05 06:59:32 +03:00
Code Issues Projects Releases Packages Wiki Activity Actions

add updated benchmarks

Browse source
This commit is contained in:
daan 2020-01-22 15:05:02 -08:00
parent 5bc1c52ae6
commit af2cfe255a
1 changed files with 117 additions and 91 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

208
readme.md
View file

@ -313,68 +313,71 @@ under your control or otherwise mixing of pointers from different heaps may occu
# Performance # Performance
Last update: 2020-01-20
We tested _mimalloc_ against many other top allocators over a wide We tested _mimalloc_ against many other top allocators over a wide
range of benchmarks, ranging from various real world programs to range of benchmarks, ranging from various real world programs to
synthetic benchmarks that see how the allocator behaves under more synthetic benchmarks that see how the allocator behaves under more
extreme circumstances. extreme circumstances. In our benchmark suite, _mimalloc_ outperforms other leading
allocators (_jemalloc_, _tcmalloc_, _Hoard_, etc), and has a similar memory footprint. A nice property is that it
does consistently well over the wide range of benchmarks.
In our benchmarks, _mimalloc_ always outperforms all other leading General memory allocators are interesting as there exists no algorithm that is
allocators (_jemalloc_, _tcmalloc_, _Hoard_, etc), and usually uses less
memory (up to 25% more in the worst case). A nice property is that it
does *consistently* well over the wide range of benchmarks.
Allocators are interesting as there exists no algorithm that is generally
optimal -- for a given allocator one can usually construct a workload optimal -- for a given allocator one can usually construct a workload
where it does not do so well. The goal is thus to find an allocation where it does not do so well. The goal is thus to find an allocation
strategy that performs well over a wide range of benchmarks without strategy that performs well over a wide range of benchmarks without
suffering from underperformance in less common situations (which is what suffering from (too much) underperformance in less common situations.
the second half of our benchmark set tests for).
We show here only the results on an AMD EPYC system (Apr 2019) -- for As always, interpret these results with care since some benchmarks test synthetic
specific details and further benchmarks we refer to the [technical report](https://www.microsoft.com/en-us/research/publication/mimalloc-free-list-sharding-in-action). or uncommon situations that may never apply to your workloads. For example, most
allocators do not do well on `xmalloc-testN` but that includes the best
industrial allocators like _jemalloc_ and _tcmalloc_ that are used in some of
the world's largest systems (like Chrome or FreeBSD).
The benchmark suite is scripted and available separately We show here only an overview -- for
more specific details and further benchmarks we refer to the
[technical report](https://www.microsoft.com/en-us/research/publication/mimalloc-free-list-sharding-in-action).
The benchmark suite is automated and available separately
as [mimalloc-bench](https://github.com/daanx/mimalloc-bench). as [mimalloc-bench](https://github.com/daanx/mimalloc-bench).
## Benchmark Results ## Benchmark Results on 36-core Intel
Testing on a big Amazon EC2 instance ([r5a.4xlarge](https://aws.amazon.com/ec2/instance-types/)) Testing on a big Amazon EC2 compute instance
consisting of a 16-core AMD EPYC 7000 at 2.5GHz ([c5.18xlarge](https://aws.amazon.com/ec2/instance-types/#Compute_Optimized))
with 128GB ECC memory, running Ubuntu 18.04.1 with LibC 2.27 and GCC 7.3.0. consisting of a 72 processor Intel Xeon at 3GHz
The measured allocators are _mimalloc_ (mi), with 144GiB ECC memory, running Ubuntu 18.04.1 with LibC 2.27 and GCC 7.4.0.
Google's [_tcmalloc_](https://github.com/gperftools/gperftools) (tc) used in Chrome, The measured allocators are _mimalloc_ (mi, tag:v1.4.0, page reset enabled)
[_jemalloc_](https://github.com/jemalloc/jemalloc) (je) by Jason Evans used in Firefox and FreeBSD, and its secure build as _smi_,
[_snmalloc_](https://github.com/microsoft/snmalloc) (sn) by Liétar et al. \[8], [_rpmalloc_](https://github.com/rampantpixels/rpmalloc) (rp) by Mattias Jansson at Rampant Pixels, Google's [_tcmalloc_](https://github.com/gperftools/gperftools) (tc, tag:gperftools-2.7) used in Chrome,
[_Hoard_](https://github.com/emeryberger/Hoard) by Emery Berger \[1], Facebook's [_jemalloc_](https://github.com/jemalloc/jemalloc) (je, tag:5.2.1) by Jason Evans used in Firefox and FreeBSD,
the system allocator (glibc) (based on _PtMalloc2_), and the Intel thread the Intel thread building blocks [allocator](https://github.com/intel/tbb) (tbb, tag:2020),
building blocks [allocator](https://github.com/intel/tbb) (tbb). [rpmalloc](https://github.com/mjansson/rpmalloc) (rp,tag:1.4.0) by Mattias Jansson,
the original scalable [_Hoard_](https://github.com/emeryberger/Hoard) (tag:3.13) allocator by Emery Berger \[1],
the memory compacting [_Mesh_](https://github.com/plasma-umass/Mesh) (git:51222e7) allocator by
Bobby Powers _et al_ \[8],
and finally the default system allocator (glibc, 2.7.0) (based on _PtMalloc2_).
![bench-r5a-1](doc/bench-r5a-1.svg) ![bench-c5-18xlarge-a](doc/bench-c5-18xlarge-2020-01-20-a.svq)
![bench-r5a-2](doc/bench-r5a-2.svg) ![bench-c5-18xlarge-b](doc/bench-c5-18xlarge-2020-01-20-b.svq)
Memory usage: Any benchmarks ending in `N` run on all processors in parallel.
Results are averaged over 10 runs and reported relative
to mimalloc (where 1.2 means it took 1.2× longer to run).
The legend also contains the _overall relative score_ between the
allocators where 100 points is the maximum if an allocator is fastest on
all benchmarks.
![bench-r5a-rss-1](doc/bench-r5a-rss-1.svg) The single threaded _cfrac_ benchmark by Dave Barrett is an implementation of
![bench-r5a-rss-1](doc/bench-r5a-rss-2.svg) continued fraction factorization which uses many small short-lived allocations.
All allocators do well on such common usage, where _mimalloc_ is just a tad
faster than _tcmalloc_ and
_jemalloc_.
(note: the _xmalloc-testN_ memory usage should be disregarded as it The _leanN_ program is interesting as a large realistic and
allocates more the faster the program runs). concurrent workload of the [Lean](https://github.com/leanprover/lean)
theorem prover compiling its own standard library, and there is a 7%
In the first five benchmarks we can see _mimalloc_ outperforms the other speedup over _tcmalloc_. This is
allocators moderately, but we also see that all these modern allocators
perform well -- the times of large performance differences in regular
workloads are over :-).
In _cfrac_ and _espresso_, _mimalloc_ is a tad faster than _tcmalloc_ and
_jemalloc_, but a solid 10\% faster than all other allocators on
_espresso_. The _tbb_ allocator does not do so well here and lags more than
20\% behind _mimalloc_. The _cfrac_ and _espresso_ programs do not use much
memory (~1.5MB) so it does not matter too much, but still _mimalloc_ uses
about half the resident memory of _tcmalloc_.
The _leanN_ program is most interesting as a large realistic and
concurrent workload of the [Lean](https://github.com/leanprover/lean) theorem prover
compiling its own standard library, and there is a 8% speedup over _tcmalloc_. This is
quite significant: if Lean spends 20% of its time in the quite significant: if Lean spends 20% of its time in the
allocator that means that _mimalloc_ is 1.3× faster than _tcmalloc_ allocator that means that _mimalloc_ is 1.3× faster than _tcmalloc_
here. (This is surprising as that is not measured in a pure here. (This is surprising as that is not measured in a pure
@ -383,19 +386,23 @@ outsized improvement here because _mimalloc_ has better locality in
the allocation which improves performance for the *other* computations the allocation which improves performance for the *other* computations
in a program as well). in a program as well).
The _redis_ benchmark shows more differences between the allocators where The single threaded _redis_ benchmark again show that most allocators do well on such workloads where _tcmalloc_
_mimalloc_ is 14\% faster than _jemalloc_. On this benchmark _tbb_ (and _Hoard_) do did best this time.
not do well and are over 40\% slower.
The _larson_ server workload allocates and frees objects between The _larsonN_ server benchmark by Larson and Krishnan \[2] allocates and frees between threads. They observed this
many threads. Larson and Krishnan \[2] observe this behavior (which they call _bleeding_) in actual server applications, and the benchmark simulates this.
behavior (which they call _bleeding_) in actual server applications, and the Here, _mimalloc_ is quite a bit faster than _tcmalloc_ and _jemalloc_ probably due to the object migration between different threads.
benchmark simulates this.
Here, _mimalloc_ is more than 2.5× faster than _tcmalloc_ and _jemalloc_
due to the object migration between different threads. This is a difficult
benchmark for other allocators too where _mimalloc_ is still 48% faster than the next
fastest (_snmalloc_).
The _mstressN_ workload performs many allocations and re-allocations,
and migrates objects between threads (as in _larsonN_). However, it also
creates and destroys the _N_ worker threads a few times keeping some objects
alive beyond the life time of the allocating thread. We observed this
behavior in many larger server applications.
The [_rptestN_](https://github.com/mjansson/rpmalloc-benchmark) benchmark
by Mattias Jansson is a allocator test originally designed
for _rpmalloc_, and tries to simulate realistic allocation patterns over
multiple threads. Here the differences between allocators become more apparent.
The second benchmark set tests specific aspects of the allocators and The second benchmark set tests specific aspects of the allocators and
shows even more extreme differences between them. shows even more extreme differences between them.
@ -404,46 +411,62 @@ The _alloc-test_, by
[OLogN Technologies AG](http://ithare.com/testing-memory-allocators-ptmalloc2-tcmalloc-hoard-jemalloc-while-trying-to-simulate-real-world-loads/), is a very allocation intensive benchmark doing millions of [OLogN Technologies AG](http://ithare.com/testing-memory-allocators-ptmalloc2-tcmalloc-hoard-jemalloc-while-trying-to-simulate-real-world-loads/), is a very allocation intensive benchmark doing millions of
allocations in various size classes. The test is scaled such that when an allocations in various size classes. The test is scaled such that when an
allocator performs almost identically on _alloc-test1_ as _alloc-testN_ it allocator performs almost identically on _alloc-test1_ as _alloc-testN_ it
means that it scales linearly. Here, _tcmalloc_, _snmalloc_, and means that it scales linearly. Here, _tcmalloc_, and
_Hoard_ seem to scale less well and do more than 10% worse on the _Hoard_ seem to scale less well and do more than 10% worse on the multi-core version. Even the best industrial
multi-core version. Even the best allocators (_tcmalloc_ and _jemalloc_) are allocators (_tcmalloc_, _jemalloc_, and _tbb_) are more than 10% slower as _mimalloc_ here.
more than 10% slower as _mimalloc_ here.
The _sh6bench_ and _sh8bench_ benchmarks are The _sh6bench_ and _sh8bench_ benchmarks are
developed by [MicroQuill](http://www.microquill.com/) as part of SmartHeap. developed by [MicroQuill](http://www.microquill.com/) as part of SmartHeap.
In _sh6bench_ _mimalloc_ does much In _sh6bench_ _mimalloc_ does much
better than the others (more than 2× faster than _jemalloc_). better than the others (more than 1.5× faster than _jemalloc_).
We cannot explain this well but believe it is We cannot explain this well but believe it is
caused in part by the "reverse" free-ing pattern in _sh6bench_. caused in part by the "reverse" free-ing pattern in _sh6bench_.
Again in _sh8bench_ the _mimalloc_ allocator handles object migration The _sh8bench_ is a variation with object migration
between threads much better and is over 36% faster than the next best between threads; whereas _tcmalloc_ did well on _sh6bench_, the addition of object migration causes it to be 10× slower than before.
allocator, _snmalloc_. Whereas _tcmalloc_ did well on _sh6bench_, the
addition of object migration caused it to be almost 3 times slower
than before.
The _xmalloc-testN_ benchmark by Lever and Boreham \[5] and Christian Eder, The _xmalloc-testN_ benchmark by Lever and Boreham \[5] and Christian Eder, simulates an asymmetric workload where
simulates an asymmetric workload where some threads only allocate, and others only free -- they observed this pattern in
some threads only allocate, and others only free. The _snmalloc_ larger server applications. Here we see that
allocator was especially developed to handle this case well as it
often occurs in concurrent message passing systems (like the [Pony] language
for which _snmalloc_ was initially developed). Here we see that
the _mimalloc_ technique of having non-contended sharded thread free the _mimalloc_ technique of having non-contended sharded thread free
lists pays off as it even outperforms _snmalloc_ here. lists pays off as it outperforms others by a very large margin. Only _rpmalloc_ and _tbb_ also scale well on this benchmark.
Only _jemalloc_ also handles this reasonably well, while the
others underperform by a large margin.
The _cache-scratch_ benchmark by Emery Berger \[1], and introduced with the Hoard The _cache-scratch_ benchmark by Emery Berger \[1], and introduced with
allocator to test for _passive-false_ sharing of cache lines. With a single thread they all the Hoard allocator to test for _passive-false_ sharing of cache lines.
With a single thread they all
perform the same, but when running with multiple threads the potential allocator perform the same, but when running with multiple threads the potential allocator
induced false sharing of the cache lines causes large run-time induced false sharing of the cache lines can cause large run-time differences.
differences, where _mimalloc_ is more than 18× faster than _jemalloc_ and Crundal \[6] describes in detail why the false cache line sharing occurs in the _tcmalloc_ design, and also discusses how this
_tcmalloc_! Crundal \[6] describes in detail why the false cache line
sharing occurs in the _tcmalloc_ design, and also discusses how this
can be avoided with some small implementation changes. can be avoided with some small implementation changes.
Only _snmalloc_ and _tbb_ also avoid the Only the _tbb_, _rpmalloc_ and _mesh_ allocators also avoid the
cache line sharing like _mimalloc_. Kukanov and Voss \[7] describe in detail cache line sharing completely, while _Hoard_ and _glibc_ seem to mitigate
the effects. Kukanov and Voss \[7] describe in detail
how the design of _tbb_ avoids the false cache line sharing. how the design of _tbb_ avoids the false cache line sharing.
## On 24-core AMD Epyc
For completeness, here are the results on a
[r5a.12xlarge](https://aws.amazon.com/ec2/instance-types/#Memory_Optimized) instance
having a 48 processor AMD Epyc 7000 at 2.5GHz with 384GiB of memory.
The results are similar to the Intel results but it is interesting to
see the differences in the _larsonN_, _mstressN_, and _xmalloc-testN_ benchmarks.
![bench-r5a-12xlarge-a](doc/bench-r5a-12xlarge-2020-01-16-a.svq)
![bench-r5a-12xlarge-b](doc/bench-r5a-12xlarge-2020-01-16-b.svq)
## Peak Working Set
The following figure shows the peak working set (rss) of the allocators
on the benchmarks (on the c5.18xlarge instance).
![bench-c5-18xlarge-rss-a](doc/bench-c5-18xlarge-2020-01-20-rss-a.svq)
![bench-c5-18xlarge-rss-b](doc/bench-c5-18xlarge-2020-01-20-rss-b.svq)
Note that the _xmalloc-testN_ memory usage should be disregarded as it
allocates more the faster the program runs. Similarly, memory usage of
_mstressN_, _rptestN_ and _sh8bench_ can vary depending on scheduling and
speed. Nevertheless, even though _mimalloc_ is fast on these benchmarks we
believe the memory usage is too high and hope to improve.
# References # References
@ -453,14 +476,12 @@ how the design of _tbb_ avoids the false cache line sharing.
the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX). Cambridge, MA, November 2000. the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX). Cambridge, MA, November 2000.
[pdf](http://www.cs.utexas.edu/users/mckinley/papers/asplos-2000.pdf) [pdf](http://www.cs.utexas.edu/users/mckinley/papers/asplos-2000.pdf)
- \[2] P. Larson and M. Krishnan. _Memory allocation for long-running server applications_.
- \[2] P. Larson and M. Krishnan. _Memory allocation for long-running server applications_. In ISMM, Vancouver, B.C., Canada, 1998. In ISMM, Vancouver, B.C., Canada, 1998. [pdf](http://citeseer.ist.psu.edu/viewdoc/download?doi=10.1.1.45.1947&rep=rep1&type=pdf)
[pdf](http://citeseer.ist.psu.edu/viewdoc/download;jsessionid=5F0BFB4F57832AEB6C11BF8257271088?doi=10.1.1.45.1947&rep=rep1&type=pdf)
- \[3] D. Grunwald, B. Zorn, and R. Henderson. - \[3] D. Grunwald, B. Zorn, and R. Henderson.
_Improving the cache locality of memory allocation_. In R. Cartwright, editor, _Improving the cache locality of memory allocation_. In R. Cartwright, editor,
Proceedings of the Conference on Programming Language Design and Implementation, pages 177–186, New York, NY, USA, June 1993. Proceedings of the Conference on Programming Language Design and Implementation, pages 177–186, New York, NY, USA, June 1993. [pdf](http://citeseer.ist.psu.edu/viewdoc/download?doi=10.1.1.43.6621&rep=rep1&type=pdf)
[pdf](http://citeseer.ist.psu.edu/viewdoc/download?doi=10.1.1.43.6621&rep=rep1&type=pdf)
- \[4] J. Barnes and P. Hut. _A hierarchical O(n*log(n)) force-calculation algorithm_. Nature, 324:446-449, 1986. - \[4] J. Barnes and P. Hut. _A hierarchical O(n*log(n)) force-calculation algorithm_. Nature, 324:446-449, 1986.
@ -468,17 +489,22 @@ how the design of _tbb_ avoids the false cache line sharing.
In USENIX Annual Technical Conference, Freenix Session. San Diego, CA. Jun. 2000. In USENIX Annual Technical Conference, Freenix Session. San Diego, CA. Jun. 2000.
Available at <https://github.com/kuszmaul/SuperMalloc/tree/master/tests> Available at <https://github.com/kuszmaul/SuperMalloc/tree/master/tests>
- \[6] Timothy Crundal. _Reducing Active-False Sharing in TCMalloc._ - \[6] Timothy Crundal. _Reducing Active-False Sharing in TCMalloc_. 2016. CS16S1 project at the Australian National University. [pdf](http://courses.cecs.anu.edu.au/courses/CSPROJECTS/16S1/Reports/Timothy_Crundal_Report.pdf)
2016. <http://courses.cecs.anu.edu.au/courses/CSPROJECTS/16S1/Reports/Timothy_Crundal_Report.pdf>. CS16S1 project at the Australian National University.
- \[7] Alexey Kukanov, and Michael J Voss. - \[7] Alexey Kukanov, and Michael J Voss.
_The Foundations for Scalable Multi-Core Software in Intel Threading Building Blocks._ _The Foundations for Scalable Multi-Core Software in Intel Threading Building Blocks._
Intel Technology Journal 11 (4). 2007 Intel Technology Journal 11 (4). 2007
- \[8] Paul Liétar, Theodore Butler, Sylvan Clebsch, Sophia Drossopoulou, Juliana Franco, Matthew J Parkinson, - \[8] Bobby Powers, David Tench, Emery D. Berger, and Andrew McGregor.
_Mesh: Compacting Memory Management for C/C++_
In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'19), June 2019, pages 333-–346.
<!--
- \[9] Paul Liétar, Theodore Butler, Sylvan Clebsch, Sophia Drossopoulou, Juliana Franco, Matthew J Parkinson,
Alex Shamis, Christoph M Wintersteiger, and David Chisnall. Alex Shamis, Christoph M Wintersteiger, and David Chisnall.
_Snmalloc: A Message Passing Allocator._ _Snmalloc: A Message Passing Allocator._
In Proceedings of the 2019 ACM SIGPLAN International Symposium on Memory Management, 122–135. ACM. 2019. In Proceedings of the 2019 ACM SIGPLAN International Symposium on Memory Management, 122–135. ACM. 2019.
-->
# Contributing # Contributing